Data is the lifeblood of modern business, but managing it can be a nightmare. Traditional ETL (Extract, Transform, Load) processes often involve brittle scripts, complex orchestration tools, and a lack of visibility that can make debugging a painful, time-consuming ordeal. What if you could treat your data pipelines just like any other piece of modern software—version-controlled, testable, and scalable?
Enter Business-as-Code. With the .do SDK, you can transform complex data pipelines into simple, powerful Services-as-Software. This post will walk you through how to leave fragile scripts behind and build robust, automated ETL processes using our intuitive Typescript SDK.
If you've ever managed a data pipeline, these challenges probably sound familiar:
The .do SDK addresses these issues by reframing the problem. Instead of a loose collection of scripts, we define a data pipeline as an Agentic Workflow—a clear, declarative set of steps executed by a software agent.
Let's build a common ETL process: extracting daily sales data from an e-commerce platform (like Shopify), enriching it with customer information, and loading it into a data warehouse (like BigQuery) for analysis.
With the .do SDK, this entire process is defined in a single, readable Typescript file.
First, we define the Workflow. This is the core of Business-as-Code. It's a simple, ordered list of tasks. Each do statement represents a modular, independent action that can be developed and tested in isolation.
Notice how the workflow reads like a business plan. This clarity is crucial for maintainability and collaboration between teams.
The Agent is the executor. It's a worker that holds one or more workflows and is responsible for running them. You can create specialized agents for different business functions, such as a dataAgent, a customerOnboardingAgent, or a financeAgent.
Before deploying, you can run the entire workflow on your local machine. The .do SDK is designed for local-first development, complete with mocking utilities to simulate API calls. This allows you to validate your logic and catch errors early.
Executing the workflow is a single, clean function call.
With just a few lines of declarative code, you've defined a testable, maintainable, and highly readable ETL pipeline.
This is where the magic happens. Once you're satisfied with your workflow, you deploy it to the .do cloud platform with a simple CLI command.
Instantly, your sales.process.nightly workflow becomes a secure, scalable API endpoint.
This means:
Your ETL process is no longer a hidden script running on a cron job. It's a first-class, version-controlled Service-as-Software that can be called, monitored, and managed like any other microservice in your stack.
Stop wrestling with brittle scripts and start building data pipelines the modern way. The .do SDK provides the developer-first experience needed to turn complex business processes into simple, powerful, and scalable services.
Ready to build your first agentic workflow?
Transform your business logic into powerful software today with sdk.do.
Q: What is the .do SDK?
A: The .do SDK is a Typescript library that allows developers to programmatically define, manage, and execute agentic workflows. It's the primary tool for building Business-as-Code on the .do platform.
Q: Who is the SDK for?
A: The SDK is designed for developers who want to automate complex business processes, integrate services, and deliver them as robust, version-controlled software. If you know Typescript, you can start building powerful agents immediately.
Q: How do I get started with the SDK?
A: Getting started is simple. Install the package via npm (npm install @do-sdk/core), import the necessary modules, and start defining your first agent and workflow. Check our documentation for comprehensive guides.
Q: Can I test my workflows locally?
A: Yes, the .do SDK comes with a local testing environment and mocking utilities. This allows you to develop and debug your workflows entirely on your local machine before deploying them to the .do cloud platform.
Q: How do my workflows become APIs?
A: Once you deploy a workflow using the SDK, the .do platform automatically exposes it as a secure, scalable API endpoint. This transforms your business logic (Business-as-Code) into a consumable service (Service-as-Software) without any extra configuration.
import { Agent, Workflow } from '@do-sdk/core';
// Define a robust ETL workflow for processing sales data
const etlSalesData = new Workflow({
name: 'sales.process.nightly',
steps: [
// 1. Extract: Pull new paid orders from a Shopify store
{ do: 'shopify.extract.orders', params: { status: 'paid', since: 'yesterday' } },
// 2. Transform: Enrich customer data with location info from a geo-service
{ do: 'geo.enrich.customer', input: '{{steps[0].output.customers}}' },
// 3. Transform: Standardize product data format for analytics
{ do: 'format.products', input: '{{steps[0].output.line_items}}' },
// 4. Load: Push the transformed data to a BigQuery data warehouse
{ do: 'bigquery.load.records', table: 'processed_sales' },
],
});
// Create an agent responsible for data processing workflows
const dataAgent = new Agent({ workflows: [etlSalesData] });
// Execute the workflow on-demand or via a scheduler
async function runNightlyETL() {
console.log('Starting nightly sales data ETL...');
const result = await dataAgent.execute('sales.process.nightly');
console.log(`ETL process completed successfully!`);
console.log(result);
}
runNightlyETL();