The most broken part of data pipelines is the handoff, and I'm fixing that

A thing that has always felt broken to me about data pipelines is that the people building the actual logic are usually data scientists, researchers, or analysts, but once the workload gets big enough, it suddenly becomes DevOps responsibility.

And to be fair, with most existing tools, that kind of makes sense. Distributed computing requires a pretty technical background.

So the workflow usually ends up being:

• build the pipeline logic in Python
• prove it works on a smaller sample
• hit the point where it needs real cloud compute
• hand it off to someone else to figure out how to actually scale and run it

The handoff sucks, creates bottlenecks, and leaves builders at the mercy of DevOps.

The person who understands the workload best is usually the person writing the code. But as soon as it needs hundreds or thousands of machines, now they’re dealing with clusters, containers, infra, dependency sync, storage mounts, distributed logs, and all the other headaches that comes with scaling Python in the cloud.

That is a big part of why I’ve been building Burla.

Burla is an open source cloud platform for Python developers. It’s just one function:

from burla import remote_parallel_map my_inputs = list(range(1000)) def my_function(x): print(f"[#{x}] running on separate computer") remote_parallel_map(my_function, my_inputs) 

That’s the whole idea. Instead of building a pile of infrastructure just to get a pipeline running at scale, you write the logic first and scale each stage directly inside your Python code.

remote_parallel_map(process, [...]) remote_parallel_map(aggregate, [...], func_cpu=64) remote_parallel_map(predict, [...], func_gpu="A100") 

https://i.redd.it/ekxmil3epfrg1.gif

It scales to 10,000 CPUs in a single function call, supports GPUs and custom containers, and makes it possible to load data in parallel from cloud storage and write results back in parallel from thousands of VMs at once.

What I’ve cared most about is making it feel like you’re coding locally, even when your code is running across thousands of VMs

When you run functions with remote_parallel_map:

• anything they print shows up locally and in Burla’s dashboard
• exceptions get raised locally
• packages and local modules get synced to remote machines automatically
• code starts running in under a second, even across a huge amount of computers

A few other things it handles:

• custom Docker containers
• cloud storage mounted across the cluster
• different hardware per function

Running Python across a huge amount of cloud VMs should be as simple as calling one function, not something that requires additional resources and a whole plan.

Burla is free and self-hostable --> github repo

And if anyone wants to try a managed instance, if you click "try it now" it will add $50 in cloud credit to your account.