Backtests that once ran overnight now finish before your coffee cools. We turned a single-box bottleneck into a 1,000-worker grid chewing through 20 years of tick data in minutes.
A single backtest over 20 years of ticks pinned one machine for hours, and a full parameter sweep meant overnight queues. Researchers were rate-limited by hardware, not ideas, every new strategy idea cost a night.
The fix was to stop scaling the box and start scaling out. We reframed each backtest as an embarrassingly parallel task and let Ray spray them across an elastic Kubernetes fleet.
Throughput lives and dies on I/O. Tick history sits in S3 as Parquet partitioned by symbol and day, so each worker reads only the shard it owns, no cross-talk, no shared bottleneck.
Polars replays those shards in a vectorized, columnar loop, keeping the Python interpreter out of the hot path. The grid scales near-linearly: double the workers, roughly halve the wall-clock.
| Component | Technology | Why |
|---|---|---|
| Orchestration | Ray | Pythonic task fan-out and fault tolerance without bespoke queue glue. |
| Data engine | Polars | Columnar, multi-threaded replay, 10x faster than pandas, zero-copy. |
| Storage | S3 (Parquet) | Cheap, partitioned, shard-local reads scale linearly with workers. |
| Scheduling | Kubernetes | Autoscaling worker pool that bursts to 1,000+ then drains to zero. |
| Language | Python | One language from research notebook to production grid. |
Kubernetes bursts to 1,000+ pods for a sweep and drains to zero when idle, so cost tracks usage.
Polars streams Parquet shards through a columnar engine with no per-row Python overhead.
Researchers submit a grid of strategy params and get a ranked leaderboard back in one run.
Ray retries failed shards automatically, so a single spot-node eviction never sinks a sweep.
S3 partitioning by symbol and day means each worker reads exactly the data it needs.
Every sweep persists inputs, params and metrics to S3 for audit and exact re-execution.
“We stopped scaling the machine and started scaling the idea, days of backtesting collapsed into minutes.”
The grid turned backtesting from an overnight chore into an interactive tool. With 1,000+ workers replaying 20 years of tick data and 5,000+ backtests running per day, researchers now iterate in a tight feedback loop instead of waiting on a single overloaded box. Ray, Polars and an autoscaling Kubernetes fleet over S3 deliver near-linear scaling and cost that tracks demand, the same hardware that once gated discovery now disappears into the background.