I used my brain instead of AI - now CI runs 70% faster

Serhii Shchoholiev · October 2026

Your codebase size has been growing exponentially since you adopted coding agents. The number of tests is higher than it ever was when you used to write code by hand. So your CI has become extremely slow, and you wait hours to merge your PRs. Product is asking you to ship a new bespoke feature for a customer. Being a high-performing Product Engineer, you solve the issue in the most productive way:

/goal speed up my CI

After profiling your CI, a coding agent tells you that you have too many tests and the way to solve it is sharding. It quickly adds a flag to your test runner --shard=1/16. CI takes 10 minutes, a quick stamp, and it's on main.

The next day, you see CI running for 15 minutes, then 40, then 10. Something doesn't make sense. Turns out, test shards aren't split based on duration, but rather on some obscure hash that changes shard distribution every time you add a new test file. You end up with one shard taking 15 min, while the rest finish in under 2.

The solution isn't to switch models or reasoning levels - it's to use your brain.

Defining the problem

This is where LeetCode starts, but you have to design the problem first. Our problem is called multiprocessor scheduling. We need to spread processes (tests) across multiple processors (CI runners). All implementations start with designing the interface - in our case, the function definition. We have many parameters in play: tests, durations, number of runners, vCPU and RAM on runners, architecture of those runners, execution time, and many others. Easy to get lost, so I broke down the problem.

Sorting tests into shards

Starting simple, just sorting tests equally into a given number of shards.

fn sort_tests(tests: [int], shards: int) -> shards: [[int]]

I decided to solve it by using a max heap for tests and a min heap for shards (workers). I would key workers by the total time of tests assigned to them. Then I would loop through tests and add them to the worker with the shortest test duration.

The complexity turned out decent: O(n log n)

The issue with this algorithm is that as the number of tests grows, the execution time grows with it, meaning over time we will end up with CI as slow as it was before.

Making each shard run under the target time

Our compute can scale almost indefinitely, so we can remove the number of shards from the function definition and instead add a target execution time. This allows us to keep test execution under a defined limit by just adding more runners as we add more tests.

fn sort_tests(tests: [int], time: int) -> shards: [[int]]

We first need to compare our target time to the larger number between the longest test duration and the target test time. Tests are atomic, meaning we cannot split a single test across multiple machines, so we can reduce the number of shards in cases where the longest test is greater than the target time. Then we use a heuristic to estimate the number of shards: total test duration divided by the target time. But this produced an edge case. As we greedily fill up our shards, we might go over the target time. A quick fix is to increment the number of shards and rerun our sort. Very inefficient solution.

Each overflow added a full pass through our algorithm, bringing Big O to O(n² log n).

Unravelling the chain of edge cases

To recover from our downgrade to n², we can stop rerunning the whole algorithm and just add one more shard to the heap with a new value. This unravels another edge case: we might add an extra shard when tests can still fit by reorganizing existing shards. Instead of somehow reshuffling the tests - we switch to filling every shard to full capacity. Meaning we now take the shard that has the least capacity left that can still fit our candidate test. If such a shard doesn't exist - we add a new one to the heap. Shards will now be keyed by remaining capacity instead of total time.

One caveat: with the new approach of finding the tightest-fitting shard, a heap is not an optimal data structure. Each lookup can require scanning all shards, bringing the complexity to O(n²).

Optimizing Data Structures

We need a data structure that can do 3 operations efficiently:

1. Find the smallest remaining capacity ≥ the test’s duration
2. Remove a shard
3. Insert the updated shard

When I need cheap search - I think about a hash map. It fits 2 out of 3 requirements. For the search, when we don't have an exact match for the remaining capacity - we have to iterate over the full map, wiping out all the benefits of a hash map. My next thought was to use a sorted array. Binary search would run at O(log n) complexity, but removal and insertion require iterating over the full array. This is how we arrive at a balanced binary search tree. It runs all three of our operations at O(log n) - a perfect balance between the previous data structures.

Back to O(n log n).

Implementation

I decided to implement this algorithm in Rust for 2 reasons:

1. To not introduce latency into CI
2. A binary can run on any runner without extra dependencies

This was the first time in the last year that I wrote code by hand again. I defined the interface, then, following TDD, I built unit tests. Eventually, I wrote the core algorithm. It felt refreshing... It was slow, but I was getting dopamine as I declared a variable, searched the Rust docs for a BST and found the structure, and called the functions - I was constantly getting dopamine by solving small puzzles. Very different from vibecoding, where you prompt an agent, come back in an hour, give some feedback, come back later and see the result and get dopamine.

Benchmarking

We calculated the complexity to be O(n log n), but this is on paper. Now we need to make sure it holds in a production setting. My codebase has 45,000 unit tests, and after digging around the internet, I found a Google blog from 2017 mentioning that they have 4.2 million tests in CI. So I picked a range of tests from 1K to 10 million.

You can see on the chart that the throughput of our algorithm does not match O(n log n) complexity, so it's time to optimize it.

Hyper-optimizing with AI

We got unit tests to ensure the logic stays the same and a benchmark to have a metric to optimize. Time to use AI agents. I had an agent iterate for a few hours to increase the throughput of our algorithm while keeping the tests and benchmark untouched. The result: almost 5× higher throughput at 10 million tests.

Integrating in CI

There are a few changes we need to implement in the pipeline:

1. Track test durations
2. Use the new binary to shard the tests

Most testing frameworks already collect timing data. Since test runners execute whole test files, those are the units we’ll distribute across shards. Aggregate the timing data per file into JSONL. Store it as a GitHub Actions artifact. After each run, use the new measurements to update each file’s running average.

Test {
    id: string                    // Test file path
    duration_ms: positive integer  // Estimated duration in milliseconds
}

During CI execution, download the binary and timing data in a preparation job. Then run the binary with your target shard duration:

tests-sharder --target-ms 60000 test-durations.jsonl > shard-plan.jsonl

The output contains one JSONL record per shard, listing its assigned test files. Use the number of returned shards to spin up runners, and upload the plan as a GitHub Actions artifact. Each runner downloads the same plan and runs only the files assigned to its shard. For Vitest, a custom sequencer reads the assignments. For Playwright, pass the assigned files through a test list.

I’ve published the full implementation on my GitHub. You can download the released binary straight into your pipeline: tests-sharder.

Cost

You might think more runners mean higher costs. Not exactly. GitHub Actions bills you per minute of compute, so total execution time is what really matters. But there is an edge case: GitHub rounds up to the next whole minute. If your shard runs for 2 min 5 sec, you pay for 3 min. In that case, your cost will increase, but you can fix this by adjusting the target shard duration in the binary.

This was a very fun exercise that let me feel the dopamine I used to get by solving problems end to end and making them as performant as possible. It also once again confirmed my view on vibecoding:

You can achieve higher quality, faster than before if you don't outsource thinking and don't have unrealistic deadlines.

After solving the sharding issues, one of the engineers on the team bet me a drink to get CI running under 5 min; another eng challenged me to get it under 3 min. For context: we have over 3 million lines of code and all types of testing: unit, integration, browser, etc. This led to a much deeper optimization, which I will cover in the next article.