API

TestShards

Split a Julia test suite across CI jobs, and record what each piece did.

The shardable units are whatever runtests.jl includes — not files matching a naming convention, and not a list you maintain. @shard shadows include inside its block, so every include call is observed at the moment it happens; a unit computed by a loop over readdir is seen exactly like a literal one.

using MyPackage, TestShards

TestShards.@shard begin
    include("core/test_a.jl")
    for f in readdir("solver"; join = true)
        include(f)                       # computed includes are units too
    end
    TestShards.@unit "stateful" begin    # one unit: same shard, in this order
        include("stateful/01_setup.jl")
        include("stateful/02_use.jl")
    end
end

Two properties make this safe, and both come from every shard observing the whole sequence and skipping what is not its own:

• Nothing is silently dropped. A unit that no shard claims cannot exist — assignment is a total function of the observed sequence, computed identically in every shard.
• Identity is shard-independent. A unit is (key, index) where index is its position in the full sequence, so records from different shards merge into one ordered report.

Within a shard, units run in observed order. Across shards order is not preserved — that is what parallelism means — so anything order-dependent belongs in one @unit.

Each unit's @testset tree is captured with its per-testset timings and outcomes, giving a file → testset hierarchy that a reporting layer can render directly (one page per unit, one section per testset). Attach evidence to the running testset with evidence!.

Sharding never changes what a bare Pkg.test() means: with no environment set, everything runs.

Named operations for building and reading the testset a unit runs in — see register_unit_provider!. nothing when no tool has registered, which is the default and is the case in every run that does not load one.

Assigned <: Ownership

Ownership by computation: every shard packs the same timing history the same way, so a unit belongs to exactly one shard and no shard has to ask anyone.

That determinism is the package's central guarantee — and, under a congested queue, precisely what is wrong: a shard that starts five minutes late still owns its share, and everyone else waits for it. Determinism is what prevents a late runner from taking less.

Bottleneck

What is setting this run's wall clock. One of QueueBound, FixedCostBound, FloorBound or WorkBound.

The same numbers mean opposite things at different scales, and that is the whole reason this is a type rather than a sentence in the manual. A 150s suite with a 49s per-shard cost and a 130s start window is being destroyed by both; a 40-minute suite with the identical figures is barely inconvenienced. A reader can work that out from the raw numbers — but then every consumer repository has to work it out again, and the ones that get it wrong get it wrong silently. So bottleneck decides, and remedy and usable_shards dispatch on the answer.

BudgetBound <: Bottleneck

The split would use more shards than the account can run at once, so the surplus queues behind the first budget of them instead of adding parallelism.

Shard counts are chosen per repository; hosted runners are budgeted per organisation. Every other regime here is a fact about the suite, and this one is not a fact about the suite at all — it is why a knee of ten is the wrong number to act on when eight jobs is what the account can actually deliver. Measured on QAtlasHub: two repositories, sixteen shards and eight, 91% of the org's CI on a busy day, and a peak of 27 concurrent jobs between them.

budget has to be told to diagnose; nothing in a test run can observe it.

Claimed <: Ownership

Ownership by claim: a shard sweeps the whole observed sequence and runs whatever nobody has taken yet. A shard that starts late claims what is left; if nothing is left it exits, having wasted only its own startup instead of delaying everyone.

The primitive is creating a git ref, which is an atomic compare-and-swap on the server: POST /repos/:repo/git/refs creates it or returns 422 because someone else already did. No external service and no new secret — contents: write is already granted for the timing history.

git push was the obvious implementation and is the wrong one: every shard would push the same commit, so a push to an existing ref pointing at that same commit is a no-op that SUCCEEDS, and every shard would believe it had won. Creating a ref fails whatever the sha is.

It buys work stealing at the price of one round trip per unit, and it introduces a failure mode static assignment does not have: a shard that claims a unit and then dies leaves a hole. That is why completeness is not optional — a run that silently skips a unit is exactly the green-but-wrong outcome this package refuses everywhere else.

min_seconds keeps cheap units on Assigned: below it, a round trip costs more than the unit does. Mixing is safe — static assignment is total over the units it covers, so every unit is still owned exactly once.

Completeness

Whether the shards between them ran every unit they observed, exactly once.

Under Assigned this is a theorem: assignment is a total function of the observed sequence, so a unit no shard claims cannot exist. Under Claimed it is not — a shard that claims a unit and then dies or is cancelled leaves the unit claimed and never run, and the merged records are short by one with nothing to say so. That is the green-but-wrong outcome this package refuses everywhere else, which is why claiming does not ship without this check.

It is worth running under both. It costs nothing, and it turns the guarantee from something the design argues into something each run demonstrates.

Diagnosis

What the timing history says about the shape of the suite, rather than about any one run.

walls is the predicted wall clock per shard count under the model wall(N) = fixed + max_bin(N): a shard pays a fixed cost (checkout, depot restore, precompile) that does not shrink when you add shards, plus the load of its heaviest bin. knee is the smallest N at which walls stops improving — past it, more shards buy nothing and cost a fixed price each.

The floor is the single heaviest unit: no split of the suite across jobs can finish sooner than that, so floor_unit is the only place where more parallelism can come from.

observed is the same run as measured rather than modelled, when the shards reported their windows. The model's whole premise is that the shards overlap; observed is what says whether they did.

FixedCostBound <: Bottleneck

A shard spends longer getting ready than testing. Splitting further multiplies the setup and buys almost nothing; the cost itself has to come down.

FloorBound <: Bottleneck

The heaviest single unit is what is left. No split across jobs beats it, so it has to be cut in two — split_here names where.

LcovFile

One source file's coverage, merged across the shards that reported it.

branches maps (line, block, branch) to the number of times it was taken, or nothing for lcov's -: reached by no shard at all.

Observation

What the shards of one run actually did in absolute time, as opposed to what the model predicts.

The model behind Diagnosis assumes the shards run concurrently. effective is the number of shards that assumption was worth: the work done divided by the wall clock it took. It equals the shard count only when they truly overlap, and falls towards 1 — or below it, since each shard re-pays first-use compilation — as the queue spreads them out.

Ownership

How a shard decides whether a unit is its to run: Assigned or Claimed.

The two are answers to different questions, and which one is right depends on something no suite can know about itself — whether the runners start together. Observation measures that, and QueueBound is the diagnosis that says Claimed is worth its cost here.

QueueBound <: Bottleneck

The shards did not run at the same time, so the wall clock is set by the last one to start. Nothing about the split can fix this — see Observation.

Section

One @testset, with its nested sections. duration is wall-clock from Julia's own testset bookkeeping, so it is per-section rather than per-file.

ShardWindow

When one shard ran, in absolute time, and how much of that window it spent on units.

The per-unit durations say how the work divides; they cannot say whether the shards ran at the same time. Under a congested queue they do not, and then the wall clock is set by the last shard to start rather than by the heaviest bin. This is the record that makes that visible: merged across shards it gives the start window, the observed wall clock and, by subtraction, the fixed cost each shard actually paid.

started and finished are epoch seconds from the runner's own clock, so a spread of a second or two between shards is noise rather than a queue effect.

The window runs from the job's start — CI reports it through TESTSHARDS_JOB_START — to the moment the test process ends, which is as far as a shard can see: it is not running when its job finishes. Work the job does after the tests — processing coverage, uploading artefacts — is therefore outside it, and measured here that tail is about half the fixed cost on this suite.

CI closes the gap from outside by stamping the job's end into an ended-*.tsv that load_shards folds back in. Without that file finished is the process end and fixed_cost is a lower bound.

UnitRecord

One shardable unit: what ran, where, for how long, and what it established.

index is the unit's position in the FULL observed sequence, identical in every shard, so records merged from separate jobs sort back into source order.

WorkBound <: Bottleneck

Nothing is in the way: the work still divides, and more shards would still make the run finish sooner. This is the regime the whole design assumes, and the only one in which raising the shard count is the right move.

_claim(c::Claimed, index) -> Bool

Try to become the owner of index. true iff this process created the ref.

201 is a win and 422 is a loss — both are answers. Anything else is the network or the token, which is NOT an answer: it is retried, and if it never answers the shard errors rather than guessing. Guessing "no" silently drops a unit; guessing "yes" runs it twice.

testshards-coverage-s3 → s3; anything else → "".

_facts(d) -> Vector{Pair{String,String}}

The diagnosis as label/value pairs, in report order.

Both renderers walk this. Adding a fact to one of them and forgetting the other is not a hypothetical: budget shipped showing in the plain-text summary under one name and in the Markdown one under another, and peak reached CI in the Markdown only. One list, one set of names, and a new fact appears in both or in neither.

A JSON array of items, each written by write!(io, item). Used for sections and for the section lists hanging off a record, which is the same shape twice.

One line of records-*.jsonl: the unit, its counts, and its testset tree.

Load of the heaviest bin when LPT-packing timings into n shards.

The heaviest bin at n, recovered from the wall curve so the timings need not be kept.

_ownership(env) -> Ownership

Claimed when TESTSHARDS_CLAIM is set AND everything it needs is there, otherwise Assigned.

Asking for claiming without the means is an ERROR, not a downgrade. Falling back quietly would turn "the token was not passed" into "the shards ran what they were assigned", which is a working run with the wrong strategy — and the whole point of the input is to compare the two.

The historical assignment, and round-robin over what the history has not seen.

Take it if nobody else has. Cheap units are left to Assigned: below the threshold a round trip costs more than running the unit would, and mixing is safe because static assignment is total over the units it still covers.

_owns(ctx, key) -> Bool

Does this shard run key? Manual mode consults the explicit list. Auto mode consults the history-derived assignment, and falls back to round-robin over the unknown units in observation order — so a test file added since the last recorded run is still guaranteed to land in exactly one shard.

Include a file as part of the current unit, without opening a new one.

_run(ctx, key, body)

Observe a unit, and run body if this shard owns it. The observation counter advances either way — that is what keeps index, and the round-robin fallback, identical across shards.

The testset is pushed and popped by hand rather than via @testset so that the tree can be read back even when the unit failed: a top-level @testset throws before returning its result. Failure is re-signalled once, at the end of the whole block.

Seconds, to one decimal, the way every line of every report wants them.

src/Foo.jl.123.cov → src/Foo.jl.123-s3.cov, which Foo.jl.*.cov still matches.

assign(timings, n) -> Dict{String,String}

Longest-processing-time bin packing of the KNOWN units: heaviest first into the least-loaded shard. Computed identically in every shard from the same history, so no coordination is needed and no unit can land in two shards.

Units absent from the history are not here; they are assigned on sight, round-robin over the order they are observed in (see _owns), which is also identical everywhere.

bottleneck(d::Diagnosis) -> Bottleneck

Which of the five regimes this suite is in.

They are tested in the order below, because that is the order in which fixing one exposes the next. A queue-bound run tells you nothing about its balance — the balance was never given a chance to matter — so there is no point reporting the floor at it.

1. QueueBound — the shards spent a large share of the run waiting for the last one to START. Needs shards to detect; without windows a run cannot know this happened to it.
2. FixedCostBound — the per-shard fixed cost exceeds the heaviest bin, i.e. a shard spends more of its life getting ready than testing.
3. FloorBound — the requested shard count is at or past the knee, so the heaviest single unit is what remains.
4. WorkBound — otherwise.

BudgetBound sits between the first two: it is the only one that is not a fact about the suite, and like the queue it invalidates what follows, because shards that queued cannot tell you whether the split was good.

The queue test is measured, not modelled, and it used to be the other way round: "the observed wall clock is well above the predicted one". That reads as a queue problem and is not one. critical_path is built on fixed, and fixed is a lower bound — the shard windows close when the test process exits, so per-shard work after the tests is outside them (see ShardWindow). An understated fixed understates the prediction, and the gap that opens gets blamed on the queue. Caught on this package's own CI: eight shards started 2s apart and were still called QueueBound, on a 16.7s gap the start window could account for at most 2s of.

So the question is asked directly. If the last shard started 2s after the first, the queue did not set a 70s wall clock, whatever the model expected.

bottlenecks(d::Diagnosis) -> Vector{Bottleneck}

Every regime whose condition holds, in the order bottleneck tests them.

More than one is usually true. A run can be queue-bound, past its knee, and asking for more shards than the account runs, all at once — those are three independent facts about it. bottleneck returns the first, and which one comes first is a policy, not a measurement. This is the set it was chosen from.

Nothing here is hidden behind that choice. remedy and usable_shards dispatch on any Bottleneck, so a caller that wants a different rule writes it:

bs = TestShards.bottlenecks(d)
# "tell me what the SUITE says, whatever the account can supply"
TestShards.usable_shards(TestShards.FloorBound(), d)
# "I care about the account first"
TestShards.BudgetBound() in bs && TestShards.remedy(TestShards.BudgetBound(), d)

The default order is defended in bottleneck. It is a reasonable rule and it is not the only one, which is why the alternatives stay reachable rather than being resolved away.

completeness(windows, ran) -> Completeness

Check the run against itself: every shard reports how many units it OBSERVED, and each reports which positions it took. The two must reconcile.

completeness_cli(args = ARGS) -> Int

shards.tsv ran.tsv — 0 if the run covered its suite, 1 if it did not. The CI gate calls this, which is the only reason claiming is safe to offer at all.

completeness_report(c) -> String

The check as Markdown, naming what is missing rather than only that something is.

current() -> Union{Nothing,ShardContext}

The @shard block currently executing, or nothing outside one.

diagnose(timings; n = 8, fixed = 0.0, sections = Dict(), shards = ShardWindow[],
         budget = 0) -> Diagnosis

Answer three questions the raw numbers do not: how many shards this suite can actually use, what is stopping it from using more, and where to cut to move that limit.

fixed is the per-shard cost that does not shrink with more shards: a shard's job wall clock minus the time its units took. It is what makes over-sharding expensive rather than merely useless. Pass shards and it is measured rather than guessed — each window carries exactly that subtraction, and their mean becomes fixed unless an explicit non-zero fixed overrides it. shards also decides whether the model's premise held: see Observation.

budget is how many jobs the ACCOUNT can run at once, which nothing in a test run can observe and which is not a fact about the suite at all — see BudgetBound. Left at 0 it changes nothing and is claimed nothing about.

diagnose_cli(args = ARGS) -> Int

timings.tsv [sections.tsv [shards.tsv]] [--shards N] [--fixed SECONDS] [--ends FILE] [--budget JOBS], printing the Markdown report. Used by the CI collect step so every run says where the suite is badly shaped.

The files are positional and in that order, because each one only adds detail to the answer: the timings alone say how many shards the suite can use, the sections say where to cut the unit that limits it, and the shard windows say whether the shards actually ran at the same time.

diagnose_report(d) -> String

The diagnosis as Markdown, for a CI job summary.

evidence!(; kwargs...)

Attach evidence to the @testset currently running — what was checked, to what tolerance, on what grounds — so the report can state it without a reader going to the source:

@testset "inflation preserves the tiling" begin
    err = norm(inflate(t) - reference)
    evidence!(; tolerance = 1e-12, achieved = err, oracle = "closed-form inflation matrix")
    @test err < 1e-12
end

Values that are not Real, Bool, or AbstractString are stored as their string() form, so anything is safe to pass. Outside a @shard block this is a no-op, which keeps a test file runnable on its own.

evidence(ts) -> Dict{String,Any}

What evidence! recorded against a testset — or an empty dictionary when it recorded nothing.

It is keyed on the testset object, so it works whichever type that testset is, including one a register_unit_provider! provider handed out. That is the point of exposing it: a provider whose tool renders a suite can put what a test established into its output, beside whether the test passed, and it does not have to reach into this package's internals to do so.

The evidence for a whole subtree is reachable by walking the testset's own children — a provider's type knows its nesting, and this reads one node at a time.

The part of a shard's window that no split of the suite can remove: checkout, depot restore, precompilation, and the sandbox Pkg.test builds before the first unit runs.

Exact when CI supplied the job's end (see ShardWindow and load_ends); a lower bound without it, because the window then stops at the test process rather than at the job.

(lines, hits) for one file, or summed over several.

load_ends(path) -> Dict{String,Float64}

Read merged ended-*.tsv rows, shard <TAB> epoch seconds: when each shard's JOB finished, as opposed to when its test process did.

A shard cannot record this itself — it is not running any more. CI writes it after the work that follows the tests, which on this repository is coverage processing and is not small. One number per shard, so the only format knowledge outside this file is a printf.

load_ran(path) -> Vector{Tuple{Int,String,String}}

Read merged ran-*.tsv rows: position, shard, unit key. Malformed rows are skipped — but note that a row skipped here reads as a MISSING unit downstream, which fails the run rather than passing it quietly. That is the safe direction for this particular file.

load_sections(path) -> Dict{String,Vector{Pair{String,Float64}}}

Read a sections-*.tsv (unit <TAB> section path <TAB> seconds) into per-unit section lists. Malformed rows are ignored, like load_timings: a diagnosis must never be the thing that fails a run.

load_shards(path; ends = "") -> Vector{ShardWindow}

Read merged shard-*.tsv rows (shard, started, finished, units, unit seconds, observed). Malformed rows are ignored, like load_timings: a diagnosis must never be the thing that fails a run.

ends is an optional load_ends file, and it fixes a real understatement rather than adding a nicety. The finished a shard can write for itself is when its test PROCESS ended; everything the job does afterwards — processing coverage, uploading artefacts — is outside it. Measured here that tail is about half the fixed cost on this suite and about a tenth of it on a large one, so without ends fixed_cost is a lower bound, and the FixedCostBound test built on it is biased towards saying no.

A shard present in ends gets the later of the two timestamps; one absent keeps its own, so a partial file degrades the measurement rather than corrupting it.

load_timings(path) -> Dict{String,Float64}

Read a "key\tseconds" TSV. A missing file or a malformed row is IGNORED: the timing plane is advisory, and a truncated history must degrade the balance, never abort the run.

matrix_json(n) -> String

The GitHub Actions matrix: include: array for n shards.

It carries only labels. Which units a shard runs is decided inside the shard, from the timing history, so the planning job never loads the package under test and never needs to know what the suite contains.

merge_lcov(paths) -> Vector{LcovFile}

Merge lcov tracefiles into one record per source file, in the order the files were first seen.

Merging is a UNION, and that is the whole subtlety. Every shard loads the whole package but runs only part of the suite, so each report marks only the lines that shard executed; a line missed by seven shards and hit by the eighth is covered. Concatenating the tracefiles instead leaves N records for one source file, which reads as N times the line count against a fraction of the hits: this repository's own CI reported 54.5% that way against a true 94.8%, and Codecov rejected the report outright.

Line hits, function hits and branch counts are summed per key; LF/LH and the rest are recomputed by write_lcov rather than trusted from the inputs.

Malformed lines are skipped rather than raising. Coverage is a byproduct of the run, so a truncated report must degrade the number, never fail the suite that produced it.

merge_lcov_cli(args = ARGS) -> Int

out.info in1.info in2.info ... — merge and write, printing a Markdown summary.

Used by the CI collect step, which is why this lives in the package at all: as a step script it was unreachable from Pkg.test(), so nothing could catch it reporting 54.5% for a suite that covered 94.8%.

observe(windows) -> Union{Nothing,Observation}

Fold the shards' windows into the run-level figures. nothing for an empty list, so a caller can pass whatever CI collected without checking first.

peak_concurrency(windows) -> Int

The largest number of shards alive at the same instant.

Requesting N jobs is not acquiring N runners. This counts the ones the scheduler actually granted at once, which is the ceiling on any parallelism the split could have delivered.

register_unit_provider!(; name, open, close = _noop, fold)

Register the testset a unit runs in. Called from a package extension's __init__ — an assignment rather than a method override, so nothing is overwritten at precompile time.

• open(key::String) returns the AbstractTestSet for a unit, or nothing to decline it and leave the default in place. Decline unless the tool's capture is actually running: a suite that merely depends on the tool must not have its testset type changed underneath it.
• close(ts) runs after the testset is popped. A tool that attaches a finished testset to its parent does it here (Test.finish), which is the only moment at which it can.
• fold(ts) returns the counts and structure as plain data — see unit_fold. This is what keeps the balancing history and the completeness verdict correct when the testset is not ours, and it is the first thing to test: the same suite must yield the same numbers whichever testset type ran it.

Only one provider can be registered: two tools cannot both own the type of one testset, and letting the last one win would surface as a mysteriously empty report rather than as an error.

remedy(d::Diagnosis) -> String

What to do about it, in one sentence, chosen by dispatch on bottleneck rather than left to the reader.

restore_counters(parts, dest = ".") -> Vector{String}

Put every shard's raw coverage counters back where their sources are, tagged so they cannot collide. Returns the paths written.

Julia writes Foo.jl.<pid>.cov beside Foo.jl, and CoverageTools finds them by globbing Foo.jl.*.cov. Shards run on different machines, so their PIDs can be equal — s1 and s4 both producing TestShards.jl.1234.cov would have one silently overwrite the other, and the lost shard's coverage would simply not appear. The shard label goes into the name to prevent that, in the glob's wildcard where CoverageTools still matches it.

parts is the artifact download directory: one subdirectory per shard, named ...coverage-<shard>, each holding the counter files under their original relative paths.

This exists in the package rather than in the workflow because it is the step where coverage can go missing without anything failing — and the last thing to hold that job silently reported 54.5% for a suite that covered 94.8% for as long as it existed.

unit_fold(ctx, ts) -> Section

The unit's Section tree. A DefaultTestSet is read directly; any other testset type goes through the registered provider's fold, which returns plain data —

(; name, duration, npass, nfail, nerror, nbroken, sections)

with sections a vector of the same shape (duration in seconds; the fields may be omitted and default to zero). Plain data rather than a Section, so a provider living in another package does not have to name this package's types.

usable_shards(d::Diagnosis) -> Int

How many shards are worth starting under the regime bottleneck selected — which is not always the knee, and is not always what you want asked.

Under QueueBound it is the number of runners the scheduler actually granted at once; under BudgetBound it is the account's limit; otherwise it is the knee.

It answers one question under one policy. Call it with a regime to ask a different one — usable_shards(FloorBound(), d) for what the suite could use regardless of what the account supplies, usable_shards(BudgetBound(), d) for the reverse. bottlenecks says which are true at once, and this deliberately does not reconcile them: the budget is a constraint on the account, the knee is a property of the suite, and which of the two should give way is a decision about the repository that the diagnosis is in no position to make.

How long this shard's job was alive.

write_lcov(path, files)

Write merged LcovFiles back out as a tracefile, recomputing every summary line (LF/LH, FNF/FNH, BRF/BRH) from the merged records so they cannot disagree with them.

write_records(ctx, dir)

Write this shard's records as JSONL (one unit per line) plus its timings as TSV.

Two files because they have different consumers and different lifetimes: the TSV is the planner's history and is merged into a single ledger, while the JSONL is the report's input and is merged into one ordered document. Both key on the same unit key, so they always agree.

@shard begin ... end

Run the block with include shadowed, so each include is a shardable unit. See the module docstring for the whole picture.

The test root — what unit keys are relative to — is the directory of the file this macro is written in, so keys are stable no matter where CI runs from.

@unit "name" begin ... end

Treat everything in the block as ONE shardable unit: it runs in a single shard, in the order written. This is the only way to keep order-dependent files together, since across shards order is not preserved.

It is also the manual-mode handle: in manual mode a shard is given unit names to run, so naming a unit is how you assign work by hand rather than by measured time.