Getting started

Install

pkg> add TestShards

TestShards is used from your test suite, so it belongs in your test dependencies:

[extras]
TestShards = "acceef1d-f5e0-4fe4-a546-818dc56ce7b2"

[targets]
test = ["Test", "TestShards"]

Wrap your suite

Take whatever test/runtests.jl already does and put it inside @shard:

using MyPackage, TestShards

TestShards.@shard begin
    include("core/a.jl")
    include("core/b.jl")
    for f in readdir(joinpath(@__DIR__, "solver"); join = true)
        include(f)
    end
end

Each include becomes one shardable unit. Nothing else changes: run Pkg.test() and the whole suite runs, in order, as before.

Wire up CI

name: CI
on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    permissions:
      contents: write        # the timing history is pushed to a `ci-timings` branch
    uses: QAtlasHub/TestShards.jl/.github/workflows/sharded-tests.yml@main
    with:
      shards: 8
    secrets:
      CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

If your Codecov token does not arrive

Check the collect job for this warning:

No CODECOV_TOKEN reached this workflow, so the upload will be rejected during processing

A secret does not always survive the call into a reusable workflow: the token was present in the caller's own job and absent inside the called workflow, with the secret named rather than inherited — the form makes no difference.

The failure is quiet by construction. Codecov rejects a tokenless upload during processing, which happens after the uploader has already succeeded at queueing it, so the step goes green and coverage simply stops arriving.

You are in it when your repository is private and this workflow is in another organisation. Measured across four callers of one reusable, same named secrets: block, one variable at a time:

Neither condition alone predicts it — it is the combination. The failing caller's token was verifiably present in its own job, so it is the call losing it and not a missing secret, and secrets: inherit versus a named secret makes no difference.

There is a second, unrelated way to end up with no token, worth ruling out first: an organisation secret does not reach a private repository on a free plan at all, so such a repo has an empty token in every job of its own. That one is fixed by giving the repository its own secret, not by moving the upload.

Nothing is lost, because the merge has already happened: the report is published as the testshards-lcov artifact. Upload it from your own repository, where your token works, in a job of its own — and tell the workflow to stop trying, so it does not warn you every run about a token you deliberately did not send:

  test:
    uses: QAtlasHub/TestShards.jl/.github/workflows/sharded-tests.yml@main
    with:
      shards: 8
      coverage-upload: false     # this repository sends it, below
    # no `secrets:` block — it would not arrive, and not sending it is what silences the warning

  coverage:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: QAtlasHub/TestShards.jl/actions/upload-coverage@main
        with:
          codecov-token: ${{ secrets.CODECOV_TOKEN }}

An action runs inside your job, so the token never crosses anything. codecov/codecov-action is the proof that this works — another organisation's action, receiving that same token without trouble, in the very run whose reusable-workflow call did not.

Give it its own job. It checks out the repository — Codecov needs the tree to build the report's file network, and an upload without one is rejected during processing, silently — and a checkout cleans the workspace.

Calling it twice from one workflow

A matrix over Julia versions is two calls, and they share a run. Without a discriminator they collide on upload-artifact, which rejects a duplicate name — so the second call fails on an upload after its tests passed. Give each its own prefix:

jobs:
  ci:     { uses: ..., with: { shards: 8, artifact-prefix: 'v1' } }
  ci-lts: { uses: ..., with: { shards: 8, artifact-prefix: 'lts', julia-version: '1.10', record-timings: false } }

record-timings: false on the secondary leg matters more than the prefix does, and fails more quietly. Both legs would otherwise record to the same branch having measured different configurations: whichever finishes last wins, and every later run is planned against a history that is half one version and half the other. Nothing fails; the balance just stops meaning what it says.

If you set a prefix and upload coverage yourself, pass the matching artifact to the action: artifact: v1-lcov.

Inputs

Every input has a default, so shards is the only one most suites ever set.

shards: 1 — the suite whole

shards: 1 does not mean "one of one shard". It means run the suite whole, in one job: no split, no shard id, nothing skipped. Everything that is not the split still happens — the exactly-once check, the merged coverage report, the diagnosis that tells you how many shards this suite could actually use.

It is what to write when a suite has one unit, when it is too small to be worth splitting, or when you adopt this for the report and want the split later. Ask for the split by raising the number; nothing else about the call changes.

This repository's own CI runs one, on every commit, next to its eight-way call — so the sentence above is demonstrated rather than asserted.

submodules — for a project that resolves through one

If your Manifest.toml carries a dev path into a submodule, or any part of the suite lives in one, set submodules: true — or recursive, exactly as you would on your own actions/checkout. It is passed to every checkout this workflow runs.

Without it the checkout simply does not fetch what the manifest points at, and every shard dies with

Build path for MyDep does not exist: …/submodules/MyDep.jl

which reads like a broken manifest and is not. The manifest is fine; the tree under it is incomplete. If the sources your coverage report covers live in a submodule too, pass the same value to actions/upload-coverage as well — Codecov builds the file network from the checked-out tree.

registries — for a project that resolves from an overlay

Each line is either a name Pkg knows or a clone URL. Each is added only if it is not already reachable — re-adding an existing registry errors rather than doing nothing — and then every registry in every depot is refreshed:

    with:
      registries: |
        General
        https://github.com/my-org/MyRegistry.git

Registries are depot-level, so this reaches inside Pkg.test's sandbox.

Set it even for a package that resolves entirely from General, if you run on a persistent self-hosted depot. Present is not the same as current, and the two fail differently: a missing registry stops the resolve, while a stale one resolves fine and simply cannot see a version registered since it was last pulled.

The workflow exposes an All shards passed job — that is the one to require in branch protection, rather than the individual shards, whose names change with shards.

Run one shard locally

Sharding is driven entirely by environment variables, so you can reproduce any CI shard:

TESTSHARDS_ID=s3 TESTSHARDS_N=8 julia --project -e 'using Pkg; Pkg.test()'

Or run an explicit selection, which is usually what you want when chasing one failure:

TESTSHARDS_UNITS="core/a.jl,solver/heavy.jl" julia --project -e 'using Pkg; Pkg.test()'

With none of them set, everything runs.