API reference

SweepRunner

HPC experiment runtime for Julia.

Wraps ParamIO.jl and DataVault.jl with a unified run! that handles parallel dispatch, advisory locking, .done rollups, structured event logging, and retry. Designed to replace the recurring "glue" layer that every HPC research project re-invents.

Modules

Each file is one concern, one module-scope piece. They can be used independently.

Quick start

using ParamIO, DataVault, SweepRunner

spec  = ParamIO.load("config.toml")
keys  = ParamIO.expand(spec)
vault = DataVault.Vault("config.toml"; run="phase1")

SweepRunner.init_workers!(mode=:auto)
work_fn = key -> Dict{String,Any}("x" => compute(key))
SweepRunner.run!(work_fn, vault, keys)

Design constraints

1. work_fn is a pure function — no IO, no globals, no logging. All of that lives in the runtime.
2. There is no per-item println API. Use log_event for structured events. Per-item prints are the reason FiniteTemperature.jl used to generate 300 MB job logs.
3. The Manifest is monotonic — keys are only added, never removed. Re-computing the same root is considered a contract violation; branch into a new outdir instead.
4. Multi-master coordination is entirely delegated to DataVault.acquire_running!, which uses POSIX link() for atomic "create iff not exists" on NFS. No flock, no central service, no separate locks/ tree — .running itself is the lock.

See also

• ParamIO.canonical — the stable string form used by Manifest as a directory-safe key identity.
• DataVault.Vault, DataVault.save!, DataVault.is_done, DataVault.acquire_running! — the storage + lock layer run! delegates to.

atomic_write(write_fn, path)

Write path atomically: the target either does not exist or contains the complete output — never partially-written content.

Internally:

1. Writes to "$path.tmp.$(getpid()).$(rand(UInt32))" via write_fn(io).
2. Flushes + fsyncs the tmp file (best-effort on filesystems that don't support fsync via ccall).
3. mv(tmp, path; force=true) — POSIX rename(2) is atomic, including on NFS, so concurrent readers see either the old content or the new.

The parent directory of path is created if missing. If write_fn throws an exception, the tmp file is cleaned up and the target path is left untouched.

Returns

The path argument unchanged, so the function is chainable:

p = atomic_write("out/data.jld2") do io
    write(io, payload_bytes)
end
# p === "out/data.jld2"

Notes

• The tmp filename includes getpid() and a random UInt32 so two processes racing on the same path do not clobber each other's tmp.
• This helper is file-format agnostic: callers pass any IO-consuming closure. For JLD2 payloads, Manifest uses a similar path-level dance because JLD2.jldsave needs a path, not an IO.
• On a filesystem where fsync is not available, the ccall is caught and ignored; rename atomicity still holds.

See also: atomic_touch for empty-file markers.

atomic_touch(path)

Create an empty file at path atomically. Convenience wrapper over atomic_write for sentinel files like .done markers.

atomic_touch(joinpath(vault.outdir, "stage1", "done.marker"))

Semantics are identical to atomic_write(_ -> nothing, path) — the parent directory is created, the file appears atomically, and no tmp residue is left behind.

EventLog(path::AbstractString)

Append-only JSONL event log with a thread-safe per-EventLog lock.

Each call to log_event writes one JSON object as a single line. Concurrent writes from multiple tasks are serialized through a ReentrantLock held per PATH, so several EventLog objects on one file share it. Concurrent writes from multiple processes (separate masters) rely on POSIX O_APPEND atomicity, which is guaranteed for single write syscalls of length < PIPE_BUF (4 KiB); log_event composes each line as a single String and issues one write to an unbuffered descriptor to stay within that guarantee.

Fields

• path::String — target JSONL file. Parent directory is created lazily on first log_event.
• lock::ReentrantLock — the per-path lock at construction time. log_event resolves the lock from path rather than reading this field, so an EventLog that arrived on a worker by deserialization (which skips the constructor) still serialises against its siblings there.

Event kinds used by run!

run! emits the following kind values (as strings in the JSON):

:key_start and :lock_busy are emitted at :debug level and are suppressed unless the EventLog is created with min_level=:debug (see RunOpts.log_level); their totals still appear in :stage_done. Downstream analysis (jq, DataFrame-based) can filter and aggregate over these kinds without ever parsing freeform text.

Example

log = EventLog("out/events.jsonl")
log_event(log, :stage_start; stage=:phase1, todo=3600)
# ... work ...
log_event(log, :stage_done; stage=:phase1, done=3600, err=0)

log_event(log, kind; kwargs...)

Append one JSON object to log.path with fields ts (ISO-8601 local time), kind (the Symbol converted to String), and any additional key/value pairs passed via kwargs.

The line is built in full (including the trailing newline) as a single String and written with one write syscall to an UNBUFFERED append-mode descriptor. Both halves matter: the syscall is what POSIX O_APPEND atomicity applies to, so cross-process writes do not tear each other's lines, and an IOStream would flush on its own boundaries instead.

log_event(log, :key_done; stage=:phase1, key="N=8;J=1.0;#sample=1", secs=12.3)

produces one line like:

{"ts":"2026-04-13T14:23:51.123","kind":"key_done","stage":"phase1","key":"N=8;J=1.0;#sample=1","secs":12.3}

Returns nothing.

merge_event_logs(dir; output="events_merged.jsonl") -> String

Merge all per-master event log files (events_*.jsonl) in dir into a single sorted file. Returns the output path.

Each master writes to its own events_<host>_<pid>.jsonl; this function collects and sorts all lines by their ts field for post-hoc analysis.

Manifest(stage, root, complete)
Manifest(stage, root)

Rollup index for a single stage under a vault root. complete holds ParamIO.canonical strings of every key known to be finished.

Fields

• stage::Symbol — stage label, e.g. :phase1. Used to pick the sub-directory inside root.
• root::String — the vault root (typically vault.outdir) combined with the project name. See manifest_root for the convenience overload.
• complete::Set{String} — in-memory set of canonical key strings, rehydrated from the JLD2 file on load.

On-disk format

<root>/<stage>/manifest.jld2

JLD2 layout: one field complete::Vector{String}. A Vector is used (not a Set) so older JLD2 versions deserialize cleanly; load_manifest rehydrates to a Set{String} for O(1) lookup.

Invariants

• Monotonic. Once a key is added via add_complete! it is never removed by this package. Re-computing the same root with different work is considered a contract violation — branch into a new outdir.
• Atomic updates. save_manifest uses a tmp + rename dance so a crash mid-write cannot leave a half-written manifest on disk.
• Corrupt file = empty. If a manifest file exists but cannot be parsed by JLD2, load_manifest returns an empty Manifest. The worst outcome is re-running already-complete keys, which is idempotent thanks to the per-key .running lock and the is_done re-check in run!.

Typical use from inside run!

m = load_manifest(vault)
todo = todo_keys(m, collect(keys))
isempty(todo) && return :skip
for key in todo
    # ... do work ...
    add_complete!(m, key)
end
save_manifest(m)

See also

load_manifest, save_manifest, add_complete!, is_complete, todo_keys, manifest_path, manifest_root.

manifest_path(root, stage) -> String

Return the conventional on-disk location "$root/$stage/manifest.jld2". Pure function; does not touch the filesystem.

load_manifest(root, stage) -> Manifest
load_manifest(vault::DataVault.Vault) -> Manifest

Read manifest.jld2 if present; otherwise return an empty Manifest.

A corrupted manifest (any JLD2.load failure) is treated as empty. This prefers making progress over refusing to run — the runtime's is_done re-check after lock-acquire makes accidental re-processing harmless.

The second form is a convenience defined in Run.jl that derives (root, stage) from a DataVault.Vault (see manifest_root).

load_manifest(vault::DataVault.Vault) -> Manifest

Convenience overload of the two-argument load_manifest that derives (root, stage) from a DataVault.Vault:

load_manifest(manifest_root(vault), Symbol(vault.run))

save_manifest(m) -> String

Atomically write m.complete to manifest_path(m.root, m.stage) and return that path.

Internally uses a JLD2.jldsave to a sibling tmp path followed by mv(..., force=true), so a crash in the middle leaves either the old manifest (unchanged) or the new one — never a truncated JLD2.

m.complete is sorted on save so the on-disk Vector has a deterministic order, making jld2 files easier to diff for debugging.

add_complete!(m, key) -> Set{String}

Add ParamIO.canonical(key) to the in-memory m.complete set. Not persisted until save_manifest is called.

This is a one-way operation: the Manifest API does not expose a remove_complete! (see the monotonic invariant in the Manifest docstring).

is_complete(m, key) -> Bool

true if ParamIO.canonical(key) is in m.complete. This is the per-key check used by todo_keys; it never touches the filesystem.

todo_keys(m, keys) -> Vector{DataKey}

Return the subset of keys that are not yet in m.complete. This is the work list for the next run!; a fully-done run returns an empty vector, which triggers the early-skip path.

Cost is O(length(keys)) hash lookups — independent of filesystem state. On 3600 keys this takes ~6–12 ms in the benchmark.

manifest_root(vault) -> String

Return the directory under which run! and load_manifest look for this vault's manifest.jld2 — one manifest per (project, run).

The layout is:

<vault.outdir>/manifest/<project_name>/<vault.run>/manifest.jld2

Pure function; does not touch the filesystem.

merge_and_save_manifest!(m) -> String

Reload the on-disk manifest, merge its completed keys into m, then atomically persist. This prevents the "last writer wins" problem when multiple masters call save_manifest concurrently — without the merge, each master would overwrite the others' newly completed keys.

There is a small TOCTOU window between the load_manifest and the save_manifest; in the worst case a concurrent saver's keys are not included in this write, but they will be re-processed on the next invocation (safe due to the is_done re-check after lock-acquire).

init_workers!(; mode=:auto, master_blas=1, launch_timeout=300.0,
                worker_timeout=300, verbose=true) -> Symbol

Bootstrap worker processes / threads according to mode, and return the mode actually used (useful when mode=:auto).

Modes

Timeouts (relevant to :slurm / :distributed)

• launch_timeout::Real = 300.0 — seconds the master will wait for SlurmClusterManager.SlurmManager to produce worker addresses via srun. Large jobs (≳ 100 workers) with cold NFS package caches need a substantially larger value than the SlurmManager default (60s).
• worker_timeout::Integer = 300 — value exported as JULIA_WORKER_TIMEOUT so every freshly spawned Julia worker waits up to this many seconds for the master to send its first handshake message. The built-in Distributed default is 60s, which is too tight when 100+ workers race each other through _include_from_serialized on a shared depot.

Both defaults (300s) handle the 128-worker i8cpu case on ISSP System B comfortably. Set lower values only for local debugging.

Idempotent: calling multiple times with worker processes already present does not double-add. BLAS thread settings are always (re)applied.

detect_mode() -> Symbol

Inspect the environment to pick a default init_workers! mode:

• :slurm if SLURM_JOB_ID is present in ENV,
• :distributed if JULIA_SLURM_N_WORKERS > 0 (multi-worker without a Slurm job — e.g. a local addprocs smoke test driven by the same env var the batch scripts set),
• :threads if Threads.nthreads() > 1,
• :sequential otherwise.

This is what init_workers!(mode=:auto) delegates to. Callers rarely need to invoke detect_mode directly; it is public mainly for tests.

verify_workers!()

Probe each Distributed worker for hostname, Julia threads, BLAS threads, and CPU affinity. Prints a summary table, then ONE @warn carrying how many workers have BLAS.get_num_threads() > 1.

That setting is reported, not diagnosed. It is a known cause of OpenBLAS segfaults in multi-process Julia, but on a 2-site TDVP workload (10 sites, chi=20) ms/step was flat from 1 to 36 threads and ~250 completed keys at blas=16 produced no segfault, so the warning does not claim the setting is wrong here. It used to fire per worker: 287 lines on a 72-node allocation, interleaved with the rows of the table above it.

Ported from FiniteTemperature.jl Parallel/Slurm.jl::print_worker_identities.

RunOpts(; workers=:auto, max_attempts=3, stale_after=600.0,
         heartbeat_interval=60.0, stop_flag=nothing, deadline=nothing)

Execution options for run!.

Fields

• workers::Symbol = :auto — dispatch mode. :auto fans out over the Distributed WorkerPool(workers()) via pmap when nprocs() > 1, and runs sequentially otherwise. :sequential forces the sequential path even when worker processes are present (useful for debugging a serialization issue).

• max_attempts::Int = 3 — per-key retry budget. Set to 1 to disable retry (a failed work_fn is logged as :error instead of :gave_up).

• stale_after::Float64 = 600.0 — seconds before another master can reclaim a held lock as stale. Passed through to DataVault.acquire_running!.

• heartbeat_interval::Float64 = 60.0 — how often the per-lock heartbeat task refreshes DataVault's .running file. Enforced to be < stale_after (otherwise a live holder's lock could be reclaimed mid-work).

• log_level::Symbol = :info — event-log verbosity. At :info (default) the high-churn per-key :lock_busy and :key_start events are suppressed (their totals still ride in the :stage_done summary), keeping the JSONL log O(computed keys) instead of O(masters × keys) under multi-master contention. Set :debug to log them (e.g. to debug lock contention).

• stop_flag::Union{String,Nothing} — path to a sentinel file. When isfile(stop_flag) becomes true, run! and run_loop! stop dispatching new keys and return early. This is the infra equivalent of FiniteTemperature.jl's STOP_NOW_$JOB_ID mechanism, typically created by a SIGUSR1 signal handler in the batch script 60 s before Slurm kills the job.

  Defaults to ENV["SWEEPRUNNER_STOP_FLAG"], because the batch script that traps the signal and the driver that passes the option are different files, and the only thing they can agree on without one importing the other is the environment. Leaving the name to the caller meant every driver had to remember a variable this package never mentions; a driver that misspells it gets no error and no graceful stop, only a killed job. Pass stop_flag=nothing explicitly to opt out.

  Granularity: the flag is read between keys, not inside one. A key already in work_fn runs to completion, so the time between raising the flag and run! returning is bounded by the longest key, which the caller usually cannot predict.

• deadline::Union{Float64,Nothing} = nothing — an absolute time() past which no new key is handed out. The same mechanism as stop_flag with the same in-key granularity, and the reason to have both is that a deadline is set in ADVANCE: a batch job can subtract its longest expected key and the time its summary needs from the end of its allocation, where a flag raised reactively 60 s before the wall clock cannot buy back a key that runs for ten minutes.

  RunOpts(deadline = time() + 25 * 60)   # stop dispatching 5 min before a 30 min job ends

• defer_poll::Float64 = 30.0 — seconds run! waits before re-dispatching keys whose work_fn threw DataVault.ArtifactBusy (an artifact being built by another worker or job), when the previous pass made no progress. A deferred key costs no attempt.

Example

opts = RunOpts(max_attempts=5, stale_after=900.0, heartbeat_interval=30.0,
               stop_flag="/path/to/STOP_NOW_12345")
SweepRunner.run!(work_fn, vault, keys; opts)

run!(work_fn, vault, keys; opts=RunOpts(), load=nothing, observe=true) -> NamedTuple

Run work_fn(key) -> Dict for every key in keys, persisting through vault. Writes a structured JSONL event log at joinpath(vault.outdir, "events_<hostname>_<pid>.jsonl") — one file per master, so concurrent masters never contend on a single log.

load names the module(s) the worker processes need beyond the always-loaded seam (ParamIO/DataVault/SweepRunner) — typically the package or module that defines work_fn and the types it touches. Accepts a Module, Symbol, String, or a collection of them (e.g. load=MyModel or load=[MyModel, Statistics]). Under :distributed/:slurm, run! usings these in Main on every worker before fan-out, so a compute script no longer has to hand-roll the for w in workers(); remotecall_fetch(…, :(using …)); end broadcast. It is a no-op on the master (nprocs() == 1) and idempotent, so it is safe even when a project still broadcasts by hand.

Early skip (todo 10): on startup a stage-level Manifest is loaded. Keys already in the manifest are skipped — when all keys are done, the second run-through takes O(1) filesystem operations regardless of length(keys).

Source observations

With observe=true (the default) the master and every worker call DataVault.observe_sources before any key is dispatched, and each .done a process writes carries that process's token (observation=<token>). The observation records what the source looked like at run! start and its binding — how far the code that process had loaded was checked against it — so a marker never claims more than was checked. An observation that fails does not stop the run: the event log says why, and that process's markers read observation=unknown, as they do with observe=false.

Affinity

affinity is key -> value, and turns the fan-out from "any free worker takes the next key" into "a free worker PREFERS a key whose affinity value it has already handled". Pass it when work_fn memoises something per group in worker-local state, so a worker that stays on a group pays the load once instead of once per key.

run!(work_fn, vault, keys; affinity = k -> param(k, "system.L"))

A preference, not a partition: a worker is never idle while a key is pending, so a 200-key group does not serialise onto the worker that opened it. When a worker has nothing from its own groups left it takes from the group with the most work outstanding, which spreads workers over groups.

Only affects the pmap path; the sequential path already visits keys in order.

Returns (; stage, done, err, busy, gave_up, stop, skipped, total, stopped_by). stopped_by is :flag, :deadline, or nothing: a stage that finished every key reports nothing even if the deadline passed while its last key ran, since no key was ever held back by it. The full-done early exit returns the same field set rather than a shorter one.

Contract:

• work_fn is expected to be a pure function: given a DataKey, return a Dict payload to persist via DataVault.save!.
• Exceptions in work_fn are caught and logged; the corresponding key's .done file is not written, so re-runs will pick it up.
• The stage label used for logging is Symbol(vault.run).
• Manifest is monotonic: saved at end-of-stage with every newly completed key.

Parallel dispatch

If nprocs() > 1 (i.e. init_workers!(mode=:distributed|:slurm) has added worker processes), run! automatically fans out over the Distributed WorkerPool(workers()) via pmap. Each worker runs the per-key lock-acquire → work_fn → DataVault.save! → mark_done! pipeline independently. All filesystem operations (the .running lock, atomic JLD2 write, JSONL event log) are already NFS-safe, so concurrent workers inside one master are structurally consistent with multi-master operation.

If only the master is active (nprocs() == 1), run! falls back to the sequential loop from todo 11. This means the same compute.jl script is valid in three modes:

1. No init_workers! call at all → sequential on the master.
2. init_workers!(mode=:distributed) with addprocs(n) → local pmap fan-out.
3. init_workers!(mode=:slurm) inside a SLURM job → cluster fan-out.

Multi-master locking (several separate julia processes writing to the same vault) continues to work underneath either path because the lock layer (DataVault's .running) uses POSIX link() / atomic rename only.

run_loop!(work_fn, vault, keys; opts=RunOpts(), max_empty_rounds=3,
          idle_sleep=30.0, load=nothing, prerequisite=nothing) -> NamedTuple

Work-stealing loop that repeatedly calls run! until there is no more work to do. This is the infra equivalent of FiniteTemperature.jl's _work_loop driver.

The loop exits when:

• max_empty_rounds consecutive rounds produce zero new completions AND leave nothing held by a sibling, or
• opts.stop_flag is raised, or opts.deadline has passed.

A round that completes nothing but finds keys :lock_busy does NOT count toward max_empty_rounds until opts.stale_after has been waited out. Those keys are either being worked on by a live sibling, or held by one the wall clock killed, and stale_after is what separates the two: past it, acquire_running! reclaims the lock on the next attempt. Returning before then leaves the campaign short and reports nothing, because max_empty_rounds * idle_sleep (90 s by default) is an order of magnitude under stale_after (600 s).

Default parameters (max_empty_rounds=3, idle_sleep=30.0) are the battle-tested values from FiniteTemperature.jl.

load is forwarded verbatim to every run! call (see its docstring) — name the work module(s) the workers need and the loop handles the per-round broadcast.

Prerequisite

run! locks the KEY, so no two workers compute the same key. Work shared BETWEEN keys has to live inside work_fn, and there it has no protection at all: every worker that wants a setup not yet on disk builds it itself.

Pass a Prerequisite and that setup becomes its own key space, run to completion by run_prerequisite! before the dependent stage starts. It then gets the same locking, resume and provenance as any other stage, and its cost is recorded in its own payload instead of landing on whichever dependent key happened to run first.

run_loop!(work_fn, vault, keys;
          prerequisite = Prerequisite(prep_fn, prep_vault, derived_keys),
          opts = opts)

If the prerequisite does not complete, the dependent stage does NOT start, and the returned prerequisite field says why. Running it anyway would spend the allocation on keys whose setup is known to be missing.

SweepRunner does not know which dependent key needs which prerequisite key. The dependency is one level deep and resolved inside work_fn, so this is "all of the prerequisite, then all of the dependents", not a DAG.

affinity is forwarded verbatim to every run! call.

Returns (; ran, rounds, done, busy, stopped_by, prerequisite). busy is how many keys the last round found held by a sibling, so a caller can tell "everything is done" from "someone else still has work out". ran is false exactly when a prerequisite blocked the stage.

artifact_affinity(vault, name) -> Function

An affinity for run! that groups keys by the artifact name they need — its ParamIO.artifact_identity — so a worker keeps drawing keys that reuse the artifact it just built or loaded, and distinct artifacts start on distinct workers.

SweepRunner.run!(work_fn, vault, keys; affinity = artifact_affinity(vault, :ground_state))

Pair with DataVault.artifact!(...; wait=false) inside work_fn when keys outnumber workers: a key whose artifact is mid-build then throws DataVault.ArtifactBusy, run! defers it without spending an attempt, and the worker takes another key instead of blocking.

owner_token() -> String

The identity of ONE acquisition, stamped into .running by run! so a sibling can ask about it later. host:pid:nonce from DataVault, with :slurm<jobid> appended inside a Slurm allocation.

Call it per acquisition and do not cache it. The nonce is what tells a master's current hold from a hold it lost and retook, and a token broadcast once per process cannot make that distinction.

The Slurm field is what makes the question answerable ACROSS hosts: /proc only works for a holder on this machine, and the master of the job that was killed is usually somewhere else.

holder_liveness(owner) -> Symbol

:alive, :dead, or :unknown for the holder named by an owner_token.

:dead is only ever returned on positive evidence that the process is gone. Everything else is :unknown, including every error path: a wrong :dead would hand a live master's key to someone else, which is the one outcome the lock exists to prevent.

Two sources, in order:

1. a Slurm job id, when this process is ITSELF inside a Slurm allocation. The job is absent from the queue, so it has finished, been cancelled, or hit its wall clock.
2. the pid, when the holder is on THIS host and /proc exists. A recycled pid reads as :alive, which is the safe direction.

Prerequisite(work_fn, vault, keys; opts=nothing)

Work that a later stage's keys share. Its three fields are run!'s three arguments, because that is what it becomes: its own key space, its own vault, its own payloads.

opts overrides the dependent stage's RunOpts for the prerequisite alone, which is usually about stale_after: the shared setup is typically the slow half, and a lock reclaimed mid-build is the thing this exists to prevent.

stop_flag and deadline are not stage knobs: they bound the JOB, and a stage may not loosen a bound the job set. deadline therefore takes the TIGHTER of the two, and stop_flag takes the caller's whenever it has one. A prerequisite cannot redirect or outlive either.

That asymmetry is deliberate for stop_flag: RunOpts resolves its default from ENV["SWEEPRUNNER_STOP_FLAG"], so an opts here that never mentions stop_flag can still carry one, and === nothing does not mean "the author left it unset" for that field.

Build keys by projecting the dependent key space onto the axes the setup actually depends on (ParamIO.project), so the two spaces cannot drift apart by hand.

run_prerequisite!(p; opts=RunOpts(), load=nothing, poll=30.0) -> NamedTuple

Run p until EVERY one of its keys is done, and report whether that happened:

(; complete, remaining, done, waited, rounds, stopped_by)

complete is the only field a caller has to read. The rest say why not: remaining keys are undone, waited counts the rounds spent purely waiting for a sibling master.

This is a barrier, not a work loop, and the difference is what it does when it has nothing left to take. run_loop! stops after max_empty_rounds empty rounds, which is right when the keys are independent. Here the dependent stage cannot start until the setup exists, so a round that finds every remaining key locked by a sibling SLEEPS and goes again.

It terminates on: every key done; no progress AND no key held by a sibling (a genuine failure); opts.stop_flag; opts.deadline. A live sibling building a slow setup is waited for, which is the point; a dead one is bounded by stale_after, after which its lock is reclaimable.

Finding(layer, severity, where, message)

One reason a campaign cannot (or should not) start.

layer is the study's own vocabulary — :config, :grid, :injective, :crossphase in the original — and exists so that a refusal is attributable. Layering that cannot distinguish two different causes is decoration; a study's tests should assert that a deliberately broken config is rejected by the RIGHT layer, not merely rejected.

PreflightReport(findings) -> PreflightReport

Everything preflight found, in the order the layers were run.

Carries findings::Vector{Finding} and layers, the display order. isempty means the campaign is clean; launchable is the gate to branch on, because a report can be non-empty and still launchable when every finding is a warning.

launchable(r) -> Bool

True when nothing at :error severity was found. Warnings do not block.

Callers should gate on this and exit non-zero rather than printing a verdict for a human to grep: a grep over output reports a parse error as success.

check_injective!(findings, layer, label, name, paths)

Assert that paths has no duplicates, i.e. that distinct parameter points map to distinct files.

This is the check worth having. Under a content-blind float rendering (%.2f), h = 0.002 and 0.004 both render h0.00, so two points share a directory — and not only their observables. They share the status marker, which means the second point is reported complete without ever running. A sweep of 168 points then finishes as 132 with no error anywhere.

Pass the status paths as well as the data paths. Checking only the data half catches the overwrite and misses the skipped work, which is the worse of the two.

check_opens!(findings, layer, label, open_fn) -> Union{Any,Nothing}

Run open_fn() and, if it throws, record the failure instead of propagating it.

Opening a vault is itself a thing that fails — most commonly a DataVault log.toml conflict when two stages of one campaign share a project_name but declare different path_keys. A driver that opens several vaults in a row dies at startup on that, so it belongs in the report next to everything else rather than as a stack trace out of the checker.

representative_keys(keys) -> Vector{DataKey}

One key per distinct parameter point.

ParamIO.expand enumerates (point × sample), but a path is a property of the point — the sample only varies the filename. Counting collisions over all keys would therefore report n_samples false collisions for every real one.

on_grid(x, step; atol=1e-9) -> Bool

Is x a multiple of step?

A value that is not is one the evolution steps past: the artefact named after it is never written, and the failure surfaces much later as a missing file in a downstream stage, long after the compute is spent.