ParallelManager.jl

HPC experiment runtime for Julia — multi-master safe, crash-recoverable, Pkg.test()-fast.

Wraps ParamIO.jl and DataVault.jl with a unified run! that handles parallel dispatch, advisory locking, .done rollups, structured event logging, and retry.

Pages

• Quick start
• • Install
  • Hello world
  • Phase chaining without Stage/DAG
  • Parallel execution
  • SLURM
• Architecture
• • Layers
  • Why a separate layer?
  • Module map
  • Key identity: canonical(::DataKey)
  • The run! pipeline
  • Concurrency model
  • Why no println
  • Why no Stage / DAG
• API reference
• • AtomicIO
  • EventLog
  • Manifest
  • InitWorkers
  • Run
• Guides
• • 1. Analyzing the event log
  • 2. Running multiple masters
  • 3. Recovering from a crashed master
  • 4. Incremental re-runs
  • 5. Debugging a single key
  • 6. Custom retry policy
  • 7. Running without SLURM
  • 8. Cleaning a corrupted manifest

Highlights

• Multi-master safe — several julia processes can hit the same vault root without double-executing any key (DataVault .running advisory lock).
• Crash recovery — kill -9 a master mid-run and the next run! picks up where it left off via heartbeat-based stale lock reclaim.
• Early skip — full-done re-runs take O(1) filesystem operations (a single manifest.jld2 read), not O(N) per-key .done stats. 3600-key warm re-run ≈ 3.5 ms.
• Structured events — JSONL event log atomic across concurrent writers; per-item println is a non-goal, by design.
• One entry point for all parallel modes — init_workers!(mode=:auto) dispatches to :sequential / :threads / :distributed / :slurm depending on environment.
• Pure work functions — your physics is a plain (DataKey) -> Dict, IO/locking/logging live in the runtime.

30-second tour

using ParamIO, DataVault, ParallelManager

spec  = ParamIO.load("config.toml")
keys  = ParamIO.expand(spec)
vault = DataVault.Vault("config.toml"; run="phase1")

ParallelManager.init_workers!(mode=:auto)
work_fn = key -> Dict{String,Any}("x" => compute(key))
ParallelManager.run!(work_fn, vault, keys)

Re-running the same script after completion emits :skip_complete and exits in milliseconds regardless of length(keys).

Pain points it answers

See also

• ParamIO.jl — config TOML parsing and DataKey enumeration
• DataVault.jl — Vault struct, atomic JLD2 save, .done markers
• templateHPC.jl — clone-to-start scaffold that wires all three together