ParamIO.jl

ParamIO — config TOML を読み込み、DataKey のリストに展開する

ファイル構成:

• core/types.jl データ構造とエラー型
• core/load.jl TOML 読み込みと継承マージ
• core/expand.jl Cartesian 展開と sweep 順序制御
• core/project.jl キー空間の座標射影
• core/artifacts.jl セル間で共有する中間成果物の識別 ([artifacts.<name>])
• core/format.jl DataKey からパス文字列を生成
• util/grid.jl {start,stop,length|step} grid spec → sweep リスト展開
• util/flatten.jl サブテーブルのフラット化、ドット記法分解
• util/path_keys.jl path_keys の自動解決と検証

AmbiguousPathKeyError

Raised when a plain leaf name (e.g. "N") appears in multiple groups (e.g. both system.N and model.N) and the user has not disambiguated with dotted notation.

ArtifactSpec

An intermediate result that many sweep points share, declared in [artifacts.<name>]:

[artifacts.ground_state]
depends_on = ["run.U", "run.D", "run.cutoff"]   # the parameters it is a function of
version    = 2                                   # bump when the code that builds it changes
per_sample = false                               # true if it also depends on the sample index

Its identity is the key projected onto depends_on plus version — see artifact_key and artifact_identity. A parameter left out of depends_on does not invalidate it, so an artifact that reads a knob it does not declare is silently reused across that knob's values.

Fields: name, depends_on (resolved to dotted keys), version, per_sample.

AxisFloatFmt

Per-axis float rendering decision produced by build_axis_formats.

• fallback == false: render every value of the axis with %.<precision>f — the MINIMAL uniform fixed precision that renders the axis's whole value set both LOSSLESSLY and INJECTIVELY.
• fallback == true: render each value as its own shortest round-trippable decimal (string(v)), which is injective by construction. Used only when no fixed precision ≤ 17 can represent the axis losslessly (e.g. an extreme scale spread such as [1.0, 1e-18]).

ConfigSpec

Parsed representation of a config TOML.

Fields:

• study: project-level metadata
• path_keys: ordered keys used to build directory paths (dotted or plain)
• paramsets: flattened [[paramsets]] blocks; each is a Dict{String,Any} where sub-table keys are prefixed as "group.leaf"
• sweep_order: optional explicit sweep ordering for expand. If empty, path_keys is used as the default sweep order.
• float_format: how float path segments are rendered — "fixed2" (default: the legacy %.2f) or "auto" (content-aware, per-axis injective+lossless; see build_axis_formats / format_path). Set via [datavault] float_format. Never affects canonical.
• artifacts: [artifacts.<name>] tables, by name — see ArtifactSpec. Empty when the config declares none.

DataKey

A single point in the parameter space, including sample index. params keys match the dotted/plain scheme used in the config's path_keys.

DiagnosticReport

Result of diagnose: a static, non-throwing audit of a parameter grid.

Fields:

• n_points: number of DISTINCT param points (post-dedup; == length(expand(spec)) ÷ n_samples)
• n_samples: study.total_samples (each point is repeated this many times by expand)
• duplicates: (point, multiplicity) for every assignment the raw per-block enumeration yields ≥2× (overlapping value lists / redundant [[paramsets]])
• collisions: (path, points) for every format_path output ≥2 DISTINCT points map to — a real run would silently overwrite one with the other
• invisible_params: names of params that vary (>1 distinct value) but are NOT covered by path_keys — the ROOT CAUSE of a collision
• ambiguous: captured AmbiguousPathKeyErrors (surfaced into the report, never thrown)
• ok: true iff every check is clean (all of the above empty)

StudySpec

Project-level metadata extracted from [study] in a config TOML.

_Literal(value)

A flatten-time marker meaning "value is a fixed parameter value, not a swept axis". expand classifies it as fixed (it is not an AbstractArray) and unwraps it to value in every DataKey, so a list can be a value ({const = [1.0, 0.5]}) instead of a sweep.

_axis_format(vals::Vector{Float64}) -> AxisFloatFmt

Choose the injective+lossless rendering for ONE float axis, given its DISTINCT value set. Precision = the max over values of _needed_decimals (so every value is lossless at that uniform precision; losslessness ⇒ injectivity, since two distinct values that both round-trip cannot share a string). If any value needs more than _MAX_FIXED_DECIMALS, or the uniform precision is somehow not injective, fall back to per-value shortest strings. Injectivity is @asserted.

_cartesian_product(flat, order) -> Vector{Dict{String,Any}}

All Cartesian combinations of array-valued keys. Scalars stay fixed.

order specifies the outermost-to-innermost iteration order. Sweep keys not in order are appended at the end in sorted order, so the result is always deterministic regardless of Dict iteration order.

_check_canonical_key(k)

Guard the canonical grammar: keys must not contain the reserved delimiters ; or = (or a newline), otherwise two distinct DataKeys could serialize to the same string and collide on one on-disk identity. Config-sourced keys are always group.leaf (delimiter-free); this protects hand-built DataKeys.

_covered_by_path_keys(param_key, path_keys) -> Bool

true when some entry of path_keys selects param_key: an exact (dotted or top-level) match, or a plain (dot-free) pathkey equal to the param's leaf. Mirrors `formatpath's plain-leaf lookup, so a param is "visible" here iffformat_path` would put it in the directory name.

_differing_keys(pts) -> Vector{String}

The param keys whose value is not constant across pts — i.e. the axes on which a set of path-colliding points actually differ. These are exactly the params that should have appeared in path_keys to keep the points apart.

_expand_grid(v) -> v

Expand a grid spec into the explicit sweep list it stands for; pass any non-grid value through unchanged. Three forms (start/stop inclusive):

{start=1.0,  stop=4.0, length=31}                 # linspace: 31 points  → Vector{Float64}
{start=16,   stop=128, step=16}                   # 16:16:128            → Vector{Int}
{start=1e-3, stop=1.0, length=7, scale="log"}     # 7 log-spaced points  → Vector{Float64}

length (point count, ≥ 2) and step are mutually exclusive — exactly one is required. The linear length form is always Float64 (a linspace); the step form follows Julia's a:s:b, so an all-integer step keeps Int. scale="log" needs length (a geometric grid has no constant step) and start, stop > 0.

Errors loudly on a malformed spec — a typo'd key (lenght) leaves the table with neither length nor step, so a mistyped grid raises rather than silently degrading to a fixed table-valued parameter.

_flatten_block(block) -> Dict{String,Any}

Flatten one [[paramsets]] block: sub-tables become dotted top-level keys.

Example: {"system" => {"N" => [24,48], "chi" => 40}} → {"system.N" => [24,48], "system.chi" => 40}

Leaf-table specs are resolved here (see grid.jl), so the rest of the pipeline sees ordinary values: a grid ({start, stop, length|step}) becomes a swept list, a const ({const = X}) becomes a fixed _Literal(X). A spec is treated as a leaf, not a sub-table to descend into — _is_spec distinguishes it from a parameter namespace.

_flatten_value(v) -> v

Resolve a leaf value: a const spec → a _Literal (fixed), a grid spec → an explicit list (swept), anything else unchanged.

_is_const(v) -> Bool

true when v is a const spec: a table whose only key is const. {const = X} pins X as one fixed value (never swept) — the explicit counterpart to list ⇒ sweep.

_is_grid(v) -> Bool

true when v is a grid spec: a table carrying start and stop whose keys are all drawn from start/stop/length/step/scale.

The all-keys rule is what keeps an ordinary parameter namespace that merely contains a start/stop parameter from being mistaken for a grid — e.g. [paramsets.window] with start=0.0; stop=1.0; dt=0.01 has a non-grid key (dt), so it stays a namespace and flattens normally.

_is_spec(v) -> Bool

true when v is any leaf-table spec — a grid ({start, stop, …}, a swept list) or a const ({const = X}, a fixed value). _flatten_block treats a spec as a leaf, not a namespace to descend.

_load_raw(path, inherit, chain) -> Dict{String,Any}

Parse the TOML at path, recursively merging [base] inherit parents so multi-level (grandparent and deeper) chains compose correctly. chain tracks the absolute paths visited so far and is used to detect inheritance cycles.

_lookup_name(name, available) -> String | Vector{String} | Nothing

The outcome of resolving name, without deciding what to do about it:

• a String — the key it selects;
• a Vector{String} — the groups a plain leaf appears in, when there is more than one;
• nothing — nothing matched.

Split out from _resolve_name so that the caller which treats "absent" and "ambiguous" as ordinary outcomes does not have to reach them through a catch. A try around a raise is a filter that cannot say what it caught: it takes InterruptException and a genuine bug with the same hand as the two conditions it means to allow.

_merge_configs(parent, child) -> Dict{String,Any}

Merge two raw TOML dicts; child overrides parent. [[paramsets]] arrays are concatenated (parent first).

_parse_artifacts(raw, flat_blocks) -> Dict{String,ArtifactSpec}

Read the [artifacts] table. Refuses what would make the identity wrong rather than merely unusual: an unknown field (a typo of depends_on would otherwise declare an artifact of NOTHING, shared by every cell), an empty depends_on, and a name that some [[paramsets]] block does not carry (the projection is then undefined on that block's keys).

_resolve_name(name, available) -> String

The key in available that name selects. Throws AmbiguousPathKeyError when a plain leaf sits in more than one group, and an error naming what is available when nothing matches.

Returns the KEY, not the value, so callers that need the group prefix (or the value, or neither) all get what they need from one rule.

_safe_format_path(key, path_keys) -> Union{String,Nothing}

format_path(key, path_keys), returning nothing instead of throwing (e.g. a path_key absent from a heterogeneous block). A point with no formattable path cannot participate in a collision, so it is simply skipped — keeping diagnose non-throwing.

_split_dotted(k) -> (group, leaf)

Split a dotted key like "system.N" into ("system", "N"). A plain key like "N" becomes ("", "N").

_validate_path_keys(path_keys, flat_blocks)

Check that every path_key exists in at least one paramset block, either as exact match (dotted) or unambiguous plain leaf name.

artifact_identity(spec, name, key) -> String

The artifact's on-disk identity as a stable string:

<name>@v<version>|<canonical(artifact_key(spec, name, key))>

Deterministic across Julia versions and insertion orders, because canonical is. DataVault hashes this for a directory name and records it in full beside the artifact.

artifact_key(spec, name, key) -> DataKey

The point in the artifact's own key space that key needs: key projected onto the artifact's depends_on, with the sample index kept only if it declared per_sample = true (otherwise 1). Two cells get the same artifact exactly when this is equal.

artifact_keys(spec, name) -> Vector{DataKey}

Every distinct artifact point the sweep needs — for building them ahead of the cells that read them, or counting them.

build_axis_formats(spec::ConfigSpec) -> Dict{String,AxisFloatFmt}

Compute, once from expand(spec), the content-aware float format for every FLOAT path_key axis. Integer / string / other axes get no entry (they are formatted exactly as in fixed2). The result is passed to the 3-arg format_path to render auto-mode paths.

Each entry guarantees: distinct values of that axis → distinct, round-trippable path segments (the anti-collision guarantee fixed2 cannot make).

canonical(key::DataKey) -> String

Return a string representation of key that is:

• Deterministic — same key always yields the same string, regardless of how params was constructed.
• Order-independent — permuting insertion order of params does not change the result (fields are sorted by key name).
• Stable across Julia versions — does not depend on hash, which is only guaranteed stable within a Julia version.

This is intended for use as a directory-safe identity by downstream packages that need a lookup key (e.g. SweepRunner.Manifest).

Schema(固定。変更不可)

<k1>=<v1>;<k2>=<v2>;...;#sample=<n>

• Keys are sorted lexicographically.
• Values are formatted per _canonical_value:
  • Int, Bool: decimal digits / true / false
  • Float64, Float32: repr(v) (round-trippable, e.g. "1.5", "NaN")
  • String: double-quoted, inner " and \ escaped
  • Symbol: :name
  • Nothing: nothing
  • Anything else: repr(v) as a last resort
• Sample index is always appended as #sample=<n> so it cannot collide with user params.

Examples

julia> using ParamIO

julia> k = DataKey(Dict{String,Any}("N" => 8, "J" => 1.0), 3);

julia> canonical(k)
"J=1.0;N=8;#sample=3"

julia> canonical(DataKey(Dict{String,Any}("N" => 8, "J" => 1.0), 3)) ==
       canonical(DataKey(Dict{String,Any}("J" => 1.0, "N" => 8), 3))
true

diagnose(spec::ConfigSpec) -> DiagnosticReport
diagnose(config_path::AbstractString; inherit=true) -> DiagnosticReport

Statically audit the parameter grid a config expands to — BEFORE any run is submitted — and return a DiagnosticReport. Pure logic: performs zero filesystem writes. The config_path form only reads the TOML through load; the ConfigSpec form touches the filesystem not at all.

Four checks, each a field on the report:

1. duplicate points — the raw per-block Cartesian enumeration yields the same assignment ≥2× (overlapping value lists or redundant [[paramsets]]). expand silently dedups these; here they are surfaced with their multiplicity.
2. path collisions — two DISTINCT points map to the same format_path(key, path_keys), so a real run would silently overwrite one result with the other.
3. invisible params — params taking >1 distinct value across the grid that are NOT covered by path_keys. This is the ROOT CAUSE of (2): a swept axis with no directory segment to separate its points. The single most useful line in the report.
4. ambiguous path keys — an AmbiguousPathKeyError (a plain leaf shared by multiple groups) that would otherwise be thrown by load is captured into the report instead of thrown.

ok is true iff all four checks are clean.

Example

report = ParamIO.diagnose("config.toml")   # loads + audits, never writes
report.ok || show(report)                  # inspect the offending detail

expand(spec; sweep_order=nothing) -> Vector{DataKey}

Expand all [[paramsets]] blocks via Cartesian product, deduplicate across blocks, and return one DataKey per (param_point × sample).

Sweep ordering

The Cartesian product is evaluated outermost-to-innermost in a deterministic order. The default order is, in priority:

1. The sweep_order keyword argument (if provided)
2. spec.sweep_order (set via [datavault] sweep_order in the TOML)
3. spec.path_keys
4. Sorted leftover keys

Sweep keys not listed in the chosen ordering are appended at the end in sorted order, so the result is always deterministic.

Deduplication

Two blocks that produce the same point yield one key, so length(expand(spec)) is not the product of the axis lengths whenever blocks overlap. The first block to produce a point also fixes its position, which is what makes a small leading block a way to order a long acquisition: put the slice to close first at the top, overlapping a later broader block, and only the position survives.

Sameness is canonical, the on-disk directory identity, not Dict equality: 1 and 1.0 get different directories and are both kept. expand_report returns how many were collapsed, which is what separates "my new block added nothing" from "my new block was not read".

Example

spec = ParamIO.load("config.toml")
keys = ParamIO.expand(spec)                                 # uses path_keys order
keys = ParamIO.expand(spec; sweep_order=["model.h", "system.N"])  # explicit

expand_report(spec; sweep_order=nothing) -> NamedTuple

expand's keys together with what deduplication removed to get them:

• keys, points: the keys, and the distinct parameter points behind them (length(keys) == points * spec.study.total_samples);
• duplicates: points dropped for repeating an earlier one;
• per_paramset: one (; produced, kept, duplicate) per [[paramsets]] block, in block order.

per_paramset[i].kept == 0 is a block that contributed nothing, which a total alone cannot tell from a block that was never read.

format_path(key, path_keys, axis_formats) -> String

auto-mode path builder: identical to the 2-arg format_path except float segments are rendered with the content-aware, provably-injective per-axis format in axis_formats (from build_axis_formats) instead of %.2f. Integer / string / other segments are byte-for-byte identical to fixed2. A float axis with no entry in axis_formats degrades to the fixed2 rendering.

format_path(key, path_keys) -> String

Build a compact path segment from a DataKey.

Examples:

• plain key "N", value 24 → "N24"
• dotted key "system.N", value 24 → "sysN24" (3-char group prefix)
• float value → two decimal places: "g0.50"

load(path; inherit=true) -> ConfigSpec

Read a TOML config and return a ConfigSpec. If [base] inherit = "..." is present, merge the parent file first (parent [[paramsets]] come first in the union).

The optional [datavault] sweep_order key may be set in the TOML to override the default sweep enumeration order (which is path_keys).

param(key, name) -> Any
param(key, name, T) -> T

The value name selects in key, resolved by the same rule as a path_key: an exact match on the dotted name, else a unique match on the leaf. See _resolve_name.

work_fn is otherwise written as Float64(key.params["system.kbT"]), and both halves of that are a hazard.

A misspelled name is a bare KeyError naming only the string that was wrong. It is raised inside the work function, which is running on a worker, dispatched by run! — so the config that has the right spelling and the function that has the wrong one are in different files, and nothing compares them until a point is computed. param reports what the key does carry instead.

The type is the other half. TOML gives 2 as Int64 and 2.0 as Float64, so a config edited from [2.0, 2.5] to [2, 3] changes what reaches the kernel without changing anything a reader would look at.

param(key, name, T) converts and then, for numbers, checks the value survived it, so any T that would have altered it is refused rather than returned. convert alone is not enough: it raises for Int from 2.5, but silently rounds for Float64 from 2^53 + 1 and for Float32 from 2.1 — the same silent change of number, one type down from the one this exists to prevent.

The check stops at the numeric tower on purpose. A type with no == falls back to identity, so applying it further would refuse a correct convert(::Type{Celsius}, ::Real) for returning a different object rather than a different value. Outside numbers, param gives you whatever convert gives you.

a  = param(key, "system.a", Float64)
n  = param(key, "numerics.nsteps", Int)
kT = param(key, "kbT", Float64)          # leaf form, when it is unambiguous

project(spec, axes; total_samples=spec.study.total_samples) -> ConfigSpec

The spec over axes alone: every other parameter is dropped, so the keys expand returns from it are the distinct values of key -> (axes...) across the original sweep.

Names resolve as they do for param: exact dotted match, else a unique leaf. An axis absent from any one [[paramsets]] block is refused, because the projection is then undefined on that block's keys rather than merely narrower.

DataKey.sample is not a parameter, so no entry of axes can select it; total_samples is how a projection that does not depend on the sample index collapses it to 1.

states = expand(project(spec, ["system.L", "model.lambda", "thermal.beta"]))

project(key::DataKey, axes; sample=key.sample) -> DataKey

key restricted to axes: the same parameters under the same dotted names, every other one dropped. Names resolve as for param. The key-level counterpart of project(spec, axes).

resolve_path_keys(flat_blocks) -> Vector{String}

Auto-detect path_keys from flattened paramset blocks (sorted). Raises AmbiguousPathKeyError when the same leaf name appears in multiple groups and the caller has not supplied explicit dotted notation.