Release decisions

The payoff for marking anything at all: "changing an experimental name is not breaking" stops being an argument in a pull request and becomes a function call.

Snapshot, then compare

At release time, write down what the package promises and commit it:

ExperimentalAPI.write_snapshot("api.toml", MyPackage)

module = "MyPackage"
version = "0.4.2"
stable = ["adapt", "measure"]

[experimental.render_report]
reason = "reads Test's internal result tree"
tracking = "https://example.invalid/issues/12"

On the next release, compare:

d = ExperimentalAPI.compare(ExperimentalAPI.read_snapshot("api.toml"), MyPackage)
ExperimentalAPI.isbreaking(d) && error("breaking: $(d.removed_stable) removed, $(d.demoted) demoted")

What counts as breaking

move	breaking?
`removed_stable`	yes	a settled name is gone
`demoted`	yes	a settled name is now called experimental
`removed_experimental`	no	this is what marking a name buys
`promoted`	no	experimental → settled; the direction all of this is for
`added_stable`, `added_experimental`	no

Removing a name you had declared experimental lands in removed_experimental and does not make the release breaking. That is the whole contract, and it is worth being precise about why it is legitimate: the notice was given in the source, at the definition, before the removal, in a form a downstream author's own test suite could have read. It is not a retroactive excuse.

demoted is the mirror image and is counted as breaking for the same reason. Taking a name you shipped as settled and calling it experimental afterwards does not un-promise it; it changes what callers were told after they believed it.

Names, not signatures

Warning

compare reads name sets. A name present in both snapshots whose arguments, keyword arguments or return type changed is a breaking change, and this cannot see it. Read the diff as a floor on breakage, never as a clearance.

This is not a defect to be fixed later by making the snapshot signature-aware — or at least, not only that. Signature stability is a strictly harder question (a method table is not a set of names, and "compatible" is not a set operation), and a tool that answered it approximately would be worse than one that declines: a green light nobody should have trusted.

Which is also why the snapshot layer of this package is itself declared @experimental. The file format above was a guess, and making it signature-aware changed it.

The method-level half

The snapshot carries two more keys, and compare_methods reads them:

stable_methods = ["adapt(::Model, ::Grid)", "measure(::Model)"]

[experimental_methods."fetch_value(::Heisenberg, ::Energy)"]
reason = "extrapolated from a finite-size sweep; no reference value"

The key is a signature a human can read in a committed file, and it carries the argument types and the keyword names — so a method whose arguments or keywords changed reads as one key removed and another added, and isbreaking says so. That closes most of the blind spot above. What stays open is stated as a constant rather than left to be discovered: compare_methods_sees_keywords is true for names and the docstring says defaults are invisible, because a default lives in the body and moves nothing.

Note also what stable does not do: a name with four methods of which one is marked stays in the covenant. Declaring one dispatch path unsettled is not a licence to remove the name.

When a mark may go

A mark that can only ever be added is a decoration. until= is what makes it a work item:

@experimental(
    "no reference value yet",
    since = v"0.1.0",
    tracking = "https://example.invalid/issues/12",
    until = () -> isfile(joinpath(@__DIR__, "..", "test", "refs", "energy.toml")),
    energy(β) = 2β,
)

ready_to_promote calls that predicate; promotable lists the marks whose exit condition is met. A mark with no until= is false always — reporting it "ready" because it happens to be covered by a test would be inventing a criterion the author did not state — and those are reported separately by marks_without_exit, which is the different finding they are.

age reads since on the axis a version bump is breaking along, and stale_since is the list a CI job turns into a nag. exceeds_mark_cap is the ratchet, in the shape test_surface's skip already has.

Provenance next to the result

ExperimentalAPI.stamp("figures/energy_sweep.provenance.toml") do
    sweep(model; βs = 0.05:0.05:2.0)
end

The end state this is all for: a figure's directory says which unvalidated code paths produced it, in plain TOML with the reason, the counts and stamp_versions beside them. A year later the package may not resolve, and a provenance record nobody can open is not one.