API — checking and releasing

Audit

What audit found. The name fields are sorted Vector{Symbol}, and every name in surface appears in exactly one of foreign, documented, unaccounted or declared-and-not-documented — the partition partition_holds checks.

unaccounted is the one a test asserts is empty. dangling needs no external oracle to be wrong: it is the module contradicting itself.

aqua_compatible_names(m::Module) -> Vector{Symbol}

The names Aqua.test_undocumented_names reports and this package's audit accounts for.

The two tools ask different questions, and both are legitimate. Aqua's is two-valued: a public name has a docstring or it fails. audit has a third answer — a mark — so a name that is declared unsettled is still undocumented but is not unaccounted.

This is the difference, computed, so a project running both can see exactly which names it would have to argue about. Empty means the two agree, which is the state a package should be aiming for: a mark records that a shape is unsettled, and that is never a reason to say nothing about what the name does.

audit(m::Module; methods = true) -> Audit

Report the public names of m that are missing prose, a mark, or both — and the methods it contributed to other modules' generics, which no name-level check can see.

julia> audit(Pinax).undocumented
15-element Vector{Symbol}:
 :completeness_overview
 :dump_test_report
 ⋮

A mark is not a substitute for a docstring. The two are independent accounts of a name and both are owed: the docstring says what it does, the mark says whether the shape is settled. A public name that carries a mark and no prose appears in undocumented exactly as one with neither does, and test_surface fails on it.

Also reports dangling: marks on names that are not public. That check needs no reference implementation to be right, because the module is disagreeing with itself.

methods = false skips the method-level search, which is the expensive half — it scans the generics of every loaded module. The name-level answer is unchanged by it.

What this cannot see

• Signatures of its own names. A name that stays present while its arguments change is invisible here; compare_methods is the release-side answer.
• Prose quality. A docstring exists or it does not; isdocumented reads no further.
• Names public only inside an extension, which is a separate module — reported in extensions and audited by passing it to audit in its own right. Avoidable altogether: declare the function and its docstring in the parent and let the extension add only the method, which is what this package does for test_surface.

See test_surface to run this as a test, and snapshot to carry the result into a release decision.

contributed_methods(m::Module) -> Vector{Method}

The methods m defines on generics it does not own.

audit's foreign bucket says "this name is bound elsewhere, so documenting it is not our problem". A method we wrote on such a name is the exact opposite: it is entirely our problem, and it is invisible to names(m).

extends_base(mm::Method) -> Bool

Whether mm extends a generic owned by Base or Core.

The line test_surface draws by default when it asks whether a contributed method is accounted for. A method on Base.show or Base.== implements a protocol whose documentation is Base's, and requiring a docstring on each of them trains a project to turn the whole check off. A method on another package's generic is a downstream extension only this package can describe — fetch_value(::Heisenberg, ::Energy) is surface in a way show(io, ::Audit) is not.

Stated as a rule about who owns the generic, not as a list of names: a list of interface functions is not closed under the ones Julia adds next.

isdocumented(m::Module, name::Symbol) -> Bool
isdocumented(m::Method) -> Bool

Whether name, or the method m, has a docstring.

A re-exported name counts: the lookup follows the binding to the module the name actually comes from, so a package that puts a dependency's name on its own surface is not asked to re-document it. audit still reports those separately as foreign, because who owns a name and who documented it are different questions.

Docstrings are keyed by signature, so the method form is answerable and is not the same question: a documented generic with an undocumented method is normal, and for a package whose surface is methods on somebody else's generic it is the only question there is.

This answers whether prose exists, never whether it is any good. A docstring reading "TODO" is documented as far as this package is concerned.

own_methods(m::Module) -> Vector{Method}

Every method m defines, on its own generics and on other modules' alike.

Julia indexes methods by generic, not by module, so this is a search: the generics bound in m (including the ones it imported to extend), the generics m's own marks name, and the exported-or-public names of every loaded module. That last set is what catches Base.show(io, ::Widget), written with a qualified name that leaves no binding in m.

partition_holds(a::Audit) -> Bool

Whether every public name landed in exactly one bucket.

foreign, documented, unaccounted, and the declared-but-undocumented remainder must cover surface exactly once. The invariant is stated in Audit and is easy to break from the outside — every new field is one more thing a reader assumes partitions the surface — so it is checkable rather than only written down.

stable(m::Module) -> Vector{Symbol}

The public surface of m minus the names that are wholly @experimental — the names whose removal or renaming is a breaking change.

The covenant, in other words. snapshot writes this down so two releases can be compared; isbreaking is what reads the comparison.

"Wholly" is the load-bearing word. A name is out of the covenant when a whole-name declaration covers it, or when every method behind it is marked. A name with four methods of which one is marked stays in — the author declared one dispatch path unsettled, which is not a licence to remove the name. See stable_methods for the finer unit.

stable_methods(m::Module) -> Vector{Method}

The methods m defines that carry no mark — the covenant at the unit a call site actually reaches.

The name-level stable cannot see this: fetch(::Ising, ::Energy) and fetch(::Heisenberg, ::Energy) are one name and two promises. Covers both m's own generics and the methods it contributed to somebody else's; see contributed_methods.

surface(m::Module) -> Vector{Symbol}

The public surface of m: every name it exports or declares public, sorted, excluding the module's own name.

This is names(m), given a name — since Julia 1.11 that list is precisely "exported or public", which is the surface a stability claim has to be made about. Names reachable only as M.internal_thing are not here, and are nobody's promise.

unaccounted_methods(m::Module) -> Vector{Method}

The methods m contributed to other modules' generics that carry neither a docstring nor a mark.

The method-level twin of audit(m).unaccounted, and the finding for a package whose surface is methods rather than names. A name-level audit reports unaccounted = [] for such a package while having looked at none of them.

Verification

How much of one marked definition the current run exercised.

missing is not zero. A run without --code-coverage has nothing to say about how much of a definition ran, and reporting 0.0 there would flag every marked definition in every ordinary run.

Whether it ran at all is a different question, and this package already answers it exactly: the probe. A marked definition whose flag never fired is 0.0 whatever the line counters say — which is not a refinement but a correction. Measured on 1.14.0-DEV.3115: --code-coverage now emits a counter for the definition line of a method nothing ever called, so a one-line definition comes back "fully covered" on the strength of having been defined. Up to 1.12 that line had no counter at all, so the two versions disagree about the same file, and only one of them can be read as "the suite ran this".

coverage(m::Module, name::Symbol) -> Union{Float64,Missing}

The fraction of name's definition that this run executed, or missing if coverage is off.

Partial coverage comes back partial: a definition whose error branch is never taken is not verified, and reporting it as 1.0 would be the false pass this whole package exists to remove.

coverage_enabled() -> Bool

Whether this process was started with --code-coverage.

The question coverage has to ask before reporting a number: without it there are no counts at all, and the honest answer is missing.

flush_coverage() -> Union{Dict,Nothing}

Write this process's coverage counters out and read them back, or nothing if coverage is off.

Julia writes them at exit, which is too late for a test to assert on. The same C entry point the exit hook uses is called here instead, into a temporary file that is read and removed.

stale_marks(m::Module) -> Vector{Mark}

The marks whose recorded line no longer holds a @experimental declaration.

A mark records where it was written. An edit above it moves the code and not the record, and every join keyed on that line — coverage here, and anything downstream reading mk.file/mk.line — then describes the wrong lines silently. Re-loading the module fixes it, which is why this reports rather than repairs: a stale mark means the module on disk and the module in memory differ.

unverified(m::Module) -> Vector{Mark}

The marks whose definitions this run never executed at all.

The worst case, and the one worth a separate verb: code that is both unvalidated and untested.

Does not need --code-coverage. The signal is the probe the mark already emits, which is exact and costs nothing; coverage adds the partial fraction, which is a different question. A mark on a struct, a const or a name list carries no probe and is not listed — there is nothing to enter.

verification(m::Module) -> Vector{Verification}

One Verification per mark in m, sorted by name.

Data, not a printout — the same convention audit follows. Flushes the process's coverage counters once and joins them against every mark, so asking about twenty marks costs what asking about one does.

age(m::Module, name::Symbol, current::VersionNumber) -> Union{Int,Missing}
age(mk::Mark, current::VersionNumber) -> Union{Int,Missing}

How many breaking releases the mark has been standing for, or missing if it records no since.

Counted on the axis a version bump is breaking along, which under Julia's 0.x convention is the minor component and after 1.0 the major one. since = v"0.1.0" seen from v"0.9.0" is 8.

since exists so a mark cannot quietly become permanent. This is the function that reads it; see stale_since for the list form a CI job asks for.

exceeds_mark_cap(m::Module, cap::Int) -> Bool

Whether m carries more than cap marks.

The ratchet, in the shape test_surface's skip already has: a number checked into the repository that can be lowered and not raised. It is a count and not a list because the list is experimental(m), and a project that wants the finer gate should assert on that instead.

promotable(m::Module) -> Vector{Mark}

Every mark in m whose exit condition is met — the work item list, in the direction of done.

ready_to_promote(m::Module, name::Symbol) -> Bool
ready_to_promote(mk::Mark) -> Bool

Whether the declaration's own exit condition is met.

Not "is this marked" but "may this stop being marked", answered by the thing that knows: the until= predicate the author wrote next to the reason.

@experimental(
    "no reference value yet",
    since = v"0.1.0",
    tracking = "https://example.invalid/issues/12",
    until = () -> isfile(joinpath(@__DIR__, "..", "test", "refs", "energy.toml")),
    energy(β) = 2β,
)

false for a mark with no until=, always — a mark whose exit was never written down cannot be retired mechanically, and reporting it "ready" because it happens to be covered by a test would be inventing a criterion the author did not state. Those marks are reported separately by marks_without_exit rather than being quietly lumped in with the ones still standing.

A predicate that throws counts as not ready: an exit condition that cannot be evaluated has not been met.

stale_since(m::Module, current::VersionNumber; releases::Int = 2) -> Vector{Mark}

The marks that have been standing for releases breaking releases or more.

The report a CI job turns into a nag. Marks with no since are not listed here — they are a different finding, and marks_without_exit plus a since-less mark is what a package that never intended to retire anything looks like.

docstring_note(m::Module, name::Symbol) -> Union{String,Nothing}
docstring_note(mk::Mark) -> String

The admonition a docs build should render above name's docstring, or nothing if the name is settled.

Markdown, in Documenter's !!! warning form, carrying the reason, the version it has been unsettled since, and the tracking link — the three things a reader needs and the author has already written once:

julia> println(docstring_note(MyPkg, :provisional))
!!! warning "Experimental"
    the r,s branch has no reference value

    Unsettled since v0.2.0. Tracking: <https://example.invalid/issues/9>.

nothing for a settled name is the point: a renderer that annotates everything says nothing. The Documenter extension turns this into an @experimental block; nothing stops a project splicing it in itself.

marks_markdown(m::Module) -> String

Every mark in m as one markdown block, for a docs page that wants the list in one place.

Sorted by name, with the reason, the version and the tracking link. A module with no marks renders a sentence saying so rather than nothing, because "this page is empty" and "this build failed to find anything" look identical otherwise.

Deliberately heading-free. A docs builder registers heading anchors in an earlier pass than the one that expands a block like this, so a heading spliced in here arrives after the pass that was supposed to see it — Documenter's HTML writer asserts on exactly that.

compare_methods_sees_keywords

true. A signature key carries the keyword names a method declares, so adding, removing or renaming one moves the key and compare_methods reports it.

Stated as a constant rather than left to be discovered, because the neighbouring limit is real and has to be stated with it: keyword defaults live in the method body and are invisible here, so changing tol = 1e-8 to tol = 1e-6 moves nothing. Names yes, defaults no.

Diff

The result of compare: how a module's public surface moved between two snapshots.

isbreaking is the one-bit reading of the first two rows.

MethodDiff

Diff at method granularity — the same six buckets, keyed by signature rather than by name, as produced by compare_methods.

A signature key carries the argument types and the keyword names, so a method whose arguments or keywords changed reads as one key removed and another added. That is the blind spot compare admits to and this closes; see compare_methods_sees_keywords for the part that stays open.

compare(old::AbstractDict, new::Union{Module,AbstractDict}) -> Diff

Say mechanically what moved between two snapshots — and therefore whether the change is breaking.

d = compare(read_snapshot("api.toml"), MyPackage)
isbreaking(d) && error("breaking: $(d.removed_stable) removed, $(d.demoted) demoted")

A removed name that was declared @experimental in old lands in removed_experimental and does not make the change breaking. That is the whole contract: the mark was the notice, and it was given in the source, at the definition, before the removal.

demoted — a name that was stable and is now marked experimental — counts as breaking. Retroactively withdrawing a promise is a change to what callers were told, not a correction of it.

compare_methods(old::AbstractDict, new::Union{Module,AbstractDict}) -> MethodDiff

compare at method granularity: what moved between two snapshots' stable_methods and experimental_methods.

The unit a call site actually reaches. fetch(::Ising, ::Energy) and fetch(::Heisenberg, ::Energy) are one name and two promises, and removing the second is breaking exactly when it was not marked.

Reads the same snapshot file compare does — the two keys mirror the name-level pair — so a repository commits one file and a release script asks it both questions.

isbreaking(d::Union{Diff,MethodDiff}) -> Bool

Whether d removes a settled name or method, or demotes one — the two moves that break callers who were told the truth.

This is the release gate: a true here means the version bump is major (or, under Julia's 0.x convention, a minor bump). Because a diff reads names and signatures and not behaviour, false means no breakage of this kind was found, not nothing broke.

read_snapshot(path::AbstractString) -> Dict{String,Any}

Read a TOML file written by write_snapshot.

snapshot(m::Module) -> Dict{String,Any}

Write down what m currently promises: its stable names and its experimental ones with their reasons, and the same split at method granularity.

The result is plain Dict/String/Vector data, so TOML.print accepts it as-is (write_snapshot does that). Take one at each release; hand the old one and the new module to compare.

module = "MyPackage"
version = "0.4.2"
stable = ["adapt", "measure"]
stable_methods = ["adapt(::Model, ::Grid)", "measure(::Model)"]

[experimental.render_report]
reason = "reads Test's internal result tree"
tracking = "https://example.invalid/issues/12"

[experimental_methods."fetch_value(::Heisenberg, ::Energy)"]
reason = "numerically delicate; no reference value"

methods = false writes the name half only, which is what a package with no foreign methods and a slow method search wants.

stamp(path::AbstractString, f) -> String
stamp(path::AbstractString, r::Record) -> String

Run f, and write next to its result a record of which unvalidated code paths produced it.

The end state this package is for: a figure's directory says what the number came out of, in plain TOML that a reader a year later can open without resolving the package that wrote it.

stamp("figures/energy_sweep.provenance.toml") do
    sweep(model; βs = 0.05:0.05:2.0)
end

Returns path. The file carries every mark the run entered, its reason, how often it was entered, and the versions of the packages the marks came from — see stamp_versions.

stamp_versions() -> Dict{String,Any}

The version of every loaded package, for stamp to write beside a result.

energy being experimental in v0.3 says nothing about v0.9, so a provenance record that names marks without naming versions describes a state nobody can get back to.

write_snapshot(path::AbstractString, m::Module) -> String

snapshot m and write it to path as TOML. Returns path.

Committing this file at each release is what gives the next release something to compare against; a repository with no committed snapshot can be told what is experimental now, but not what changed.

ExperimentalAPI

Say, at the definition site, that a name is not settled yet — and find out, without asking, when a run went through it.

public decides who may call a name. Whether the name is finished is the orthogonal question, and the place it is usually answered is a sentence in a docstring that no tool reads. That matters most where it is least visible: a long numerical run finishes, hands back a number, and nothing in the result says which of the code paths behind it had never been validated.

using ExperimentalAPI

@experimental "convergence not established below β ≈ 0.1" energy(m::Model) = m.β * correction(m)

The five questions this answers

The first is the reason to have any of it. It is asked after a run, about the run, by somebody who is not the author — which is exactly the question a docstring is structurally unable to answer.

Three layers, and what each costs

• Detection is on always and costs one short-circuit read in the body: measured at 1.03x on one thread and 0.985x on eight, for 10M calls of a numeric body — see overhead_when_detecting. A flag written once and read thereafter stops dirtying its cache line, which a counter (3.76x at eight threads, and losing 40% of its increments to races unless atomic) does not.
• Recording (record) counts, captures call paths and attributes time. It costs something, which is why it is a call and not a default, and each record reports its own overhead.
• Analysis (reach) is static and answers about code rather than about a run. Its answer is three-valued: :depends, :clean, and :unknown for a call site it could not pin to a method. Reporting :unknown as :clean is not a weaker claim, it is a false one.

Only a definition with a body carries a flag. A mark written as a name list, or attached to a struct, a const or a module, is a declaration — queryable, audited and analysable, but not observed at run time. See @experimental for the form-by-form table.

What this is not

• Not visibility. public (Julia 1.11) already decides who may call a name. A name can be public and experimental, or public and settled; those are independent axes.
• Not deprecation. @deprecate points the other way — a settled name on its way out.
• Not type stability. Unrelated axis, different tooling.
• Not a documentation generator. The prose belongs to the author; audit only ever checks whether prose exists, never whether it is any good.

Scope of the check

audit compares names(M) — exported and public names — against two independent accounts: a docstring, and a mark. They are not alternatives; the docstring is owed either way. Because names(M) cannot see a method a package contributed to somebody else's generic, the audit also reports contributed_methods — for a package whose surface is such methods, a clean name audit reports nothing while covering nothing.

See compare for the same limit on the release side, and compare_methods for the finer unit.

test_surface(m::Module; skip = Symbol[], require_tracking = false, max_marks = nothing,
             methods = true, outputlevel = 0) -> Audit

Assert, as a @testset, that every public name of m has a docstring, that every mark applies to a name that is actually public, and that every method m contributed to another module's generic is accounted for.

A mark is not an alternative to prose. @experimental records that a shape is unsettled, which is never a reason to say nothing about what the name does, so a marked-but-undocumented name fails this test exactly as an unmarked one does. There is no switch to turn that off; skip is the only escape, and it is per-name, visible, and can only shrink.

Available once Test is loaded (it lives in a package extension, so ExperimentalAPI itself never pulls Test into a runtime dependency). Put it in runtests.jl:

using MyPackage, ExperimentalAPI, Test

ExperimentalAPI.test_surface(MyPackage)

Returns the Audit on the normal return path whether the testset passed or not. outputlevel ≥ 1 also prints it.