Archeion.jl

Where a record keeps its machine face, relative to the record directory.

Per-record list of the files the last deposit wrote: the only files a later deposit may remove.

Environment variable naming the registry root when neither an argument nor a config gives one.

The registry's identity file, at the root of the registry repo.

A backend that mirrors to/from a local directory (root). Used when a host's deploy target is local (hostname dispatch) and as a credential-free transport in tests.

One run's registry metadata. Keyword-constructed; id/project/title/gallery required.

A backend that moves the registry to/from its host. Implement pull_file(t, relpath, local) and push_dir(t, dir; delete).

Snapshot of a source tree's git + environment state at capture time.

active() -> Dict

The host's initialized config (the non-secret active.toml): socket, project, content_dir, db, out, host. Errors if the host hasn't been initialized. This is what makes a per-project environment fall back to the terminal default with no explicit config.

add_search(site) -> Bool

Run Pagefind over the assembled static site at site, producing a client-side search index. Call this AFTER build_index and after the galleries are in place. Returns true on success; warns and returns false if npx is unavailable or Pagefind fails.

True if a responsive agent is listening on this host's socket.

build_dashboard(dg; out, title="") -> path

Render the registry dashboard from a digest to out/index.html, and return its path. Cards are the records; the strip above them is the registry's state.

Separate from build_index, which renders a plain card index from Records alone (what discover produces over DataVault outdirs, where there is no agent.json to read).

build_index(records; out, title="Archeion") -> path

Render the cross-run index over records (a vector of Record) to out/index.html and return its path. Each card links to that record's gallery. This does not render or copy the galleries themselves — they are expected to already exist under out (or at the hrefs the records carry).

capture_repro(srcdir, dest; config=nothing, strict=false, env="") -> ReproBundle

Snapshot the reproducibility bundle of the project at srcdir into dest/repro/: the git commit + dirty flag, Project.toml, Manifest.toml, the Julia version, an optional config file, and a runnable reproduce.sh recipe.

env is the directory whose Project.toml / Manifest.toml are snapshotted, and defaults to srcdir. They are the same directory only when the code that ran and the environment that ran it live together. A study whose render has its own env (projects/X/report/) must pass it, or the bundle pins an environment that never ran while claiming it did; reproduce.sh then names that directory in its --project=.

With strict=true, a dirty git tree raises (refuse to record a non-reproducible run); otherwise the dirty state is recorded and a warning is emitted. A config path that does not exist is warned about (not silently dropped), and a config whose filename would collide with the environment snapshot (Project.toml/Manifest.toml) raises.

create_registry(root; name="", description="", repo="", git=true) -> String

Create a registry at root and return its path: Archeion.toml (identity), a README.md saying what the directory is, and, unless git = false, a git repository with those files committed. repo is the remote URL; when given it is recorded and set as origin.

Refuses to overwrite an existing registry. To adopt a directory that already holds records, create the registry and then reindex; nothing about a record depends on when the registry was declared.

Remove the host default (stops resolving to it; does NOT stop a running agent — Ctrl-C its pane).

deploy(site; config=nothing, delete=true) -> Bool

Mirror the static site at site to a private host over FTPS using a resolved deploy config. config is resolved by _resolve_deploy_config, in order: the explicit config arg (if it exists) → deploy.local.toml in the current directory → ENV["ARCHEION_DEPLOY"] → the machine-global default <ARCHEION_HOME>/deploy.toml (default ~/.archeion/deploy.toml). That lets ONE 0600 file outside every repo drive deploy for every project on a machine — no per-project config, no password prompt. A machine-global config (the last two sources) that is group- or other-readable/writable gets a prominent @warn naming the file and the chmod 600 fix (never a hard error — permissions, not encryption, are what make the secret "hard to see from outside"). Uploads via lftp, passing credentials in a 0600 temp script (password never in process args). With delete=true the remote mirrors the local tree exactly (removes stale remote files). Returns true on success.

deploy(; site="", delete=false) -> Dict

Push a local content tree to the (agent-held, hidden) remote via the agent — no config/creds in the caller, the agent does the FTP. site is the local docroot-level dir to push; when omitted, the agent falls back to the config's [archeion].content_dir. Passing site lets any project's STAGE deploy through the machine-global agent (creds stay in the agent), mirroring the explicit deploy(site; config) form — so the config-file and machine-global paths coexist over the same [ftp] config.

deposit(dir; project, source, title="", doc=nothing, root="", config=nothing, srcdir="",
        strict=false, summary="", tags=String[], data_keys=String[], thumbnail=nothing,
        date="", index=true, search=false) -> (; dir, record, index, pruned)

Deposit the built directory dir into the registry as one record.

dir is any directory with an index.html at its top: a Pinax render output, but equally a slide deck or a hand-written page. Pass doc (a rendered Pinax document) only to take the title from it.

Written under <root>/<project>/<source>/:

• the contents of dir, copied file by file;
• repro/, when srcdir is given – see capture_repro for what it snapshots, for env (the directory whose Project.toml / Manifest.toml ran, when it is not srcdir itself) and for strict, which refuses a source tree with uncommitted changes;
• record.toml – the Record, and the only file the index reads.

Files a PREVIOUS deposit wrote and this one did not are removed, and nothing else in the record directory is touched: a sidecar (an annotation store, a note, a PDF someone dropped in) survives a re-render.

An untitled record is refused. Every card in the index is labelled by the title, so a record without one cannot be told from any other, and Pinax.render leaves it empty unless asked.

Heavy data is NOT copied: reference it through data_keys (DataVault keys), the same way Record does.

When the registry root is a git repository, the deposit is committed (the record directory and the index, by explicit path, never git add -A) and the short SHA comes back as commit. Re-depositing an unchanged render stages nothing and returns "" rather than failing. commit = false writes the files and leaves staging alone.

digest(root=registry_root()) -> Dict{String,Any}

Everything the registry knows about itself, in one structure: the registry's identity, per-record facts (from record.toml and, when the record has one, its agent.json), and the totals a reader wants before opening anything.

The totals are deliberately the ones that describe state, not size: how many records cannot be reproduced from what they recorded (unknown_commit + dirty), and how many carry no machine face at all. A registry that only counts its records cannot tell you it is rotting.

Written to index.json by reindex, and used to render the dashboard.

discover(outdirs; out, title="Archeion") -> Vector{Record}

End-to-end cross-project aggregation: discover all (project, study, run) under outdirs, write per-run summaries + the cross-run index into out, add Pagefind search, and return the Records. outdirs are DataVault output directories (each containing .datavault/).

feedback_md(db, rid) -> String

A clean Markdown digest of a record's human annotations — status, tags, the project's PARA bucket / description / tags, and the discussion. Drop it into an LLM prompt next to the agent.json so the steering loop sees the human's feedback. Says so plainly when nothing has been annotated yet.

ingest(doc; db, project, source, runs=[], html_path="", pdf_path="", date="", git="") -> NamedTuple

Ingest a rendered Pinax doc (get it via Pinax.current_document() after Pinax.render) as one registry record. runs is a vector of (project, run) (the DataVault runs the doc renders). The record id = slug(project)/slug(source) (stable); project is stored canonically as slug(project) too, so the viewer's project key (URL/FK/id) never drifts. Figures use Pinax's stable ids. Annotation- preserving: importance/archived/tags/bookmarks/comments are never overwritten.

initialize(config_enc="config.enc")

REPL-ONLY (a human at a terminal). Prompt for the password (hidden), decrypt config_enc in memory, write the NON-SECRET host default ~/.archeion/active.toml (active project + local working paths + socket — NO target/creds), then BLOCK serving the agent (run this in a dedicated terminal/tmux pane).

After this, on the same host, deploy() / pull() / status() / publish(...) work with no config — they resolve paths from active.toml and delegate the credentialed FTP to this agent. The decrypted target + creds live ONLY in this process's memory; nothing on disk or any socket op reveals them.

is_registry(root) -> Bool

Whether root holds an Archeion registry, i.e. an Archeion.toml at its top.

lock_config(plain_path; out="config.enc", shred=false) -> out

REPL-ONLY. Encrypt a plaintext config TOML (which carries the FTP creds + deploy target) into out, prompting twice for a password (hidden). After this, the plaintext is no longer needed — pass shred=true to delete it (overwrite then remove). The encrypted out is safe to keep in the tree; without the password it reveals nothing, and the LLM (non-interactive) can never decrypt it.

master_ledger(outdirs) -> Vector{Dict{String,String}}

Concatenate DataVault.build_master_ledger across several project outdirs. Each row is a ledger entry enriched by DataVault with project_name / run / log_toml; this is the cross-project ledger DataVault intentionally leaves to a higher layer.

pages_status(slug) -> (; private, enabled, url)

Ask GitHub whether owner/name is private and whether it serves a Pages site. Uses the gh CLI, so it sees private repositories the caller can see.

This exists because the obvious signal is wrong: pushing to gh-pages succeeds whether or not Pages is enabled, so a green deploy reports that the push happened and never that the site is reachable.

project_annotations(db, name) -> NamedTuple

A project's app-owned filing: (; exists, para, description, tags, todos). para is the PARA bucket (Projects/Areas/Resources/Archives), description the human note, tags the project tags, todos a (; body, done) checklist. name is the canonical slug. exists is false if unknown.

publish(doc; config, project, source, site, runs=[], html_dir="",
        db=joinpath(site, "data", "archeion.db"), content_dir=dirname(db),
        remote_db=nothing, delete=true) -> NamedTuple

The annotation-SAFE registry round-trip for a REMOTE registry: pull the live DB → ingest doc into it (content UPSERTs by stable id; the human annotations already in it — comments / tags / status / project notes — are preserved by the content/annotation split) → push site (the DB + figures) back. Use this instead of ingest + deploy when humans annotate on the remote, so a redeploy never clobbers their work. On the first publish (no remote DB yet) it ingests fresh and pushes. Returns the ingest result.

publish(; doc, source, project=active project, html_dir=active out, delete=false) -> NamedTuple

The annotation-safe round-trip with NO config in the caller: pull the live DB via the agent → ingest doc into the active content tree (preserving existing annotations) → push via the agent. The creds never leave the agent.

publish_pages(root=registry_root(); remote="origin", branch="gh-pages", check=true,
              search=true, message="") -> String

Publish the registry at root as a static site on branch of remote, and return the URL GitHub serves it at (empty when check = false, since that is the answer only GitHub has).

The site IS the registry: the same index.html, the same record directories. What the publish adds is .nojekyll (Pages runs Jekyll otherwise, which drops paths beginning with an underscore) and, with search = true, the Pagefind index.

It refuses in three cases, each of which otherwise produces a green publish that serves nothing or destroys something:

• the repository is private (Pages needs a paid plan there, and no plan below Enterprise gives an access-controlled site) – a private registry is read from its clone instead;
• Pages is not enabled on the repository;
• branch exists and is not this registry's site, which is what a Documenter gh-pages looks like.

The branch is force-pushed with a single commit: it is derived from the registry, so its history would only duplicate main's.

pull(config; out=joinpath(tempdir(), "archeion.db"), remote_db=nothing) -> local_db_path

Fetch the LIVE registry DB from the remote host to out, so the annotation readers (record_comments, record_annotations, feedback_md, …) see the web app's latest human annotations. Backend-neutral — dispatches on [archeion.remote].kind. remote_db overrides the DB's remote (docroot-relative) path.

pull(; dest=<active content_dir>/data/archeion.db) -> dest

Fetch the live registry DB via the agent (credentialed) to a local dest. The DB content is non-secret; the creds that fetch it stay in the agent. Distinct from pull(config; out) (the explicit form).

read_deploy_target(config) -> DeployTarget

Read FTPS credentials + remote target from a gitignored TOML config with an [ftp] table (host, user, password, remote_dir, optional tls/tls_verify). The password may instead come from ENV["ARCHEION_FTP_PASSWORD"] (preferred — keeps it out of files). config must already be a concrete, resolved path — see deploy/_resolve_deploy_config for how that path is discovered (explicit arg → CWD deploy.local.toml → ARCHEION_DEPLOY env → machine-global ~/.archeion/deploy.toml).

read_record(dir) -> Record

Read a Record from dir/record.toml. Raises with the offending field and file path if a required field (id/project/title/gallery) is missing.

read_records(root=registry_root()) -> Vector{Record}

Every record under root, newest first. A record is any directory holding a record.toml, at any depth, so the <project>/<source>/ layout deposit writes is a convention rather than a requirement: a record placed by hand, or by a tool this package has never heard of, is picked up by being on disk.

Returns an empty vector when root does not exist yet.

record_annotation_list(db, rid) -> Vector{NamedTuple}

Every comment/annotation on a record, each WITH its LOCATION — the traceable comment list. One entry per annotations row (oldest-first): (; id, kind, page, target, anchor, body_md, author, created_at), kind ∈ record/figure/section/passage, target = figure id / section heading ("" for record/passage), anchor = the passage's text-quote JSON ("" otherwise). So an LLM can read each comment and exactly where it points from one query.

record_annotations(db, rid) -> NamedTuple

The full app-owned annotation of a record: (; exists, importance, archived, title, project, tags, comments, annotations). importance is 0..3 (shared "notable"); archived is a Bool — together the human's "status". comments is the record-level discussion (back-compat); annotations is the FULL location-tagged list (record/figure/section/passage — see record_annotation_list). exists is false when the id isn't in the registry (everything else defaulted).

record_comments(db, rid) -> Vector{@NamedTuple{author::String, body_md::String, created_at::String}}

The web-app discussion on a record (oldest-first). Empty when the record has no comments.

record_dirs(root=registry_root()) -> Vector{String}

Absolute path of every record directory under root, sorted. A record is any directory holding a record.toml; this is what read_records walks, exposed separately because a caller that needs the directory (to reach agent/agent.json, repro/, or a figure) cannot recover it from a Record.

record_tags(db, rid) -> Vector{String}

The record's app-owned tags (sorted).

record_versions(db, rid) -> Vector{@NamedTuple{version::Int, title::String, ingested_at::String, git_commit::String, date::String}}

The content version history of a record (oldest-first) — one entry per distinct content state (ingest appends one when title/body_md/run-set/git change). title is the minimal snapshot (the record's title at that version). Empty for records ingested before versioning.

records_from_outdirs(outdirs; site) -> Vector{Record}

Discover every (project, study, run) under the DataVault outdirs (each containing a .datavault/ anchor), render a Pinax-styled summary page for each under site/<project>/<run>/, and return the corresponding Records (provenance from the log.toml + ledger).

registry_info(root=registry_root()) -> (; name, uuid, repo, description, root)

Identity of the registry at root, read from Archeion.toml. Raises when root is not a registry. deposit does not require one: it writes into a plain directory just as happily, and commits only when the root is a git repository. This is the reader for the declared identity, used when a registry is replicated or published.

registry_root(; root="", config=nothing) -> String

Absolute path of the registry root, resolved in this order:

1. the explicit root argument;
2. [archeion] root in config (a path to a TOML file, or an already-parsed Dict);
3. ENV["ARCHEION_REGISTRY"];
4. ~/registry.

The default deliberately sits OUTSIDE any repository: a deposit that landed inside a git checkout would be swept into someone's next commit. Set the environment variable once per machine, or [archeion] root per project, and every call agrees without passing paths around.

reindex(root=registry_root(); title="", search=false) -> path

Rebuild the registry's two faces from the records currently on disk, and return the path of the human one:

• root/index.html, the dashboard: a card per record, over a strip of the registry's own state (how many records, how many cannot be reproduced, how many carry no machine face);
• root/index.json, the digest: the same content as data, so a program reads the whole registry once instead of opening every record.

The title defaults to the registry's declared name (registry_info). With search=true, refresh the Pagefind index too (add_search).

Idempotent and cheap: it reads the record.toml files and re-renders one page. Call it after anything changes the tree, including a record you added by hand.

remote_slug(root=registry_root(); remote="origin") -> String

owner/name of the registry's remote. Raises when the remote is missing or is not a github.com URL. The URL itself never appears in the error: a push URL can carry a token.

status(db; io=stdout, print=true) -> NamedTuple
status(; config=nothing, io=stdout, print=true) -> NamedTuple   # config=nothing → the host's agent; else pull `config`

A one-call overview of the whole registry — so the current state is queried, not hand-inspected with raw SQLite. Returns (; projects, records, figures, comments, items) where items is one (; project, para, id, title, importance, archived, figures, runs, comments, updated_at) per record (grouped by PARA bucket then project). With print=true it also writes a grouped table to io.

status(db) reads a local DB (a path or an open SQLite.DB). status(; config=...) fetches the deployed DB via pull (see transport.jl) and summarizes the LIVE registry in one call.

sync(root=registry_root(); remote="origin", pull=true, push=true) -> Bool

Bring the registry at root level with its remote: git pull --rebase, then git push. Returns whether every requested step succeeded; a failure warns with git's own output rather than raising, because a registry that cannot reach its remote is still perfectly readable locally.

transport(config) -> RemoteTransport

Build the remote transport from config's [archeion.remote].kind (default "ftps"). FTPS reads its credentials from [ftp] (see read_deploy_target). Other backends plug in via a new kind branch.

view_config(enc_path="config.enc")

REPL-ONLY. Prompt for the password (hidden) and print the decrypted config to the terminal. This is the ONLY way to read the config back, and it works solely in an interactive REPL — an automated/LLM session (non-interactive) is refused, and there is no socket/agent op that reveals the config.

write_basic_auth(site; user, password, realm="Archeion") -> nothing

Write .htaccess + .htpasswd into site for HTTP Basic auth (Apache/Lolipop), making the deployed site private. The password is hashed with openssl passwd -apr1. (Alternative: configure Basic auth via Lolipop's access-restriction panel and skip this.)

write_record(rec, dir) -> path

Write rec to dir/record.toml, creating dir if needed. Returns the file path.