Module http 

Centralized HTTP client wrapper. All Source impls fetch through here.

Security defaults per docs/SECURITY.md:

• rustls TLS only (no openssl, no native-tls — enforced by deny.toml)
• HTTPS-only redirect policy (file://, data://, http:// rejected)
• Per-source redirect host allowlist (docs/REDIRECT_ALLOWLIST.md)
• Body size cap (crate::PDF_MAX_BYTES = 100 MB)
• Per-request timeouts (connect 10s, read 60s, total 300s)
• PDF magic-byte check on the first 5 bytes (%PDF-)
• User-Agent: doiget/<version> (+https://github.com/QAtlasHub/doiget)

See docs/SECURITY.md §1.2-1.3 / §1.10 and docs/REDIRECT_ALLOWLIST.md.

Architectural note: per-source reqwest::Client

reqwest::redirect::Policy::custom receives only an Attempt value, which exposes the next URL and previous URL chain but not the original request’s headers. That makes the “tag the request with X-Doiget-Source and inspect it from inside the redirect closure” approach infeasible on reqwest 0.13.x. Instead, HttpClient holds one [reqwest::Client] per source — each client’s redirect closure captures that source’s SourceAllowlist so cross-source confusion is impossible by construction.

Structs

HttpClient

Workspace-wide HTTP client with the security defaults applied.

ProbeOutcome

Per-source allowlist entry. Matches the schema in What a HttpClient::probe observed (issue #407).

SourceAllowlist

docs/REDIRECT_ALLOWLIST.md §2.

Enums

HttpError

Errors that can arise during HTTP fetches.

Functions

discovery_allowlist

Always-compiled allowlist for the discovery search call path (ADR-0031).

fulltext_allowlist

Always-compiled allowlist for the full-text extraction call path (ADR-0032).

init_tls

Public entry point for callers that build their own reqwest::Client outside of HttpClient and need the process-default TLS provider installed first (ADR-0020 Amendment 1).

is_transparent_resolver

Whether host is a DOI resolver rather than a content host (#533).

oa_publisher_allowlist

Hard-coded Phase 1 allowlist for the synthetic "oa-publisher" source — the publisher / preprint / repository hosts to which Unpaywall’s best_oa_location.url (or url_for_pdf) typically resolves.

preprint_allowlist

Allowlist for the preprint finders that are not fetch sources. Each has its own gate, and each is asked only when the content leg found nothing; none of them serves content.

pubmed_allowlist

Always-compiled allowlist for PubMed id resolution (#500, ADR-0061): a PMID / PMCID is turned into its DOI by NCBI E-utilities. Not a fetch source either – it answers “which DOI is this”, never with content – so it stays out of tier_1_allowlist and out of every fetch plan.

software_allowlist

Always-compiled allowlist for software citations (#614, ADR-0058): doiget cite / verify on a GitHub repository or release URL. Kept out of tier_1_allowlist because it is not a fetch source – a fetch plan’s source list is read from that one, and GitHub must never appear in it. Only the CLI registers it; no MCP tool cites a URL.

tier_1_allowlist

Hard-coded Phase 1 allowlist for Tier 1 sources. Sourced from docs/REDIRECT_ALLOWLIST.md §3.

tier_2_allowlist

Hard-coded Phase 4 allowlist for Tier 2 metadata sources (OpenAlex, Semantic Scholar, DOAJ). Sourced from docs/SOURCES.md §1 (the Tier 2 table) and docs/REDIRECT_ALLOWLIST.md §3 (same redirect-allowlist policy as Tier 1, distinct source keys).

tier_3_allowlists

Every Tier-3 TDM allowlist this build actually compiled in.