core.url

soul-sidesoulstack.networknetworkroot required

Download a file over an https URL with checksum verification and an SSRF guard.

core.url.fetched downloads the file at url to path and converges mode/owner/group to the declaration. It is secure-by-default: https:// only, an SSRF guard on the resolved IP, TLS chain verification, and checksum verification in a temporary file before the atomic rename — a wrong hash never materializes. Idempotency goes by checksum, and without one by SHA-256; each security control is removed by a separate opt-out flag. No shell.

Requirements

Rootrequired
Sidesoul-side
Collectionsoulstack.network
Categorynetwork

Capabilitiesnetwork_outboundfs_write_root

States

fetched

core.url.fetched — The file at url is materialized at path with the given mode/owner/group.

Changes when

The content was re-downloaded or an attribute was changed (mode/owner/group). The branching is in the "fetched behavior by branch" table.

Stays unchanged when

A checksum/SHA-256 match with no attribute changes.

Parameters

Param	Type	Required / default	Description
`url`	`string`	required	Source (https:// only; SSRF guard on the resolved IP).
`path`	`string`	required	Target file path.
`checksum`	`string`	optional	Expected hash in the form "sha256:<hex>"/"sha1:<hex>" (supply-chain).
`mode`	`string`	optional	Permissions in octal form, e.g. "0644".
`owner`	`string`	optional	Owner (user name).
`group`	`string`	optional	Owning group (group name).
`headers`	`map<string>`	optional	HTTP request headers (values are secret, kept out of output). If-None-Match/If-Modified-Since here give a conditional GET (304 → no-op).
`timeout`	`string`	optional	Request timeout (Soul Stack duration, e.g. "300s").
`allow_http`	`bool`	optional · false	Allow http:// (downgrade risk). Does NOT open SSRF — the dial guard is enforced separately (allow_private).
`insecure_skip_verify`	`bool`	optional · false	Skip TLS chain verification (self-signed / internal CA). MITM risk.
`allow_private`	`bool`	optional · false	Lift the SSRF guard: allow dialing metadata/loopback/RFC1918 (a legitimate internal endpoint).

Example — Fetch a GitHub release with a checksum check

- name: Fetch node_exporter tarball with checksum
  module: core.url.fetched
  params:
    url: "https://github.com/prometheus/node_exporter/releases/download/v1.9.0/node_exporter-1.9.0.linux-amd64.tar.gz"
    path: /tmp/node_exporter-1.9.0.tar.gz
    checksum: "sha256:abc...def"
    mode: "0644"

Output

Field	Type	Description
`path`	`string`
`url`	`string (echo without headers)`
`sha256`	`string (SHA-256 of the written/matched content)`
`size`	`int (bytes)`
`changed`	`bool`
`fetched`	`true`

Reference

fetched behavior by branch

The logic depends on whether checksum is set and whether the file already at path matches.

Condition	Action	changed
checksum set + file exists and matched by hash	content is not downloaded; mode/owner/group are converged to the declaration (convergence)	true only if an attribute was changed
checksum set, file missing / hash did not match	download to temp → verify against checksum → atomic rename; mismatch → failed, temp is removed, the target path is not touched	true
checksum not set + file exists and matched by SHA-256	download to temp → compare SHA-256 → no write; mode/owner/group are reconciled	true only if an attribute was changed
checksum not set + content differs / file missing	download to temp → atomic rename	true

Notes

Conditional GET: If-None-Match / If-Modified-Since in headers give a conditional GET — the server may answer 304 Not Modified (no body is transferred). A 304 with an existing local file → no-op (mode/owner/group are converged to the declaration, output over the existing file); a 304 without a local file → failed (a stale validator with no cache, nothing to download). The 304 is checked before the general 2xx check.
Permissions depend on the target path: writing to a system directory requires root, to a user-writable path (e.g. /tmp) it does not. The manifest declares network_outbound + fs_write_root.
checksum: sha256 and sha1 are supported; md5 is deliberately unsupported (weak for supply-chain). output.sha256 is always SHA-256, even if checksum was given as sha1.
Lifting any opt-out flag (allow_http / allow_private / insecure_skip_verify) puts a line in output warnings — host only, without the full URL and headers (query/path and headers can carry secrets).
Spawns no subprocesses: downloading, hashing, and writing happen in-process, without a shell.

Security & defaults

By default https:// only — http://, file://, and tricks like https://\nhttp://evil are rejected (the scheme is checked case-insensitively via url.Parse). allow_http permits http://, but file:// / ftp:// stay forbidden and the SSRF guard is not weakened.
SSRF guard: a dial to metadata (169.254.169.254), loopback, RFC1918, link-local/CGNAT is blocked by the actually resolved IP — this closes direct SSRF (stealing cloud-metadata IAM credentials) and DNS rebinding (the dial goes to the already-checked IP, with no second resolve). A host with a pair of A records "public + metadata" is blocked entirely. Lifted only by allow_private.
Checksum verify BEFORE publishing: the download always goes to a temporary file in the path directory, the hash is computed on the fly, and the verify against checksum happens before the rename. On a mismatch the temp is removed and the target path is not touched — a wrong hash never materializes.
TLS — the system trust store by default; lifted by insecure_skip_verify (self-signed / internal CA, MITM risk). Redirects: an https→http downgrade is rejected (allowed only with allow_http; a redirect to a non-http(s) scheme is always blocked, as is a dial to private addresses on a hop without allow_private).
headers are sensitive-by-construction (ADR-010 §7.4): the values are never logged and never appear in output/register, including in the url echo field.