Version/0 48 1 rc2#3823
Merged
Merged
Conversation
If the directory exists, the depot thinks it's a full artifact, but a failed unpack does not create a runtime.json file, which raises a different error. This masks the original unpacking error.
Remove partially-unpacked directory if the unpack failed.
Updated to Go 1.26.4.
… WebSocket The build-log-streamer WebSocket previously opened with only an Origin header, so the server had no way to authorize the stream. When the caller is authenticated, offer the platform JWT via Sec-WebSocket-Protocol as 'bearer.<jwt>' alongside the real 'build-log-streamer.activestate.com.v1' subprotocol the server echoes back (its allow-list contains only the real subprotocol, so the token never appears in the upgrade response). The browser WebSocket API can't set custom request headers, so the dashboard carries the JWT the same way -- both clients stay symmetric. Also send the versioned State Tool User-Agent on the Upgrade so the server can see which client versions are connecting. The token is threaded as a plain string from the runtime options into Connect, so pkg/runtime gains no authentication dependency; an empty token means anonymous (unchanged behavior). Includes a real-handshake test that asserts the offered subprotocols, the negotiated subprotocol, and the User-Agent the server receives.
…-fixes Forward platform JWT and version User-Agent on the build-log-streamer…
Remediate CVEs.
…NG-1631) New leaf package providing content encryption for private ingredient artifacts under a single organization-wide AES-256 key. - Encrypt/Decrypt stream the v1 payload format with bounded memory; each chunk is sealed with a fresh CSPRNG nonce, with the chunk index and a final-chunk flag bound into the AEAD so reorder/truncation are detected. - The header (magic marker, version, key id, key fingerprint, chunk size) is hashed into every chunk's AAD, so tampering or stripping it fails closed. - Fingerprint plus a header-only pre-flight CheckKey reject a wrong key before the body is read. - Decrypt writes to a sibling temp file and renames on success, so a failed decrypt never leaves partial or unauthenticated plaintext at the destination. Tests cover round-trip across sizes, a fail-closed tamper suite, nonce uniqueness, wrong-key pre-flight, a v1 golden vector, and an import-boundary check (stdlib + internal/errs only). The package is CGO-free so pkg/runtime's reachable set stays CGO-free. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the key-custody backend for private ingredients: a provider that fetches the organization's single AES-256 key from the customer-hosted HTTPS key service, validates it, and caches it for the run. - internal/runbits/orgkey: the Provider, the v1 org-key contract validation (schema/org/algorithm/encoding, base64 decode, fingerprint match via artifactcrypto), and the HTTPS backend (https-only, TLS 1.2+ with a configured CA or pinned cert, optional mTLS, bearer token from an env var or file, bounded timeout, no redirects). The key is held in memory for the run with an opt-in 0600 on-disk cache for headless/offline/CI. - Config options for the URL, CA, mTLS cert/key, bearer-token source, and the on-disk cache opt-in, registered in the package's init() and settable via state config. - subshell scrubs the configured bearer-token env var from child process environments. - pkg/runtime gains a WithDecryptionKey option as the consume-side injection seam, and orgkey exposes a header pre-flight key check. The custody backend lives caller-side and is not imported by pkg/runtime, which stays CGO-free. Wiring into the publish and consume flows lands with ENG-1634 and ENG-1635. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ENG-1631: Add content encryption package (single org key, chunked AES-256-GCM)
…1640) Private ingredient wheels come from untrusted sources, so their extraction must not be able to write or point anywhere outside the destination directory. Unarchive now takes a WithUntrustedSource option. When set, every entry path, symlink target, and hardlink target is confined under the destination root and anything that would escape aborts extraction. It is off by default: trusted Platform artifacts may legitimately contain absolute symlinks (for example into /usr/share), so their extraction is unchanged. The ENG-1635 decrypt path will extract private wheels WithUntrustedSource; per- user isolation of decrypted content and the decrypt temp dir are handled there. testfile.tar.gz is a contained fixture backing the successful-extraction test; the previous fixture, which has a root-level escaping symlink, is renamed to testfile-escapes.tar.gz and backs the rejection test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…path Detect encrypted private-ingredient payloads by envelope magic during the unpack stage, decrypt them under the org key supplied via the options seam, and extract the inner tar.gz archive in place before the artifact is committed to the depot. Decrypted artifacts are marked private so they survive LRU eviction. Extraction routes through the shared untrusted-source sanitizer. A wrong key or corrupt payload fails closed with an error naming the artifact. A missing key (no key service configured, or the key service unreachable) fails open: the artifact is skipped, not committed to the depot, and the runtime hash is left unsaved so the next update retries once a key is available. Skipped artifacts fire ArtifactInstallSkipped so the install progress bar still completes. ENG-1635 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the organization key is unavailable, encrypted private ingredients are skipped during runtime setup. pkg/runtime cannot output to the user and the skip set lives in a transient setup object, so the skip is reported by way of the existing event stream: ArtifactInstallSkipped now carries the artifact name, a small collector handler in runbits accumulates them, and the caller prints a single warning naming the skipped ingredients after the update completes. This also covers the case where no key service is configured at all, which was previously skipped silently. ENG-1635 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Warn on unexpected on-disk cache stat errors (permission denied, bad path) instead of silently treating them as a cache miss; a genuinely missing file stays a normal silent miss. This matches the behavior the doc comment already describes. Set Proxy: http.ProxyFromEnvironment on the key service HTTP transport. A hand-built http.Transport does not inherit the default proxy behavior, so HTTPS_PROXY/NO_PROXY were ignored and a customer-hosted key service behind a corporate proxy could be unreachable. ENG-1632 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add internal/python/wheel, a pure-Go routine that turns a local source
directory into a spec-compliant, byte-reproducible py3-none-any wheel. It only
relocates and zips files; it never executes the source it packs.
Pack reads core metadata from pyproject.toml's [project] table (caller values
override), mirrors the source tree into the wheel root (excluding the top-level
pyproject.toml, __pycache__, *.pyc/*.pyo, and version-control directories), and
emits {name}-{version}.dist-info with METADATA, WHEEL, and a PEP 376 RECORD.
Determinism comes from the stdlib archive/zip writer with fixed timestamps,
sorted entries, and fixed mode bits, so the same tree always produces the same
bytes. Compiled, platform-specific files (.so/.pyd/.dll/.dylib, matched
case-insensitively) are rejected rather than silently packed, and there is no
partial artifact on any failure path.
ENG-1633
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TestPackIsDeterministic ignored os.ReadFile errors, so two unreadable wheels would compare equal as nil and pass. Fail the test if either read errors. ENG-1633 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The untrusted-source guard used filepath.IsAbs to reject absolute symlink
targets, but that is host-specific: a Unix-style "/etc/passwd" reads as
relative on Windows (no drive letter), so it slipped past the check and got
folded into the extraction root, escaping rejection.
Replace it with hasAbsoluteTarget, which treats a leading separator ("/" or
"\") or a volume name ("C:") as rooted on any platform, and add a
backslash-rooted test case that exercises this on every OS.
ENG-1640
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ENG-1632: Add org key custody backend (client-hosted HTTPS key service)
ENG-1640: Confine untrusted archive extraction under the destination root
…unavailable org key.
ENG-1635: Decrypt and install encrypted private artifacts in the runtime setup path
Add a --build <srcdir> flag to state publish that, on the customer's own machine, packs the source tree into a wheel, encrypts it under the organization key, wraps the ciphertext together with a cleartext runtime.json into a tar.gz, and uploads only that wrapped artifact. Publishing without --build is unchanged. The org key is fetched and validated before anything is built, so a private publish fails closed (nothing is uploaded) when no key service is configured or the key is unavailable. The plaintext wheel lives only in a temp dir that is removed on every path. --build requires a private/<org> namespace. The wheel package is split so resolution and packing are distinct: ResolveMetadata reads pyproject.toml and applies --name/--version overrides, and Pack takes the resolved metadata and builds the wheel. artifactcrypto.Encrypt now rejects an over-long header (driven by the key id length) rather than emitting a payload its own reader would reject. The genesis/timeless publish flag is left as a TODO until the platform supports it (ENG-1641); until then --build is not genesis-stamped. ENG-1634 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A --build artifact is encrypted under the project org's key, so it must be published under that same org or its consumers cannot decrypt it. Validate the resolved namespace (flag, meta file, or default) against private/<owner> in one place, after namespace resolution, rather than only checking the flag for a private/ prefix. Also wrap the wheel.Pack error with errs.Wrap (it is not an input error), fix a stale cleanup comment, and close the gzip reader in the test helper. ENG-1634 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This feature existed previously, but was lost in a runtime refactor.
ENG-1633: Implement pure-Python wheel packer for local source trees
TestPublishBuildEncrypted drives the full private-ingredient round trip: `state publish --build` packs a pure-Python source tree into a wheel, encrypts it under the org key, and publishes it under a unique random name; `state install --ts=now` then resolves, downloads, and decrypts it. The test proves decryption by reading the decrypted wheel back out of the depot and finding a sentinel string that only the plaintext source contains. To avoid standing up an HTTPS key service, the org key is supplied to both the publish (encrypt) and install (decrypt) sides through a new environment variable carrying the key-service contract JSON. The orgkey provider reads it only after Configured() passes and validates it through the same path as a fetched contract, so org-binding, fingerprint, and key-length checks all still apply. The contract is stripped from child process environments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CI failure surfaced no diagnostics because the assertion lived in an inner helper. Inline the check into the test body so a failure dumps the spawned command output and state logs. Also search the whole depot (any wheel or file) for the sentinel rather than a fixed install/*.whl glob, and log the depot's top two levels, since the on-disk layout depends on how the artifact is packaged on install. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A --build publish now declares a dependency on builder/private-builder. Without a builder dependency the platform falls back to the noop-builder, which ingests the upload but produces an empty artifact, leaving nothing installable on the consume side. The wrapped tar.gz now carries only the encrypted payload; the private-builder generates the runtime.json on ingest, so we no longer write or wrap one. The build unit test is updated to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nd logs." This reverts commit 0a8e9bc.
The private-builder nests the encrypted payload under the artifact's install directory rather than leaving it at the top level, so the consume side never found it: the search only scanned top-level files, and even once found the decrypted archive was extracted to the artifact root instead of where the runtime.json points. The result was an installed-but-never-decrypted artifact. findEncryptedPayload now searches the artifact recursively, locating the payload by its conventional name and confirming the envelope magic, and the decrypted archive is extracted alongside the ciphertext so it lands under the install dir. The payload filename is now a single shared constant (artifactcrypto.PayloadFilename) used by both the publish (produce) and runtime (consume) sides, so it can no longer drift between them. The integration test's decryption check is factored into reusable suite helpers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stop walking the depot once the sentinel is found rather than scanning every remaining file, and skip a wheel zip entry whose read fails instead of testing partial bytes. Both are confined to the integration test's verification helpers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ENG-1634: Add --build to state publish: build, encrypt, wrap, upload
Verify downloaded artifact checksums.
After decrypting a private ingredient the consume path now installs the wheel into a self-contained site-packages directory and exposes it on PYTHONPATH via the artifact's runtime.json, rather than leaving the wheel file unextracted, so the package is importable in the runtime. A new internal/python/wheelinstall package extracts the wheel (confining entries to the destination) and writes the PEP 376 INSTALLER marker. The consume path gates the install on a .whl payload and logs anything else, leaving room for other payload types. The integration test now verifies the package imports via `state exec python3`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… (ENG-1793)
The private artifact's PYTHONPATH entry must share the separator and inherit
directives of the Python runtime's own PYTHONPATH entry, or the cross-artifact
environment merge fails ("incompatible separator or inherit directives"). Use a
canonical ":" separator and inherit=false to match it, rather than
os.PathListSeparator and inherit=true.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ENG-1643: Add end-to-end integration test for encrypted publish and install
ENG-1793: Install decrypted private-ingredient wheels at consume time
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.