Fix resource.enc race when functions share a bundle#6750
Conversation
When multiple `aws.Function` resources share a pre-built bundle
directory (via `bundle:`), Pulumi runs their `Runtime.Build` RPCs
concurrently. Each RPC encrypts the function's links and writes them
to `{bundle}/resource.enc` — so the writes race. The last writer wins,
but partial / interleaved writes can also leave a truncated or mixed
ciphertext that then fails AES-GCM authentication on the Lambda's
first cold start (surfaces as a `Decipheriv` error during SDK init).
Namespace the file by `FunctionID` when a bundle is shared
(`resource-{FunctionID}.enc`) and point the Lambda at its own file
via `SST_KEY_FILE`. The default (non-bundle) path keeps writing
`resource.enc` into each function's own artifact directory, so
existing deployments are unaffected.
Covered by new tests in `pkg/runtime/runtime_test.go`.
vimtor
left a comment
There was a problem hiding this comment.
this fixes the issue, but leaks other resource files since the whole bundle folder is uploaded to the lambda. we should probably split them by directory instead of file name, or maybe exclude the other resources from the uploaded zip file
…nID>/ @vimtor pointed out that the previous revision (`resource-<FunctionID>.enc` at the bundle root) fixes the concurrent-write race but still leaks every sibling's encrypted envelope into each Lambda's zip — they share the same app-level encryption key, so a deployed function could read its siblings' linked secrets/URLs. Move the per-function file under a `.sst/<FunctionID>/` subdirectory and teach the zipper to exclude every other function's subtree: * `pkg/runtime/runtime.go` writes to `{bundle}/.sst/<FunctionID>/resource.enc` (with mkdir) when a shared bundle is used; the non-bundle path keeps writing `resource.enc` at the per-function artifact root, unchanged. * `platform/src/components/aws/function.ts` sets `SST_KEY_FILE = .sst/<name>/resource.enc` to match, and filters the bundle glob so each function's zip carries only its own `.sst/<name>/` subtree. Filter is scoped to the bundle iteration only — copyFiles entries can't contain a `.sst/<id>/` subtree. * Tests assert the new layout for both the single- and multi-function cases.
|
Thanks @vimtor — pushed a follow-up that adopts the "split by directory" option you suggested first. Each function now writes to const sstDir = `.sst${path.sep}`;
const ownDir = `.sst${path.sep}${name}${path.sep}`;
const filtered =
item.from === bundle
? found.filter((file) => !file.startsWith(sstDir) || file.startsWith(ownDir))
: found;So each Lambda's zip carries only its own envelope — sibling envelopes (encrypted with the same app-level key) are dropped at zip time. The non-bundle path is unchanged (each function still gets its own artifact directory with Existing tests updated for the new layout, both subtests still pass: |
Problem
When multiple
aws.Functionresources share a pre-built bundle directory via thebundle:option, Pulumi runs theirRuntime.BuildRPCs concurrently. Each RPC encrypts the function's links and writes them to{bundle}/resource.enc, so the writes race.The last writer wins for the happy case, but partial / interleaved writes can leave a truncated or mixed ciphertext on disk — we see consistent trailing byte corruption after the 16-byte AES-GCM auth tag. The Lambda then fails to authenticate the ciphertext on its first cold start and the SDK throws at init:
Root cause
pkg/runtime/runtime.goCollection.Build:With
input.Bundle != "",result.Out == input.Bundleand the filename is constant, so N concurrent builds all race on the same path.In the non-bundle path, each function gets its own
.sst/artifacts/{FunctionID}-src/directory, so there's no collision.Fix
Namespace the file by
FunctionIDwhen a shared bundle is in play (resource-{FunctionID}.enc) and point each Lambda at its own file viaSST_KEY_FILE:pkg/runtime/runtime.go— writeresource-{FunctionID}.encwheninput.Bundle != "", keepresource.encotherwiseplatform/src/components/aws/function.ts— setSST_KEY_FILEto the per-function filename whenargs.bundleis setThe default (non-bundle) path is unchanged, so existing deployments keep the old filename.
Minor downside: each function's uploaded zip contains every
resource-{FunctionID}.encin the shared bundle dir (Lambda only reads its own viaSST_KEY_FILE). This is a few extra KB per function and avoids more invasive changes to the bundle-packaging code path. Happy to explore trimming if reviewers prefer.Test coverage
Adds
TestCollectionBuildEncryptedResourceFileWithBundletopkg/runtime/runtime_test.go:Bundle != ""and checksresource-{FunctionID}.encis written while the legacyresource.encis not.Real-world repro
This is hitting us in production (grasp-gg). We share a single pre-built bundle across ~40 Lambdas (Turbo-cached handlers, one-shot esbuild,
bundle:+ relativehandler:). After every deploy, a subset of Lambdas cold-start withDecipheriverrors until they're re-invoked or re-deployed. Byte-level inspection of the deployed zip consistently shows 6 garbage bytes after the expected ciphertext+tag — the telltale sign of an interrupted partial write landing on top of a complete one.Happy to iterate on naming / scope / packaging if maintainers want a different shape.