Sketch: provider-agnostic StorageService + S3/MinIO (Azure Blob replacement)#3921
Draft
TaprootFreak wants to merge 7 commits into
Draft
Sketch: provider-agnostic StorageService + S3/MinIO (Azure Blob replacement)#3921TaprootFreak wants to merge 7 commits into
TaprootFreak wants to merge 7 commits into
Conversation
Sketch for replacing Azure Blob Storage with an S3-compatible object store: - StorageService abstraction (same 6-method surface as the previous service) - S3StorageService (MinIO via @aws-sdk/client-s3), with Object Lock (Compliance mode) applied to configured compliance buckets - MockStorageService for local development - createStorageService() factory; the three call sites (KYC, support, fiat-output) switch from 'new AzureStorageService(...)' to the factory - s3 config block (secrets via env, per-env endpoint/url via compose, invariant settings as code constants) WIP/draft: Azure service kept for cutover fallback; dep installed via package-lock only (no local node_modules), CI validates the build.
- Drop per-object Object Lock from S3StorageService; WORM is enforced server-side via the bucket's default retention (Compliance mode, provisioned at setup) — robust against a client that forgets it - s3 config block reduced to connection settings only - Document S3StorageService as a protocol client (on-prem MinIO, not AWS cloud); abstraction stays provider-agnostic - Fix Prettier formatting (mock storage line wrap) that failed the format check
- Keep Blob/BlobContent/BlobMetaData required (structurally identical to the previous types) so downstream DTOs compile unchanged - Point the remaining type imports (kyc-file dto/mapper, support-issue controller/service) at storage.service instead of the deprecated azure module - S3 listBlobs fetches per-object metadata via HeadObject (S3 listings carry no content-type, unlike the Azure listing it replaces)
…anup) - Restore LOC dummy-file fallback in MockStorageService.getBlob (parity with the old mock; LOC document reads return bytes again) - Fail-fast S3 config validation (endpoint/region/keys/publicUrl required; publicUrl must end with '/') instead of an opaque SDK error at first use - URL-encode CopyObject CopySource (copyBlobs no longer corrupts keys with spaces/special chars) - Move blobUrl/blobName/encodeKey to the StorageService base (dedup); base now owns the container - Delete the now-unused AzureStorageService, drop @azure/storage-blob and the azure storage config block (appInsights kept) - Soften WORM doc to 'enforced via bucket default retention, provisioned in infra' - Document S3 metadata lowercasing / created==updated; remove unused logger - Add storage.service.spec.ts for the blobUrl/blobName round-trip invariant - Document the factory-over-DI rationale (per-container, runtime EP2 container)
- Guard empty GetObject body in getBlob (clear error instead of NPE) - copyBlobs uses a key-only listing (no per-object HeadObject fan-out) - blobName throws a clear error for URLs outside the container - Move spec into __tests__/ per repo convention; add container-mismatch case - Restore original 'pg' position in package.json (npm had reordered it) - Document that eager KYC/support providers make S3 config a boot-time fail-fast
- storage.service.spec.ts now sets up Config via TestUtil.provideConfig in beforeAll, so blobUrl no longer throws on an undefined Config singleton; also assert the full public-URL prefix (trailing-slash contract) - Shorten WORM doc comment (drop the dangling infra-RFC reference) - Fix stale 'azure-storage' wording in the local KYC seed script
… (Säule 2) (#3934) * Anchoring core: Merkle tree + OpenTimestamps wrapper (Säule 2, stage 1) - merkle.ts: sha256, buildMerkleRoot, merkleInclusionProof, verifyMerkleProof (SHA-256, parent=sha256(left||right), odd-node duplicated, 0=throw/1=leaf) - opentimestamps.service.ts: stamp/upgrade/verify wrapper over the public calendars - merkle.spec.ts: 19 unit tests (roots, per-leaf inclusion proofs, tamper detection) - add opentimestamps dependency * Anchoring: DB entities + ArchiveService (Säule 2, stage 2) - ArchiveBatch / ArchiveFile entities (IEntity base, BaseRepository pattern), migration for both tables + unique(bucket,name) + batch FK - ArchiveService: recordHash (idempotent), anchorPending (Merkle root over pending hashes -> batch -> OTS stamp), upgradeBatches (Bitcoin attestation), verifyDocument (hash match + Merkle inclusion proof + OTS verify) - ArchiveModule; archive.service.spec.ts (round-trip + tamper, real Merkle, mocked OTS) * Anchoring: scheduler, upload-path hash capture, WORM provisioning (Säule 2, stage 3) - Capture SHA-256 on compliance uploads: kyc-document (kyc) + fiat-output (EP2) call archiveService.recordHash after uploadBlob; support bucket untouched - ArchiveScheduler (@DfxCron + process guard): daily anchorPending, hourly upgradeBatches; ARCHIVE_ANCHOR/ARCHIVE_UPGRADE process entries - scripts/storage/provision-bucket.ts: create bucket with Object Lock + versioning + COMPLIANCE default retention (idempotent); note for dynamic EP2 - wire ArchiveModule into kyc + fiat-output modules; scheduler spec * Anchoring: fix opentimestamps transitive uuid resolution The opentimestamps lib pulls request->request-promise->uuid via the legacy 'uuid/v4' subpath, which the repo-wide uuid@9 override breaks (ERR_PACKAGE_PATH_NOT_EXPORTED). Scope a uuid@3.4.0 override under 'request' only (nested override) so its calendar HTTP works, without touching the app's uuid@9. Surfaced by the m5me integration test. * Anchoring: address review (anchored-hash guard, best-effort capture, upgrade persistence) - recordHash: never overwrite the hash of an already-anchored file; identical -> no-op, differing -> hard error+log (deterministic blob names could otherwise silently rewrite an anchored leaf and break verifyDocument) - KYC + EP2: recordHash is best-effort (try/catch + log), never rolls back the completed upload; EP2 sets reportCreated before anchoring to avoid a WORM re-PUT retry trap - upgradeBatches: persist upgraded .ots bytes whenever they change, even while still pending; confirm only on a real Bitcoin attestation - verify result exposes confirmed vs pending; verifyDocument loads batch via relations (removed dead branch); tests extended * Anchoring/storage: strong unit test coverage for new modules - s3-storage.service.spec (aws-sdk-client-mock): config validation, listBlobs pagination + per-key HeadObject, getBlob empty-body guard, uploadBlob, copyBlobs CopySource encoding -> 100% - mock-storage.service.spec: roundtrip, prefix filter, dummy-file fallbacks -> 100% - storage.factory.spec: LOC->Mock, otherwise->S3 -> 100% - opentimestamps.service.spec (lib mocked): stamp/upgrade/verify, confirmed vs pending -> 100% - archive.service.spec: closed upgrade-no-change + verifyDocument confirmed branches - kyc-document + fiat-output wiring specs: recordHash called with sha256, best-effort (upload not rolled back; reportCreated set before anchoring) - add aws-sdk-client-mock devDependency * Anchoring/storage tests: harden against silent regressions - archive.service.spec fakeStore now honors relations:['batch'] (batch only hydrated when requested) + explicit assertion that recordHash/verifyDocument request the relation -> removing it in prod now fails the suite - fiat-output spec asserts invocationCallOrder uploadBlob < reportCreated < recordHash (WORM re-PUT deadlock guard) - fiat-output spec covers the routeId -> sell.id fallback Mutation-checked: each guard fails when the corresponding prod behavior is removed * Anchoring: clear ESLint warnings (unused import + stale eslint-disable)
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Was
Design-Sketch (WIP, Draft) für die Ablösung von Azure Blob Storage durch einen S3-kompatiblen Object-Store (MinIO).
Führt eine provider-agnostische
StorageService-Abstraktion ein und implementiert sie gegen S3. Der dfx-api ist der erste Konsument; die Abstraktion ist generisch und liesse sich auch von anderen Diensten nutzen.Änderungen
storage/storage.service.ts— abstrakteStorageServicemit exakt der bisherigen 6-Methoden-Oberfläche (listBlobs,getBlob,uploadBlob,copyBlobs,blobUrl,blobName).blobUrl-Form bleibt identisch →blobNamereversibel, persistierte URLs konsistent.storage/s3-storage.service.ts—S3StorageService(MinIO via@aws-sdk/client-s3,forcePathStyle). Object Lock (Compliance-Mode) wird auf konfigurierte Compliance-Buckets angewandt — definierte, verlängerbare (nie verkürzbare) Retention.storage/mock-storage.service.ts— In-Memory-Mock für lokale Entwicklung (aus dem alten Mock-Mode portiert).storage/storage.factory.ts—createStorageService(container)(LOC → Mock, sonst → S3).new AzureStorageService(...)→createStorageService(...)): KYC, Support, fiat-output.config.ts—s3-Block (Secrets via env, per-Env-Endpoint/URL via compose, invariante Settings als Code-Konstanten).@aws-sdk/client-s3als Dependency.WORM ist bewusst nicht Teil der Methoden-Signaturen — der Service wendet Retention bucket-getrieben an.
Bewusst (noch) nicht in diesem Sketch
AzureStorageServicebleibt vorerst (read-only Fallback für den Daten-Cutover); wird danach entfernt, ebenso@azure/storage-blobund derazure-Config-Block.support_message.fileUrl/kyc_log.pdfUrl) ist ein separater Schritt.listBlobsliefert keinecontentType/Metadaten (S3-Listing tut das nicht; HeadObject nur falls ein Aufrufer es braucht — aktuell keiner).Hinweis
Sketch/WIP zur Design-Review, nicht merge-fertig. Dep wurde via
package-lock --package-lock-onlyergänzt (kein lokalesnode_modules); die CI validiert den Build.