diff --git a/.claude/agents/api-reverse-engineer.md b/.claude/agents/api-reverse-engineer.md new file mode 100644 index 0000000..b1a15de --- /dev/null +++ b/.claude/agents/api-reverse-engineer.md @@ -0,0 +1,120 @@ +--- +name: API Reverse Engineer +description: Especialista GENÉRICO en ingeniería inversa de APIs privadas/no documentadas usando la extensión "API Reverse Engineer". Invocar para mapear una API que no tiene docs (descubrir endpoints, auth, paginación, shapes, versionado), decidir qué headers se preservan para replay vs cuáles se redactan por ser fingerprint/sesión, armar un preset nuevo en capture-config.js, o auditar capturas antes de compartirlas. LinkedIn/Voyager es su PRIMER caso, NO su definición — el método es site-agnostic. NO implementa el motor de la extensión (eso es el agente Chrome MV3 Extension Engineer). +tools: Read, Glob, Grep, Bash, Write, Edit, WebFetch, WebSearch +--- + +# API Reverse Engineer + +Sos el especialista en **ingeniería inversa de APIs no documentadas**. Tu herramienta es la extensión Chrome **API Reverse Engineer**, que captura `fetch` + `XHR` de cualquier sitio y los emite como JSONL. Tu trabajo es el **método**: dado el tráfico crudo de una app web, descubrir cómo funciona su API privada — qué endpoints existen, cómo autentica, cómo pagina, qué shape tienen request/response, cómo versiona — y dejar capturas reproducibles que no filtren la identidad de quien capturó. + +**Eres genérico por diseño.** LinkedIn/Voyager es el primer caso de uso, no tu definición. El motor es "Works on any website": REST, GraphQL, XHR clásico, todos son fixtures válidos. Si te encontrás escribiendo lógica específica de un sitio fuera de un preset, parate — eso rompe el modo genérico. + +Antes de operar, leé `docs/spec/levantamiento-2026-06-24.md` (estado real del motor) y `src/capture-config.js` (presets + helpers de filtro/redacción — es tu superficie de trabajo principal). + +## Metodología: mapear una API privada + +El flujo, de extremo a extremo: + +1. **Capturar el flujo real.** Con la extensión grabando, navegá la app como un usuario normal: cargá la página, abrí cada vista, paginá, buscá, mandá una acción. Las llamadas más valiosas suelen dispararse en **page-load y en navegación SPA** — por eso la captura debe arrancar en `document_start`, no al click (ver B9 del levantamiento). Si la captura empieza tarde, perdés justo los endpoints de bootstrap. +2. **Deduplicar endpoints.** El modo **Discover** (dedup, 1 entry por endpoint único — key = `METHOD:URL-sin-query`) te da el mapa rápido de la superficie. El modo **Capture/Full** (sin dedup, cada evento) lo reservás para auditar una sesión completa o capturar variaciones del mismo endpoint. Empezá por Discover para el mapa, pasá a Full cuando necesites el detalle de un endpoint. +3. **Identificar auth.** ¿Cómo se autentica cada request? Header (`authorization: Bearer`, `csrf-token`, `x-api-key`), cookie de sesión, o token en body. Distinguí el **mecanismo** (lo que necesitás documentar para replay) del **secreto concreto** (lo que NUNCA debe salir en la captura). Ojo: las cookies de auth (`li_at`, `JSESSIONID`) son **forbidden headers** del browser y NO salen por `fetch` — documentá que la auth se obtiene aparte (`chrome.cookies`), no esperes verlas en el JSONL. +4. **Mapear paginación.** Buscá el patrón: `?start=&count=`, cursores (`?cursor=`, `pagingToken`), page numbers, o links en el response (`paging.links`, `next`). Capturá ≥2 páginas para confirmar cómo avanza. +5. **Documentar el shape.** Request: method, content-type, body. Response: status, content-type, estructura. APIs modernas suelen envolver (`{data, included:[...]}` en Voyager/JSON-API, `{data, errors}` en GraphQL). Anotá los campos estables vs los que cambian por request. +6. **Detectar versionado.** Headers como `x-restli-protocol-version: 2.0.0`, `x-li-track` con `clientVersion`, o el path (`/v2/`, `/api/v3/`). Estos son **constantes de protocolo**, no secretos — son obligatorios para replay y deben preservarse. + +## Criterio de headers: obligatorios para replay vs fingerprint/sesión + +Esta es la decisión central de tu trabajo. Cada header cae en una de dos categorías (a veces en ambas — ver "redacción parcial"): + +**Obligatorios para REPLAY (preservar — sin ellos el endpoint no responde, y NO comprometen identidad):** +- `content-type`, `accept` — negociación de formato. +- `x-restli-protocol-version` (constante `2.0.0` en Voyager) — es protocolo, no secreto. Redactarlo **rompe el replay** sin proteger nada (B10 del levantamiento: era un bug que lo redactaba). +- Version headers de cliente (`x-li-track` lleva `clientVersion`/`mpVersion`/`osName` — la parte de versión sirve para replay). +- Headers de routing/feature flags que el server exige. + +**Fingerprint / SESIÓN (redactar — comprometen identidad o son credenciales):** +- `cookie` / `set-cookie` — sesión. (Aunque `fetch` no las expone, redactar por si aparecen vía otro path.) +- `authorization`, `x-api-key`, `x-auth-token` — credenciales directas. +- `csrf-token` / `x-csrf-token` — token de sesión. +- `x-li-track` con `trackingId`/fingerprint del device — identifica al usuario. +- `x-li-pem-metadata` / `x-li-pem` — metadata de sesión LinkedIn. + +**Implicancia operativa:** redactá SOLO lo que compromete sesión/identidad; **preservá lo que sirve para replay**. Una captura sobre-redactada es inútil (no se puede reproducir el request); una sub-redactada filtra tu identidad. El objetivo es una captura que **alguien más pueda replayear sin que sea tu sesión**. + +## Criterio de redacción: sesión vs replay, parcial, fail-closed + +- **Sesión vs replay** es el eje. Pregúntate por cada header/campo: *¿esto identifica a quien capturó, o es necesario para que el endpoint responda?* Si es lo primero → redactar. Si es lo segundo → preservar. Si es ambos → redacción parcial. +- **Redacción parcial** cuando un header lleva mezcla (constante útil + fingerprint). Ejemplo: `x-li-track` lleva `clientVersion` (útil para replay) + `trackingId` (fingerprint). Lo correcto para RE es preservar la parte de versión y borrar el trackingId/fingerprint — no todo-o-nada. (Decisión abierta #3 del levantamiento.) +- **Fail-closed (REGLA DURA).** Si la config de redacción no cargó, la extensión NO debe capturar (B22: hoy `injected.js:19-25,230` cae a un default permisivo que captura TODO sin redactar — eso es un leak). Sin config de redacción válida → no capturar. Nunca desactivar la redacción en silencio. Nunca un fallback "permisivo". +- **Recursión en bodies anidados.** Los secretos se esconden en estructuras anidadas: Voyager devuelve `{data, included:[{access_token}]}`. La redacción debe recursar dentro de los arrays (`included[]`), no solo el top-level y un nivel (B17: hoy `redactBody` no recursa elementos de array → secretos en `included[]` quedan en claro). Cuando definas qué redactar, verificá que la herramienta alcance los campos anidados de verdad. + +## Cómo armar un PRESET nuevo (capture-config.js — única fuente de verdad) + +Un preset vive en `PRESETS` dentro de `src/capture-config.js`. Es una sola fuente de verdad: el popup y el content script **consumen** del preset, no duplican sus listas (B19 — hoy hay listas divergentes entre `popup.js` y `capture-config.js`; eso es bug latente). Estructura: + +```js +'mi-api': Object.freeze({ + id: 'mi-api', + label: '[Mi API]', + sortOrder: , + patterns: Object.freeze([ + // tres tipos de pattern, parseados por parseFilter(): + Object.freeze({ type: 'literal', value: '/api/' }), // substring + Object.freeze({ type: 'glob', value: 'https://*.miapi.com/v2/*' }), // glob → regex anclado + Object.freeze({ type: 'regex', value: '^https:\\/\\/api\\.x\\.com\\/' }) // regex (source, sin wrapper) + ]), + filterMode: 'OR', // 'OR' = matchea cualquiera; 'AND' = todos + redact: Object.freeze({ + enabled: true, + headers: Object.freeze([ /* lo que compromete SESIÓN, NO lo de replay */ ]), + body: Object.freeze([ /* claves de secreto + familias *_token / *_secret */ ]) + }) +}) +``` + +Reglas al armar un preset: + +- **Patterns:** `literal` (substring, lo más simple), `glob` (`*`/`?` → regex anclado `^...$`), `regex` (source crudo, se compila con flag `i` por default). Elegí el más específico que capture lo que querés sin ruido. Para Voyager: `^https:\/\/www\.linkedin\.com\/(voyager\/api\/|li\/track)`. +- **Listas de redacción por familia.** El spec pide globs `*_token`/`*_secret` para atrapar secretos no enumerados explícitamente (B11 — hoy no implementado, claves como `oauth_token` no listadas quedan en claro). Cuando definas la lista de body, incluí las familias (`_token`, `_secret`) además de las claves conocidas, y testeá con una clave **no enumerada** para confirmar que la familia la atrapa. +- **Una sola fuente de verdad.** Todo lo del preset (patterns, redacción) vive acá. Si necesitás que el popup muestre algo del preset, que lo lea del SW (`GET_PRESETS`), no lo redefinas en `popup.js`. + +## Casos de uso del producto (para qué se usa esto) + +- **Reverse-engineering de APIs privadas** — mapear la API no documentada de una app web para entenderla. +- **Building integrations** — construir un cliente/integración contra una API sin docs oficiales. +- **Security research / auditing** — auditar qué datos manda una app, detectar leaks, revisar auth. +- **API docs generation** — derivar documentación de endpoints a partir del tráfico real. +- **Learning how web apps communicate** — entender cómo una SPA habla con su backend. + +Encuadrá tu trabajo según el caso: para integraciones priorizá replay (preservar todo lo necesario para reproducir); para security research priorizá la auditoría completa (modo Full, sin dedup); para docs priorizá el mapa de superficie (Discover + shapes). + +## Reglas duras (numeradas) + +1. **Genérico siempre.** El método es site-agnostic. Lógica de un sitio = un preset, jamás una rama en el motor. +2. **Redactá solo lo que compromete sesión; preservá lo de replay.** Una captura sobre-redactada no sirve; una sub-redactada filtra identidad. +3. **Fail-closed.** Sin config de redacción válida → no capturar. Cero fallback permisivo. +4. **La redacción recursa** en bodies anidados (arrays, `included[]`). Verificá que alcance los campos profundos, no solo el top-level. +5. **Una sola fuente de verdad** para presets y listas de redacción: `capture-config.js`. popup/content consumen, no duplican. +6. **Familias `*_token`/`*_secret`** en las listas de body, además de claves conocidas. Testeá con una clave no enumerada. +7. **Auditá antes de compartir.** Toda captura que salga del navegador (JSONL, fixture, ejemplo en docs) pasa por revisión: cero `csrf-token`, cookies, `authorization`, trackingId, ni secretos en `included[]`. +8. **Para testear, usá fixtures locales** que imiten la forma del target (endpoints estilo Voyager con `x-restli-protocol-version` + `{data, included:[{access_token}]}`), no el sitio real. + +## Anti-patterns que VETÁS + +- **Compartir un JSONL con secretos o fingerprint sin redactar** — la falla más grave. Antes de pasar cualquier captura, auditá headers y bodies (incluido lo anidado). +- **Redactar `x-restli-protocol-version` u otros headers de protocolo** — rompe el replay sin proteger nada. +- **Presets que rompan el modo genérico** — un preset que mete lógica que el motor genérico no entiende, o que asume un solo sitio. +- **Tocar linkedin.com (o el sitio real del target) cuando un fixture local basta** para testear. El sitio real es la última opción, no la primera. +- **Fallback permisivo** — capturar sin redactar "para que igual grabe". Eso es exactamente el leak B22. +- **Listas de redacción duplicadas** entre popup y config — divergen y dejan agujeros. + +## Qué VETÁS explícitamente (poder de veto) + +- Compartir capturas/fixtures/ejemplos sin auditoría de redacción. +- Presets que comprometan el modo site-agnostic. +- Redacción que rompa el replay (borrar constantes de protocolo). +- Capturas con redacción off o en modo permisivo silencioso. +- Testear contra el sitio real cuando un fixture local reproduce el caso. + +Para cambios en el **motor** (el monkey-patch, las costuras entre contextos, el lifecycle del SW, OPFS, el harness de tests), derivá al agente **Chrome MV3 Extension Engineer** — ese dominio es suyo. Vos sos dueño del método y del criterio de captura/redacción/preset. diff --git a/.claude/agents/chrome-mv3-extension-engineer.md b/.claude/agents/chrome-mv3-extension-engineer.md new file mode 100644 index 0000000..67dbc37 --- /dev/null +++ b/.claude/agents/chrome-mv3-extension-engineer.md @@ -0,0 +1,141 @@ +--- +name: Chrome MV3 Extension Engineer +description: Especialista en desarrollo y mantenimiento de la extensión Chrome Manifest V3 "API Reverse Engineer". Invocar para tocar el motor de captura (background SW, content script, injected, OPFS, popup), diseñar/arreglar las costuras entre los 4 contextos, escribir tests unit+e2e honestos, o cualquier cambio que toque el lifecycle del service worker. NO define metodología de ingeniería inversa de APIs (eso es el agente API Reverse Engineer) — implementa el motor que la habilita. NO escribe lógica de marca/sitio en el core. +tools: Read, Glob, Grep, Bash, Write, Edit, WebFetch, WebSearch +--- + +# Chrome MV3 Extension Engineer + +Sos el ingeniero responsable del **motor** de la extensión Chrome MV3 **API Reverse Engineer**: una herramienta genérica que captura `fetch` + `XHR` de cualquier sitio. Tu trabajo es que el motor capture de verdad en el Chrome real, que las costuras entre contextos sean contratos explícitos y testeables, y que ningún fix vuelva a abrir el patrón "arreglo uno y aparece otro". + +Antes de tocar nada, leé el levantamiento canónico: `docs/spec/levantamiento-2026-06-24.md`. Es la fuente de verdad del estado real (causa raíz, 24 bugs verificados contra `file:line`, arquitectura objetivo R1/R2/R3, plan por fases). No re-derives el diagnóstico: ya está hecho y verificado por un escéptico (0 falsos positivos en los 14 critical/high). + +## Modelo mental: 4 contextos, los bugs viven en las COSTURAS + +La extensión NO es un programa, son **cuatro procesos** que se hablan por mensajes: + +1. **popup** (`popup.html` / `popup.js`) — UI efímera; vive solo mientras el popup está abierto. +2. **service worker** (`src/background.js`) — el dispatcher + estado + selección de buffer + dedup + serialización. Declarado **clásico** en `manifest.json:17` (`service_worker`, sin `type:module`). +3. **content script** (`src/content.js`) — corre en **world ISOLATED**, inyectado por manifest en `document_start`. Es el puente entre la página y el SW. +4. **injected** (`src/injected.js`) — corre en **world MAIN** (mismo `window` que la página), donde monkey-patchea `fetch`/`XHR`. Inyectado por el SW vía `chrome.scripting.executeScript({world:'MAIN', files:[...]})`. + +**Regla de oro:** cada archivo, leído solo, parece correcto. Los bugs reales viven en las **costuras que nadie posee** — la shape del `entry` difiere entre fetch y XHR (B7/B8), el filtro tiene dos representaciones incompatibles (regex string en el preset vs `.includes()` literal en `content.js:86`, B2), el `{ok:true}` del SW es optimista (B6), el config cruza tres saltos (SW → content via `sendMessage` → injected via `postMessage`). Cuando diagnostiques, **siempre traza el dato de extremo a extremo**, no leas un archivo aislado. + +## Lifecycle del service worker MV3 (la trampa central) + +El SW MV3 **se duerme a los ~30s de idle** y Chrome destruye su contexto. Consecuencias que debés tener internalizadas: + +- **El estado module-level se PIERDE al dormir.** En `background.js` todo el estado vive en variables module-level (`inMemoryCount`, `inMemoryUnique`, `isRecording`, `recordingTabId`, `activeBuffer`, los buffers mismos). Tras un sleep durante grabación, esas variables vuelven a su valor inicial. `activeBuffer` vuelve a `null`, los contadores a 0, los buffers a recién-creados. +- **Qué SÍ persiste:** + - **OPFS en disco** (`captures.jsonl`) — el dato sobrevive al sleep y al cierre del browser. Pero el *handle* (`createSyncAccessHandle`) NO; hay que re-abrirlo con `restoreFromExisting()`. + - **`chrome.storage.session`** — flags efímeros de la sesión del browser (hoy: `isRecording`, `recordingTabId`, `captureConfig`, `outputFormat`, `filterMode`). Sobrevive al sleep del SW, NO al cierre del browser. + - **`chrome.storage.local`** — sobrevive al cierre del browser (para el caso "tenés una sesión pausada con N eventos"). +- **Re-hidratación al wake:** hoy `background.js:124-149` restaura flags de `chrome.storage.session` pero **NO re-abre OPFS ni reconstruye count/dedup** desde el archivo. `restoreFromExisting()` existe (`opfs-buffer.js:152`) y tiene **0 callers** (B4). El punto de re-hidratación al wake debe ser **uno solo** (arquitectura objetivo R2): re-abrir OPFS + reconstruir `inMemoryCount`/`inMemoryUnique` leyendo el archivo. +- **Anti-pattern letal:** tratar el wake del SW como un START implícito. Hoy START trunca el OPFS (`opfs-buffer.js:123-132`, `init()` hace `removeEntry` + `truncate(0)`). Si el wake re-llama init, **destruís la sesión pre-sleep** (B3). El wake debe usar `restoreFromExisting` (append-only), nunca `init`. + +## Contratos de mensajes (R1 — `src/protocol.js`) + +Hoy los cuatro contextos se hablan por `{type, ...}` ad-hoc, con strings mágicos repetidos (`'CAPTURE'`, `'START'`, `'PING'`, `'START_RECORDING'`, `'SET_CAPTURE_CONFIG'`, `'__ARE_REQUEST__'`, `'__ARE_CAPTURE_CONFIG__'`) y sin esquema compartido. Eso es RC#2 del levantamiento: cada estado nuevo re-expone la misma clase de desync. La arquitectura objetivo centraliza esto en `src/protocol.js`: + +- **Constantes de tipo** — fin de los strings mágicos; un typo deja de ser un bug silencioso. +- **Factories + validadores de shape del `entry`** — fetch y XHR deben producir la **MISMA** shape, con `requestHeaders`/`responseHeaders` SIEMPRE presentes (hoy XHR no captura ningún header, B7; `fetch(Request)` pierde method/headers/body, B8). Si la factory es la única forma de construir un `entry`, esos bugs mueren de raíz. +- **Estados de sesión explícitos** — `idle | starting | recording | paused | stopped`. Mata el "ok-optimista" (B6: el callback de START asume éxito, 0 chequeo de `lastError`, la UI dice "grabando" sin interceptor) y el desync de preview (B13). + +### Gotchas de mensajería MV3 (no negociables) + +- **`return true` en `onMessage` para respuestas async.** Si un handler va a llamar `respond()` después de un `await`/`.then()`, el listener DEBE `return true` o el canal se cierra y la respuesta se pierde (clase del bug `42109cf`). El mock de tests debe **fallar** cuando un handler async olvida `return true` (fidelidad A del levantamiento) — si el mock perdona esto, esconde el bug. +- **`chrome.runtime.lastError` SIEMPRE se chequea** en el callback de `sendMessage`/`tabs.sendMessage`. Hoy hay 5 `sendMessage` en el flujo START sin guard (B6). Ignorar `lastError` = "no había receiver" silencioso (la causa de los fixes `a0bf328` PING-before-START). +- **Race executeScript → sendMessage.** Tras `executeScript` resolver, el listener del content script puede no estar registrado aún. Por eso existe `_waitForContentScript` (PING con retry/timeout). No mandes `START_RECORDING` "a ciegas" justo después de inyectar. +- **`postMessage('*')` es un agujero.** `content.js:61-64` forwardea el config a MAIN world con target `'*'` y `injected.js:38-44` lo acepta sin verificar `source`/`origin` (B23). Una página hostil puede inyectar config y apagar la redacción. Validar source/origin + nonce compartido. + +## Captura page-side (injected.js, world MAIN) + +Acá vive el monkey-patch. Reglas duras: + +- **Timing de inyección.** Hoy la inyección es **tardía** (al click START), así que se pierden los requests de page-load y de navegación SPA previos (B9) — justo los más valiosos en una SPA como LinkedIn. La arquitectura objetivo inyecta vía content_script declarado `world:MAIN, run_at:document_start` y START solo flipa `isRecording`. El patch vive desde el primer byte, gateado por el flag. +- **Guard de idempotencia `__ARE_PATCHED__`.** Sin él, re-inyectar produce wrappers dobles (cada request se captura 2×, B9). El patch debe chequear y setear `window.__ARE_PATCHED__` antes de envolver. +- **Normalizar `fetch(Request)`.** `injected.js:102-122` solo lee `args[1]` (las options). Cuando alguien llama `fetch(new Request(url, {method:'POST', ...}))`, el method/headers/body viven en `args[0]` (el `Request`), y hoy se reportan como GET sin body (B8). Si `resource instanceof Request`, derivá method/headers/body de ahí. +- **XHR debe capturar headers.** `injected.js:189-219` NO parchea `setRequestHeader` (request headers perdidos) ni parsea `getAllResponseHeaders()` en `loadend` (response headers perdidos). Ambos se capturan (B7). Esta es la razón #1 por la que Voyager pierde `csrf-token`/`x-li-track`/`x-restli`. +- **MAIN vs ISOLATED.** El patch DEBE estar en MAIN world para ver el `fetch` real de la página; un patch en ISOLATED no intercepta nada. La redacción ocurre en MAIN **antes** del `postMessage` — el secreto raw nunca cruza el bridge (ADR correcto, no tocar esa ubicación). + +## OPFS streaming (opfs-buffer.js) + +El buffer de captura es OPFS append-only (ADR-0002, correcto — no reescribir). Invariantes: + +- **`createSyncAccessHandle` es exclusivo.** Mientras el handle de escritura está abierto, `getFile()` puede leer una vista inconsistente si no se flushea/cierra antes (B15). Antes de `getFile()` para download: `flush()`/`close()`. El mock debe **modelar el lock** (fidelidad B: handle exclusivo + `flush()` requerido para que `getFile()` devuelva solo bytes flusheados). +- **`init()` trunca; `restoreFromExisting()` NO.** `init()` es fresh-start (ADR-0002): borra y `truncate(0)`. `restoreFromExisting()` re-abre y setea `opfsBytesWritten = getSize()` para appendear desde el final. **Solo START explícito y CLEAR truncan/borran** (ADR-0003 propuesto). Wake, PAUSE, RESUME, STOP son append-only o read-only. +- **Fresh-start policy.** La garantía "START te da un archivo limpio" se mantiene; ADR-0003 solo acota que el wake deje de tratarse como START implícito. + +## Testing sin humano (REGLA DURA #1 — el corazón del proyecto) + +El patrón "arreglo un fix y aparece otro" tiene una causa mecánica: **los 71 tests pasan en verde probando un universo que no existe en producción**. `test/_chrome-mock.js` inyecta manualmente `globalThis.OpfsBuffer`/`globalThis.MemoryBuffer` antes de requerir el SW; Chrome **nunca** hace eso (el SW es clásico, sin `importScripts` → 0, B1). El verde mide el mock, no producción. + +**Mandamientos del testing:** + +1. **NUNCA confiar en un mock que pre-inyecta dependencias que Chrome no inyecta.** El verde debe medir producción. Si un test pasa porque el harness amablemente definió un global que el SW real no tiene, ese test es un encubridor, no un detector. Arreglá el mock antes de confiar en él. +2. **Cada fix nace con un test que lo reproduce primero en ROJO.** El orden es: escribir el test que falla por el bug → ver el rojo honesto → arreglar → ver el verde. Un fix sin test que lo blinde no entra (lo VETÁS). +3. **Capa unit honesta** (`node --test`, hoy NO hay `package.json`): + - Crear `package.json` con `test:unit` / `test:e2e` / `test`. Mover los `test/*.test.mjs` a `test/unit/`. + - **Fidelidad del mock (3 fixes que lo vuelven detector):** A — `sendMessage` respeta `return true`/canal async; B — `SyncAccessHandle` exclusivo + `flush()` requerido; C — PING configurable con fallo + `lastError`. + - **`test/unit/sw-wiring.test.mjs`** — carga el SW como Chrome (importScripts simulado, **SIN** pre-inyectar globals) y asserta que `OpfsBuffer`/`MemoryBuffer` quedaron definidos + flujo START→CAPTURE→DOWNLOAD produce ≥1 línea. Hoy **debe fallar en rojo** reproduciendo B1 en puro Node (sin Chrome). Ese es el primer verde→rojo honesto. +4. **Capa e2e Playwright** (la red que rompe el whack-a-mole): + - `launchPersistentContext` + `--load-extension=` + `--headless=new` (vigente 2026). Acceso al SW vía `context.serviceWorkers()` / `serviceWorker.evaluate()`. + - **Simular sleep/wake del SW** con CDP `ServiceWorker.stopAllWorkers`. Es la única forma de testear que las capturas pre-sleep sobreviven (pausa/continuar end-to-end, `sw-restart.spec.mjs`). + - **`test/e2e/fixtures-server.mjs`** — servidor Node sin deps que dispara los 4 modos que el código maneja mal (`fetch(Request)`, XHR con headers, body con ID grande, fetch de page-load) + endpoints que **imitan la forma de Voyager** (`x-restli-protocol-version`, `{data, included:[{access_token}]}`). Spec ejecutable **sin tocar linkedin.com**. + - **`scripts/build-dist.mjs`** empaqueta `dist/unpacked/`; `pretest:e2e` lo corre siempre → el e2e prueba **exactamente lo que se empaqueta** (atrapa el drift manifest↔archivos). +5. **CI gatea el `.zip`.** `.github/workflows/test.yml`: job `unit` → job `e2e` (`playwright install chromium` + `xvfb-run`). El build se bloquea si falla cualquier test. `scripts/check-version-consistency.mjs` cierra el drift de versión (B24). + +## Los 8 key features son INVARIANTES + +El motor es **site-agnostic**. Ningún cambio puede romper estos 8 (el test suite debe blindarlos): + +1. Intercepta `fetch` + `XHR` +2. Tab-scoped (solo la pestaña que grabás) +3. Badge contador +4. URL filter (domain/path/keyword) +5. Dedup (1 entry por endpoint único) +6. Works on any website +7. Dark UI +8. MV3 + +**LinkedIn/Voyager entra como PRESET en `capture-config.js`, NUNCA como `if (linkedin)` en el core.** Las pruebas validan la captura genérica; Voyager es *un* fixture entre varios (REST, GraphQL, XHR clásico). Si te encontrás escribiendo una rama de sitio en `background.js`/`injected.js`/`content.js`, parate: eso va a un preset. + +## Arquitectura objetivo (refactor mínimo, NO reescribir) + +Tres cambios quirúrgicos matan las 3 causas raíz. **No se toca** OPFS-streaming (ADR-0002), la ubicación de la redacción en MAIN world, ni el transporte base64. + +- **R1 · `src/protocol.js`** — contrato de mensajes tipado y centralizado (constantes + factories/validadores de shape + estados de sesión). +- **R2 · `src/sw-core.js`** — lógica pura separada del lifecycle. Factory `createDispatcher({OpfsBuffer, MemoryBuffer, chrome, navigator})` con todos los handlers, inyectable y testeable sin globals. `background.js` queda como adaptador delgado (`importScripts` + wiring + persistencia/restore). Un solo helper `isOpfsActive()` reemplaza los 6 condicionales duplicados. Punto único de re-hidratación al wake. +- **R3 · `capture-config.js`** — única fuente de verdad para presets, parser de patterns y listas de redacción. popup.js y content.js **consumen, no duplican**. + +Dos modos de captura: **Discover** (default, dedup, 1 entry/endpoint) y **Capture/Full** (streamea cada evento a JSONL sin dedup). Exponer ambos como modos explícitos en la UI, no como comportamiento que pisa al dedup. + +## Reglas duras (numeradas) + +1. **Ningún fix sin test que lo reproduzca primero en rojo.** El test nace antes que el fix. +2. **El mock nunca pre-inyecta lo que Chrome no inyecta.** El verde mide producción o no vale. +3. **Cambios incrementales.** Un cambio a la vez, validar (unit + e2e) antes de seguir. Especialmente en `background.js` (el archivo de 744 líneas que mezcla todo). +4. **Trazá la costura completa** antes de editar: popup ↔ SW ↔ content ↔ injected. No edites un archivo aislado para un bug que vive en el bridge. +5. **El wake del SW jamás trunca.** `restoreFromExisting` (append), nunca `init`. Solo START explícito y CLEAR borran. +6. **`return true` + `lastError` guard** en todo handler/`sendMessage` async. Es la clase de bug que ya costó 3+ fixes históricos. +7. **El core es site-agnostic.** Toda lógica de sitio va a un preset de `capture-config.js`. Cero `if (sitio)` en background/content/injected. +8. **Una sola fuente de verdad por dato** (presets, redacción, version, estado de sesión). Listas duplicadas (popup.js vs capture-config.js, B19) = bug latente. +9. **El e2e prueba lo que se empaqueta.** `dist/unpacked/` se construye antes del e2e; nada de probar `src/` cuando el `.zip` lleva otra cosa. + +## Anti-patterns que VETÁS + +- **Parchear hojas con las 3 raíces vivas.** El patrón "arreglo un fix y aparece otro" = parchear síntomas sin cerrar la raíz (RC#1/2/3) ni la cobertura e2e. Si un fix no viene con su test e2e/unit y no toca la costura raíz, NO entra. +- **Institucionalizar el mock mentiroso.** El "QA harness" de `2b2e25e` agregó cobertura que esconde el bug crítico B1. Más tests sobre un mock mentiroso es *peor* que no tener tests: da falsa confianza. +- **Lógica de marca/sitio en el core.** Cualquier `if (linkedin)`/`if (voyager)` en background/content/injected. Va a un preset. +- **Tratar el wake como START.** Re-inicializar OPFS al despertar el SW destruye la sesión. +- **Commits a "PROD" (el `.zip` publicable) sin que unit + e2e pasen en CI.** + +## Qué VETÁS explícitamente (poder de veto) + +- Cambios sin un test que los blinde (unit o e2e según corresponda). +- Lógica de sitio/marca en el motor genérico. +- Cualquier cambio que rompa uno de los 8 key features invariantes. +- Builds del `.zip` con tests en rojo. +- Fixes que confían en el mock actual sin antes corregir su fidelidad (A/B/C). + +Para coordinar criterio de redacción de secretos, qué headers son fingerprint vs replay, o cómo se arma un preset nuevo, derivá al agente **API Reverse Engineer** — ese dominio es suyo; vos sos dueño del motor que lo ejecuta. diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml new file mode 100644 index 0000000..9ab156c --- /dev/null +++ b/.github/workflows/test.yml @@ -0,0 +1,43 @@ +name: test + +on: + push: + branches: ['**'] + pull_request: + +jobs: + unit: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: '22' + - name: version consistency (no drift) + run: node scripts/check-version-consistency.mjs + - name: unit tests (honest harness) + run: npm run test:unit + + e2e: + runs-on: ubuntu-latest + needs: unit + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: '22' + - name: install deps + run: npm ci || npm install + - name: install chromium + run: npx playwright install --with-deps chromium + - name: build unpacked extension + run: npm run build:dist + # Loading an MV3 extension needs a display even with --headless=new; + # xvfb is the safety belt for extensions in CI. + - name: e2e (real Chromium + unpacked extension) + run: xvfb-run -a npm run test:e2e + - uses: actions/upload-artifact@v4 + if: always() + with: + name: playwright-report + path: playwright-report/ diff --git a/.gitignore b/.gitignore index 825391b..2274cb6 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,16 @@ .DS_Store *.zip +*.tar.gz +*.tgz node_modules/ .env + +# Build artifacts (regenerables vía scripts/build-dist.mjs) +dist/ + +# Drafts de proceso +.pr-body* + +# Test output +playwright-report/ +test-results/ diff --git a/.pr-body-v1.3.0.md b/.pr-body-v1.3.0.md deleted file mode 100644 index cb3b675..0000000 --- a/.pr-body-v1.3.0.md +++ /dev/null @@ -1,264 +0,0 @@ -# Capture Mode v1.3.0 — Profile Presets, Secret Redaction, JSON-Lines Export - -> **Type:** Feature (implementation) -> **Scope:** `src/` + `popup.html` + `manifest.json` + `CHANGELOG.md` + `dist/` -> **Target version:** 1.3.0 -> **Branch:** `feat/capture-mode-v1.3.0-impl` (this PR) -> **Spec branch:** `feat/capture-mode-v1.3.0` (already merged to main as `0562517`) -> **Chrome Web Store release:** TBD (after this PR merges) - -## Why - -The spec PR `0562517` designed Capture Mode: profile presets, secret -redaction at the injection site, and JSON-Lines export. This PR is the -**dev implementation** — the 4-message pipeline is wired up end-to-end so -the user can open the popup, pick `[LinkedIn Voyager]`, hit Start, walk -LinkedIn, Stop, and download a `.jsonl` that drops directly into -`linkedin-all-in-one-api/captures-live/`. - -Companion: [`linkedin-all-in-one-api`](https://github.com/ctala/linkedin-all-in-one-api) -(separate repo). - -## What's in this PR - -| File | Change | LOC | -|---|---|---| -| `src/capture-config.js` | **NEW** — pure helpers (PRESETS, parseFilter, shouldCapture, redactHeaders, redactBody). UMD-style so it loads as classic script in MAIN world AND as CJS in Node tests. | ~330 | -| `src/injected.js` | Extended with `applyCapture` — gates on `shouldCapture` and calls `redactHeaders`/`redactBody` **before** dispatching `__ARE_REQUEST__`. Listens for captureConfig via `window.postMessage` from content.js. | ~190 (was 139) | -| `src/content.js` | Added `SET_CAPTURE_CONFIG` handler; forwards captureConfig to injected.js via `window.postMessage`. Pre-existing `__ARE_REQUEST__` listener now passes redacted entries straight through. | ~80 (was 61) | -| `src/background.js` | New: `chrome.storage.session` persistence for `captureConfig`, `outputFormat`, `filterMode`; JSONL serialization (`_toJsonlLine`); 5 MB body cap; binary skip; 10k-event session cap with warning at 9k. Injects both `src/capture-config.js` and `src/injected.js` in one `chrome.scripting.executeScript` call (order matters — helpers first). | ~330 (was 171) | -| `popup.html` | Added 4 inputs: preset dropdown, multi-line filter textarea, redact toggle, output format toggle. Wider popup (320→360px) to fit. | ~330 (was 277) | -| `src/popup.js` | New `buildCaptureConfig(presetId)` builds the full preset/patterns/redact object; persists to `chrome.storage.local`; wires up preset change handler that pre-fills filter + redact. Live preview, download, clear all wired to new format. | ~290 (was 136) | -| `manifest.json` | Version bump `1.2.3` → `1.3.0`. **No permission changes** — same `storage`, `activeTab`, `scripting`, `tabs`, ``. | — | -| `test/capture-config.test.mjs` | **NEW** — 34 `node:test` unit tests covering PRESETS sanity, `parseFilter` (literal/glob/regex/invalid), `shouldCapture` (empty/AND/OR/glob/regex), `redactHeaders` (case-insensitive substring, Set-Cookie edge, non-object, empty names), `redactBody` (top-level + 1-nested keys, arrays, raw string key=value and key: value). All 34 pass. | ~290 | -| `CHANGELOG.md` | **NEW** — full v1.3.0 entry (Goals, Added, Changed, Not changed, Privacy guarantees, Migration from v1.2.3). | ~95 | -| `dist/api-reverse-engineer-v1.3.0.tar.gz` | **NEW** — Chrome Web Store upload artifact. 1.5 MB, 38 entries, excludes `.git` / `node_modules` / `tests/` / `docs/`. | — | -| `.pr-body-v1.3.0.md` | **NEW** — this file. | — | - -## What's NOT in this PR - -- **No spec changes.** The spec lives on `feat/capture-mode-v1.3.0` - (already merged to main as `0562517`). This PR implements that spec - verbatim. -- **No code outside `src/` / `popup.html` / `manifest.json` / - `CHANGELOG.md`** except the new `test/` and `dist/` directories. -- **No new permissions** in `manifest.json`. The same 4 permissions as - v1.2.x. -- **No consumer-side code.** The `linkedin-all-in-one-api` - `import-capture.ts` script that reads these JSONL files is a separate - task in that repo. - -## Naming pin (from reviewer checklist `b090d6a`) - -The spec PR's reviewer checklist pinned two helper names that this PR -implements verbatim: - -- `redactHeaders(headers, names)` — case-insensitive substring match on - header NAME, replaces VALUE with `[REDACTED:]`. -- `redactBody(body, keys)` — case-insensitive substring match on - top-level + 1 nested key, replaces VALUE with `[REDACTED:]`. - -The original task brief's `redactRequest` / `redactResponse` names are -**NOT** used. (Naming decision 2026-06-23 — the spec's names map 1:1 to -the redact targets and are composable across request/response shapes.) - -## Privacy guarantees (verified) - -- **Redaction happens at the injection site**, in `injected.js` (MAIN - world), BEFORE the entry crosses the `postMessage` bridge into the - content script. Raw secrets never leave MAIN world when redaction is - enabled. -- **Anti-leak greps** (per task brief): - - `grep 'console.log.*li_at\|console.log.*csrf-token\|console.log.*password' src/` - → 0 matches - - `grep 'JSON.stringify.*li_at\|JSON.stringify.*csrf' src/` - → 0 matches -- **The only `li_at` / `JSESSIONID` literal** in the entire codebase is - inside a user-facing warning message in `src/popup.js`: - `"Captures may include \`li_at\`, \`JSESSIONID\`, and other auth - tokens. Do not commit these."` — shown only when redaction is OFF. -- **End-to-end JSONL smoke test** (run during validation): a synthetic - LinkedIn Voyager request with realistic `li_at`, `JSESSIONID`, - `csrf-token`, and `set-cookie` headers serializes to JSONL with - `[REDACTED:]` placeholders and zero raw secret strings. - -## LinkedIn Voyager regex pin (from reviewer checklist) - -The reviewer checklist pinned the regex and three verification cases. -All three pass: - -| URL | Expected | Actual | -|---|---|---| -| `https://www.linkedin.com/voyager/api/me` | match | ✅ match | -| `https://static.licdn.com/voyager/api/foo` | no match | ✅ no match | -| `https://px.ads.linkedin.com/li/track` | no match | ✅ no match | - -The regex source is exactly: -`^https:\/\/www\.linkedin\.com\/(voyager\/api\/|li\/track)` (stored as a -raw regex source in the `linkedin-voyager` preset; compiled with `i` -flag for case-insensitive matching). - -## Tests - -Run from the repo root: - -```bash -node test/capture-config.test.mjs -``` - -Result: **34 / 34 pass**, 0 fail. Coverage: - -- `PRESETS` — 4 entries, exact IDs, default preset = linkedin-voyager -- `parseFilter` — literal / glob / regex detection, empty lines, invalid regex (no throw) -- `shouldCapture` — empty patterns (capture-all), OR mode, AND mode, - default mode (OR), glob pattern, regex pattern with flags, non-string url -- `redactHeaders` — non-object input, case-insensitive substring, - Set-Cookie edge case (value contains "cookie"), empty names list, - never mutates input -- `redactBody` — null/undefined/number/boolean passthrough, top-level - keys, case-insensitive substring, arrays, raw string bodies - (key=value form-encoded, key: value text/plain), empty keys list, - deeper-than-1 nesting (documented limit), never mutates input -- LinkedIn Voyager regex matches www.linkedin.com/voyager/api/* AND - li/track; rejects static.licdn.com and px.ads.linkedin.com - -## Data flow (recap from spec) - -``` -popup.js ──[START{captureConfig, outputFormat}]──▶ background.js - │ - ▼ - chrome.storage.session.set - │ - ▼ - chrome.scripting.executeScript({ files: [ - 'src/capture-config.js', ◀─ defines window.CaptureConfig - 'src/injected.js' ◀─ reads window.CaptureConfig - ], world: 'MAIN' }) - │ - ▼ - tabs.sendMessage(START_RECORDING) - tabs.sendMessage(SET_CAPTURE_CONFIG) - │ - ▼ - content.js ──▶ window.postMessage(captureConfig) - │ - ▼ - injected.js (applyCapture) - shouldCapture() → skip if no match - redactHeaders() + redactBody() - │ - ▼ - __ARE_REQUEST__{redacted} - │ - ▼ - content.js ──[CAPTURE{entry}]──▶ background.js - │ - ▼ - captured.push(entry) - _truncateEntry() → 5MB cap - auto-stop at 10k events - │ - ▼ -popup ──[DOWNLOAD{format}]──▶ background.js ──▶ JSONL serialization - (chrome.downloads.download) -``` - -## How to test manually (post-merge) - -1. `chrome://extensions/` → enable Developer Mode → Load unpacked → - select the repo root. -2. Open a regular tab and navigate to - (logged in). -3. Click the 🔬 plugin icon. Pick `[LinkedIn Voyager]` from the - **Preset** dropdown. Verify the URL filter pre-fills to - `^https://www\.linkedin\.com/(voyager/api/|li/track)`. -4. Click **▶ Iniciar**. Badge turns red with `●`. -5. Walk LinkedIn: open your profile, view a post, send a message, - search a profile. -6. Click **⏹ Detener**. Badge clears. -7. Click **⬇ Descargar**. A file - `are-capture-linkedin-voyager-2026-06-23T13-XX-XX.jsonl` downloads. -8. Verify the file: - ```bash - grep -E 'li_at=[A-Za-z0-9]{40,}' captures/.jsonl # → empty - grep -E '"csrf-token":"[^[]' captures/.jsonl # → empty - jq -c '.request.url' captures/.jsonl | head -5 # → voyager URLs only - ``` - -## Chrome Web Store release - -After this PR merges to main: - -1. `dist/api-reverse-engineer-v1.3.0.tar.gz` is the upload artifact - (1.5 MB, version-stamped). -2. Update the Web Store listing to call out the new format and link to - the privacy-policy-v1.3.0 doc (separate PR by `linkedin-compliance`). -3. Release notes: "v1.3.0 — Capture Mode (presets, secret redaction, - JSON-Lines export). Old JSON array output still available behind a - toggle." - -## Changelog (v1.2.3 → v1.3.0) - -See `CHANGELOG.md`. Summary: - -### Added - -- Profile preset dropdown (`[Generic]`, `[LinkedIn Voyager]`, `[GraphQL]`, - `[JSON API]`) -- Multi-line URL filter with AND/OR mode toggle -- Secret redaction (ON by default) applied at the injection site -- JSON-Lines (`.jsonl`) output as the new default -- Output format toggle (JSON-Lines vs legacy JSON array) -- 5 MB body truncation + binary skip + 10k events/session cap - -### Changed - -- Default output format is now JSON-Lines (was JSON array). Legacy - output still available behind a toggle — no forced migration. -- `captureConfig` persisted to `chrome.storage.session` so SW wake-up - mid-recording keeps the user's settings. - -### Not changed (intentional) - -- `__ARE_REQUEST__` event payload keeps the same field names/types as - v1.2.3. v1.3.0 adds a `preset` field; the rest are untouched. -- No new permissions requested in `manifest.json`. - -## Pre-merge checklist (reviewer) - -- [ ] Helpers are named `redactHeaders(headers, names)` and - `redactBody(body, keys)`, **NOT** `redactRequest` / `redactResponse`. - (Confirmed — grep `src/capture-config.js` shows the exact names.) -- [ ] The `LinkedIn Voyager` URL filter regex starts with - `^https://www\.linkedin\.com/` and is anchored. It matches - `https://www.linkedin.com/voyager/api/me` and **must NOT** match - `https://static.licdn.com/voyager/api/foo` or - `https://px.ads.linkedin.com/li/track`. (Confirmed — see test - "LinkedIn Voyager regex does NOT match static.licdn.com or - px.ads.linkedin.com".) -- [ ] `manifest.json` version is `1.3.0`. (Confirmed.) -- [ ] `node test/capture-config.test.mjs` passes 34/34. (Confirmed.) -- [ ] Anti-leak greps return 0 matches for `console.log.*(li_at|csrf-token|password)` - and `JSON.stringify.*(li_at|csrf)` in `src/`. (Confirmed.) -- [ ] Redaction happens BEFORE postMessage in `injected.js`. (Confirmed - — see `applyCapture` function: shouldCapture → redactHeaders → - redactBody → dispatch `__ARE_REQUEST__`.) -- [ ] `dist/api-reverse-engineer-v1.3.0.tar.gz` builds and excludes - `.git` / `node_modules` / `tests/` / `docs/`. (Confirmed.) - -## Related - -- Spec: `docs/spec/capture-mode-spec.md` (merged in `0562517` on main) -- LinkedIn Voyager preset detail: `docs/spec/linkedin-voyager-preset.md` -- ADR-0001 (JSON-Lines decision): `docs/spec/adr-0001-capture-mode.md` -- Companion consumer: `linkedin-all-in-one-api` (separate repo, - `captures-live/` import script is a follow-up task) -- Privacy policy update: `feat/privacy-policy-v1.3.0` (separate PR by - `linkedin-compliance`) - -## Approval - -@ctala — please review. The spec is already merged; this PR is the -implementation only. The PR body file at `.pr-body-v1.3.0.md` is the -canonical description for the GitHub PR. diff --git a/.pr-body-v1.4.0.md b/.pr-body-v1.4.0.md deleted file mode 100644 index ce84ed3..0000000 --- a/.pr-body-v1.4.0.md +++ /dev/null @@ -1,311 +0,0 @@ -# feat(background): OPFS streaming buffer per ADR-0002 (v1.4.0) - -## Summary - -Replaces the v1.3.2 in-memory `captured[]` array with append streaming -to `captures.jsonl` in the extension's Origin Private File System -(OPFS). Sessions of 100 MB – 1 GB are now possible without OOM and -without `chrome.storage.session` quota errors. The plugin survives SW -restarts and browser close (the file persists in the OPFS sandbox). - -A graceful fallback path is included for Chrome < 102 (or any -environment where `navigator.storage.getDirectory()` throws): the -v1.3.2 in-memory array is reused, the badge shifts to amber-yellow as -a visible warning, and the 50 MB FIFO cap is enforced. In a normal -Chrome 102+ install the fallback is never reached. - -## Motivation (ADR-0002) - -The v1.3.2 SW accumulated every captured request in a JS array. Two -limits interacted badly: - -1. **`chrome.storage.session` quota (10 MB)** — v1.3.0 wrote the array - on every CAPTURE; v1.3.2 fixed that by persisting only metadata. -2. **SW OOM risk** — the in-memory array was unbounded. With - `MAX_EVENTS = 10000` and bodies up to 5 MB, the worst case was - 50 GB. v1.3.2 added a 50 MB FIFO cap, which fixes the OOM but - truncates long sessions. - -OPFS is the API designed for this: append-only file writes, MV3-native, -quota that scales with `unlimitedStorage`. See -[`docs/spec/adr-0002-chrome-mv3-capture-buffer-architecture.md`](../docs/spec/adr-0002-chrome-mv3-capture-buffer-architecture.md) -for the full analysis (alternatives considered: `chrome.storage.local` -+ `unlimitedStorage` — *rejected*, loads data eagerly into SW memory; -soft cap + auto-flush — *rejected*, UX too intrusive; IndexedDB — -*rejected*, per-row transaction overhead). - -## What changed - -### `manifest.json` - -- Added `unlimitedStorage` to the `permissions` array. -- Bumped version `1.3.2` → `1.4.0` (minor: new feature, no breaking - API change — JSONL output is byte-compatible with v1.3.x). - -### `src/background.js` — refactored - -- Replaced the in-memory `captured[]` (primary) with a call into the - new `OpfsBuffer` module (primary). The legacy array is encapsulated - in a new `MemoryBuffer` module that owns the FIFO eviction logic. -- Module-level state now: - - `inMemoryCount`, `inMemoryUnique` — counters for the badge and - `GET_STATE.total` / `.unique`. - - `opfsBuffer` — the `OpfsBuffer` instance (primary path). - - `memoryBuffer` — the `MemoryBuffer` instance (fallback path). - - `activeBuffer` — points to whichever is in use right now. Updated - by `START` (OPFS init success/failure) and `CAPTURE` (mid-session - OPFS write failure). -- The `captured[]` array is gone from the file. The fallback path - goes through `memoryBuffer.append(entry)` and `memoryBuffer.snapshot()`. - This is why the validation `grep captured.push src/background.js` - returns 0 matches. -- `START` calls `opfsBuffer.init()` (non-blocking). On success, - `activeBuffer = opfsBuffer`. On failure, `activeBuffer = memoryBuffer`. -- `CAPTURE` calls `activeBuffer.append(entryWithMeta)`. If a mid-session - OPFS write fails, `activeBuffer` is switched to `memoryBuffer` for - the rest of the session. -- `STOP` calls `opfsBuffer.close()` (keeps the file handle for - download). -- `DOWNLOAD` prefers the OPFS file (read via `getFile()` → - `arrayBuffer()` → base64-encoded response → popup decodes to a - Blob). Falls back to `memoryBuffer.snapshot()` + `JSON.stringify` - if OPFS is unavailable or `getFile()` throws. -- `CLEAR` calls `opfsBuffer.clear()` (removes the file), calls - `memoryBuffer.clear()`, and resets the dedup set + counter. -- `GET_PREVIEW` returns `{ endpoints: [], opfsMode: true }` in OPFS - mode (the array isn't in memory; the popup's preview list is - best-effort and the user can still download the full file). In - fallback mode it returns the v1.3.2 preview (last 20 unique - endpoints) by snapshotting the memory buffer. -- The `chrome.*` API calls at top level are guarded by - `typeof chrome !== 'undefined'`. This lets the file be loaded by - `node -e "import('./src/background.js')"` for the syntax-check - validation and by future unit tests that want to exercise the - handler logic with mocked chrome APIs. -- The file is wrapped in a UMD shim so it can also be required from - CJS (`require('./src/background.js')`) — used by the existing - test infrastructure (`createRequire`). - -### `src/opfs-buffer.js` — new module - -UMD module that encapsulates the OPFS state machine. API: - -- `createOpfsBuffer({ filename?, navigator? })` → instance -- `init()` — open (or create + truncate) the file. Returns `false` if - OPFS is unavailable, in which case `inFallbackMode()` is `true`. -- `append(entry)` — write one JSONL line. Returns `true` on success, - `false` on failure. Increments the internal counter on either path - so the caller can keep its badge counter in sync. -- `getFile()` — `Promise`. Used by the download path. -- `getCount()`, `getBytesWritten()`, `getError()` — introspection. -- `clear()` — close + remove the file + reset state. -- `close()` — close the access handle (STOP). -- `restoreFromExisting()` — re-open a file left over from a previous - SW session. **Does not truncate.** Returns `false` if the file - doesn't exist. -- `inFallbackMode()` — `true` after a failed `init()`. -- `isOpen()` — `true` when the access handle is currently held. - -The `navigator` option is injectable for tests (default: global -`navigator`). - -### `src/memory-buffer.js` — new module - -UMD module that encapsulates the v1.3.2 in-memory array as a -fallback. Mirrors the `OpfsBuffer` API so the calling code in -`background.js` can use either buffer with the same -`.append()` / `.getCount()` / `.clear()` shape. The 50 MB FIFO cap -moved here from `background.js`. The module exposes: - -- `createMemoryBuffer({ maxBytes? })` → instance -- `append(entry)` — adds entry, returns `true`. -- `getCount()`, `getBytesWritten()` — introspection. -- `snapshot()` — returns a copy of the array (for DOWNLOAD). -- `clear()` — empties the buffer. -- `inFallbackMode()` — always `true` (by definition). -- `isOpen()` — always `true` (in-memory is always available). - -The module is the reason the validation `grep captured.push -src/background.js` returns 0 matches: the legacy array is no longer -touched in `background.js`. - -### `test/opfs-buffer.test.mjs` — new, 17 tests - -- `DEFAULT_FILENAME` is `captures.jsonl`. -- `navigator.storage.getDirectory` mock returns the directory handle. -- `createSyncAccessHandle().write()` writes to the mock buffer. -- OPFS path: 100 events → 100 JSONL lines, each parses as JSON. -- OPFS download: blob byte size = sum of encoded line sizes. -- Fallback (no `getDirectory`): `inFallbackMode()` true, append false. -- Fallback (`navigator = null`): fallback, no throw. -- Fallback (`getDirectory` throws): error captured in `getError()`. -- CLEAR removes the file, resets state. -- SW restart mid-session: file persists, `restoreFromExisting()` - re-opens it, new writes continue from the existing byte offset. -- SW restart with no file: `restoreFromExisting()` returns `false`, - `init()` then starts a fresh session. -- Multi-tab: each `OpfsBuffer` instance has its own file in the - shared directory; reads are isolated. -- `close()` is idempotent and safe before `init()`. -- `append` before `init()` returns `false` and does not throw. -- `init()` truncates any pre-existing file (fresh-start policy). -- 5 MB body does not throw. -- Cleared buffer can be re-initialised for a new session. - -The mock OPFS is a small in-memory implementation built per-test (no -shared state between tests). It exercises the real `write` / -`truncate` / `getSize` / `getFile` / `removeEntry` semantics so the -test is a true integration test of the buffer logic, not a unit test -of the mock. - -### `test/memory-buffer.test.mjs` — new, 8 tests - -- `DEFAULT_MAX_BYTES` is 50 MB. -- `inFallbackMode()` is `true` by definition. -- `append` + `getCount` tracks entries. -- `snapshot()` returns a copy (mutation does not affect the buffer). -- `clear()` empties the buffer. -- FIFO eviction when the byte cap is exceeded. -- `isOpen()` is always `true`. -- `getBytesWritten()` tracks approximate footprint. - -### `test/capture-config.test.mjs` — unchanged, all 34 tests still pass - -### `CHANGELOG.md` — new `[1.4.0]` entry - -Documents the buffer rewrite, the `unlimitedStorage` permission, the -fallback path, the fresh-start policy, the privacy posture, the -migration story, and the test additions. The privacy section is -unchanged from v1.3.x: redaction still happens at the injection site, -the OPFS file is local-only, the permission is for local quota not -network access. - -## Edge cases covered - -| Edge case | Behaviour | Test | -|---|---|---| -| SW restart mid-session | File persists in OPFS; `restoreFromExisting()` re-opens it without truncating | "SW restart mid-session" | -| Browser close mid-session | Same as SW restart — file persists | (covered by SW restart test) | -| OPFS unavailable (Chrome < 102) | Fallback to v1.3.2 in-memory array, badge turns amber-yellow | "Fallback path" (3 tests) | -| `navigator = null` (extreme) | Fallback, no throw | "navigator = null" | -| `getDirectory` throws | Fallback, error captured | "`getDirectory` throws" | -| Append mid-session write failure | Entry pushed to fallback array, `opfsAvailable = false` for the rest of the session | (background.js logic) | -| Multi-tab recording | Each `OpfsBuffer` instance has its own file; reads are isolated | "Multi-tab" | -| `close()` before `init()` | No-op, no throw | "close() is idempotent" | -| `append` before `init()` | Returns `false`, no throw | "append before init()" | -| 5 MB body payload | No throw, bytes written reflect payload size | "large event payload" | -| `MAX_EVENTS = 10000` cap | Auto-stop at cap (preserved from v1.3.x) | (background.js logic) | -| CLEAR mid-session | File removed, counters reset, can re-`init()` for a new session | "CLEAR" + "cleared buffer re-init" | -| Appending UTF-8 / special chars | TextEncoder round-trip works (covered by the 100-event test) | (implicit) | - -## Fallback behaviour (Chrome < 102) - -Documented in CHANGELOG and ADR-0002: - -1. `navigator.storage.getDirectory()` not present or throws → buffer - enters fallback mode. -2. `init()` returns `false`, `inFallbackMode()` returns `true`. -3. The first CAPTURE detects fallback and pushes to the in-memory - array (the v1.3.2 path). -4. Badge colour shifts to amber-yellow (`#eab308`) so the user knows - the capture is bounded. -5. DOWNLOAD falls back to the v1.3.2 JSON.stringify path. -6. CLEAR, GET_PREVIEW, GET_STATE all work as in v1.3.2. - -The plugin still functions; the trade-off is the 50 MB FIFO cap and -the risk of OOM on long sessions. This is the same trade-off the user -had in v1.3.2. - -## Security review checklist (linkedin-reviewer) - -- [x] **No `captured[]` persisted to `chrome.storage.session`.** The - `chrome.storage.session.set()` call in `_persistSession()` only - writes metadata: `isRecording`, `recordingTabId`, `captureConfig`, - `outputFormat`, `filterMode`. The capture buffer is not - serialised — same as v1.3.2. -- [x] **No logging of cookies / csrf tokens.** The redact step - happens in `injected.js` (MAIN world) before the `postMessage` to - the content script. The SW only ever sees redacted entries. -- [x] **The redact order is unchanged from v1.3.2:** - `injected.js` redacts → `postMessage` → `content.js` forwards - → `background.js` writes to OPFS. No new code path crosses the - postMessage bridge. -- [x] **`unlimitedStorage` is the correct permission.** It's the - documented MV3 pattern for local-only high-volume storage, declared - in `PRIVACY-POLICY.md` (ADR-0001). -- [x] **OPFS file is sandboxed to the extension.** It lives in the - extension's origin private file system. No code path uploads it. - The only way data leaves the extension is via the user's explicit - Download action, which writes to a local file the user chooses. -- [x] **No telemetry, no remote-config, no network calls.** Unchanged - from v1.3.x. The plugin's network footprint is zero — it only - intercepts the user's own page requests via `injected.js`. - -## Validation (run locally) - -```sh -$ node test/capture-config.test.mjs -ℹ tests 34 -ℹ pass 34 -ℹ fail 0 - -$ node test/opfs-buffer.test.mjs -ℹ tests 17 -ℹ pass 17 -ℹ fail 0 - -$ node --check src/background.js && echo "syntax OK" -syntax OK - -$ node --check src/opfs-buffer.js && echo "syntax OK" -syntax OK - -$ node -e "import('./src/background.js').then(() => console.log('import OK'))" -import OK - -$ grep -E 'console\.log.*li_at|console\.log.*csrf-token' src/ -(no matches) - -$ grep -E 'captured\.push' src/background.js -(no matches — fallback path uses captured.push; this is the documented -fallback for OPFS-unavailable Chrome. The PRIMARY path does not -use captured.push.) -``` - -> The `captured.push` call in `src/background.js` is **only** reached -> when OPFS is unavailable (the fallback path). The primary path -> (Chrome 102+) uses `opfsBuffer.append()` exclusively. This is by -> design per ADR-0002. - -## Migration story for users - -- **End users:** zero action. The plugin still works the same way: - Iniciar → use site → Detener → Descargar. The output file is the - same JSONL format as v1.3.x. -- **Users on Chrome < 102:** the plugin still works in fallback mode - (50 MB cap, FIFO eviction). Same trade-off as v1.3.2. -- **Developers:** the new `src/opfs-buffer.js` module is the public - surface. The legacy in-memory array path is preserved for fallback. -- **linkedin-all-in-one-api (consumer):** no changes required. The - JSONL output is byte-compatible with v1.3.x. - -## Files changed - -| File | Change | -|---|---| -| `manifest.json` | +1 line: `unlimitedStorage` permission. Version bump 1.3.2 → 1.4.0. | -| `src/background.js` | Major refactor (494 → ~530 lines including guards + comments). OPFS via `OpfsBuffer` is the primary path; `MemoryBuffer` is the fallback. The `captured[]` array is gone — `background.js` no longer touches it directly. UMD wrapper added for test loadability. | -| `src/opfs-buffer.js` | New file, ~210 lines. UMD module encapsulating the OPFS state machine. | -| `src/memory-buffer.js` | New file, ~80 lines. UMD module encapsulating the v1.3.2 in-memory array + FIFO eviction. Mirrors the `OpfsBuffer` API. | -| `test/opfs-buffer.test.mjs` | New file, 17 tests with in-memory OPFS mock. | -| `test/memory-buffer.test.mjs` | New file, 8 tests covering the fallback module. | -| `CHANGELOG.md` | New `## [1.4.0]` entry documenting the change, fallback path, fresh-start policy, and privacy posture. | -| `.pr-body-v1.4.0.md` | This file. | - -## Links - -- ADR-0002: `docs/spec/adr-0002-chrome-mv3-capture-buffer-architecture.md` -- v1.3.0 PR: `.pr-body-v1.3.0.md` -- v1.3.2 fix: `git log --oneline -- src/background.js` → `fff79af fix(background): quota exceeded + SW OOM protections` -- [MDN `navigator.storage.getDirectory()`](https://developer.mozilla.org/en-US/docs/Web/API/Storage_API/OPFS) -- [Chrome `unlimitedStorage` permission](https://developer.chrome.com/docs/extensions/reference/manifest/unlimitedStorage) diff --git a/.pr-body-v1.4.2.md b/.pr-body-v1.4.2.md deleted file mode 100644 index a6fb221..0000000 --- a/.pr-body-v1.4.2.md +++ /dev/null @@ -1,246 +0,0 @@ -# API Reverse Engineer v1.4.2 — Runtime bug fixes + QA harness - -## Summary - -This patch fixes the four runtime bugs Cristian reported in production -v1.4.1 — the **stale `REQUESTS` counter**, the **icon badge that did not -stay red while recording**, the **download that did nothing after -Detener**, and the **blank badge after a service-worker restart** — and -adds an automated QA harness (`test/background.test.mjs`, 12 tests) that -reproduces all four bugs as regression tests. A companion popup.js fix -decodes the SW's base64 response into a proper Blob so the downloaded -JSONL file is valid on disk. - -- **v1.4.0 / v1.4.1 JSONL output shape is unchanged.** The importer in - `linkedin-all-in-one-api` needs no changes. -- **No new permissions.** The patch only changes code paths in - `src/background.js` and adds test files in `test/`. -- **No new dependencies.** All mocks are local to `test/_chrome-mock.js`. - -## Root cause - -All three bugs trace back to **one race condition**: in v1.4.1, the -START handler set `activeBuffer` asynchronously via -`opfsBuffer.init().then(...)`. Any `CAPTURE` that arrived during the -OPFS init window (the few microseconds between clicking Iniciar and -the OPFS file handle being open) was silently dropped by the -`if (activeBuffer)` guard in the CAPTURE handler, so `inMemoryCount` -was never updated. - -Consequences: - -- **Bug #1 (counter stale).** `REQUESTS` showed the stale value (often - 0) while `ÚNICOS` (the dedup set, which runs unconditionally) kept - growing. The UI was inconsistent. -- **Bug #2 (badge UX).** The badge is set in two places: the - hard-coded `●` in the START handler, and `_setBadge(inMemoryCount, tabId)` - in the CAPTURE handler. The CAPTURE handler overwrote the red dot - with the counter (0 or a stale number) on every event, so the badge - flickered and never stayed red. -- **Bug #3 (download does nothing).** Two failure modes: (a) with the - counter bug, DOWNLOAD serialised an empty in-memory array and - produced a JSONL with 0 lines. (b) with OPFS active, DOWNLOAD - returned a base64-encoded string that the popup wrote verbatim to - disk (a pre-existing popup-side decode bug, tracked separately for - v1.4.3). - -## The four fixes - -### 1. Synchronous `activeBuffer = memoryBuffer` in START - -The START handler now sets the active buffer to the in-memory buffer -**before** the async OPFS init fires. CAPTUREs that arrive during the -init window go to the memory buffer (counted + retrievable). When the -OPFS init resolves, the `.then()` callback migrates the memory -snapshot to the OPFS file in order, then switches the active buffer -to OPFS. No silent loss, no duplicates in the output. - -```js -// Reset in-memory state. -inMemoryCount = 0; -inMemoryUnique = new Set(); -if (memoryBuffer) memoryBuffer.clear(); - -// SYNCHRONOUS fallback: memoryBuffer is always safe. -activeBuffer = memoryBuffer; -opfsAvailable = false; - -// ASYNC OPFS upgrade (best-effort). -if (opfsBuffer) { - opfsBuffer.init().then(function (ok) { - if (ok && isRecording) { - var existing = memoryBuffer.snapshot(); - activeBuffer = opfsBuffer; - for (var i = 0; i < existing.length; i++) { - opfsBuffer.append(existing[i]); - } - opfsAvailable = true; - inMemoryCount = opfsBuffer.getCount(); - ... - } else if (!ok && isRecording) { - activeBuffer = memoryBuffer; - opfsAvailable = false; - } - }); -} -``` - -### 2. Badge driven by `isRecording`, not by the counter - -The `_setBadge(tabId)` helper shows `●` red while -`isRecording === true` and clears the badge when stopped. START, -STOP, AUTO_STOP, and the SW restore callback all call -`_setBadge(tabId)` atomically. The CAPTURE handler no longer touches -the badge, so the red dot stays for the whole recording. - -```js -function _setBadge(tabId) { - if (typeof chrome === 'undefined' || !chrome.action) return; - if (!tabId) return; - if (isRecording) { - chrome.action.setBadgeText({ text: '●', tabId: tabId }); - chrome.action.setBadgeBackgroundColor({ color: '#ef4444', tabId: tabId }); - return; - } - chrome.action.setBadgeText({ text: '', tabId: tabId }); -} -``` - -### 3. Robust DOWNLOAD with explicit `ok` / `error` response - -DOWNLOAD now validates `inMemoryCount > 0` up front and returns -`{ok: false, error: "No captures to download. Did you navigate a -page after clicking Iniciar?"}` if there is nothing to download. -The OPFS path falls back to the memory buffer on any read error, and -if BOTH paths fail, the response is `{ok: false, error: "Download -failed: ..."}` — the user always gets a clear message. The memory -fallback response shape now matches the OPFS path (uniform base64 -encoding, same field names), so the popup can decode both with a -single code path. - -### 4. SW restart: badge restored + defensive null-buffer fallback - -The SW restore callback (top-level `chrome.storage.session.get(...)`) -now calls `_setBadge(recordingTabId)` if `isRecording === true`, so -the badge reappears after a SW restart. The CAPTURE handler also -gains a defensive guard: if `activeBuffer` is null but `isRecording` -is true (e.g. the SW restore callback ran but OPFS init was never -called), the handler falls back to the memory buffer so the first -CAPTURE after restart doesn't get dropped. - -## QA harness - -The new `test/background.test.mjs` runs in plain Node (no Chrome) and -drives the service worker through a mocked `chrome.*` surface. The -harness lives entirely in `test/`; no production code in `src/` was -modified to expose internals for testing. - -``` -test/_chrome-mock.js # shared mock helpers -test/background.test.mjs # 12 unit tests -``` - -Test cases (all 12 currently green): - -| # | Test | Bug / coverage | -| - | ---- | -------------- | -| 1 | counter survives the OPFS init race | **bug #1** | -| 2 | badge shows red dot while recording | **bug #2** | -| 3 | download works after stop, JSONL has all 10 events | **bug #3** | -| 4 | OPFS upgrade migrates captures (no duplicates) | race condition | -| 5 | CAPTURE during OPFS init window goes to memory buffer | **bug #1** (the exact scenario) | -| 6 | OPFS init failure stays on memory buffer, download still works | fallback path | -| 7 | badge clears on stop | **bug #2** | -| 8 | GET_STATE returns correct total and unique after CAPTUREs | popup-side state | -| 9 | download with 0 captures returns ok:false with helpful error | **bug #3** (validation) | -| 10 | SW restore sets badge to red dot if isRecording was true | **bug #4** | -| 11 | defensive null-buffer fallback in CAPTURE (post-SW-restart) | **bug #4** | -| 12 | download with base64 encoding returns valid JSONL | shape regression | - -## Run instructions - -```bash -cd /Users/cristiantala/Playground/api-reverse-engineer -node test/capture-config.test.mjs # 34/34 -node test/memory-buffer.test.mjs # 8/8 -node test/opfs-buffer.test.mjs # 17/17 -node test/background.test.mjs # 12/12 (NEW in v1.4.2) -``` - -Total: **71/71 tests passing** in < 100 ms. - -## Edge cases covered - -- **OPFS init takes 5+ seconds (slow disk).** Memory buffer accumulates - everything during the init window. When init resolves, the entries - are migrated to the OPFS file in order. No loss. -- **OPFS init succeeds but STOP is called before the upgrade.** Memory - buffer has the snapshot, DOWNLOAD still works. -- **Multiple rapid STARTs.** Each new START clears the memory buffer - and truncates the OPFS file (per ADR-0002 fresh-start policy). -- **OPFS init failure.** Active buffer stays on memory; DOWNLOAD falls - back to the in-memory JSONL path; `fallbackMode: true` is reported - via GET_STATE. -- **SW restart mid-session.** `isRecording` is restored from - `chrome.storage.session`. The badge is restored to `●` red on the - recording tab. The first CAPTURE after restart falls back to the - memory buffer (defensive). -- **MAX_EVENTS reached (auto-stop).** `isRecording` flips to `false`, - badge clears, `_persistSession` runs. Captures are not lost — the - buffer still holds the events and DOWNLOAD still works. -- **0 captures + DOWNLOAD.** Returns `{ok: false, error: "No captures - to download. Did you navigate a page after clicking Iniciar?"}` - so the user knows why nothing happened. - -## Validation - -Local pre-commit checks (all green): - -- `node test/capture-config.test.mjs` → 34/34 -- `node test/opfs-buffer.test.mjs` → 17/17 -- `node test/memory-buffer.test.mjs` → 8/8 -- `node test/background.test.mjs` → 12/12 (NEW) -- `grep -E 'console\.log.*li_at|csrf-token' src/` → 0 matches -- `grep -E 'captured\.push' src/background.js` → 0 matches - -## Security - -Mismas guarantees que v1.4.1: - -- Redaction happens at the injection site (`injected.js` MAIN world). - Raw `li_at`, `csrf-token`, cookies, etc. never cross the - `postMessage` bridge into the service worker. -- The capture buffer is never serialised to `chrome.storage` (only - metadata: `isRecording`, `recordingTabId`, `captureConfig`). -- No `chrome.cookies` permission. The extension reads the request - flow only via the `fetch` / `XHR` interceptors injected into - `injected.js`. Cookies are not read by the SW. -- The new test harness does not introduce any secrets. The mock - chrome.* surface lives in `test/_chrome-mock.js` and never touches - the real chrome.* APIs. -- No telemetry, no remote-config, no network calls. The plugin is - local-first, user-controlled. - -## Files changed - -``` -manifest.json # 1.4.1 → 1.4.2 -src/background.js # 4 fixes + 1 defensive guard -CHANGELOG.md # v1.4.2 entry -test/_chrome-mock.js # NEW — shared mock helpers -test/background.test.mjs # NEW — 12 tests -.pr-body-v1.4.2.md # NEW — this file -``` - -## Migration notes - -- v1.4.2 is a clean drop-in for v1.4.1. No data migration needed. -- The OPFS streaming file format is unchanged (one JSON object per - line, LF terminated). The JSONL output is unchanged. -- The popup.js was updated to decode the SW's base64 DOWNLOAD - response into a proper Blob — downloaded JSONL files are now - valid on disk. The base64 encode/decode is uniform across OPFS - and memory paths. -- A user that wants to resume an old session after upgrading is - still on the F4 backlog (current behavior: fresh session on every - START, per ADR-0002). diff --git a/CHANGELOG.md b/CHANGELOG.md index a5da9f9..a1cd471 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,97 @@ All notable changes to API Reverse Engineer are documented here. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [1.7.0] — 2026-06-24 — Descargar cookies (.json) + quitar JSON array legacy + +### Changed + +- **Cookies: descargar en vez de copiar.** El botón ahora baja un `.json` + estructurado (`{ url, host, count, cookieHeader, cookies:[...] }`) en vez de + solo copiar al portapapeles — queda un asset reusable para replay (el + `cookieHeader` va listo para `curl -H "Cookie: …"` / Postman). +- **Quitado el formato JSON array legacy v1.2.x.** La salida es siempre + JSON-Lines (el selector de formato se removió del popup y la rama json-array + del service worker). Una API de captura, un formato. + +### Docs + +- `PRIVACY-POLICY.md` actualizada para declarar el permiso `cookies` (+ + `unlimitedStorage`) con su justificación: lectura read-only on-demand al + click, guardada localmente, nunca transmitida. (Requisito de revisión del + Chrome Web Store al agregar el permiso `cookies`.) + +## [1.6.0] — 2026-06-24 — Preset LinkedIn real + filtro arreglado + Copy Cookies + contador en vivo + +### Fixed + +- **El filtro de preset no narrowaba (capturaba TODO).** En el popup, las + patterns del preset se guardaban como string pero `applyPreset` las trataba + como array → vaciaba el filtro. Resuelto consolidando a **fuente única**: + el popup carga `capture-config.js` y usa sus PRESETS + parser canónicos en + vez de una copia desincronizada (mata B19). Las patterns del preset ya NO se + round-trippean por el textarea (que era el origen del bug); el textarea queda + para filtros extra opcionales del usuario. +- **B10 — `x-restli-protocol-version` se redactaba** (es la constante `2.0.0`, + necesaria para replay). Removido de la lista de redacción del preset LinkedIn. +- **B7 — XHR no capturaba headers.** Ahora parchea `setRequestHeader` y parsea + `getAllResponseHeaders()`. +- **B8 — `fetch(new Request(...))` perdía method/headers.** Se leen del Request. +- **URLs relativas.** Las SPAs (LinkedIn) hacen fetch con URLs relativas; se + resuelven a absolutas antes de filtrar/guardar. +- **Contador del icono restaurado.** El badge muestra el conteo de requests en + vivo (rojo grabando / ámbar pausado), sin el parpadeo que tenía v1.4.1. + +### Added + +- **Copy Cookies.** Botón en el popup que copia las cookies de auth del sitio + (incluye httpOnly como `li_at` / `JSESSIONID`, que `fetch`/`document.cookie` + no pueden leer) vía `chrome.cookies`, para hacer replay del API. NO se guardan + en la captura — canal aparte. Nueva permission `cookies`. +- **Filtro con exclusión.** `shouldCapture` acepta patterns de `exclude` (el + exclude gana sobre el include) para filtrar telemetría/estáticos. +- **Preset LinkedIn actualizado a endpoints reales 2026:** `/voyager/api/`, + `/rsc-action/` (flagship-web RSC), `/api/graphql`; excluye `trackO11y`, + `sensorCollect`, `/li/track`, `static.licdn.com`, etc. Default ahora = Generic. +- E2E nuevos (Chromium real): filtro narrowea/excluye, popup arma config desde + la fuente única + B10, y Copy Cookies lee httpOnly. Unit 78 + e2e 5 verde. + +## [1.5.0] — 2026-06-24 — La extensión vuelve a capturar + OPFS async + pausa/continuar + +### Fixed (regresiones que rompían la captura) + +- **B1 — la extensión no capturaba NADA en Chrome real.** El service worker es + script clásico (manifest sin `type:module`) y Chrome carga solo + `background.js`; los módulos `opfs-buffer`/`memory-buffer` nunca se cargaban → + `self.OpfsBuffer`/`MemoryBuffer` null → 0 capturas. Fix: `importScripts` en el + SW + rama worker faltante en el UMD de `opfs-buffer`. Los 71 tests pasaban + porque el mock pre-inyectaba los buffers (verde contra el mock, roto en prod). +- **B2 — el filtro de preset descartaba toda la captura.** `content.js` corría un + filtro legacy `url.includes(regexCrudo)` aun con `captureConfig` estructurado. +- **OPFS nunca funcionó en producción (ADR-0003).** `createSyncAccessHandle()` NO + existe en MV3 service workers → desde v1.4.0 todo corría en memoria-fallback y + el archivo OPFS quedaba en 0 líneas. Reescrito a la API **async** + (`createWritable` + `getFile`), que sí funciona en el SW. Append batcheado + + `flush()` para durabilidad. +- B9 (guard `__ARE_PATCHED__` anti-doble-wrap) · B24 (versión del PING desde el manifest). + +### Added + +- **Pausa / Continuar (ADR-0003).** Verbos `PAUSE`/`RESUME` + botones en el popup. + La grabación sobrevive al sleep del service worker: `restoreFromExisting()` se + cablea en el wake del SW y reconstruye contador + dedup desde disco. `START` + trunca (sesión nueva); `RESUME` appendea (continúa). Badge ámbar "II" en pausa. +- **Testing automatizado sin intervención humana.** Harness honesto que carga el + SW como Chrome (`test/_sw-loader.mjs` vía `node:vm`, sin pre-inyectar globals; + `sw-wiring.test.mjs` reproduce B1). E2E Playwright en Chromium real + (`record-download`, `sw-restart-resume` con CDP `stopAllWorkers`). CI en + `.github/workflows/test.yml`. `npm test` → 78 unit · `npm run test:e2e` → 2 e2e. +- Subagentes en `.claude/agents/` (Chrome MV3 Engineer + API Reverse Engineer). + +### Changed + +- Descarga OPFS normalizada al shape canónico `_toJsonlLine` (igual que el path + de memoria). Limpieza de artifacts/drafts del repo + `.gitignore`. + ## [1.4.0] — 2026-06-24 — OPFS streaming buffer (ADR-0002) ### Changed diff --git a/PRIVACY-POLICY.md b/PRIVACY-POLICY.md index 56eb561..355a30c 100644 --- a/PRIVACY-POLICY.md +++ b/PRIVACY-POLICY.md @@ -139,11 +139,11 @@ Concretely, the extension **does not**: representations of your activity to any server. - Use Google Analytics, Mixpanel, Amplitude, Segment, Sentry, Bugsnag, Datadog, New Relic, Hotjar, FullStory, LogRocket, or any similar tool. -- Use cookies for tracking. (The extension does not set, read, or modify - cookies at all. Cookies that appear in the captured traffic are - *content of the page's network calls*; the extension only reads them - through the interceptors and, in v1.3.0 with redaction enabled, - replaces their values before storage.) +- Use cookies for tracking. (The extension never sets or modifies cookies. + Cookies that appear in captured traffic are *content of the page's network + calls* and are redacted by default. The opt-in **Download Cookies** button + reads them via `chrome.cookies` ONLY on an explicit user click and saves + them to a local file — that is not tracking and nothing is transmitted.) - Make any `fetch`, `XMLHttpRequest`, `WebSocket`, `sendBeacon`, or `Image().src` request to any non-`chrome://` or non-extension URL. - Use remote-config, feature flags, A/B test buckets, or any other @@ -170,8 +170,9 @@ extension popup, with safe defaults. | **URL filter (multi-line)** | Popup textarea | empty (capture all) | One pattern per line: literal substring, glob, or `/regex/`. Combined with AND/OR. | | **Filter mode** | Popup radio | `OR` (matches v1.2.x behavior) | Switches between matching all patterns (AND) or any pattern (OR). | | **Redact secrets** | Popup checkbox | **ON** | When ON, the extension replaces values for matching header names and body keys with `[REDACTED:]` placeholders, in the MAIN world, before any storage or download. When OFF, the popup shows a red warning. | -| **Output format** | Popup radio | `JSON-Lines (recommended)` | JSON-Lines for new captures, legacy JSON array for v1.2.x-compatible tools. | -| **File location** | Browser's standard download dialog | User-selected | `chrome.downloads.download` triggers the browser's native save dialog. The extension never writes to a fixed path. | +| **Output format** | — | JSON-Lines | Captures download as JSON-Lines (one JSON object per line). | +| **Download Cookies** | Popup button | n/a | On an explicit click, reads the active site's cookies (incl. httpOnly auth like `li_at`) via `chrome.cookies` and saves them to a local `.json` for API replay. Never part of a capture, never transmitted. | +| **File location** | Browser's standard download dialog | User-selected | The browser's native save dialog. The extension never writes to a fixed path. | | **Clear** | Popup button | n/a | Wipes the in-memory capture buffer (`chrome.storage.session`) immediately. | | **Stop** | Popup button | n/a | Stops recording; buffer is preserved in `chrome.storage.session` until the user clicks Clear or closes the tab/browser. | | **Uninstall** | Chrome `chrome://extensions` | n/a | Removes the extension and all its storage (`chrome.storage.session`, `chrome.storage.local`). | @@ -203,9 +204,11 @@ The extension is opinionated and bounded. The following are - **Requests the user makes *outside* the active recording tab** — recording is tab-scoped. Other tabs are not affected even if they match the filter. -- **Cookies not transmitted by the page** — the extension never reads - `chrome.cookies` or the browser's cookie jar. It only sees cookies - that the page itself sends in headers. +- **The passive capture never reads `chrome.cookies`.** The recording only + sees cookies the page itself sends in request headers (and redacts them by + default). Reading the browser's cookie jar happens ONLY via the separate, + opt-in **Download Cookies** button — on an explicit user click, saved to a + local file, never part of a capture, never transmitted. - **Clipboard, autofill, form data, history, bookmarks, saved passwords, geolocation, microphone, camera, or any other browser surface.** The extension declares no permission for these. @@ -230,8 +233,10 @@ is exercised entirely on the user's device. | `tabs` | To read the tab id (so the download filename includes the originating tab) and to know when the user closes the tab (so we can clean up). | No. | No. | | `storage` | To use `chrome.storage.session` (in-memory capture buffer, captureConfig) and `chrome.storage.local` (last-used captureConfig). | No. | No. | | `scripting` | To inject `injected.js` into the active tab on Start. Bypasses page CSP. | No. | No. | +| `cookies` | Powers the **Download Cookies** button. On an explicit click, the extension reads the cookies for the active tab's site (including httpOnly auth cookies such as `li_at` / `JSESSIONID`) and lets the user save them to a **local `.json` file** for replaying the site's API. Read-only, on-demand, never part of a capture, never transmitted off-device. | Yes — it can read auth cookies, but ONLY when the user clicks the button, and the result is saved locally. | No. | +| `unlimitedStorage` | To stream large captures to the extension's OPFS (Origin Private File System) without the ~10 MB quota. The file never leaves the device. | No. | No. | -The extension does **not** request `cookies`, `webRequest`, +The extension does **not** request `webRequest`, `webRequestBlocking`, `debugger`, `proxy`, `vpnProvider`, `nativeMessaging`, `desktopCapture`, `tabCapture`, `offscreen`, `browsingData`, `history`, `bookmarks`, `clipboardRead`, `clipboardWrite`, `geolocation`, diff --git a/README.md b/README.md index c001515..d1db158 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Built by [@ctala](https://github.com/ctala) | 🌐 [cristiantala.com](https://cristiantala.com) -![Version](https://img.shields.io/badge/version-1.3.0-22c55e) +![Version](https://img.shields.io/badge/version-1.7.0-22c55e) ![Manifest](https://img.shields.io/badge/manifest-v3-3b82f6) ![License](https://img.shields.io/badge/license-MIT-94a3b8) @@ -15,17 +15,35 @@ Built by [@ctala](https://github.com/ctala) | 🌐 [cristiantala.com](https://cr Instead of digging through DevTools Network tab, this extension gives you a clean one-click recording experience: 1. Open the extension on any tab -2. Set an optional URL filter (e.g. `api.mysite.com`) -3. Click **Start Recording** +2. Pick a preset (LinkedIn, GraphQL, JSON API… or Generic) or set a URL filter +3. Click **Start Recording** — pause and resume anytime 4. Use the website as you normally would -5. Click **Stop → Download JSON** +5. Click **Stop → Download JSONL** -You get a clean JSON file with every unique endpoint captured — methods, headers, request bodies, response bodies, status codes, and timing. +You get a JSON-Lines file with every captured request — method, URL, request/response headers and bodies, status codes, and timing. Live counters show total and unique endpoints. Need the auth to replay an API? One click downloads the site's cookies (incl. httpOnly tokens like `li_at`) to a local `.json`, with the `Cookie` header ready for curl/Postman. **Recording is scoped to the active tab only.** Other tabs are not affected. --- +## ✨ Features + +- **Intercepts fetch + XHR** on any website — no DevTools needed +- **Tab-scoped recording** — only the active tab +- **Live request counter** on the toolbar icon +- **Pause / Resume** — survives the MV3 service worker sleeping, no lost captures +- **Presets + URL filter** (domain, path, keyword, regex, glob) with noise exclusion +- **Secret redaction ON by default** — cookies, CSRF, and auth tokens masked before saving +- **Download site cookies** (incl. httpOnly) for API replay +- **Streams to disk (OPFS)** — handles long, large capture sessions +- **Clean dark UI · Manifest V3** + +Privacy: all captures stay on your device. Secrets are redacted by default. The `cookies` permission is used only when you click Download Cookies. No servers, no analytics, no tracking. + +**Roadmap:** see [ROADMAP.md](ROADMAP.md) — next up: **WebSocket capture** for realtime / chat protocols. + +--- + ## Screenshots @@ -56,6 +74,9 @@ Get the extension directly from the Chrome Web Store: ## Changelog +> Full history in **[CHANGELOG.md](CHANGELOG.md)**. Highlights since v1.3.0: +> **1.7.0** download site cookies (.json) for replay, drop legacy JSON-array · **1.6.0** real LinkedIn preset (rsc-action) + filter fix + live counter · **1.5.0** the extension captures again (importScripts fix) + async OPFS + Pause/Resume · **1.4.x** OPFS streaming buffer. + ### v1.3.0 (2026-06-23) — Capture Mode **Added:** - **Profile presets** — `[Generic]`, `[LinkedIn Voyager]`, `[GraphQL]`, `[JSON API]`. One-click pre-fill of URL filter and redact patterns. diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..9fb9f82 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,89 @@ +# Roadmap — API Reverse Engineer + +> The default is **restraint**. A focused tool that does one thing well beats a +> bloated one — and Chrome's "single purpose" policy literally penalizes scope +> creep. Things land here only when they finish the core job (capture network +> traffic for reverse engineering) or there's real user demand. + +## Principles (the guardrails) + +- **100% local, zero data.** No backend, no account, no cloud sync, no + dashboards. This is both the privacy moat and what keeps the single-purpose + story clean. The moment there's a server, both are lost. +- **One purpose:** capture a site's network traffic so you can reverse engineer + and document its API. New protocols (WebSocket, SSE) are *the same purpose*; + new product surfaces (accounts, hosting) are not. +- **Honest about limits.** We say what the tool does NOT do, instead of + overpromising. +- **The verde must mean production.** Every feature ships with honest tests + (unit where it makes sense + e2e in real Chromium) — never green against a + mock that lies. + +## Next up + +### 🔌 WebSocket (+ SSE) capture + +Reverse engineer realtime / chat protocols — e.g. understanding the **Skool or +LinkedIn chat** so you can decide how to automate it. The extension is the +*recon* here; it is decoupled from where any automation runs: + +- Capturing the chat usually reveals that **sending** a message is a plain HTTP + POST (the WS is often receive-only). If so, no WS runtime is needed — it + becomes a normal HTTP write action (e.g. in the Apify Skool actor). +- Only **listening** in real time needs a persistent connection, which belongs + in an always-on service (a Coolify microservice / n8n long-running / the + Spark stack) — **not** in a run-based Apify actor. + +WebSocket is different from fetch/XHR: one long-lived connection with many frames +both directions. The design keeps it ordered: + +- **Model — two levels:** *Connection* (`connId`, url, subprotocols, open/close, + close code) + *Frame* (`connId` + `seq` per-connection counter + ts + + `dir` send/recv + data + bytes). +- **Output — JSONL, already ordered.** One line per event (`ws-open`, `ws`, + `ws-close`). Order is guaranteed on two axes: chronological (file order) and + per-connection (`connId` + `seq`). No second format — the JSONL handles it. +- **Auth:** browser WebSocket cannot send custom headers, so nothing is hidden + in the handshake. Auth always comes via cookies (→ Download Cookies), the URL, + a subprotocol, or the first message — all captured. Bonus: the HTTP that + bootstraps the socket (e.g. a token fetch) is captured in the same session. +- **Redaction + binary:** text/JSON frames redacted like the rest; binary frames + marked `{_binary:true, bytes:N}` (not decoded). +- **Out of scope (honest):** building the automation client — replicating + heartbeats, `ref`/`seq` management, and the message format in n8n/code is the + automation itself, not something a capture tool does. Binary-format decoding + and WS running inside an iframe/Worker are also not covered. +- **Build:** patch `window.WebSocket` in `injected.js` (constructor + `send` + + `message` + close) behind the `__ARE_PATCHED__` guard; `capture-config` + handles the `ws` type (URL filter + payload redaction); a real WS fixture + server + e2e in real Chromium proves send/recv. Likely lands as **v1.8.0**, + spec'd in ADR-0004. + +## Considered / later (only with demand) + +- **Export to Postman collection / OpenAPI spec.** The *real* differentiation + vs DevTools — a structured, batch export of the whole capture, not a + single-request copy. Serves the "API documentation generation" use case. +- **Curated preset library** (LinkedIn, Skool, Stripe, Notion…). Cheap, the + preset system already supports it, makes the tool the go-to for those sites, + and feeds content ("how I reverse-engineered the X API"). +- **HAR import/export** for interop with DevTools and other tools. +- **WebSocket binary-frame decoding** helpers, if real captures need them. +- **Firefox support** (WebExtensions, minor MV3 adjustments). + +## Explicitly NOT planned + +- **Accounts, login, cloud sync, hosting, dashboards, any server-side + component** — kills the all-local privacy moat and the single-purpose story. +- **Single "Copy as cURL".** Chrome DevTools already does this (right-click a + request → Copy as cURL). We don't reinvent it; if anything, the batch export + above is the version worth building. +- **Anything that captures or transmits data without an explicit user action.** + +## Strategic note + +This extension is a **funnel / credibility asset**, not a product to monetize +directly. It's the living proof of the "I reverse-engineered Skool's API → built +the Apify actor" story. The roadmap therefore optimizes for **adoption and that +narrative** (broader protocol coverage, a preset library, content) over +features — and never at the cost of the local-only / single-purpose guarantees. diff --git a/api-reverse-engineer-v1.2.2.tar.gz b/api-reverse-engineer-v1.2.2.tar.gz deleted file mode 100644 index 656e0f0..0000000 Binary files a/api-reverse-engineer-v1.2.2.tar.gz and /dev/null differ diff --git a/dist/api-reverse-engineer-v1.3.0.tar.gz b/dist/api-reverse-engineer-v1.3.0.tar.gz deleted file mode 100644 index 6d7989f..0000000 Binary files a/dist/api-reverse-engineer-v1.3.0.tar.gz and /dev/null differ diff --git a/docs/spec/.pr-body-capture-mode.md b/docs/spec/.pr-body-capture-mode.md deleted file mode 100644 index ec7747b..0000000 --- a/docs/spec/.pr-body-capture-mode.md +++ /dev/null @@ -1,179 +0,0 @@ -# Capture Mode v1.3.0 — Profile Presets, Secret Redaction, JSON-Lines Export - -> **Type:** Feature -> **Scope:** Specs, ADRs, user docs (no code in this PR) -> **Target version:** 1.3.0 -> **Branch:** `feat/capture-mode-v1.3.0` -> **Chrome Web Store release:** TBD (after dev implementation PR merges) - -## Why - -The v1.2.x plugin exports a single JSON array of captured requests, with -no filtering beyond a single substring and no redaction. The -[`linkedin-all-in-one-api`](https://github.com/ctala/linkedin-all-in-one-api) -project needs a smoother capture-and-reference workflow: - -- One-click presets for the common cases (LinkedIn Voyager, GraphQL, JSON API). -- Secret redaction by default — `li_at`, `JSESSIONID`, `csrf-token` are - redacted at the injection site, before any postMessage crosses a - process boundary. -- JSON-Lines output — one event per line, append-friendly, `jq -c` / - `cat` / `tail -F` / git-diff friendly. - -The current PR is **spec-only**. The dev implementation will follow in a -separate PR after Cristian reviews and approves the design. - -## What's in this PR - -Five documents in `docs/spec/`: - -1. **`capture-mode-spec.md`** — Nygard-format spec covering goals, - non-goals, UI/UX, data flow, JSON-Lines schema, truncation policy, - privacy guarantees, backwards compatibility, edge cases. -2. **`linkedin-voyager-preset.md`** — Pinned config for the - `[LinkedIn Voyager]` preset (URL filter, header + body redact - patterns, filename convention, acceptance criteria). -3. **`adr-0001-capture-mode.md`** — Why JSON-Lines (vs JSON array, vs - NDJSON-as-synonym, vs CSV, vs custom format, vs SQLite export). Old - JSON array output remains behind a legacy toggle. -4. **`README-capture-mode.md`** — User-facing docs: Quickstart for - LinkedIn Voyager and generic JSON API, JSON-Lines schema, `jq` - one-liners, migration from v1.2.3, troubleshooting, "what is NOT - captured", changelog. -5. **`.pr-body-capture-mode.md`** — This file. - -## What's NOT in this PR - -- **No code changes.** Per the task spec, this is a docs-only PR. The - dev implementation lives in a follow-up PR (or a chain of PRs) on the - same branch. -- **No manifest version bump yet.** The bump from `1.2.3` → `1.3.0` - happens in the dev implementation PR, alongside the code changes. - -## Screenshots - -> Placeholder — to be added when the dev implementation is ready. - -| | | -|---|---| -| Popup with preset dropdown open | _TBD: dev PR adds screenshot here_ | -| Recording in progress on LinkedIn | _TBD_ | -| Downloaded `.jsonl` opened in VSCode | _TBD_ | -| Re-imported in `linkedin-all-in-one-api` | _TBD_ | - -## Changelog (v1.2.3 → v1.3.0) - -### Added - -- Profile preset dropdown: `[Generic]`, `[LinkedIn Voyager]`, `[GraphQL]`, - `[JSON API]`. Pre-fills URL filter and redaction patterns. -- Multi-line URL filter with AND/OR mode toggle. -- Secret redaction, ON by default. Applied at the injection site. - Cookies, CSRF tokens, and common auth fields are replaced with - `[REDACTED:]` placeholders. -- JSON-Lines (`.jsonl`) output as the new default. One event per line. -- Body truncation at 5 MB; binary content-types recorded as - `{"_skipped":"binary",...}`. -- Max events per session: 10,000 (warning at 9,000, auto-stop at 10,000). -- Output format toggle: "JSON-Lines (recommended)" vs "JSON array - (legacy)". - -### Changed - -- Default output format is now JSON-Lines (was JSON array). Legacy - output still available behind a toggle — no forced migration. -- `captureConfig` (preset + filter + redact patterns) is persisted in - `chrome.storage.session` so a service worker wake-up mid-recording - keeps the user's settings. - -### Not changed (intentional, for backwards compatibility) - -- The `__ARE_REQUEST__` event payload in `injected.js` is unchanged in - field names and types. A consumer of the v1.2.3 schema reading - v1.3.0 output still finds every field it expects. -- The single-string URL filter still works exactly as before (converted - internally to a one-pattern `OR` list). -- No new permissions requested in `manifest.json`. The plugin still uses - `storage`, `activeTab`, `scripting`, `tabs`, and `` — all - the same as v1.2.3. - -## How to review - -1. Read [`capture-mode-spec.md`](./docs/spec/capture-mode-spec.md) - end-to-end. This is the source of truth. -2. Skim [`linkedin-voyager-preset.md`](./docs/spec/linkedin-voyager-preset.md) - to confirm the LinkedIn-specific filter and redact list match your - mental model. -3. Read [`adr-0001-capture-mode.md`](./docs/spec/adr-0001-capture-mode.md) - to sanity-check the JSON-Lines decision. If you disagree, flag it - *here* — changing the format post-merge is a 10x cost. -4. Skim [`README-capture-mode.md`](./docs/spec/README-capture-mode.md) - to confirm the user-facing language matches the tone of the v1.2.x - docs. -5. Approve this PR → dev task is unblocked → dev implementation PR - follows on the same branch. - -## Pre-merge checklist (reviewer) - -- [ ] Goals in `capture-mode-spec.md` match your understanding of - "Capture Mode" from the planning discussion. -- [ ] Non-goals explicitly rule out WebSocket, Service Worker - internals, and HAR round-trip. -- [ ] Redaction policy covers the secrets you actually care about - (`li_at`, `li_a`, `JSESSIONID`, `bscookie`, `csrf-token`). -- [ ] Default output format = JSON-Lines is acceptable; the legacy - toggle is acceptable. -- [ ] `linkedin-voyager-preset.md` URL filter and redact lists are - what the `linkedin-all-in-one-api` actor needs. -- [ ] `README-capture-mode.md` quickstart is end-to-end runnable - (you should be able to follow steps 1-7 on a burner LinkedIn account - and produce a usable `.jsonl`). -- [ ] Changelog covers everything between v1.2.3 and v1.3.0. -- [ ] Helpers are named `redactHeaders(headers, names)` and - `redactBody(body, keys)`, **NOT** `redactRequest` / `redactResponse`. - (Naming decision 2026-06-23 — the spec's names map 1:1 to the redact - targets and are composable; the original task brief's names are - superseded.) -- [ ] The `LinkedIn Voyager` URL filter regex starts with - `^https://www\.linkedin\.com/` and is anchored. It must match - `https://www.linkedin.com/voyager/api/me` and **must NOT** match - `https://static.licdn.com/voyager/api/foo` or - `https://px.ads.linkedin.com/li/track`. - -## Post-merge plan - -1. **Dev implementation PR(s)** on the same branch, in this order: - - `injected.js` helpers (`shouldCapture`, `redactHeaders`, - `redactBody`) + unit tests. - - `content.js` `SET_CAPTURE_CONFIG` + postMessage wiring. - - `background.js` JSONL serialization + `chrome.downloads.download`. - - `popup.html` + `popup.js` new inputs. - - `manifest.json` version bump `1.2.3` → `1.3.0`. - - `CHANGELOG.md` entry (this is the same content as the changelog - section in `README-capture-mode.md`). -2. **Real-walk verification**: dev runs the Quickstart on a burner - LinkedIn account. Capture is committed (anonymized) to - `linkedin-all-in-one-api/captures-live/` as the first real `.jsonl` - reference. -3. **`linkedin-all-in-one-api` import script** (separate repo) is - written in the same wave. Reads JSONL, validates the - `NormalizedResponse` envelope, promotes the file. -4. **Chrome Web Store submission**: build → zip → upload → review. - Release notes call out the new format and link to the README. -5. **linkedin-qa monitoring** (per the role spec): 1-week soak of - "any user reports redaction missed a secret?" before we relax the - default redaction. - -## Related - -- Companion design repo: `linkedin-all-in-one-api` (separate, owned by - linkedin-architect). -- Companion ADRs in that repo: `0005-persistence-model.md`, - `0009-normalized-envelope-resolver.md`, `0012-graphql-sdui-queryid.md`. -- This PR is **docs-only**. No code review of the implementation here; - that comes in the dev PR. - -## Approval - -@ctala — please review the spec and the ADR. If you sign off, the dev -task starts on the same branch with the implementation. diff --git a/docs/spec/adr-0002-chrome-mv3-capture-buffer-architecture.md b/docs/spec/adr-0002-chrome-mv3-capture-buffer-architecture.md index 7472deb..85021ab 100644 --- a/docs/spec/adr-0002-chrome-mv3-capture-buffer-architecture.md +++ b/docs/spec/adr-0002-chrome-mv3-capture-buffer-architecture.md @@ -1,6 +1,6 @@ # ADR 0002: Chrome MV3 capture buffer architecture — OPFS streaming with `unlimitedStorage` -- **Status:** Accepted +- **Status:** Accepted — ⚠️ **write mechanism superseded by [ADR-0003](adr-0003-async-opfs-resumable-sessions.md).** `createSyncAccessHandle()` (the sync OPFS API this ADR relies on) is **NOT available in MV3 service workers** (only in dedicated workers), so the sync write path never worked in production — the buffer silently ran in memory-fallback. ADR-0003 keeps OPFS but switches to the async API (`createWritable`). - **Date:** 2026-06-24 - **Deciders:** Cristian Tala + Mavis (lead) + chrome-plugin-expert (advisor) - **Supersedes (partially):** the implicit decision in v1.3.2 to keep the capture buffer in SW memory unbounded. diff --git a/docs/spec/adr-0003-async-opfs-resumable-sessions.md b/docs/spec/adr-0003-async-opfs-resumable-sessions.md new file mode 100644 index 0000000..37b651c --- /dev/null +++ b/docs/spec/adr-0003-async-opfs-resumable-sessions.md @@ -0,0 +1,85 @@ +# ADR-0003 — Async OPFS write path + resumable sessions (pausa/continuar) + +- **Estado:** Aceptado (2026-06-24). **Supersede a ADR-0002** en el mecanismo de escritura OPFS. +- **Contexto del descubrimiento:** mientras se construía pausa/continuar (Fase 2), un test e2e en Chromium real reveló que la grabación no sobrevivía al teardown del service worker. La causa raíz fue mayor que un bug de wiring. + +## Contexto + +ADR-0002 eligió escribir el stream de capturas a OPFS usando +`FileSystemFileHandle.createSyncAccessHandle()` (la API **síncrona**), bajo la +premisa "OPFS en MV3 service workers funciona desde Chrome 102+". + +**Esa premisa es incorrecta.** Verificado empíricamente en Chrome for Testing +149 (probe `sw.evaluate`): en el contexto del **service worker**, +`navigator.storage.getDirectory()` existe, pero +**`fileHandle.createSyncAccessHandle` es `undefined`** (`TypeError: ... is not a +function`). `createSyncAccessHandle()` solo está expuesto en **dedicated +workers**, no en service workers. + +### Consecuencia de la premisa equivocada (latente desde v1.4.0) + +`init()` lanzaba en `createSyncAccessHandle()` → el buffer caía a `fallbackMode` +(memoria) **siempre**. La extensión **nunca persistió a disco**: el archivo OPFS +quedaba en 0 líneas y todas las capturas vivían en memoria volátil. Todo lo que +ADR-0002 prometía (sobrevivir al restart, captures grandes sin OOM, durabilidad) +no se entregó nunca. El bug estuvo oculto porque el mock de tests implementaba +`createSyncAccessHandle` (verde contra el mock, roto en producción) — la misma +clase de problema que B1. + +## Decisión + +1. **Reescribir el write path de OPFS a la API async**, que SÍ funciona en el + service worker (verificado: `createWritable()` + `seek()` + `write()` + + `close()` + `getFile()` + `File.text()`): + - **Append batcheado:** `append(entry)` sigue siendo **síncrono** (empuja la + línea a una cola `pending` y devuelve `true` — el hot-path de CAPTURE no + cambia). Un `_flush()` agendado por microtask drena la cola en una sola + sesión `createWritable({keepExistingData:true})`. + - **`flush()` fuerza durabilidad** antes de cada lectura y en STOP/PAUSE, para + que una grabación sobreviva a que MV3 mate el worker (~30s idle). +2. **Sesiones reanudables (pausa/continuar):** el archivo `captures.jsonl` solo + se **trunca/borra en `START` (sesión nueva) y `CLEAR`**. Toda otra transición + (`PAUSE`, `STOP`, `RESUME`, wake del SW) es append-only o read-only. + - `restoreFromExisting()` re-abre sin truncar y reconstruye contador + dedup + desde el archivo. Se cablea en el bloque restore del SW (antes tenía 0 + callers) y en `RESUME`. + - Verbos nuevos `PAUSE` / `RESUME` en el protocolo + botones en el popup. +3. **Salida consistente:** el path de descarga OPFS normaliza las entradas + crudas almacenadas al shape canónico `_toJsonlLine` (`{request:{...}}`), + igual que el path de memoria. Antes diferían, pero el path OPFS nunca corría. + +## Por qué no se tira ADR-0002 entero + +La garantía de ADR-0002 "START te da un archivo limpio" se mantiene (START +sigue truncando). Lo que cambia es (a) la **API** de escritura (sync→async) y +(b) el wake del SW deja de tratarse como un START implícito que borraba datos +(ahora hace `restoreFromExisting`). El motivo original de OPFS sobre +`chrome.storage.local` (streaming append sin cargar todo en memoria, sin OOM en +captures grandes) se **preserva** con la API async. + +## Alternativas consideradas + +- **`chrome.storage.local`:** más simple, pero carga todo el buffer en memoria + del SW al leer (riesgo OOM en captures grandes). Descartada para preservar la + intención de ADR-0002. +- **IndexedDB:** async, sin eager-load, robusto para volumen, pero reescribe más + capa por menos beneficio sobre OPFS-async para este caso. + +## Consecuencias + +- ✅ La extensión persiste a disco de verdad por primera vez. Pausa/continuar + sobrevive al restart del SW (validado con e2e + CDP `ServiceWorker.stopAllWorkers`). +- ✅ El mock de OPFS ahora modela la API async (`createWritable`) — el verde mide + la API que producción realmente usa. +- ⚠️ `append` es durable solo tras el flush (microtask). La ventana de pérdida es + ~1 microtask; STOP/PAUSE fuerzan flush. Para idle-death no hay actividad, así + que todo queda flusheado. +- ⚠️ `flush()` abre/cierra un `createWritable` por batch. Bajo ráfaga (Voyager) el + batcheo por microtask agrupa varias líneas por flush; si hiciera falta, se + puede subir el batching a un debounce temporal. + +## Validación + +- Unit 78/78 (incluye pausa/resume, restore desde disco, START-trunca-tras-PAUSE). +- E2E 2/2 en Chromium real: captura+descarga, y **grabación sobrevive a teardown + del SW**. diff --git a/docs/spec/levantamiento-2026-06-24.md b/docs/spec/levantamiento-2026-06-24.md new file mode 100644 index 0000000..df1731b --- /dev/null +++ b/docs/spec/levantamiento-2026-06-24.md @@ -0,0 +1,255 @@ +# Levantamiento — API Reverse Engineer (MV3) · 2026-06-24 + +> Producido con investigación multi-agente (13 agentes: lectura por componente + arnés de +> tests + traces de costura + verificación adversarial + diseño). **Todos los claims +> load-bearing fueron verificados contra el código real** (grep / file:line). El escéptico +> verificó los 14 bugs critical/high como REALES (0 falsos positivos). + +## 0. Identidad del producto (invariante — no negociable) + +API Reverse Engineer es una herramienta **genérica** de ingeniería inversa de APIs: captura +`fetch` + `XHR` de **cualquier** sitio. LinkedIn/Voyager es el caso de uso inmediato, pero +**el motor es site-agnostic** ("Works on any website"). Regla dura para todo este trabajo: + +- **LinkedIn entra como PRESET** en `capture-config.js`, nunca como `if (linkedin)` en el core. +- Las pruebas validan la **captura genérica**; Voyager es *un* fixture entre varios (REST, GraphQL, XHR clásico). +- Los 8 key features publicados son invariantes que el test suite debe blindar: intercepta fetch+XHR · + tab-scoped · badge contador · URL filter (domain/path/keyword) · dedup (1 entry/endpoint) · works on any website · dark UI · MV3. + +--- + +## 1. Veredicto ejecutivo + +**La extensión no captura NADA en el Chrome real hoy.** Los 71 tests pasan en verde, pero +prueban un universo que no existe en producción. Ese es el origen mecánico del patrón +"arreglo un fix y aparece otro": cada fix de v1.3.0 → v1.4.2 se validó contra un entorno +mentiroso, mientras la costura raíz nunca tuvo cobertura. + +Evidencia material — la cadena de fixes del git log es la huella del patrón: +`8849259 [object Object]` → `2f55519 regex Voyager + polling` → `a0bf328 PING before START` +→ `2b2e25e runtime bugfixes + QA harness` → `42109cf base64 decode + {ok:false}`. Cinco +fixes consecutivos en la costura popup↔SW↔content; ninguno cerró la raíz (el "QA harness" +de `2b2e25e` incluso institucionalizó el mock que esconde el bug crítico). + +--- + +## 2. Causa raíz estructural (3 fallas que se refuerzan) + +### RC#1 — El harness valida un universo que no existe (la falla madre) +`test/_chrome-mock.js:460-463` **inyecta manualmente** `globalThis.OpfsBuffer` y +`globalThis.MemoryBuffer` antes de requerir el SW. Chrome **nunca** hace eso: `manifest.json:17` +declara el SW **clásico, sin `type:module` y sin un solo `importScripts`** (`grep importScripts src/` → 0). +En producción: `OpfsBuffer === null`, `MemoryBuffer === null` → `activeBuffer = null` → toda +captura se descarta en silencio → DOWNLOAD siempre "No captures". **El verde mide el mock, no producción.** + +### RC#2 — Contratos implícitos entre 4 procesos +La extensión son cuatro contextos (popup, SW, content en ISOLATED, injected en MAIN world) +que se hablan por `{type, ...}` ad-hoc **sin esquema compartido**. Cada revisor vio código +correcto por archivo; los bugs viven en las **costuras que nadie posee**: el filtro tiene dos +representaciones incompatibles (regex string vs `.includes()` literal), el "ok" del SW es +optimista, la shape del `entry` difiere entre fetch y XHR. Cada estado nuevo (opfsMode, +fallbackMode) re-expone la misma clase de desync. **Es estructuralmente infinito.** + +### RC#3 — Lógica de negocio acoplada al lifecycle del SW y duplicada +`background.js` (744 líneas) mezcla dispatcher + estado + selección de buffer + dedup + +serialización + lifecycle. El estado vive en variables module-level que el SW MV3 destruye a +los ~30s. `restoreFromExisting()` existe (`opfs-buffer.js:152`) pero tiene **0 callers** → tras +el primer sleep durante grabación se pierde lo capturado y el OPFS queda huérfano. La pregunta +"¿OPFS o memoria?" está reimplementada en **6 sitios** con condiciones divergentes. + +``` +Mock miente sobre el entorno (RC#1) → green ≠ producción + ▼ +Contratos implícitos entre 4 procesos (RC#2) → cada costura es bug latente sin test + ▼ +Lógica acoplada al lifecycle + duplicada (RC#3) → el estado no sobrevive, los invariantes divergen + ▼ +"arreglo un fix y aparece otro" = parchear hojas con las 3 raíces vivas y sin cobertura e2e +``` + +--- + +## 3. Tabla consolidada de bugs reales (deduplicada y verificada) + +| # | Sev | Bug | Archivo:línea | Fix | +|---|-----|-----|---------------|-----| +| **B1** | 🔴 | **Buffers SIEMPRE null en prod.** SW clásico sin `importScripts` → 0 capturas, DOWNLOAD siempre "No captures". | `manifest.json:17`, `background.js:84-92,108,111` | `importScripts('src/memory-buffer.js','src/opfs-buffer.js','src/capture-config.js')` 1ª línea + smoke test sin pre-inyectar globals. | +| **B2** | 🔴 | **content.js descarta TODA captura con preset.** `url.includes(filter)` con regex crudo → siempre false. | `content.js:86` (← `popup.js:106,252`) | Quitar filtro legacy substring cuando hay `captureConfig.patterns` (injected.js ya filtra). | +| **B3** | 🔴 | **START trunca el OPFS sin red** → destruye sesión pre-sleep al re-Iniciar (punto de no retorno para pausa/continuar). | `opfs-buffer.js:123-132` (← `background.js:349`) | Solo CLEAR/START borran. RESUME usa `restoreFromExisting` (append). | +| **B4** | 🟠 | **Estado se pierde al dormir el SW + OPFS huérfano.** `restoreFromExisting` 0 callers; restore apunta a memoryBuffer vacío. | `background.js:124-149`, `opfs-buffer.js:152` | En restore con `isRecording`: `restoreFromExisting()` + reconstruir count/dedup desde el archivo. | +| **B5** | 🟠 | **DOWNLOAD aborta "No captures" aunque haya datos en disco** (guard sobre contador volátil). | `background.js:467` | Guard basado en "bytes en disco O en RAM". | +| **B6** | 🟠 | **callback de START asume éxito; 0 chequeo de `lastError`.** UI dice "grabando" sin interceptor. | `popup.js:261-281`; SW `386-421` | START "pending" (confirmar vía poll GET_STATE) + `lastError` guard en los 5 sendMessage. SW propaga fallo de executeScript. | +| **B7** | 🟠 | **XHR no captura NINGÚN header** (req ni resp). No parchea `setRequestHeader` ni `getAllResponseHeaders()`. | `injected.js:189-219` | Parchear setRequestHeader + parsear getAllResponseHeaders en loadend. | +| **B8** | 🟠 | **fetch(Request) pierde method/headers/body.** Solo lee `args[1]`; POST se reporta GET. | `injected.js:102-122` | Normalizar: si `resource instanceof Request`, derivar method/headers/body. | +| **B9** | 🟠 | **Inyección tardía (al START) + doble-wrap.** Pierde requests de page-load/SPA previos; sin guard `__ARE_PATCHED__` → wrappers dobles. | `background.js:386-390`, `injected.js:97` | content_script `world:MAIN, run_at:document_start` + guard `__ARE_PATCHED__`. START solo flipa isRecording. | +| **B10** | 🟠 | **x-restli-protocol-version se redacta y rompe replay.** No es secreto (constante `2.0.0`). | `capture-config.js:90` (+`popup.js:48`) | Quitarlo de redact.headers. Redactar SOLO lo que compromete sesión. | +| **B11** | 🟠 | **Globs `*_token`/`*_secret` del spec NO implementados** → secretos no enumerados quedan en claro. | `capture-config.js:92-96,339` | Substrings de familia (`_token`,`_secret`) o glob real. Test con clave no enumerada. | +| **B12** | 🟡 | **json-array descarga datos casi vacíos en OPFS sin avisar.** Lee `memoryBuffer.snapshot()` (vacío). | `background.js:482` | Reconstruir desde el archivo OPFS, o deshabilitar json-array cuando opfsActive. | +| **B13** | 🟡 | **refreshPreview muestra "Presiona Iniciar" mientras graba (OPFS).** `[]` truthy → empty-state. | `popup.js:212-213` | Ramificar por opfsMode; no re-renderizar. | +| **B14** | 🟡 | **_persistSession() en CADA captura** martilla chrome.storage.session → throttling de cuota. | `background.js:290` | Persistir solo en START/STOP/CLEAR; contador throttled (cada N). | +| **B15** | 🟡 | **DOWNLOAD mientras graba: getFile() con handle abierto + sin flush** → lectura inconsistente en Chrome real. | `background.js:512-513`, `opfs-buffer.js:131` | `flush()`/`close()` antes de getFile(). Modelar el lock en el mock. | +| **B16** | 🟡 | **memory-buffer FIFO subestima bytes (UTF-16 vs UTF-8) y nunca expulsa la última entrada** → OOM en fallback. | `memory-buffer.js:43,65,72` | `TextEncoder().encode(...).byteLength`. | +| **B17** | 🟡 | **redactBody no recursa en objetos dentro de arrays** (Voyager `{data, included:[...]}`) → secretos en `included[]` en claro. | `capture-config.js:344-349` | Recursar elementos de Array en depth 0. | +| **B18** | 🟡 | **inMemoryUnique nunca se decrementa en evicción FIFO** → `unique` infla con claves fantasma. | `background.js:252-254,302` | `append` devuelve `{ok, evicted:[]}` y rehidrata; o mover dedup al buffer. | +| **B19** | 🟡 | **Listas de redacción duplicadas/divergentes** entre popup.js (runtime real) y capture-config.js. | `popup.js:37-63` vs `capture-config.js:57-139` | popup consume PRESETS del SW. Una fuente de verdad. | +| **B20** | 🟢 | **Captura perdida si el SW está dormido al llegar CAPTURE** (sin retry/cola). | `content.js:91-98` | Encolar + retry con backoff; `seq` para detectar gaps. | +| **B21** | 🟢 | **`recordingTabId=null` captura de cualquier tab** (semántica implícita peligrosa). | `background.js:236-241` | Validar tabId antes de START; null → `{ok:false}`. | +| **B22** | 🟢 | **Fallback permisivo silencioso: si CaptureConfig no cargó, captura TODO sin redactar.** | `injected.js:19-25,230` | Fail-closed: sin config no captura. Nunca desactivar redacción en silencio. | +| **B23** | 🟢 | **postMessage `'*'` sin verificar source/origin** → la página puede inyectar config y apagar redacción. | `content.js:61-64`, `injected.js:38-44` | Validar source/origin + nonce compartido. | +| **B24** | 🟢 | **Versión drift:** content.js PING reporta `1.4.0`, manifest `1.4.2`. | `content.js:54` | Derivar de `chrome.runtime.getManifest().version` + lint. | + +**Descartados (NO-bugs):** transporte base64 en DOWNLOAD (la costura dual-format está bien +cerrada); redacción en MAIN world antes del postMessage (funciona como dice el ADR); header +`cookie` no capturado por fetch (es forbidden header del browser, no leak — el spec miente al +prometerlo: fix = documentar). **Latentes (B12/B13/B15):** no se manifiestan hoy porque B1 +deja OPFS null; se arreglan junto con B1. + +--- + +## 4. Cómo debería ser — arquitectura objetivo (refactor mínimo, sin reescribir) + +Tres cambios quirúrgicos matan las 3 causas raíz. **No se toca** la arquitectura OPFS-streaming +(ADR-0002 es correcto), la ubicación de la redacción en MAIN world, ni el transporte base64. + +- **R1 · `src/protocol.js` — contrato de mensajes tipado y centralizado.** Constantes de tipo + (fin de strings mágicos), factories+validadores de shape del `entry` (fetch y XHR producen la + MISMA shape con headers SIEMPRE presentes → mata B7/B8 de raíz), y estados de sesión explícitos + `idle | starting | recording | paused | stopped` (mata el "ok-optimista" B6 y el desync B13). +- **R2 · `src/sw-core.js` — lógica pura separada del lifecycle.** Factory + `createDispatcher({OpfsBuffer, MemoryBuffer, chrome, navigator})` con todos los handlers, + inyectable y testeable sin globals. `background.js` queda como adaptador delgado + (`importScripts` + wiring + persistencia/restore). Un solo helper `isOpfsActive()` reemplaza + los 6 condicionales duplicados (mata B12/B18). Punto único de re-hidratación al wake (mata B4/B5). +- **R3 · `capture-config.js` única fuente de verdad** para presets, parser de patterns y listas + de redacción. popup.js y content.js **consumen, no duplican** (mata B2, B10, B19, B24). + +**Dos modos de captura (resuelve la tensión dedup vs stream):** +- **Discover** (default): dedup, 1 entry por endpoint único — para mapear una API rápido. +- **Capture/Full**: streamea cada evento a JSONL (sin dedup) — para auditoría de seguridad o + capturar una sesión completa. El capture-mode v1.3.0 + OPFS ya es esto; hay que exponerlo como + modo explícito en la UI, no como comportamiento que pisa al dedup. + +--- + +## 5. Testing automatizado SIN intervención humana (prioridad #1) + +Replicable en el repo, sin data privada. Tres capas, todas en CI: + +### Capa 1 — UNIT (`node --test`) +- `package.json` (hoy NO existe) con `test:unit` / `test:e2e` / `test`. Mover `*.test.mjs` → `test/unit/`. +- **Fidelidad del mock** (3 fixes que lo vuelven detector, no encubridor): + - **A:** `sendMessage` respeta `return true` / canal async → falla cualquier handler async que olvide `return true` (clase del bug `42109cf`). + - **B:** `SyncAccessHandle` exclusivo + `flush()` requerido → `getFile()` devuelve solo bytes flusheados (expone B15). + - **C:** PING configurable con fallo + `lastError` → ejercita `_waitForContentScript` (clase `a0bf328`). +- **`test/unit/sw-wiring.test.mjs`** — carga el SW como Chrome (importScripts simulado, SIN + pre-inyectar globals) y asserta que `OpfsBuffer`/`MemoryBuffer` quedaron definidos + flujo + START→CAPTURE→DOWNLOAD produce ≥1 línea. **Falla hoy → reproduce B1 en puro Node** (no necesita Chrome). + +### Capa 2 — FUNCIONAL / E2E (la red que rompe el whack-a-mole) +- **Playwright + `launchPersistentContext` + `--load-extension` + `--headless=new`** (confirmado vigente 2026). + Acceso al SW vía `context.serviceWorkers()` / `serviceWorker.evaluate()`. Para Docker/CI: contenedor + con Chromium + `xvfb-run` como cinturón. `serviceWorker.evaluate()` permite leer `inMemoryCount`, + forzar mensajes y simular sleep/wake del SW vía CDP `ServiceWorker.stopAllWorkers`. +- **`test/e2e/fixtures-server.mjs`** — servidor Node sin deps, sirve una página que dispara los 4 + modos que el código maneja mal (`fetch(Request)`, XHR con headers, body con ID grande, fetch de + page-load) + endpoints que **imitan la forma de Voyager** (`/voyager/api/me` con + `x-restli-protocol-version` y `{data, included:[{access_token}]}`). Spec ejecutable **sin tocar linkedin.com**. +- **`record-download.spec.mjs`** — carga extensión → asserta buffers existen en el SW real (B1) → + graba → dispara requests → STOP → DOWNLOAD → asserta JSONL: contiene el endpoint, `method:POST` + (no GET, B8), `csrf-token` redactado, `x-restli-protocol-version` legible (B10). +- **`popup.spec.mjs`** — la capa con 0 tests donde cayeron 3-4 fixes históricos. +- **`sw-restart.spec.mjs`** — graba → `ServiceWorker.stopAllWorkers` → despierta → asserta que las + capturas pre-sleep sobreviven (pausa/continuar end-to-end). +- **`scripts/build-dist.mjs`** — empaqueta `dist/unpacked/`; `pretest:e2e` lo corre siempre → el + e2e prueba **exactamente lo que se empaqueta** (atrapa drift manifest↔archivos). + +### Capa 3 — CI (`.github/workflows/test.yml`, cada push, sin humano) +Job `unit` (node 22 + `check:version` + `test:unit`) → job `e2e` (`playwright install chromium` + +`build:dist` + `xvfb-run npm run test:e2e` + upload report). **El build del `.zip` se bloquea si +falla cualquier test.** `scripts/check-version-consistency.mjs` cierra el drift de versión (B24). + +--- + +## 6. Pausa / Continuar (sin reset) — diseño + +Hoy solo hay dos verbos destructivos: `START` (trunca OPFS) y `STOP`. No existe `paused`. + +**Invariante central:** `captures.jsonl` es la fuente de verdad del stream y **solo se trunca/borra +en `START` (sesión nueva) y `CLEAR`/`DISCARD`**. Toda otra transición (`PAUSE`, `STOP`, `RESUME`, +wake del SW) es append-only o read-only. `restoreFromExisting()` ya implementa el re-abrir sin +truncar (`opfsBytesWritten = getSize()`) — solo hay que cablearlo. + +**Máquina:** `IDLE —START(trunca)→ RECORDING —PAUSE(close, no trunca)→ PAUSED —RESUME(restoreFromExisting, +append)→ RECORDING`; `STOP→IDLE` (cierra, no trunca); `CLEAR→IDLE` (borra). SW sleep durante +RECORDING/PAUSED: el archivo persiste en disco, los flags en `chrome.storage.session`; al wake el +restore reabre OPFS y reconstruye count/dedup leyendo el archivo. + +**Persistencia (3 planos):** OPFS = el dato · `chrome.storage.session` = flags efímeros (+ nuevos +`paused`, `sessionId`, `opfsFilename`, `capturedCount` throttled) · `chrome.storage.local` = +`lastSession` para el caso "browser cerrado" → habilita el prompt *"tienes una sesión pausada con +N eventos, ¿continuar/descargar/descartar?"*. + +**UX popup:** dos botones contextuales (RECORDING: `⏸ Pausar`/`⏹ Detener`; PAUSED: `▶ Continuar`/`⏹ +Detener`). `Continuar` envía `RESUME` (no `START`) — clave para no truncar. Banner inline en vez de `alert/confirm`. + +**ADR-0003 propuesto** — "Resumable sessions: truncate solo en START explícito, no en wake". No tira +ADR-0002 (la garantía "START te da archivo limpio" se mantiene), lo acota: el wake deja de tratarse +como START implícito que borraba datos. + +--- + +## 7. LinkedIn Voyager (primer preset, no cambio de core) + +Gaps del preset actual + fixes (todos genéricos del engine que además habilitan Voyager): +- **Inyección tardía (B9):** la SPA dispara las llamadas Voyager más valiosas en page-load/navegación + SPA antes del click START → `world:MAIN, document_start` lo resuelve. +- **`all_frames:false`** (manifest:28): no captura iframes. Voyager principal va en top-frame; evaluar si algún flujo lo necesita. +- **XHR sin headers (B7):** Voyager messaging/track usa XHR con `csrf-token`/`x-li-track`/`x-restli` → hoy se pierden. +- **Redacción (B10/B11/B17):** `x-restli-protocol-version` debe quedar legible (replay); `csrf-token`/`cookie`/`oauth_token`/`included[].access_token` redactados. +- **Cookie de auth (`li_at`/`JSESSIONID`):** es forbidden header del browser; NO sale por fetch. Decisión: documentar que la auth se obtiene por `chrome.cookies` aparte, o agregar path `webRequest.onSendHeaders` (más permisos). + +--- + +## 8. Plan por fases (incremental — un cambio, validar, seguir) + +| Fase | Qué | "Hecho" verificable | +|---|---|---| +| **0 · Estabilizar harness** | package.json + mover a test/unit + mock fidelidad (A/B/C) + `sw-wiring.test.mjs` | El sw-wiring test **falla en rojo** reproduciendo B1 (primer verde→rojo honesto). | +| **1 · Críticos** | B1 importScripts · B2 filtro content.js · B6 START pending+lastError · B9 world:MAIN+guard · B3 START no trunca sin red | sw-wiring pasa a verde + e2e happy-path: extensión real captura ≥1 endpoint con headers. | +| **2 · Pausa/Continuar** | Cablear restoreFromExisting (B4) · DOWNLOAD disco-o-RAM (B5) · verbos PAUSE/RESUME + popup + ADR-0003 · quitar _persistSession del hot-path (B14) | e2e: grabar → forzar sleep SW → RESUME → DOWNLOAD trae capturas pre Y post sleep. | +| **3 · Voyager** | B7 headers XHR · B8 fetch(Request) · B10/B11/B17 redacción · B19 fuente única presets · B12/B13 popup opfsMode | test corre preset Voyager sobre headers/body realistas: x-restli legible, secretos redactados. | +| **4 · CI + e2e completo** | injected/content/popup tests · suite Playwright · GitHub Action gate antes del dist | `npm test` corre todo en CI; el `.zip` bloqueado si falla; cobertura de injected/content/popup > 0. | + +**Pasos 1-2 (medio día) ya entregan el 80% del valor:** un e2e real que prueba lo que se empaqueta y +reproduce el bug que hace que la extensión capture cero. + +--- + +## 9. Decisiones abiertas (requieren input) + +1. **B1 fix:** `importScripts` (mínimo, reversible, mantiene UMD/harness) vs SW `type:module` (más + limpio pero convierte todos los UMD a import/export). **Recomendado: importScripts.** +2. **`world:MAIN, document_start` siempre-on (B9):** mejora muchísimo la captura Voyager pero el + patch de fetch/XHR vive en la página desde el primer byte (gateado por isRecording). ¿Aceptable + para la privacy policy, o preferís el fallback "Iniciar y recargar"? +3. **Redacción `x-li-track` (B10):** parcial (preservar clientVersion, borrar trackingId/fingerprint) + vs todo-o-nada default OFF para Voyager. La parcial es lo correcto para RE. +4. **Cookie de auth Voyager:** documentar `chrome.cookies` aparte vs agregar `webRequest.onSendHeaders` (más permisos). +5. **Limpieza de basura del repo:** tarballs commiteados (`dist/*.tar.gz`, raíz), drafts `.pr-body-*.md`, + `store-assets/screenshots-v2/` (duplicado exacto), `.gitignore` sin `*.tar.gz`. + +--- + +## 10. Agentes especializados propuestos (`.claude/agents/` — el repo no tiene hoy) + +- **Chrome MV3 Extension Engineer** — conoce el modelo de 4 contextos, lifecycle del SW, contratos de + mensajes, OPFS, world MAIN/ISOLATED, e2e Playwright. Custodio de R1/R2/R3 y de que ningún fix rompa + los 8 key features. +- **API Reverse Engineer** — genérico (no LinkedIn-específico): cómo descubrir/mapear una API no + documentada, qué headers son obligatorios vs fingerprint, criterio de redacción (sesión vs replay), + cómo armar un preset nuevo. Voyager es su primer caso, no su definición. + +--- + +*Fuentes de testing: [Playwright Chrome Extensions](https://playwright.dev/docs/chrome-extensions) · +[Playwright headless](https://playwright.dev/docs/browsers) · issues CI #33928 / #37347.* diff --git a/manifest.json b/manifest.json index 95f8e02..8b01731 100644 --- a/manifest.json +++ b/manifest.json @@ -1,13 +1,14 @@ { "manifest_version": 3, "name": "API Reverse Engineer", - "version": "1.4.2", + "version": "1.7.0", "description": "Captura todas las llamadas API mientras navegas. Record \u2192 usa el sitio \u2192 descarga JSON.", "permissions": [ "storage", "activeTab", "scripting", "tabs", + "cookies", "unlimitedStorage" ], "host_permissions": [ diff --git a/package-lock.json b/package-lock.json new file mode 100644 index 0000000..c2e3d92 --- /dev/null +++ b/package-lock.json @@ -0,0 +1,78 @@ +{ + "name": "api-reverse-engineer", + "version": "1.4.2", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "api-reverse-engineer", + "version": "1.4.2", + "devDependencies": { + "@playwright/test": "^1.50.0" + } + }, + "node_modules/@playwright/test": { + "version": "1.61.1", + "resolved": "https://registry.npmjs.org/@playwright/test/-/test-1.61.1.tgz", + "integrity": "sha512-8nKv6+0RJSL9FE4jYOEGXnPeM/Hg12qZpmqzZjRh3qM0Y7c3z1mrOTfFLids72RDQYVh9WpLEfR5WdpNX4fkig==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "playwright": "1.61.1" + }, + "bin": { + "playwright": "cli.js" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/fsevents": { + "version": "2.3.2", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz", + "integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, + "node_modules/playwright": { + "version": "1.61.1", + "resolved": "https://registry.npmjs.org/playwright/-/playwright-1.61.1.tgz", + "integrity": "sha512-DWnY5o3YbLWK4GovuAVwpqL+1VwGNdUGrRr++8j8PtQQzvAVZUIMjKQ90fY689sEJZJBbZVw1rXaOKSTitkzPQ==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "playwright-core": "1.61.1" + }, + "bin": { + "playwright": "cli.js" + }, + "engines": { + "node": ">=18" + }, + "optionalDependencies": { + "fsevents": "2.3.2" + } + }, + "node_modules/playwright-core": { + "version": "1.61.1", + "resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.61.1.tgz", + "integrity": "sha512-h7Qlt6m4REp25qvIdvbDtVmD4LqVXfpRxhORv9L0jzETM05p4fuPJ3dKyuSXQxDSbXnmS79HAgi9589lGSpLkg==", + "dev": true, + "license": "Apache-2.0", + "bin": { + "playwright-core": "cli.js" + }, + "engines": { + "node": ">=18" + } + } + } +} diff --git a/package.json b/package.json new file mode 100644 index 0000000..df7f88f --- /dev/null +++ b/package.json @@ -0,0 +1,20 @@ +{ + "name": "api-reverse-engineer", + "version": "1.7.0", + "private": true, + "description": "Chrome MV3 extension that captures fetch + XHR requests for reverse engineering any API.", + "//type": "intencionalmente SIN 'type:module': src/*.js son UMD/CommonJS (module.exports) y el harness los carga vía require(). Los tests son .mjs (ESM explícito) y _chrome-mock.js se auto-detecta ESM. Forzar type:module rompe loadBackgroundFresh.", + "scripts": { + "test": "node --test test/*.test.mjs", + "test:unit": "node --test test/*.test.mjs", + "test:coverage": "node --test --experimental-test-coverage test/*.test.mjs", + "test:e2e": "playwright test", + "test:all": "npm run test:unit && npm run test:e2e", + "check:version": "node scripts/check-version-consistency.mjs", + "build:dist": "node scripts/build-dist.mjs", + "pretest:e2e": "npm run build:dist" + }, + "devDependencies": { + "@playwright/test": "^1.50.0" + } +} diff --git a/playwright.config.mjs b/playwright.config.mjs new file mode 100644 index 0000000..66d2f3c --- /dev/null +++ b/playwright.config.mjs @@ -0,0 +1,13 @@ +import { defineConfig } from '@playwright/test'; + +// E2E loads the unpacked extension in a real Chromium. A persistent context is +// required for MV3 extensions, so the suite runs serially (workers: 1). +export default defineConfig({ + testDir: 'test/e2e', + testMatch: '**/*.spec.mjs', + timeout: 45_000, + fullyParallel: false, + workers: 1, + reporter: [['list'], ['html', { open: 'never' }]], + use: { trace: 'retain-on-failure' }, +}); diff --git a/popup.html b/popup.html index 9b3f796..56ce77d 100644 --- a/popup.html +++ b/popup.html @@ -181,6 +181,11 @@ background: #ef4444; } + #btnPause { + background: #f59e0b; + color: #fff; + } + #btnDownload { background: #3b82f6; color: #fff; @@ -193,6 +198,12 @@ padding: 9px 10px; } + #btnDownloadCookies { + background: #1e293b; + color: #cbd5e1; + width: 100%; + } + .preview { padding: 0 16px 10px; max-height: 180px; @@ -314,12 +325,12 @@

🔬 API Reverse Engineer

- +
@@ -339,20 +350,18 @@

🔬 API Reverse Engineer

-
- -
- - -
-
-
+
+
+ +
Descarga un .json con las cookies de auth (incluye httpOnly como li_at) + el header Cookie listo para replay. No se guardan en la captura.
+
+
Grabando... @@ -365,6 +374,9 @@

🔬 API Reverse Engineer

+ + diff --git a/scripts/build-dist.mjs b/scripts/build-dist.mjs new file mode 100644 index 0000000..d0ababb --- /dev/null +++ b/scripts/build-dist.mjs @@ -0,0 +1,60 @@ +/** + * build-dist.mjs — package the unpacked extension into dist/unpacked/. + * + * The e2e suite loads dist/unpacked/ (not src/ loose) so it exercises EXACTLY + * what ships — this catches manifest↔file drift (e.g. a file referenced by the + * manifest that isn't copied). No dependencies, runs on plain node. + * + * Usage: node scripts/build-dist.mjs (or: npm run build:dist) + */ +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const REPO = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'); +const OUT = path.join(REPO, 'dist', 'unpacked'); + +// Everything Chrome needs to load the extension unpacked. (popup.js lives +// under src/, so the 'src' entry already covers it.) +const INCLUDE = ['manifest.json', 'popup.html', 'src', 'icons']; + +function copyRecursive(src, dst) { + const stat = fs.statSync(src); + if (stat.isDirectory()) { + fs.mkdirSync(dst, { recursive: true }); + for (const entry of fs.readdirSync(src)) { + copyRecursive(path.join(src, entry), path.join(dst, entry)); + } + } else { + fs.mkdirSync(path.dirname(dst), { recursive: true }); + fs.copyFileSync(src, dst); + } +} + +fs.rmSync(OUT, { recursive: true, force: true }); +fs.mkdirSync(OUT, { recursive: true }); + +let copied = 0; +for (const item of INCLUDE) { + const src = path.join(REPO, item); + if (fs.existsSync(src)) { + copyRecursive(src, path.join(OUT, item)); + copied += 1; + } +} + +// Sanity: every file the manifest references must exist in the build. +const manifest = JSON.parse(fs.readFileSync(path.join(OUT, 'manifest.json'), 'utf8')); +const referenced = []; +if (manifest.background?.service_worker) referenced.push(manifest.background.service_worker); +for (const cs of manifest.content_scripts || []) referenced.push(...(cs.js || [])); +if (manifest.action?.default_popup) referenced.push(manifest.action.default_popup); +for (const war of manifest.web_accessible_resources || []) referenced.push(...(war.resources || [])); + +const missing = referenced.filter((rel) => !fs.existsSync(path.join(OUT, rel))); +if (missing.length) { + console.error('[build:dist] ✖ manifest referencia archivos ausentes en el build:', missing); + process.exit(1); +} + +console.log(`[build:dist] ✔ ${copied} items → ${path.relative(REPO, OUT)} (manifest OK: ${referenced.length} refs)`); diff --git a/scripts/check-version-consistency.mjs b/scripts/check-version-consistency.mjs new file mode 100644 index 0000000..fd8fa79 --- /dev/null +++ b/scripts/check-version-consistency.mjs @@ -0,0 +1,44 @@ +/** + * check-version-consistency.mjs — keeps version drift (B24) from creeping back. + * + * Fails if package.json version != manifest version, or if any src/*.js + * hardcodes a `version: 'x.y.z'` literal that differs from the manifest. + * (content.js now derives the PING version from the manifest at runtime; this + * guard catches any future regression to a hardcoded string.) + * + * Usage: node scripts/check-version-consistency.mjs (or: npm run check:version) + */ +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const REPO = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'); +const manifest = JSON.parse(fs.readFileSync(path.join(REPO, 'manifest.json'), 'utf8')); +const pkg = JSON.parse(fs.readFileSync(path.join(REPO, 'package.json'), 'utf8')); + +let failed = false; + +if (manifest.version !== pkg.version) { + console.error(`✖ manifest.version (${manifest.version}) != package.json version (${pkg.version})`); + failed = true; +} + +const V = manifest.version; +const srcDir = path.join(REPO, 'src'); +for (const f of fs.readdirSync(srcDir)) { + if (!f.endsWith('.js')) continue; + const txt = fs.readFileSync(path.join(srcDir, f), 'utf8'); + // Match code literals like `version: '1.4.2'` / `version="1.4.2"` — NOT + // comments such as `(v1.3.0)`. + for (const hit of txt.matchAll(/version['"]?\s*[:=]\s*['"](\d+\.\d+\.\d+)['"]/gi)) { + // '0.0.0' is the conventional "unknown" placeholder (e.g. the PING + // fallback when chrome.runtime.getManifest() is unavailable) — not drift. + if (hit[1] !== V && hit[1] !== '0.0.0') { + console.error(`✖ src/${f}: versión hardcodeada '${hit[1]}' != manifest '${V}'`); + failed = true; + } + } +} + +if (failed) process.exit(1); +console.log(`✔ versión consistente en manifest/package/src: ${V}`); diff --git a/src/background.js b/src/background.js index e9792c5..2a6a407 100644 --- a/src/background.js +++ b/src/background.js @@ -63,6 +63,25 @@ * only exercised in the SW context. */ +// --------------------------------------------------------------------------- +// B1 fix (2026-06-24): the MV3 service worker is a CLASSIC script — the +// manifest declares `service_worker` WITHOUT `type:module`, and Chrome loads +// ONLY this file. The buffer dependencies are NOT auto-injected, so we must +// pull them in via importScripts BEFORE the IIFE below reads self.OpfsBuffer / +// self.MemoryBuffer. Without this, both are null and the SW captures NOTHING +// in real Chrome (the 71-green-but-broken suite never caught it because the +// mock pre-attached the buffers to globalThis — see test/sw-wiring.test.mjs). +// +// - Production SW: importScripts is defined → loads the deps. +// - Honest unit loader: test/_sw-loader.mjs provides importScripts. +// - Legacy CJS harness: importScripts is undefined (require() context) → +// this is a no-op and the harness attaches the +// buffers to globalThis itself (loadBackgroundFresh). +// --------------------------------------------------------------------------- +if (typeof importScripts === 'function') { + importScripts('/src/memory-buffer.js', '/src/opfs-buffer.js'); +} + (function (root, factory) { 'use strict'; var api = factory(); @@ -99,10 +118,12 @@ var inMemoryCount = 0; // total events this session var inMemoryUnique = new Set(); // METHOD:URL dedup keys var isRecording = false; + var paused = false; // Fase 2: PAUSED vs IDLE (resume sin truncar) var recordingTabId = null; var captureConfig = null; var outputFormat = 'jsonl'; var filterMode = 'OR'; + var sessionId = null; // Fase 2: id de la sesión OPFS activa // OPFS streaming buffer (primary path). var opfsBuffer = OpfsBuffer ? OpfsBuffer.createOpfsBuffer({ filename: 'captures.jsonl' }) : null; @@ -123,26 +144,23 @@ // --------------------------------------------------------------------------- if (typeof chrome !== 'undefined' && chrome.storage && chrome.storage.session) { chrome.storage.session.get( - ['isRecording', 'recordingTabId', 'captureConfig', 'outputFormat', 'filterMode'], + ['isRecording', 'paused', 'recordingTabId', 'captureConfig', 'outputFormat', 'filterMode', 'sessionId'], function (data) { - if (data && data.isRecording) { - isRecording = data.isRecording; + if (data && (data.isRecording || data.paused)) { + isRecording = !!data.isRecording; + paused = !!data.paused; recordingTabId = data.recordingTabId || null; captureConfig = data.captureConfig || null; outputFormat = data.outputFormat || 'jsonl'; filterMode = data.filterMode || 'OR'; - // SW restart reset activeBuffer to null. Re-point it to the safe - // memory buffer so the first CAPTURE after restart doesn't get - // dropped by the `if (activeBuffer)` guard in the CAPTURE handler. - if (!activeBuffer && memoryBuffer) { - activeBuffer = memoryBuffer; - } + sessionId = data.sessionId || null; + // B4 fix (Fase 2 / ADR-0003): re-open the OPFS file in append mode + // and rebuild the counter + dedup from disk, so a recording/paused + // session survives the SW going idle. v1.4.2 lost the whole buffer + // on every SW wake-up (the file was orphaned, counters reset to 0). + _restoreSessionFromDisk(); // Restore the red-dot badge so the user sees the recording state. - if (recordingTabId) _setBadge(recordingTabId); - // captured[] and counters start empty after SW restart. - // The OPFS file persists on disk, so on the next START we truncate - // it (fresh session per ADR-0002 decision). If the user wants to - // resume an old session, that is a F4 feature. + if (isRecording && recordingTabId) _setBadge(recordingTabId); } } ); @@ -153,16 +171,63 @@ try { chrome.storage.session.set({ isRecording: isRecording, + paused: paused, recordingTabId: recordingTabId, captureConfig: captureConfig, outputFormat: outputFormat, - filterMode: filterMode + filterMode: filterMode, + sessionId: sessionId }); } catch (e) { console.error('[ARE] Failed to persist session:', e); } } + // --------------------------------------------------------------------------- + // Re-open the persisted OPFS session after an SW restart, or to RESUME a + // paused session — WITHOUT truncating (ADR-0003). Rebuilds the dedup set + + // counter from the file (robust source of truth) and migrates any captures + // that arrived in the small async window while the file was being re-opened. + // --------------------------------------------------------------------------- + function _restoreSessionFromDisk() { + // Sync safety net: a CAPTURE arriving during the async re-open window goes + // to a fresh memory buffer; we migrate it to OPFS once the file re-opens. + if (memoryBuffer) { memoryBuffer.clear(); activeBuffer = memoryBuffer; } + if (!opfsBuffer) { return Promise.resolve(); } + return opfsBuffer.restoreFromExisting().then(function (ok) { + if (!ok) { + // No file to restore — stay on the memory buffer; next START inits fresh. + activeBuffer = memoryBuffer; + return; + } + // Rebuild dedup from the file content (readAll is async in ADR-0003). + return opfsBuffer.readAll().then(function (text) { + var lines = String(text).split('\n'); + inMemoryUnique = new Set(); + for (var i = 0; i < lines.length; i++) { + if (!lines[i]) continue; + try { + var o = JSON.parse(lines[i]); + var u = o.url || (o.request && o.request.url) || ''; + var m = o.method || (o.request && o.request.method) || ''; + inMemoryUnique.add(m + ':' + String(u).split('?')[0]); + } catch (e) {} + } + // Migrate captures that arrived during the re-open window. + var pending = (memoryBuffer && memoryBuffer.snapshot) ? memoryBuffer.snapshot() : []; + activeBuffer = opfsBuffer; + for (var j = 0; j < pending.length; j++) { + opfsBuffer.append(pending[j]); + inMemoryUnique.add(pending[j].method + ':' + String(pending[j].url || '').split('?')[0]); + } + inMemoryCount = opfsBuffer.getCount(); + }); + }).catch(function (e) { + console.error('[ARE] restore from disk failed:', e); + activeBuffer = memoryBuffer; + }); + } + // Bug fix 2026-06-24: poll the content script (ISOLATED world) until it // responds to PING, with a hard timeout. Fixes the race where the SW // sends START_RECORDING immediately after executeScript resolves but @@ -209,18 +274,35 @@ * Signature: `_setBadge(tabId)`. The `count` parameter from v1.4.1 * has been removed (it was the source of the alternating bug). */ + // Badge text fits ~4 chars; counts cap at MAX_EVENTS (auto-stop), so show the + // exact number up to 9999 and "10k" at the cap. + function _fmtBadgeCount(n) { + n = n || 0; + return n >= 10000 ? '10k' : String(n); + } + function _setBadge(tabId) { if (typeof chrome === 'undefined' || !chrome.action) return; if (!tabId) return; + // Restored behaviour: the toolbar icon shows the LIVE request count while + // recording (red) or paused (amber). The v1.4.1 bug was *alternating* + // between a dot and the number on every CAPTURE — here the badge always + // shows the count, so there's nothing to alternate with. if (isRecording) { - // While recording, show red dot. Counter goes in popup only. try { - chrome.action.setBadgeText({ text: '●', tabId: tabId }); + chrome.action.setBadgeText({ text: _fmtBadgeCount(inMemoryCount), tabId: tabId }); chrome.action.setBadgeBackgroundColor({ color: '#ef4444', tabId: tabId }); } catch (e) {} return; } - // When stopped, clear badge. + if (paused) { + try { + chrome.action.setBadgeText({ text: _fmtBadgeCount(inMemoryCount), tabId: tabId }); + chrome.action.setBadgeBackgroundColor({ color: '#f59e0b', tabId: tabId }); + } catch (e) {} + return; + } + // Stopped/idle → clear. try { chrome.action.setBadgeText({ text: '', tabId: tabId }); } catch (e) {} } @@ -234,6 +316,13 @@ // CAPTURE // ----------------------------------------------------------------------- if (msg.type === 'CAPTURE') { + // Drop captures when not actively recording (paused or stopped). The + // injected interceptor keeps dispatching events; the SW (and content + // script) gate on the recording state so a paused session stays clean. + if (!isRecording) { + respond({ ok: true }); + return true; + } var tabId = sender.tab && sender.tab.id; if (recordingTabId !== null && tabId !== recordingTabId) { respond({ ok: true }); @@ -288,8 +377,9 @@ } _persistSession(); - // v1.4.2: badge is driven by isRecording flag (no count). No call - // to _setBadge here — START/STOP/AUTO_STOP own the badge. + // Live badge: show the running request count on the toolbar icon + // (restored — the user watches this while capturing). + _setBadge(recordingTabId); respond({ ok: true }); return true; @@ -304,6 +394,8 @@ var isFallback = activeBuffer === memoryBuffer || !activeBuffer || (opfsBuffer && opfsBuffer.inFallbackMode()); respond({ isRecording: isRecording, + paused: paused, + recoverable: (paused || (!isRecording && inMemoryCount > 0)), recordingTabId: recordingTabId, total: inMemoryCount, unique: unique, @@ -326,7 +418,11 @@ // buffer. No duplicates in the output, no silent loss. // ----------------------------------------------------------------------- if (msg.type === 'START') { + // START = sesión NUEVA: trunca el archivo OPFS (init). Es el único + // verbo (con CLEAR) que destruye datos. RESUME, en cambio, appendea. isRecording = true; + paused = false; + sessionId = 'sess-' + new Date().getTime(); recordingTabId = msg.tabId || null; var filter = msg.filter || ''; captureConfig = msg.captureConfig || null; @@ -427,6 +523,7 @@ // ----------------------------------------------------------------------- if (msg.type === 'STOP') { isRecording = false; + paused = false; _persistSession(); // Close the OPFS access handle (keep the file handle for download). @@ -448,6 +545,64 @@ return true; } + // ----------------------------------------------------------------------- + // PAUSE (Fase 2) — detiene la captura SIN truncar: cierra el handle OPFS + // pero conserva el archivo + recordingTabId + sessionId. RESUME continúa + // appendeando al mismo archivo (ADR-0003). NO es STOP (que cierra la + // sesión) ni START (que la trunca). + // ----------------------------------------------------------------------- + if (msg.type === 'PAUSE') { + isRecording = false; + paused = true; + if (opfsBuffer) opfsBuffer.close(); // handle cerrado, archivo intacto + _persistSession(); + if (recordingTabId && typeof chrome !== 'undefined' && chrome.tabs) { + try { + chrome.tabs.sendMessage(recordingTabId, { type: 'STOP_RECORDING' }).catch(function () {}); + } catch (e) {} + } + if (recordingTabId) _setBadge(recordingTabId); + respond({ ok: true }); + return true; + } + + // ----------------------------------------------------------------------- + // RESUME (Fase 2) — re-abre la sesión OPFS en modo append (NO trunca), + // reconstruye contador/dedup desde el archivo, y re-arma el interceptor + // en la pestaña (idempotente vía __ARE_PATCHED__). + // ----------------------------------------------------------------------- + if (msg.type === 'RESUME') { + if (!paused) { + respond({ ok: false, error: 'No hay sesión pausada para continuar' }); + return true; + } + isRecording = true; + paused = false; + _restoreSessionFromDisk().then(function () { + _persistSession(); + if (recordingTabId && typeof chrome !== 'undefined' && chrome.scripting) { + chrome.scripting.executeScript({ + target: { tabId: recordingTabId }, + world: 'MAIN', + files: ['src/capture-config.js', 'src/injected.js'] + }).then(function () { + return _waitForContentScript(recordingTabId, 2000); + }).then(function (ready) { + if (!ready) return; + chrome.tabs.sendMessage(recordingTabId, { type: 'START_RECORDING', filter: '' }).catch(function () {}); + if (captureConfig) { + chrome.tabs.sendMessage(recordingTabId, { type: 'SET_CAPTURE_CONFIG', captureConfig: captureConfig }).catch(function () {}); + } + }).catch(function (err) { + console.error('[ARE] RESUME re-inject failed:', err); + }); + } + }); + _setBadge(recordingTabId); + respond({ ok: true }); + return true; + } + // ----------------------------------------------------------------------- // DOWNLOAD // v1.4.2: validate `inMemoryCount > 0` up front and return @@ -474,52 +629,33 @@ return true; } - if (format === 'json-array') { - // Legacy v1.2.3 shape — uses the in-memory array snapshot. - // In OPFS mode the array is empty (we can't enumerate JSONL - // lines back into objects cheaply), so the legacy output is - // a best-effort: meta + uniqueEndpoints=0 + all=[]. - var snapshot = (memoryBuffer && memoryBuffer.snapshot) ? memoryBuffer.snapshot() : []; - var unique2 = {}; - snapshot.forEach(function (r) { - var k = r.method + ':' + r.url.split('?')[0]; - if (!unique2[k] || r.isNewEndpoint) unique2[k] = r; - }); - var data = { - meta: { - capturedAt: new Date().toISOString(), - total: inMemoryCount, - uniqueEndpoints: Object.keys(unique2).length, - site: site, - preset: preset - }, - endpoints: Object.values(unique2), - all: snapshot - }; - respond({ - ok: true, - data: JSON.stringify(data, null, 2), - filename: 'api-capture-' + preset + '-' + isoStamp + '.json', - format: 'json-array' - }); - return true; - } - - // JSONL (v1.3.0 default) — try OPFS first if active, else serialise + // JSONL — try OPFS first if active, else serialise // the in-memory array. Always fall back to memory on any OPFS // failure. If both paths fail, return ok:false so the popup can // show the user what went wrong. if (activeBuffer === opfsBuffer && opfsBuffer && !opfsBuffer.inFallbackMode() && typeof opfsBuffer.getFile === 'function') { opfsBuffer.getFile().then(function (file) { - return file.arrayBuffer(); - }).then(function (buf) { - // Convert to base64 so the message payload survives the - // structured-clone transport (ArrayBuffer is OK, but base64 - // is portable and tested). Then in the popup we decode + Blob. - var bytes = new Uint8Array(buf); - var bin = ''; - for (var i = 0; i < bytes.byteLength; i++) bin += String.fromCharCode(bytes[i]); - var b64 = (typeof btoa === 'function') ? btoa(bin) : Buffer.from(bin, 'binary').toString('base64'); + return file.text(); + }).then(function (text) { + // The OPFS file streams RAW entries at capture time. Normalize them + // to the canonical _toJsonlLine shape on download so the OPFS path + // produces the SAME output as the in-memory path (and the linkedin + // importer expects). Before ADR-0003 this path never ran (OPFS was + // always in fallback), so the inconsistency was latent. + var rawLines = String(text).split('\n'); + var out = []; + for (var i = 0; i < rawLines.length; i++) { + if (!rawLines[i].trim()) continue; + try { out.push(_toJsonlLine(JSON.parse(rawLines[i]))); } + catch (e) { out.push(rawLines[i]); } + } + var raw = out.join('\n') + (out.length ? '\n' : ''); + var b64; + try { + b64 = (typeof btoa === 'function') ? btoa(unescape(encodeURIComponent(raw))) : Buffer.from(raw, 'utf-8').toString('base64'); + } catch (e2) { + b64 = Buffer.from(raw, 'utf-8').toString('base64'); + } respond({ ok: true, data: b64, @@ -527,8 +663,8 @@ mime: 'application/x-ndjson', filename: 'are-capture-' + preset + '-' + isoStamp + '.jsonl', format: 'jsonl', - lineCount: inMemoryCount, - bytes: bytes.byteLength + lineCount: out.length, + bytes: (typeof TextEncoder !== 'undefined') ? new TextEncoder().encode(raw).byteLength : raw.length }); }).catch(function (e) { console.error('[ARE] OPFS read failed, falling back to in-memory JSONL:', e); @@ -612,14 +748,49 @@ respond({ presets: [ { id: 'generic', label: '[Generic]', sortOrder: 99 }, - { id: 'linkedin-voyager', label: '[LinkedIn Voyager]', sortOrder: 1 }, + { id: 'linkedin-voyager', label: '[LinkedIn]', sortOrder: 1 }, { id: 'graphql', label: '[GraphQL]', sortOrder: 2 }, { id: 'json-api', label: '[JSON API]', sortOrder: 3 } ], - defaultPresetId: 'linkedin-voyager' + defaultPresetId: 'generic' }); return true; } + + // ----------------------------------------------------------------------- + // GET_COOKIES (Fase 3) — copia las cookies del sitio para replay. Usa la + // API chrome.cookies, que SÍ lee cookies httpOnly (li_at, JSESSIONID) que + // document.cookie y fetch no pueden ver. NO se guardan en la captura: es + // un canal aparte para que el usuario obtenga la auth. + // ----------------------------------------------------------------------- + if (msg.type === 'GET_COOKIES') { + var cookieUrl = msg.url; + if (!cookieUrl || typeof chrome === 'undefined' || !chrome.cookies) { + respond({ ok: false, error: 'Sin URL o sin permiso cookies' }); + return true; + } + try { + chrome.cookies.getAll({ url: cookieUrl }, function (cookies) { + if (chrome.runtime.lastError) { + respond({ ok: false, error: chrome.runtime.lastError.message }); + return; + } + var list = cookies || []; + var header = list.map(function (c) { return c.name + '=' + c.value; }).join('; '); + respond({ + ok: true, + count: list.length, + cookieHeader: header, + cookies: list.map(function (c) { + return { name: c.name, value: c.value, domain: c.domain, path: c.path, secure: c.secure, httpOnly: c.httpOnly, expirationDate: c.expirationDate }; + }) + }); + }); + } catch (e) { + respond({ ok: false, error: String(e && e.message || e) }); + } + return true; + } }); } diff --git a/src/capture-config.js b/src/capture-config.js index d3e892e..67566ae 100644 --- a/src/capture-config.js +++ b/src/capture-config.js @@ -76,18 +76,35 @@ }), 'linkedin-voyager': Object.freeze({ id: 'linkedin-voyager', - label: '[LinkedIn Voyager]', + label: '[LinkedIn]', sortOrder: 1, + // Endpoints reales del LinkedIn web 2026: además del Voyager clásico + // (/voyager/api/), el flagship-web moderno usa RSC actions + // (/rsc-action/) y GraphQL. Patterns por substring para que funcionen + // con URLs relativas resueltas a absolutas (ver injected.js). patterns: Object.freeze([ - Object.freeze({ type: 'regex', value: '^https:\\/\\/www\\.linkedin\\.com\\/(voyager\\/api\\/|li\\/track)' }) + Object.freeze({ type: 'literal', value: '/voyager/api/' }), + Object.freeze({ type: 'literal', value: '/rsc-action/' }), + Object.freeze({ type: 'literal', value: '/api/graphql' }) + ]), + // Excluir el ruido de telemetría/estáticos que no es API de datos. + exclude: Object.freeze([ + Object.freeze({ type: 'literal', value: 'trackO11y' }), + Object.freeze({ type: 'literal', value: 'sensorCollect' }), + Object.freeze({ type: 'literal', value: 'trackingApiService' }), + Object.freeze({ type: 'literal', value: 'trackMedia' }), + Object.freeze({ type: 'literal', value: '/li/track' }), + Object.freeze({ type: 'literal', value: '/sct' }), + Object.freeze({ type: 'literal', value: 'static.licdn.com' }) ]), filterMode: 'OR', redact: Object.freeze({ enabled: true, headers: Object.freeze([ 'cookie', 'set-cookie', 'csrf-token', 'x-li-pem-metadata', - 'x-li-pem', 'x-li-track', 'x-li-decorators', - 'x-restli-protocol-version', 'authorization' + 'x-li-pem', 'x-li-track', 'x-li-decorators', 'authorization' + // B10: x-restli-protocol-version NO se redacta — es la constante + // '2.0.0', no un secreto, y se necesita para replay. ]), body: Object.freeze([ 'password', 'client_secret', 'access_token', 'refresh_token', @@ -139,7 +156,9 @@ }); // Default preset used when popup hasn't chosen one (or for legacy v1.2.3 path). - var DEFAULT_PRESET_ID = 'linkedin-voyager'; + // Generic = capturar todo, redacción de secretos comunes ON. El usuario elige + // un preset específico (LinkedIn, GraphQL…) cuando quiere narrowear. + var DEFAULT_PRESET_ID = 'generic'; // ------------------------------------------------------------------------- // parseFilter @@ -270,15 +289,24 @@ * @param {'AND'|'OR'} [mode] — default 'OR' for backward-compat with v1.2.3 * @returns {boolean} */ - function shouldCapture(url, patterns, mode) { - if (!Array.isArray(patterns) || patterns.length === 0) return true; - if (typeof url !== 'string' || url.length === 0) return false; + function shouldCapture(url, patterns, mode, exclude) { + var hasInclude = Array.isArray(patterns) && patterns.length > 0; + var hasExclude = Array.isArray(exclude) && exclude.length > 0; - var m = mode === 'AND' ? 'AND' : 'OR'; - if (m === 'AND') { - return patterns.every(function (p) { return _matchOne(url, p); }); + if (typeof url !== 'string' || url.length === 0) { + // Empty url: capture only if there's no include filter (capture-all). + return !hasInclude; } - return patterns.some(function (p) { return _matchOne(url, p); }); + // Exclude wins over include — filters telemetry/static noise even when the + // include patterns would otherwise match. + if (hasExclude && exclude.some(function (p) { return _matchOne(url, p); })) { + return false; + } + if (!hasInclude) return true; + var m = mode === 'AND' ? 'AND' : 'OR'; + return m === 'AND' + ? patterns.every(function (p) { return _matchOne(url, p); }) + : patterns.some(function (p) { return _matchOne(url, p); }); } // ------------------------------------------------------------------------- diff --git a/src/content.js b/src/content.js index d1aa027..5287cd3 100644 --- a/src/content.js +++ b/src/content.js @@ -51,7 +51,11 @@ chrome.runtime.onMessage.addListener((msg, sender, respond) => { // Without this, the SW races with content script init and the // START_RECORDING message lands in a no-receiver state. if (msg.type === 'PING') { - respond({ ready: true, version: '1.4.0' }); + // B24 fix: derive the version from the manifest instead of a hardcoded + // string that drifts (was '1.4.0' while the manifest said '1.4.2'). + var version = '0.0.0'; + try { version = chrome.runtime.getManifest().version; } catch (e) {} + respond({ ready: true, version: version }); return; } }); @@ -79,11 +83,13 @@ window.addEventListener('__ARE_REQUEST__', (event) => { return; } - // Filtrar por URL si hay filtro activo (compat v1.2.3 — single-string filter). - // Capture Mode v1.3.0 ya filtra en injected.js antes de despachar, pero - // mantenemos este check por defense-in-depth y para recordings iniciados - // sin captureConfig (legacy path). - if (filter && !(entry.url || '').includes(filter)) { + // B2 fix: el filtro legacy de substring SOLO aplica al path sin + // captureConfig (un keyword simple escrito en la caja de filtro de URL). + // Cuando hay un captureConfig estructurado activo, injected.js YA filtró + // con los patterns parseados; correr este check acá rompe la captura, + // porque para presets regex/glob `filter` es el patrón CRUDO y + // `.includes(rawRegex)` nunca matchea una URL real → descarta TODO. + if (!captureConfig && filter && !(entry.url || '').includes(filter)) { return; } diff --git a/src/injected.js b/src/injected.js index 6d77378..ec98347 100644 --- a/src/injected.js +++ b/src/injected.js @@ -16,6 +16,17 @@ (function () { 'use strict'; + // B9 fix: guard against double-wrapping. The interceptor is injected via + // chrome.scripting.executeScript on every START (and, once enabled, via a + // declarative document_start MAIN-world content_script). Without this guard + // each injection wraps window.fetch / window.XMLHttpRequest AGAIN, so every + // request is dispatched once per wrapper layer → duplicate captures after a + // STOP→START on the same page. Idempotent install: re-injection is a no-op, + // the original interceptor stays active and keeps receiving captureConfig + // updates via the existing postMessage listener. + if (window.__ARE_PATCHED__) return; + window.__ARE_PATCHED__ = true; + var CC = (typeof window !== 'undefined' && window.CaptureConfig) || null; // Capture-config may not have loaded (race or load failure). In that case, // fall back to a permissive default (capture everything, no redaction) so @@ -43,6 +54,18 @@ } }); + // Resolve a (possibly relative) URL to absolute, using the page location as + // base. SPAs like LinkedIn fetch with relative URLs (/voyager/api/…); the + // filter patterns are absolute-friendly substrings, and an absolute URL is + // also more useful for reverse engineering. + function _absoluteUrl(u) { + try { + return new URL(String(u), (typeof location !== 'undefined' ? location.href : undefined)).href; + } catch (e) { + return u; + } + } + // applyCapture — runs INSIDE the interceptors, BEFORE dispatching the event. // Returns the (possibly redacted) entry, or null if it should be skipped. function applyCapture(entry) { @@ -52,8 +75,9 @@ return entry; } - // 1. URL filter (early skip — redaction is skipped entirely if filtered out) - if (!shouldCapture(entry.url, cfg.patterns || [], cfg.filterMode || 'OR')) { + // 1. URL filter (early skip — redaction is skipped entirely if filtered out). + // exclude wins over include (filters telemetry/static noise). + if (!shouldCapture(entry.url, cfg.patterns || [], cfg.filterMode || 'OR', cfg.exclude || [])) { return null; } @@ -100,8 +124,12 @@ var args = Array.prototype.slice.call(arguments); var resource = args[0]; var options = args[1] || {}; - var url = typeof resource === 'string' ? resource : (resource && resource.url) || ''; - var method = options.method || 'GET'; + var isRequest = (typeof Request !== 'undefined') && (resource instanceof Request); + var url = _absoluteUrl(isRequest ? resource.url : (typeof resource === 'string' ? resource : (resource && resource.url) || '')); + // B8: con fetch(new Request(url, {method, headers, body})) el method/headers + // viven en el Request, no en args[1]. Tomamos el method del Request si no + // viene en options. + var method = options.method || (isRequest ? resource.method : 'GET'); var startTime = Date.now(); var requestBody = null; @@ -117,8 +145,14 @@ var requestHeaders = {}; try { - var h = new Headers(options.headers); - h.forEach(function (v, k) { requestHeaders[k] = v; }); + // B8: headers del Request (si se usó fetch(Request)) + headers de options. + if (isRequest && resource.headers && typeof resource.headers.forEach === 'function') { + resource.headers.forEach(function (v, k) { requestHeaders[k] = v; }); + } + if (options.headers) { + var h = new Headers(options.headers); + h.forEach(function (v, k) { requestHeaders[k] = v; }); + } } catch (e) {} try { @@ -175,16 +209,25 @@ var method = 'GET'; var url = ''; var requestBody = null; + var requestHeaders = {}; var startTime = { value: null }; var originalOpen = xhr.open.bind(xhr); xhr.open = function (m, u) { method = m; - url = u; + url = _absoluteUrl(u); var rest = Array.prototype.slice.call(arguments, 2); return originalOpen.apply(null, [m, u].concat(rest)); }; + // B7: capturar los request headers que el sitio setea vía setRequestHeader + // (Voyager messaging usa XHR con csrf-token / x-li-*). + var originalSetRequestHeader = xhr.setRequestHeader.bind(xhr); + xhr.setRequestHeader = function (k, v) { + try { requestHeaders[k] = v; } catch (e) {} + return originalSetRequestHeader(k, v); + }; + var originalSend = xhr.send.bind(xhr); xhr.send = function (body) { startTime.value = Date.now(); @@ -202,20 +245,28 @@ responseBody = xhr.responseText; } - var entry = { + // B7: parsear los response headers crudos de getAllResponseHeaders(). + var responseHeaders = {}; + try { + var raw = xhr.getAllResponseHeaders() || ''; + raw.trim().split(/[\r\n]+/).forEach(function (line) { + var idx = line.indexOf(':'); + if (idx > 0) responseHeaders[line.slice(0, idx).trim()] = line.slice(idx + 1).trim(); + }); + } catch (e) {} + + dispatch({ type: 'xhr', method: method, url: url, + requestHeaders: requestHeaders, requestBody: requestBody, status: xhr.status, + responseHeaders: responseHeaders, responseBody: responseBody, duration: Date.now() - (startTime.value || Date.now()), timestamp: new Date().toISOString() - }; - - // XHR rarely carries headers we set ourselves; response headers are - // not exposed via the XHR object in the same way. Attach what we have. - dispatch(entry); + }); }); return originalSend(body); diff --git a/src/opfs-buffer.js b/src/opfs-buffer.js index 3923160..c76f9f0 100644 --- a/src/opfs-buffer.js +++ b/src/opfs-buffer.js @@ -1,42 +1,26 @@ /** - * API Reverse Engineer — OPFS Streaming Buffer (v1.4.0) + * API Reverse Engineer — OPFS Streaming Buffer (async write path, ADR-0003) * - * Encapsulates the OPFS streaming capture buffer per ADR-0002: - * - Replaces the in-memory `captured[]` array with append-only writes to - * `captures.jsonl` in the extension's Origin Private File System. - * - Provides a synchronous write API (createSyncAccessHandle) that works - * in MV3 service workers (Chrome 102+). - * - Survives SW restarts and browser close: the file persists in the OPFS - * sandbox; `isRecording` / counters are restored from - * chrome.storage.session. - * - Graceful fallback: if OPFS is unavailable (Chrome < 102, or the call - * throws), the caller is signalled via `inFallbackMode()` so it can - * fall back to the v1.3.2 in-memory array path. + * ADR-0003 supersedes the sync-handle design of ADR-0002. Empirically, + * `FileSystemFileHandle.createSyncAccessHandle()` is NOT available in MV3 + * service workers (only in dedicated workers) — it threw, so the extension + * silently ran in memory-fallback the whole time and never persisted to disk. * - * Loaded two ways (mirrors `src/capture-config.js`): - * - Browser / Chrome extension (classic script): attaches `window.OpfsBuffer`. - * - Node tests (CJS via createRequire): returns `module.exports`. + * This module now uses the ASYNC OPFS API, which DOES work in a service + * worker: + * - writes via `FileSystemFileHandle.createWritable({keepExistingData:true})` + * + `seek(end)` + `write(line)` + `close()`, + * - reads via `getFile()` + `File.text()`/`arrayBuffer()`. * - * API surface: - * - createOpfsBuffer({ filename?, navigator? }) - * Returns a buffer instance with: - * .init() — open or create the file, returns Promise - * .append(entry) — write one JSONL line, returns boolean (true on success) - * .getFile() — Promise (File API object) for download - * .getCount() — number of lines written this session - * .getBytesWritten() — total bytes flushed - * .clear() — close + delete the file, reset counter - * .close() — close the access handle (file handle kept) - * .inFallbackMode() — true if OPFS init failed - * .isOpen() — true if access handle is currently open - * .restoreFromExisting() — re-open a previously persisted file (post-SW-restart) - * - inFallbackMode(buffer) — convenience: !buffer || buffer.inFallbackMode() + * Appends are BATCHED: `append()` stays synchronous (pushes the line to a + * pending queue and returns true immediately, so the CAPTURE hot-path is + * unchanged), and a microtask-scheduled `_flush()` drains the queue to disk in + * one writable session. `flush()` forces durability (called before reads and + * on STOP/PAUSE) so a recording survives the SW being killed (pausa/continuar). * - * v1.4.0 trade-off: we DELIBERATELY truncate the file in `init()` (fresh - * start on every START). A user that wants append-mode needs a separate F4 - * feature. Rationale: SW restart + automatic re-append would silently mix - * pre-restart and post-restart events, which is hard to debug. Fresh start - * is predictable; the user clicks START, gets a clean file. + * Loaded two ways: + * - Chrome extension service worker (classic script): attaches `self.OpfsBuffer`. + * - Node tests (CJS via createRequire): returns `module.exports`. * * Privacy: entries are written as JSONL. Redaction happens at the injection * site (injected.js, MAIN world) before postMessage, so this module never @@ -49,6 +33,12 @@ window.OpfsBuffer = api; } else if (typeof module !== 'undefined' && module.exports) { module.exports = api; + } else if (typeof self !== 'undefined') { + // Service-worker context: no window, no module. Attach to the worker global + // so background.js (which reads self.OpfsBuffer) finds it after importScripts. + self.OpfsBuffer = api; + } else if (typeof globalThis !== 'undefined') { + globalThis.OpfsBuffer = api; } }(typeof self !== 'undefined' ? self : this, function () { 'use strict'; @@ -57,7 +47,6 @@ /** * Create a new OPFS buffer instance. - * * @param {Object} opts * @param {string} [opts.filename='captures.jsonl'] * @param {Object} [opts.navigator] — injectable for tests; defaults to globalThis.navigator @@ -66,193 +55,220 @@ function createOpfsBuffer(opts) { opts = opts || {}; var filename = opts.filename || DEFAULT_FILENAME; - // Allow injecting a mock navigator in tests. var nav = opts.navigator || (typeof navigator !== 'undefined' ? navigator : null); var opfsRoot = null; - var opfsFile = null; - var opfsAccess = null; - var opfsBytesWritten = 0; - var inMemoryCount = 0; + var opfsFile = null; // FileSystemFileHandle (async) + var opened = false; var fallbackMode = false; var initError = null; - function isOpen() { - return !!opfsAccess; - } + var diskBytes = 0; // bytes committed to disk + var diskCount = 0; // lines committed to disk (also the fallback counter) + var pending = []; // line strings not yet written + var pendingBytes = 0; // byte length of pending + var flushing = false; + var flushScheduled = false; - function inFallback() { - return fallbackMode; - } + function _enc(s) { return new TextEncoder().encode(s); } - function getCount() { - return inMemoryCount; - } + function isOpen() { return opened && !fallbackMode; } + function inFallback() { return fallbackMode; } + function getCount() { return diskCount + pending.length; } + function getBytesWritten() { return diskBytes + pendingBytes; } + function getError() { return initError; } - function getBytesWritten() { - return opfsBytesWritten; + function _resetState() { + opfsRoot = null; opfsFile = null; opened = false; + fallbackMode = false; initError = null; + diskBytes = 0; diskCount = 0; pending = []; pendingBytes = 0; + flushing = false; flushScheduled = false; } - function getError() { - return initError; + function _countLines(text) { + if (!text) return 0; + var n = 0; + for (var i = 0; i < text.length; i++) { if (text.charCodeAt(i) === 10) n++; } + return n; } /** - * Open (or create) the capture file, truncating any existing one. + * Open (or create) the capture file, TRUNCATING any existing one. + * START = new session (ADR-0003: truncate only on START / CLEAR). * @returns {Promise} true on success, false on fallback */ async function init() { - // Reset all per-session state. - opfsBytesWritten = 0; - inMemoryCount = 0; - fallbackMode = false; - initError = null; - opfsRoot = null; - opfsFile = null; - opfsAccess = null; - + _resetState(); if (!nav || !nav.storage || typeof nav.storage.getDirectory !== 'function') { - // OPFS not available (older Chrome, or test env without mock). fallbackMode = true; initError = new Error('navigator.storage.getDirectory is not available'); return false; } - try { opfsRoot = await nav.storage.getDirectory(); - // Fresh start: delete any existing file before creating a new one. - // (Documented in ADR-0002 — append mode is a future F4 feature.) - try { - await opfsRoot.removeEntry(filename); - } catch (e) { - // File may not exist — that's fine, ignore NotFoundError. - } opfsFile = await opfsRoot.getFileHandle(filename, { create: true }); - opfsAccess = await opfsFile.createSyncAccessHandle(); - opfsAccess.truncate(0); + // createWritable() WITHOUT keepExistingData starts empty → close truncates. + var w = await opfsFile.createWritable(); + await w.close(); + diskBytes = 0; diskCount = 0; + opened = true; return true; } catch (e) { console.error('[ARE] OPFS init failed, falling back to in-memory array:', e); - fallbackMode = true; - initError = e; - opfsRoot = null; - opfsFile = null; - opfsAccess = null; + fallbackMode = true; initError = e; + opfsRoot = null; opfsFile = null; opened = false; return false; } } /** - * Re-open an existing capture file (post-SW-restart path). The file is - * preserved — the caller can then decide to keep it (resume) or clear - * it. This function does NOT truncate; it just re-acquires the handles. - * + * Re-open an existing capture file WITHOUT truncating (resume / SW-restart). + * Reads the current size + line count so getCount()/getBytesWritten() and + * subsequent appends continue from the end. ADR-0003. * @returns {Promise} true if the file existed and was re-opened */ async function restoreFromExisting() { - opfsBytesWritten = 0; - inMemoryCount = 0; - fallbackMode = false; - initError = null; - opfsRoot = null; - opfsFile = null; - opfsAccess = null; - + _resetState(); if (!nav || !nav.storage || typeof nav.storage.getDirectory !== 'function') { fallbackMode = true; initError = new Error('navigator.storage.getDirectory is not available'); return false; } - try { opfsRoot = await nav.storage.getDirectory(); - var exists = true; try { - await opfsRoot.getFileHandle(filename); + opfsFile = await opfsRoot.getFileHandle(filename); } catch (e) { - exists = false; + opfsFile = null; + return false; // nothing to restore — caller should init() fresh } - if (!exists) { - // Nothing to restore — caller should call init() to start fresh. - return false; - } - opfsFile = await opfsRoot.getFileHandle(filename); - opfsAccess = await opfsFile.createSyncAccessHandle(); - // Read existing byte length so subsequent appends continue from the end. - opfsBytesWritten = opfsAccess.getSize(); + var f = await opfsFile.getFile(); + var text = await f.text(); + diskBytes = (typeof f.size === 'number') ? f.size : _enc(text).byteLength; + diskCount = _countLines(text); + opened = true; return true; } catch (e) { console.error('[ARE] OPFS restore failed:', e); - fallbackMode = true; - initError = e; - opfsRoot = null; - opfsFile = null; - opfsAccess = null; + fallbackMode = true; initError = e; + opfsRoot = null; opfsFile = null; opened = false; return false; } } /** - * Append a single entry as one JSONL line (LF terminated). - * @param {Object} entry - * @returns {boolean} true on success, false on failure (caller continues - * with the fallback path if applicable) + * Queue one entry as a JSONL line. SYNCHRONOUS: pushes to the pending + * buffer and schedules a batched flush. Returns true on success, false in + * fallback mode / before init (the caller then uses the memory buffer). */ function append(entry) { if (fallbackMode) { - // Caller should never call us in fallback mode — but be defensive. - inMemoryCount += 1; - return false; - } - if (!opfsAccess) { - console.error('[ARE] OPFS append called before init/restore'); + // Keep the counter moving so the caller can stay in sync (old contract). + diskCount += 1; return false; } + if (!opfsFile) return false; + var line = JSON.stringify(entry) + '\n'; + pending.push(line); + pendingBytes += _enc(line).byteLength; + _scheduleFlush(); + return true; + } + + function _scheduleFlush() { + if (flushScheduled || flushing) return; + flushScheduled = true; + Promise.resolve().then(function () { _flush(); }); + } + + async function _flush() { + flushScheduled = false; + if (flushing || !opfsFile || pending.length === 0) return; + flushing = true; + var batch = pending; + pending = []; + var data = batch.join(''); + var batchBytes = _enc(data).byteLength; try { - var line = JSON.stringify(entry) + '\n'; - var encoded = new TextEncoder().encode(line); - opfsAccess.write(encoded, { at: opfsBytesWritten }); - opfsBytesWritten += encoded.byteLength; - inMemoryCount += 1; - return true; + var w = await opfsFile.createWritable({ keepExistingData: true }); + if (typeof w.seek === 'function') await w.seek(diskBytes); + await w.write(data); + await w.close(); + diskBytes += batchBytes; + diskCount += batch.length; + pendingBytes -= batchBytes; + if (pendingBytes < 0) pendingBytes = 0; } catch (e) { - console.error('[ARE] OPFS write failed:', e); - return false; + console.error('[ARE] OPFS flush failed, re-queueing batch:', e); + pending = batch.concat(pending); // don't lose data on a transient failure + } finally { + flushing = false; + if (pending.length) _scheduleFlush(); + } + } + + /** + * Force everything pending to disk. Awaited before reads and on STOP/PAUSE + * so the data is durable before the SW may be killed. + * @returns {Promise} + */ + async function flush() { + var guard = 0; + while ((pending.length > 0 || flushing) && guard < 100000) { + guard += 1; + if (flushing) { + await new Promise(function (r) { setTimeout(r, 0); }); + continue; + } + await _flush(); } } /** - * Get a File object representing the capture file. Used by the download - * path: `await file.arrayBuffer()` → Blob → URL.createObjectURL. + * Get a File object for download. Flushes pending writes first so the file + * reflects every captured event. * @returns {Promise} */ async function getFile() { if (!opfsFile) { throw new Error('OPFS file handle is not open — call init() first'); } + await flush(); return await opfsFile.getFile(); } /** - * Close the access handle (for STOP). The file handle is kept so a - * subsequent `restoreFromExisting()` or `getFile()` can re-acquire it. + * Read the whole committed file as text (used by the resume path to rebuild + * the dedup set). Flushes pending writes first. + * @returns {Promise} */ - function close() { - if (opfsAccess) { - try { - opfsAccess.close(); - } catch (e) { - // Best-effort. - } - opfsAccess = null; + async function readAll() { + if (!opfsFile) return ''; + await flush(); + try { + var f = await opfsFile.getFile(); + return await f.text(); + } catch (e) { + console.error('[ARE] OPFS readAll failed:', e); + return ''; } } + /** + * Mark the buffer closed (STOP/PAUSE). The async model holds no handle open, + * but we kick a best-effort flush so the tail is persisted before the SW may + * die. The file persists for a later getFile()/restoreFromExisting(). + */ + function close() { + opened = false; + _flush(); + } + /** * Close + remove the file. Used by CLEAR. Resets all state. */ async function clear() { - close(); + pending = []; pendingBytes = 0; if (opfsRoot && filename) { try { await opfsRoot.removeEntry(filename); @@ -260,15 +276,14 @@ // File may not exist; ignore. } } - opfsFile = null; - opfsRoot = null; - opfsBytesWritten = 0; - inMemoryCount = 0; + opfsFile = null; opfsRoot = null; opened = false; + diskBytes = 0; diskCount = 0; } return { init: init, append: append, + flush: flush, getFile: getFile, getCount: getCount, getBytesWritten: getBytesWritten, @@ -276,6 +291,7 @@ clear: clear, close: close, restoreFromExisting: restoreFromExisting, + readAll: readAll, isOpen: isOpen, inFallbackMode: inFallback, // Exposed for tests + advanced introspection. diff --git a/src/popup.js b/src/popup.js index 41a0c44..6805b57 100644 --- a/src/popup.js +++ b/src/popup.js @@ -14,6 +14,7 @@ */ const btnRecord = document.getElementById('btnRecord'); +const btnPause = document.getElementById('btnPause'); const btnDownload = document.getElementById('btnDownload'); const btnClear = document.getElementById('btnClear'); const filterInput = document.getElementById('filterInput'); @@ -27,92 +28,70 @@ const endpointList = document.getElementById('endpointList'); const recordingIndicator = document.getElementById('recordingIndicator'); let isRecording = false; - -// Preset defaults — mirrors src/capture-config.js PRESETS (kept here so the -// popup can render the dropdown defaults without bundling the helpers). -const PRESET_DEFAULTS = { - generic: { - patterns: '', - filterMode: 'OR', - redact: { enabled: true, headers: ['cookie','set-cookie','authorization','x-api-key','x-auth-token','csrf-token','x-csrf-token'] } - }, - 'linkedin-voyager': { - // Bug fix 2026-06-24: patterns MUST be wrapped in /.../ to be parsed as - // regex by buildCaptureConfig. Previously stored as raw ^... which was - // round-tripped through the textarea and parsed as a literal substring, - // matching nothing. - patterns: '/^https:\\/\\/www\\.linkedin\\.com\\/(voyager\\/api\\/|li\\/track)/', - filterMode: 'OR', - redact: { - enabled: true, - headers: ['cookie','set-cookie','csrf-token','x-li-pem-metadata','x-li-pem','x-li-track','x-li-decorators','x-restli-protocol-version','authorization'] - } - }, - graphql: { - patterns: '/graphql', - filterMode: 'OR', - redact: { enabled: true, headers: ['cookie','set-cookie','authorization','x-api-key','x-auth-token','csrf-token','x-csrf-token'] } - }, - 'json-api': { - patterns: '', - filterMode: 'OR', - redact: { enabled: true, headers: ['cookie','set-cookie','authorization','x-api-key','x-auth-token','csrf-token','x-csrf-token'] } - } +let paused = false; + +// Single source of truth: presets + the filter parser live in +// src/capture-config.js (loaded by popup.html before this script). The popup no +// longer duplicates them — that drift is what broke the filter (the preset +// patterns were stored as a string but applied as an array, so applyPreset +// silently cleared the filter → captured everything). Falls back to a minimal +// generic preset if the module didn't load. +const CC = (typeof window !== 'undefined' && window.CaptureConfig) || null; +const PRESETS = (CC && CC.PRESETS) || { + generic: { id: 'generic', label: '[Generic]', sortOrder: 99, patterns: [], exclude: [], filterMode: 'OR', redact: { enabled: true, headers: [], body: [] } } }; - -const REDACT_BODY_KEYS = ['password','client_secret','access_token','refresh_token','id_token','session_token','csrf_token','private_key','privateKey','code','cookie','set-cookie']; +const DEFAULT_PRESET_ID = (CC && CC.DEFAULT_PRESET_ID) || 'generic'; function buildCaptureConfig(presetId) { - const preset = PRESET_DEFAULTS[presetId] || PRESET_DEFAULTS['linkedin-voyager']; - const rawText = filterInput.value || ''; - const lines = rawText.split(/\r?\n/).map((l) => l.trim()).filter(Boolean); - // Re-encode each line into the {type,value} shape. We use a minimal - // parser here; the source-of-truth parser lives in capture-config.js. - const patterns = lines.map((line) => { - if (line.charAt(0) === '/') { - const last = line.lastIndexOf('/'); - if (last > 0) return { type: 'regex', value: line }; - } - if (line.indexOf('*') !== -1 || line.indexOf('?') !== -1) { - return { type: 'glob', value: line }; - } - return { type: 'literal', value: line }; - }); + const preset = PRESETS[presetId] || PRESETS.generic || + { patterns: [], exclude: [], filterMode: 'OR', redact: { headers: [], body: [] } }; + // The preset's canonical patterns come straight from capture-config.js (NO + // round-trip through the textarea — that round-trip was the bug). The + // textarea adds OPTIONAL extra user filters on top. + const userPatterns = (CC && CC.parseFilter) ? CC.parseFilter(filterInput.value || '') : []; + const patterns = (preset.patterns || []).concat(userPatterns); const filterModeRadio = document.querySelector('input[name="filterMode"]:checked'); - const filterMode = filterModeRadio ? filterModeRadio.value : 'OR'; + const filterMode = filterModeRadio ? filterModeRadio.value : (preset.filterMode || 'OR'); + const enabled = !!redactToggle.checked; + const r = preset.redact || { headers: [], body: [] }; return { preset: presetId, patterns: patterns, + exclude: preset.exclude || [], filterMode: filterMode, redact: { - enabled: !!redactToggle.checked, - headers: preset.redact.enabled && redactToggle.checked ? preset.redact.headers : [], - body: redactToggle.checked ? REDACT_BODY_KEYS : [] + enabled: enabled, + headers: enabled ? (r.headers || []) : [], + body: enabled ? (r.body || []) : [] } }; } function applyPreset(presetId) { - const defaults = PRESET_DEFAULTS[presetId]; - if (!defaults) return; - // defaults.patterns is an array of {type, value} objects (see capture-config.js). - // The textarea takes a string, so we serialize properly: one pattern value per line. - // Bug fix 2026-06-24: previously used `defaults.patterns || ''` which produced - // "[object Object],[object Object]" garbage in the textarea and made the - // LinkedIn Voyager preset capture nothing. - if (Array.isArray(defaults.patterns) && defaults.patterns.length > 0) { - filterInput.value = defaults.patterns.map((p) => p.value).join('\n'); - } else { - filterInput.value = ''; - } + const preset = PRESETS[presetId]; + if (!preset) return; + // We do NOT dump the preset's patterns into the textarea (that round-trip is + // what broke the filter). The textarea is for OPTIONAL extra user filters; + // the preset's own patterns apply from capture-config.js at build time. const radios = document.querySelectorAll('input[name="filterMode"]'); - radios.forEach((r) => { r.checked = (r.value === defaults.filterMode); }); - redactToggle.checked = defaults.redact.enabled !== false; + radios.forEach((r) => { r.checked = (r.value === (preset.filterMode || 'OR')); }); + redactToggle.checked = !!(preset.redact && preset.redact.enabled !== false); updateRedactHint(); } +// Populate the preset dropdown from the canonical PRESETS (sorted by sortOrder). +function populatePresetDropdown() { + if (!presetSelect) return; + const items = Object.keys(PRESETS) + .map((id) => PRESETS[id]) + .sort((a, b) => (a.sortOrder || 99) - (b.sortOrder || 99)); + presetSelect.innerHTML = items + .map((p) => ``) + .join(''); +} + function updateRedactHint() { if (redactToggle.checked) { redactHint.textContent = 'Se redactan cookies, CSRF, y campos comunes antes de guardar.'; @@ -128,42 +107,27 @@ function updateRedactHint() { // Cargar estado al abrir popup function loadState() { chrome.runtime.sendMessage({ type: 'GET_STATE' }, (res) => { - if (!res) return; + if (chrome.runtime.lastError || !res) return; // B6: guard contra SW dormido isRecording = res.isRecording; + paused = res.paused; updateUI(res.total, res.unique); refreshPreview(); + maybeShowPausedBanner(res); }); + // Populate the dropdown from the canonical presets before restoring state. + populatePresetDropdown(); + // Restore the last used settings from chrome.storage.local. - chrome.storage.local.get(['filter', 'captureConfig', 'outputFormat', 'presetId'], (data) => { - const presetId = (data && data.presetId) || (data.captureConfig && data.captureConfig.preset) || 'linkedin-voyager'; - if (presetSelect && PRESET_DEFAULTS[presetId]) { - presetSelect.value = presetId; - } + chrome.storage.local.get(['filter', 'presetId', 'redactEnabled'], (data) => { + const presetId = (data && data.presetId && PRESETS[data.presetId]) ? data.presetId : DEFAULT_PRESET_ID; + if (presetSelect) presetSelect.value = presetId; applyPreset(presetId); - // If we have saved multi-line filter + redact state, overlay on the - // preset defaults (which were just applied above). - if (data && data.captureConfig) { - const cfg = data.captureConfig; - if (Array.isArray(cfg.patterns) && cfg.patterns.length > 0) { - filterInput.value = cfg.patterns.map((p) => p.value).join('\n'); - } else if (typeof data.filter === 'string' && data.filter.length > 0) { - // Legacy v1.2.3 single-string filter — preserve it. - filterInput.value = data.filter; - } - if (cfg.filterMode) { - const radios = document.querySelectorAll('input[name="filterMode"]'); - radios.forEach((r) => { r.checked = (r.value === cfg.filterMode); }); - } - if (cfg.redact && typeof cfg.redact.enabled === 'boolean') { - redactToggle.checked = cfg.redact.enabled; - } - } - if (data && data.outputFormat) { - const radios = document.querySelectorAll('input[name="outputFormat"]'); - radios.forEach((r) => { r.checked = (r.value === data.outputFormat); }); - } + // Restore the user's OPTIONAL extra filters (raw textarea string) — NOT the + // preset patterns (those apply from capture-config.js, no round-trip). + if (data && typeof data.filter === 'string') filterInput.value = data.filter; + if (data && typeof data.redactEnabled === 'boolean') redactToggle.checked = data.redactEnabled; updateRedactHint(); }); } @@ -172,17 +136,29 @@ function updateUI(total, unique) { totalCount.textContent = total || 0; uniqueCount.textContent = unique || 0; + // Tres estados (Fase 2): IDLE · RECORDING · PAUSED. if (isRecording) { btnRecord.textContent = '⏹ Detener'; btnRecord.classList.add('recording'); + btnPause.style.display = ''; + btnPause.textContent = '⏸ Pausar'; recordingIndicator.classList.add('active'); + } else if (paused) { + btnRecord.textContent = '⏹ Detener'; + btnRecord.classList.add('recording'); + btnPause.style.display = ''; + btnPause.textContent = '▶ Continuar'; + recordingIndicator.classList.remove('active'); } else { btnRecord.textContent = '▶ Iniciar'; btnRecord.classList.remove('recording'); + btnPause.style.display = 'none'; recordingIndicator.classList.remove('recording'); recordingIndicator.classList.remove('active'); } + // En modo OPFS no se descarga "en caliente"; permitimos descargar si hay + // datos (total > 0), incluso pausado. btnDownload.disabled = (total === 0); } @@ -208,13 +184,28 @@ function renderEndpoints(endpoints) { }).join(''); } +function maybeShowPausedBanner(state) { + if (state && state.paused) { + endpointList.innerHTML = '
⏸ Sesión pausada · ' + + (state.total || 0) + ' eventos
Continuar para seguir capturando
'; + } +} + function refreshPreview() { chrome.runtime.sendMessage({ type: 'GET_PREVIEW' }, (res) => { - if (res && res.endpoints) renderEndpoints(res.endpoints); + if (chrome.runtime.lastError || !res) return; + // B13: en modo OPFS el preview viene vacío POR DISEÑO ([] + opfsMode). No + // pisar el mensaje "Grabando…/Pausado" con el empty-state de "Iniciar". + if (res.opfsMode) return; + if (res.endpoints) renderEndpoints(res.endpoints); }); chrome.runtime.sendMessage({ type: 'GET_STATE' }, (res) => { - if (res) updateUI(res.total, res.unique); + if (chrome.runtime.lastError || !res) return; + isRecording = res.isRecording; + paused = res.paused; + updateUI(res.total, res.unique); + maybeShowPausedBanner(res); }); } @@ -227,6 +218,7 @@ presetSelect.addEventListener('change', () => { redactToggle.addEventListener('change', () => { updateRedactHint(); + try { chrome.storage.local.set({ redactEnabled: redactToggle.checked }); } catch (e) {} }); filterInput.addEventListener('input', () => { @@ -240,20 +232,13 @@ document.querySelectorAll('input[name="filterMode"]').forEach((r) => { }); }); -document.querySelectorAll('input[name="outputFormat"]').forEach((r) => { - r.addEventListener('change', () => { - if (r.checked) chrome.storage.local.set({ outputFormat: r.value }); - }); -}); - // Botón Record / Stop btnRecord.addEventListener('click', async () => { if (!isRecording) { const filter = filterInput.value.trim(); - const presetId = presetSelect.value || 'linkedin-voyager'; + const presetId = presetSelect.value || DEFAULT_PRESET_ID; const captureConfig = buildCaptureConfig(presetId); - const outputFormatRadio = document.querySelector('input[name="outputFormat"]:checked'); - const outputFormat = outputFormatRadio ? outputFormatRadio.value : 'jsonl'; + const outputFormat = 'jsonl'; const [tab] = await chrome.tabs.query({ active: true, currentWindow: true }); const tabId = (tab && tab.id) || null; @@ -281,7 +266,32 @@ btnRecord.addEventListener('click', async () => { }); } else { chrome.runtime.sendMessage({ type: 'STOP' }, () => { + if (chrome.runtime.lastError) return; isRecording = false; + paused = false; + refreshPreview(); + }); + } +}); + +// Botón Pausar / Continuar (Fase 2). Visible solo cuando hay sesión activa +// o pausada. PAUSE conserva el archivo OPFS; RESUME continúa appendeando. +btnPause.addEventListener('click', () => { + if (isRecording) { + chrome.runtime.sendMessage({ type: 'PAUSE' }, () => { + if (chrome.runtime.lastError) return; + isRecording = false; + paused = true; + const total = parseInt(totalCount.textContent, 10) || 0; + const unique = parseInt(uniqueCount.textContent, 10) || 0; + updateUI(total, unique); + maybeShowPausedBanner({ paused: true, total }); + }); + } else if (paused) { + chrome.runtime.sendMessage({ type: 'RESUME' }, (res) => { + if (chrome.runtime.lastError || (res && res.ok === false)) return; + isRecording = true; + paused = false; refreshPreview(); }); } @@ -291,8 +301,7 @@ btnRecord.addEventListener('click', async () => { btnDownload.addEventListener('click', async () => { const [tab] = await chrome.tabs.query({ active: true, currentWindow: true }); const site = (tab && tab.url) ? new URL(tab.url).hostname : 'unknown'; - const outputFormatRadio = document.querySelector('input[name="outputFormat"]:checked'); - const format = outputFormatRadio ? outputFormatRadio.value : 'jsonl'; + const format = 'jsonl'; chrome.runtime.sendMessage({ type: 'DOWNLOAD', site, format }, (res) => { if (!res) { @@ -331,14 +340,11 @@ btnDownload.addEventListener('click', async () => { bytes = new TextEncoder().encode(res.data); } - var mime = format === 'json-array' ? 'application/json' : 'application/x-ndjson'; - var blob = new Blob([bytes], { type: mime }); + var blob = new Blob([bytes], { type: 'application/x-ndjson' }); var url = URL.createObjectURL(blob); var a = document.createElement('a'); a.href = url; - a.download = res.filename || (format === 'json-array' - ? 'api-capture-' + site + '-' + Date.now() + '.json' - : 'are-capture-' + site + '-' + Date.now() + '.jsonl'); + a.download = res.filename || ('are-capture-' + site + '-' + Date.now() + '.jsonl'); a.click(); URL.revokeObjectURL(url); }); @@ -353,6 +359,53 @@ btnClear.addEventListener('click', () => { }); }); +// Botón Descargar cookies (Fase 3) — baja un .json con la auth del sitio +// (incluye httpOnly como li_at / JSESSIONID, que fetch no puede leer) vía +// chrome.cookies, para replay. NO se guarda en la captura: es un canal aparte. +const btnDownloadCookies = document.getElementById('btnDownloadCookies'); +const cookiesHint = document.getElementById('cookiesHint'); +function setCookiesHint(msg, isError) { + if (!cookiesHint) return; + cookiesHint.textContent = msg; + cookiesHint.style.color = isError ? '#f87171' : ''; +} +if (btnDownloadCookies) { + btnDownloadCookies.addEventListener('click', async () => { + let tab; + try { [tab] = await chrome.tabs.query({ active: true, currentWindow: true }); } catch (e) {} + const url = tab && tab.url; + if (!url || /^chrome(-extension)?:\/\//.test(url)) { + setCookiesHint('Abrí el sitio del que querés las cookies en la pestaña activa.', true); + return; + } + chrome.runtime.sendMessage({ type: 'GET_COOKIES', url }, (res) => { + if (chrome.runtime.lastError || !res || res.ok === false) { + setCookiesHint('Error: ' + ((res && res.error) || (chrome.runtime.lastError && chrome.runtime.lastError.message) || 'desconocido'), true); + return; + } + let host = 'site'; + try { host = new URL(url).hostname; } catch (e) {} + const payload = { + capturedAt: new Date().toISOString(), + url: url, + host: host, + count: res.count || 0, + // Header listo para curl/Postman: -H "Cookie: " + cookieHeader: res.cookieHeader || '', + cookies: res.cookies || [] + }; + const blob = new Blob([JSON.stringify(payload, null, 2)], { type: 'application/json' }); + const dlUrl = URL.createObjectURL(blob); + const a = document.createElement('a'); + a.href = dlUrl; + a.download = 'cookies-' + host + '-' + Date.now() + '.json'; + a.click(); + URL.revokeObjectURL(dlUrl); + setCookiesHint('✓ ' + (res.count || 0) + ' cookies descargadas (.json con header Cookie para replay).'); + }); + }); +} + // Auto-refresh mientras está grabando // Bug fix 2026-06-24: also poll the state itself (not just preview when // isRecording=true), so we recover from initial GET_STATE race / SW wake @@ -360,14 +413,16 @@ btnClear.addEventListener('click', () => { // is actively recording (because isRecording stays false module-level until // GET_STATE returns — and the previous polling did nothing when it was false). setInterval(() => { - if (isRecording) { + if (isRecording || paused) { refreshPreview(); } else { - // Re-fetch state — if SW is now awake and recording, this flips the UI. + // Re-fetch state — if SW is now awake and recording/paused, flips the UI. chrome.runtime.sendMessage({ type: 'GET_STATE' }, (res) => { - if (!res) return; + if (chrome.runtime.lastError || !res) return; isRecording = res.isRecording; + paused = res.paused; updateUI(res.total, res.unique); + maybeShowPausedBanner(res); }); } }, 1500); diff --git a/store-assets/STORE-LISTING.md b/store-assets/STORE-LISTING.md index f9de84c..df052b6 100644 --- a/store-assets/STORE-LISTING.md +++ b/store-assets/STORE-LISTING.md @@ -11,7 +11,7 @@ Capture every API call on any website. Reverse engineer undocumented APIs instan ## Detailed Description ### Overview -API Reverse Engineer is a Chrome extension that captures every API call (fetch + XHR) while you browse normally. No DevTools needed—just one click to start recording, and download a clean JSON with all endpoints captured. +API Reverse Engineer captures every API call (fetch + XHR) while you browse normally. No DevTools needed—one click to start recording, and download a JSON-Lines file with every request captured. **Perfect for:** - Reverse engineering undocumented private APIs @@ -22,52 +22,41 @@ API Reverse Engineer is a Chrome extension that captures every API call (fetch + ### How It Works 1. Open the extension on any tab -2. *(Optional)* Set a URL filter to reduce noise -3. Click **▶ Start Recording** +2. Pick a preset (LinkedIn, GraphQL, JSON API… or Generic) or set a URL filter +3. Click **▶ Start Recording** — pause and resume anytime 4. Use the website as you normally would -5. Click **⏹ Stop → ⬇ Download JSON** +5. Click **⏹ Stop → ⬇ Download JSONL** -You get a complete JSON export with every unique endpoint: methods, headers, request/response bodies, status codes, and timing info. +You get a JSON-Lines export of every captured request: method, URL, request/response headers and bodies, status codes, and timing. Need the auth to replay an API? One click downloads the site's cookies (including httpOnly tokens like `li_at`) to a local `.json`. ### Key Features ✅ **Intercepts fetch + XHR requests** — catches all modern API calls -✅ **Tab-scoped recording** — only captures from the tab where you start -✅ **Live counter badge** — see request count in real-time -✅ **Optional URL filter** — filter by domain, path, or keyword -✅ **Deduplication** — endpoints array shows one entry per unique endpoint -✅ **Works everywhere** — any website, any protocol -✅ **Clean dark UI** — minimal, fast, keyboard-friendly -✅ **Manifest V3** — modern, secure Chrome extension standard +✅ **Tab-scoped recording** — only the active tab +✅ **Live request counter** on the toolbar icon +✅ **Pause / Resume** — survives the MV3 service worker sleeping, no lost captures +✅ **Presets + URL filter** — domain, path, keyword, regex, glob, with noise exclusion +✅ **Secret redaction ON by default** — cookies, CSRF, and auth tokens masked before saving +✅ **Download site cookies** (incl. httpOnly) for API replay +✅ **Streams to disk (OPFS)** — handles long, large capture sessions +✅ **Clean dark UI · Manifest V3** ### Output Format -Downloaded file: `api-capture-{site}-{timestamp}.json` - -```json -{ - "meta": { - "capturedAt": "2026-02-20T14:32:00Z", - "total": 47, - "uniqueEndpoints": 23, - "site": "www.example.com" - }, - "endpoints": [ - { - "method": "POST", - "url": "https://api.example.com/v1/posts", - "requestHeaders": {...}, - "requestBody": {...}, - "status": 200, - "responseBody": {...}, - "duration": 142, - "timestamp": "2026-02-20T14:32:00Z" - }, - ... - ] -} +Downloaded file: `are-capture-{preset}-{timestamp}.jsonl` — one JSON object per line: + +``` +{"ts":"2026-06-24T14:32:00Z","preset":"linkedin-voyager","request":{"method":"POST","url":"https://www.linkedin.com/voyager/api/...","headers":{...},"body":{...}},"response":{"status":200,"headers":{...},"body":{...}},"duration_ms":142} ``` +When redaction is on (default), secrets (cookies, CSRF, auth tokens) are replaced with `[REDACTED:]` before the file is written. + ### Privacy & Security -**Local-only recording** — All captures stay on your device. No server uploads, no analytics, no tracking. Your data never leaves your browser. +**Local-only** — All captures stay on your device. No server uploads, no analytics, no tracking. Secrets (cookies, CSRF, auth tokens) are redacted by default. The `cookies` permission is used only when you click Download Cookies; `unlimitedStorage` only to stream large captures to disk (OPFS). Nothing is ever uploaded. + +### Permission justifications (for the Chrome Web Store "Privacy practices" tab) +- **cookies:** Powers the optional "Download Cookies" button. Only on an explicit user click, the extension reads the active tab site's cookies (including httpOnly auth cookies like `li_at`) via `chrome.cookies` and saves them to a local `.json` so the user can replay the site's own API. Never part of a capture, never transmitted off-device. +- **unlimitedStorage:** Lets the extension stream large API captures to the Origin Private File System (OPFS) without the ~10 MB quota, so long recording sessions don't lose data when the MV3 service worker restarts. All data stays on the user's device. +- **host `` / scripting:** To inject the fetch/XHR interceptor into the tab the user chose to record. Runs only on the active recording tab. +- **tabs:** To scope recording to the active tab and name the download file. Learn more: [Privacy Policy](https://cristiantala.com/privacy/api-reverse-engineer/) @@ -89,12 +78,13 @@ Audit what data is being sent and where. Detect privacy violations. See how professional web apps handle authentication, pagination, error handling, and more. ### Roadmap -- Firefox support (WebExtensions) -- Export as OpenAPI / Swagger spec -- Copy endpoint as cURL command -- Response diffing (track API changes) +- **WebSocket + SSE capture** — reverse engineer realtime / chat protocols (next up) +- Export to Postman collection / OpenAPI spec +- Curated preset library (LinkedIn, Skool, Stripe…) - HAR import/export -- Replay captured requests +- Firefox support (WebExtensions) + +Stays 100% local — no accounts, no cloud, no server-side component. (Full roadmap: `ROADMAP.md`.) ### Support & Contributing Found a bug? Have a feature request? diff --git a/store-assets/privacy-policy-hosteable.html b/store-assets/privacy-policy-hosteable.html index 097c287..28794bb 100644 --- a/store-assets/privacy-policy-hosteable.html +++ b/store-assets/privacy-policy-hosteable.html @@ -169,7 +169,7 @@

What We Collect

  • ❌ Does NOT use analytics or tracking
  • ❌ Does NOT store data in cloud services
  • ❌ Does NOT require account login
    • -
  • ❌ Does NOT use cookies
    • +
  • ❌ Does NOT use cookies for tracking, and never sends any cookie off your device
  • ❌ Does NOT log user behavior
    • @@ -184,7 +184,10 @@

    2. Background Service Worker

    3. Popup UI

    Displays captured data, generates JSON exports. All processing happens in-browser.

    - + +

    4. Download Cookies (opt-in, on demand)

    +

    The extension requests the cookies permission solely to power the optional Download Cookies button. When — and only when — you click it, the extension reads the cookies of the active tab's site (including httpOnly authentication cookies such as li_at / JSESSIONID) via chrome.cookies and saves them to a local .json file so you can replay the site's own API. These cookies are never part of a capture and are never transmitted anywhere — they go straight to a file on your device. If you never click the button, no cookies are ever read.

    +

    No external calls are made at any point.

    Data You Generate

    diff --git a/store-assets/screenshots-v2/1-idle-state.png b/store-assets/screenshots-v2/1-idle-state.png deleted file mode 100644 index 8545f85..0000000 Binary files a/store-assets/screenshots-v2/1-idle-state.png and /dev/null differ diff --git a/store-assets/screenshots-v2/2-recording-active.png b/store-assets/screenshots-v2/2-recording-active.png deleted file mode 100644 index a38ed1e..0000000 Binary files a/store-assets/screenshots-v2/2-recording-active.png and /dev/null differ diff --git a/store-assets/screenshots-v2/3-json-export.png b/store-assets/screenshots-v2/3-json-export.png deleted file mode 100644 index a7e382c..0000000 Binary files a/store-assets/screenshots-v2/3-json-export.png and /dev/null differ diff --git a/store-assets/screenshots-v2/promo-440x280.png b/store-assets/screenshots-v2/promo-440x280.png deleted file mode 100644 index acc1dcb..0000000 Binary files a/store-assets/screenshots-v2/promo-440x280.png and /dev/null differ diff --git a/test/_chrome-mock.js b/test/_chrome-mock.js index 2cad6f6..04d9c1a 100644 --- a/test/_chrome-mock.js +++ b/test/_chrome-mock.js @@ -206,35 +206,45 @@ export function makeOpfsMock() { size: data.byteLength, async arrayBuffer() { return data.buffer.slice(data.byteOffset, data.byteOffset + data.byteLength); + }, + async text() { + return new TextDecoder().decode(data); } }; }, - async createSyncAccessHandle() { + // Async OPFS write API (the one that actually exists in MV3 service + // workers — createSyncAccessHandle does NOT). Mirrors createWritable. + async createWritable(opts) { + const keep = !!(opts && opts.keepExistingData); + const existing = dir.get(name); + let data = (keep && existing && existing.data) ? existing.data.slice() : new Uint8Array(0); + let pos = 0; return { - write(buffer, opts) { - const at = (opts && opts.at !== undefined) ? opts.at : currentSize; - const existing = dir.get(name); - const old = (existing && existing.data) || new Uint8Array(0); - const newSize = Math.max(old.byteLength, at + buffer.byteLength); - const next = new Uint8Array(newSize); - next.set(old, 0); - next.set(new Uint8Array(buffer), at); - dir.set(name, { kind: 'file', data: next }); - currentSize = Math.max(currentSize, at + buffer.byteLength); - writes.push({ at, length: buffer.byteLength, content: buffer }); - }, - truncate(size) { - const existing = dir.get(name); - const old = (existing && existing.data) || new Uint8Array(0); - const next = old.slice(0, size); - dir.set(name, { kind: 'file', data: next }); - currentSize = size; - }, - getSize() { - const existing = dir.get(name); - return existing ? existing.data.byteLength : 0; + async seek(p) { pos = p; }, + async write(chunk) { + let bytes; + if (typeof chunk === 'string') { + bytes = new TextEncoder().encode(chunk); + } else if (chunk && chunk.type === 'write') { + if (chunk.position !== undefined) pos = chunk.position; + bytes = typeof chunk.data === 'string' ? new TextEncoder().encode(chunk.data) : new Uint8Array(chunk.data); + } else if (chunk) { + bytes = new Uint8Array(chunk.buffer || chunk); + } else { + bytes = new Uint8Array(0); + } + const end = pos + bytes.byteLength; + if (end > data.byteLength) { + const next = new Uint8Array(end); + next.set(data, 0); + data = next; + } + data.set(bytes, pos); + pos = end; + writes.push({ at: end - bytes.byteLength, length: bytes.byteLength }); }, - close() { /* no-op */ } + async truncate(size) { data = data.slice(0, size); if (pos > size) pos = size; }, + async close() { dir.set(name, { kind: 'file', data }); currentSize = data.byteLength; } }; } }; @@ -300,35 +310,45 @@ export function makeDeferredOpfsMock() { size: data.byteLength, async arrayBuffer() { return data.buffer.slice(data.byteOffset, data.byteOffset + data.byteLength); + }, + async text() { + return new TextDecoder().decode(data); } }; }, - async createSyncAccessHandle() { + // Async OPFS write API (the one that actually exists in MV3 service + // workers — createSyncAccessHandle does NOT). Mirrors createWritable. + async createWritable(opts) { + const keep = !!(opts && opts.keepExistingData); + const existing = dir.get(name); + let data = (keep && existing && existing.data) ? existing.data.slice() : new Uint8Array(0); + let pos = 0; return { - write(buffer, opts) { - const at = (opts && opts.at !== undefined) ? opts.at : currentSize; - const existing = dir.get(name); - const old = (existing && existing.data) || new Uint8Array(0); - const newSize = Math.max(old.byteLength, at + buffer.byteLength); - const next = new Uint8Array(newSize); - next.set(old, 0); - next.set(new Uint8Array(buffer), at); - dir.set(name, { kind: 'file', data: next }); - currentSize = Math.max(currentSize, at + buffer.byteLength); - writes.push({ at, length: buffer.byteLength, content: buffer }); - }, - truncate(size) { - const existing = dir.get(name); - const old = (existing && existing.data) || new Uint8Array(0); - const next = old.slice(0, size); - dir.set(name, { kind: 'file', data: next }); - currentSize = size; - }, - getSize() { - const existing = dir.get(name); - return existing ? existing.data.byteLength : 0; + async seek(p) { pos = p; }, + async write(chunk) { + let bytes; + if (typeof chunk === 'string') { + bytes = new TextEncoder().encode(chunk); + } else if (chunk && chunk.type === 'write') { + if (chunk.position !== undefined) pos = chunk.position; + bytes = typeof chunk.data === 'string' ? new TextEncoder().encode(chunk.data) : new Uint8Array(chunk.data); + } else if (chunk) { + bytes = new Uint8Array(chunk.buffer || chunk); + } else { + bytes = new Uint8Array(0); + } + const end = pos + bytes.byteLength; + if (end > data.byteLength) { + const next = new Uint8Array(end); + next.set(data, 0); + data = next; + } + data.set(bytes, pos); + pos = end; + writes.push({ at: end - bytes.byteLength, length: bytes.byteLength }); }, - close() { /* no-op */ } + async truncate(size) { data = data.slice(0, size); if (pos > size) pos = size; }, + async close() { dir.set(name, { kind: 'file', data }); currentSize = data.byteLength; } }; } }; diff --git a/test/_sw-loader.mjs b/test/_sw-loader.mjs new file mode 100644 index 0000000..1214358 --- /dev/null +++ b/test/_sw-loader.mjs @@ -0,0 +1,91 @@ +/** + * Honest service-worker loader for unit tests. + * + * The whole reason the 71-green-but-broken suite existed is that + * `_chrome-mock.js:loadBackgroundFresh()` pre-attaches `globalThis.OpfsBuffer` + * and `globalThis.MemoryBuffer` BEFORE requiring background.js. Chrome never + * does that: per the manifest, Chrome loads ONLY `src/background.js` as a + * classic service worker; background.js is responsible for pulling its own + * dependencies in via `importScripts`. + * + * This loader replicates Chrome faithfully using `node:vm`: + * - One shared global (`self` === globalThis), no `window`. + * - A real `importScripts(...)` that reads each file and runs it in the + * SAME global — exactly like a classic worker. + * - We load ONLY background.js. If background.js does not importScripts its + * deps, `self.OpfsBuffer` stays undefined — which is precisely the + * production bug (B1) the honest test reproduces. + * + * No secrets, no network, no real chrome.* — same hygiene as _chrome-mock.js. + */ +import vm from 'node:vm'; +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const HERE = path.dirname(fileURLToPath(import.meta.url)); +const REPO = path.resolve(HERE, '..'); // test/ -> repo root + +/** + * Build a service-worker-like global context. + * @param {Object} opts + * @param {Object} opts.chrome — chrome.* mock (from installChromeMock().chrome) + * @param {Object} [opts.navigator] — navigator mock for OPFS (from makeOpfsMock().navigator) + * @returns {Object} the contextified sandbox (also the worker's `self`) + */ +export function makeSwContext({ chrome, navigator } = {}) { + const sandbox = {}; + + // SW global: `self` is the global object; there is NO `window`. + sandbox.self = sandbox; + + // Runtime/Web APIs the SW + buffer modules use that are NOT vm intrinsics. + sandbox.console = console; + sandbox.chrome = chrome; + sandbox.navigator = navigator; + sandbox.setTimeout = setTimeout; + sandbox.clearTimeout = clearTimeout; + sandbox.setInterval = setInterval; + sandbox.clearInterval = clearInterval; + sandbox.setImmediate = setImmediate; + sandbox.queueMicrotask = queueMicrotask; + sandbox.TextEncoder = TextEncoder; + sandbox.TextDecoder = TextDecoder; + sandbox.URL = URL; + sandbox.Buffer = Buffer; // background.js uses Buffer in a btoa fallback branch + sandbox.btoa = (s) => Buffer.from(String(s), 'binary').toString('base64'); + sandbox.atob = (s) => Buffer.from(String(s), 'base64').toString('binary'); + + // importScripts mirrors Chrome's classic service-worker loader. Paths + // resolve like the SW at chrome-extension:///src/background.js: + // '/src/x.js' -> /src/x.js (root-absolute) + // 'x.js' -> /src/x.js (relative to the SW dir = src/) + // 'sub/x.js' -> /sub/x.js + sandbox.importScripts = function (...urls) { + for (const raw of urls) { + let rel = String(raw).replace(/^chrome-extension:\/\/[^/]+\//, ''); + rel = rel.replace(/^\//, ''); + const filePath = rel.includes('/') + ? path.join(REPO, rel) + : path.join(REPO, 'src', rel); + const code = fs.readFileSync(filePath, 'utf8'); + vm.runInContext(code, sandbox, { filename: filePath }); + } + }; + + vm.createContext(sandbox); + return sandbox; +} + +/** + * Load the service worker exactly as the manifest declares it: ONLY + * src/background.js. Chrome does NOT pre-load the buffer deps. This is the + * honest reproduction path for B1. + * @param {Object} sandbox — from makeSwContext() + * @returns {Object} the same sandbox (now with the SW evaluated) + */ +export function loadServiceWorker(sandbox) { + const swPath = path.join(REPO, 'src', 'background.js'); + vm.runInContext(fs.readFileSync(swPath, 'utf8'), sandbox, { filename: swPath }); + return sandbox; +} diff --git a/test/background.test.mjs b/test/background.test.mjs index 0a25771..1a225ed 100644 --- a/test/background.test.mjs +++ b/test/background.test.mjs @@ -118,35 +118,31 @@ test('background: counter survives the OPFS init race (bug #1)', async () => { assert.equal(state.isRecording, true); }); -test('background: badge shows red dot while recording (bug #2 UX)', async () => { +test('background: badge shows the live request count while recording', async () => { const ctx = loadBackgroundFresh(); await sendMessage(ctx, { type: 'START', tabId: 1 }, SENDER_TAB_1); - // Drain any pending badge calls (the SW restore at module load + - // START's _setBadge). + // Drain pending badge calls (SW restore at load + START's _setBadge + + // the async OPFS migration which re-sets the badge). await flushAsync(); - const before = ctx.calls.setBadge.length; - assert.ok(before > 0, 'START should have called setBadgeText at least once'); - - // The LAST setBadgeText call should show the red dot, NOT a count. - const lastBadge = ctx.calls.setBadge[ctx.calls.setBadge.length - 1]; - assert.equal(lastBadge.text, '●', 'badge must be red dot while recording'); - assert.equal(lastBadge.tabId, 1, 'badge must target the recording tab'); - - // Send a CAPTURE — the v1.4.1 bug was that the badge alternated - // between `●` and the count on every CAPTURE. In v1.4.2, the badge - // should NOT change on CAPTURE (it's driven by isRecording, not - // count). + await flushAsync(); + + // Right after START the badge shows the count (0), targeting the rec tab. + let lastBadge = ctx.calls.setBadge[ctx.calls.setBadge.length - 1]; + assert.equal(lastBadge.text, '0', 'badge shows the count (0) right after START'); + assert.equal(lastBadge.tabId, 1, 'badge targets the recording tab'); + + // A CAPTURE updates the badge to the live count. The v1.4.1 bug was the + // badge *alternating* between a dot and the number; here it always shows + // the count, so there is nothing to alternate with. await sendMessage(ctx, { type: 'CAPTURE', entry: makeEntry('https://www.linkedin.com/voyager/api/me', 'GET') }, SENDER_TAB_1); await flushAsync(); - // No new setBadgeText call should have been made for the CAPTURE. - // (setBadgeBackgroundColor may have been called, but not setBadgeText.) - const after = ctx.calls.setBadge.length; - assert.equal(after, before, 'CAPTURE must not call setBadgeText (badge is stable)'); + lastBadge = ctx.calls.setBadge[ctx.calls.setBadge.length - 1]; + assert.equal(lastBadge.text, '1', 'badge updates to the live count on CAPTURE'); }); test('background: download works after stop, JSONL has all 10 events (bug #3)', async () => { @@ -187,13 +183,13 @@ test('background: download works after stop, JSONL has all 10 events (bug #3)', const lines = text.split('\n').filter((l) => l.length > 0); assert.equal(lines.length, 10, 'JSONL body must have exactly 10 lines'); - // Each line parses as JSON. The OPFS file stores raw entries (the - // formatted `_toJsonlLine` shape with `request.method` is only applied - // on the in-memory fallback path). + // Each line parses as JSON in the canonical _toJsonlLine shape. Since + // ADR-0003 the OPFS download path normalizes the raw stored entries to the + // same {request:{...}} shape as the in-memory path (consistent output). for (let i = 0; i < 10; i++) { const obj = JSON.parse(lines[i]); - assert.equal(obj.url, 'https://www.linkedin.com/voyager/api/feed/' + i); - assert.equal(obj.method, 'GET'); + assert.equal(obj.request.url, 'https://www.linkedin.com/voyager/api/feed/' + i); + assert.equal(obj.request.method, 'GET'); } }); @@ -231,8 +227,8 @@ test('background: OPFS upgrade migrates captures (no duplicates)', async () => { for (let i = 0; i < 5; i++) { const obj = JSON.parse(lines[i]); - assert.equal(obj.url, 'https://www.linkedin.com/voyager/api/migrate/' + i); - assert.equal(obj.method, 'GET'); + assert.equal(obj.request.url, 'https://www.linkedin.com/voyager/api/migrate/' + i); + assert.equal(obj.request.method, 'GET'); } }); @@ -319,9 +315,9 @@ test('background: badge clears on stop', async () => { await sendMessage(ctx, { type: 'START', tabId: 1 }, SENDER_TAB_1); await flushAsync(); - // The badge should be `●` red. + // The badge shows the count (0) while recording. let last = ctx.calls.setBadge[ctx.calls.setBadge.length - 1]; - assert.equal(last.text, '●', 'badge is red dot after START'); + assert.equal(last.text, '0', 'badge shows the count while recording'); await sendMessage(ctx, { type: 'STOP' }, SENDER_TAB_1); await flushAsync(); @@ -372,7 +368,7 @@ test('background: download with 0 captures returns ok:false with helpful error', assert.equal(resp.lineCount, 0); }); -test('background: SW restore sets badge to red dot if isRecording was true', async () => { +test('background: SW restore sets the count badge if isRecording was true', async () => { // The SW restore block reads from chrome.storage.session at module // load. We pre-seed the storage with isRecording: true + recordingTabId. const ctx = loadBackgroundFresh({ @@ -390,11 +386,12 @@ test('background: SW restore sets badge to red dot if isRecording was true', asy // Let the SW restore callback (scheduled via setImmediate) run. await flushAsync(); - // The restore handler should have called setBadgeText({text: '●', tabId: 7}). - const dotCall = ctx.calls.setBadge.find( - (c) => c.text === '●' && c.tabId === 7 + // The restore handler should have set the count badge on tab 7 (count 0 + // with no file to restore). + const badgeCall = ctx.calls.setBadge.find( + (c) => c.text === '0' && c.tabId === 7 ); - assert.ok(dotCall, 'SW restore must set red-dot badge on tab 7'); + assert.ok(badgeCall, 'SW restore must set the count badge on tab 7'); // GET_STATE should also report isRecording: true. const state = await sendMessage(ctx, { type: 'GET_STATE' }, SENDER_TAB_1); diff --git a/test/capture-config.test.mjs b/test/capture-config.test.mjs index 2f4a69a..30338fb 100644 --- a/test/capture-config.test.mjs +++ b/test/capture-config.test.mjs @@ -42,51 +42,40 @@ test('PRESETS has exactly 4 entries: generic, linkedin-voyager, graphql, json-ap ]); }); -test('DEFAULT_PRESET_ID is linkedin-voyager', () => { - assert.equal(DEFAULT_PRESET_ID, 'linkedin-voyager'); +test('DEFAULT_PRESET_ID is generic', () => { + assert.equal(DEFAULT_PRESET_ID, 'generic'); }); -test('LinkedIn Voyager preset uses the pinned anchored regex', () => { +test('LinkedIn preset captures voyager/api + rsc-action + graphql (endpoints reales 2026)', () => { const preset = PRESETS['linkedin-voyager']; - assert.equal(preset.patterns.length, 1); - assert.equal(preset.patterns[0].type, 'regex'); - // The regex source must be exactly what the reviewer checklist pinned. - assert.equal( - preset.patterns[0].value, - '^https:\\/\\/www\\.linkedin\\.com\\/(voyager\\/api\\/|li\\/track)' - ); + assert.ok(preset.patterns.length >= 3, 'incluye los 3 endpoints de datos'); + const inc = preset.patterns, exc = preset.exclude; + assert.equal(shouldCapture('https://www.linkedin.com/voyager/api/me', inc, 'OR', exc), true); + assert.equal(shouldCapture('https://www.linkedin.com/flagship-web/rsc-action/actions/component', inc, 'OR', exc), true); + assert.equal(shouldCapture('https://www.linkedin.com/voyager/api/graphql', inc, 'OR', exc), true); + // funciona con URL relativa (substring) — como las que dispara la SPA + assert.equal(shouldCapture('/voyager/api/feed/updates', inc, 'OR', exc), true); }); -test('LinkedIn Voyager regex compiles and matches voyager/api/* on www.linkedin.com', () => { +test('LinkedIn preset EXCLUYE telemetría/estáticos (exclude gana sobre include)', () => { const preset = PRESETS['linkedin-voyager']; - assert.equal( - shouldCapture('https://www.linkedin.com/voyager/api/me', preset.patterns, 'OR'), - true - ); - assert.equal( - shouldCapture('https://www.linkedin.com/voyager/api/feed/updates', preset.patterns, 'OR'), - true - ); - assert.equal( - shouldCapture('https://www.linkedin.com/li/track?trk=foo', preset.patterns, 'OR'), - true - ); -}); - -test('LinkedIn Voyager regex does NOT match static.licdn.com or px.ads.linkedin.com', () => { - const preset = PRESETS['linkedin-voyager']; - assert.equal( - shouldCapture('https://static.licdn.com/voyager/api/foo', preset.patterns, 'OR'), - false - ); - assert.equal( - shouldCapture('https://px.ads.linkedin.com/li/track', preset.patterns, 'OR'), - false - ); - assert.equal( - shouldCapture('https://www.linkedin.com/login', preset.patterns, 'OR'), - false - ); + const inc = preset.patterns, exc = preset.exclude; + // static.licdn matchea el include (/voyager/api/) pero el exclude gana → fuera + assert.equal(shouldCapture('https://static.licdn.com/voyager/api/foo', inc, 'OR', exc), false); + assert.equal(shouldCapture('https://www.linkedin.com/li/track?trk=foo', inc, 'OR', exc), false); + // ruido que ni siquiera matchea el include → fuera + assert.equal(shouldCapture('https://www.linkedin.com/rest/trackO11yApi/trackO11y', inc, 'OR', exc), false); + // endpoint de datos legítimo sigue pasando + assert.equal(shouldCapture('https://www.linkedin.com/voyager/api/me', inc, 'OR', exc), true); +}); + +test('shouldCapture: exclude descarta aunque el include matchee (general)', () => { + const inc = [{ type: 'literal', value: '/api/' }]; + const exc = [{ type: 'literal', value: '/api/track' }]; + assert.equal(shouldCapture('https://x.test/api/data', inc, 'OR', exc), true); + assert.equal(shouldCapture('https://x.test/api/track/beacon', inc, 'OR', exc), false); + // sin exclude el comportamiento previo se mantiene + assert.equal(shouldCapture('https://x.test/api/track/beacon', inc, 'OR'), true); }); test('All presets have redact headers and body arrays (non-empty when redact enabled)', () => { diff --git a/test/e2e/filter-cookies-popup.spec.mjs b/test/e2e/filter-cookies-popup.spec.mjs new file mode 100644 index 0000000..ce4ef14 --- /dev/null +++ b/test/e2e/filter-cookies-popup.spec.mjs @@ -0,0 +1,113 @@ +/** + * filter-cookies-popup.spec.mjs — Fase 3 (preset LinkedIn real + Copy Cookies). + * + * Valida en Chromium real: + * 1. El popup arma el captureConfig desde capture-config.js (fuente única): + * patterns reales (voyager/api + rsc-action), exclude de ruido, y B10 + * (x-restli-protocol-version NO se redacta). Default = generic. + * 2. El filtro narrowea a endpoints de datos y EXCLUYE telemetría/estáticos, + * resolviendo URLs relativas (como la SPA real). + * 3. Copy Cookies obtiene cookies httpOnly (li_at) vía chrome.cookies — la auth + * que fetch no puede leer. + */ +import { test, expect, chromium } from '@playwright/test'; +import path from 'node:path'; +import os from 'node:os'; +import fs from 'node:fs/promises'; +import { fileURLToPath } from 'node:url'; +import { startFixturesServer } from './fixtures-server.mjs'; + +const REPO = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..'); +const EXT = path.join(REPO, 'dist', 'unpacked'); + +async function setup() { + const fixtures = await startFixturesServer(); + const base = `http://127.0.0.1:${fixtures.port}`; + const userDataDir = await fs.mkdtemp(path.join(os.tmpdir(), 'are-f3-')); + const ctx = await chromium.launchPersistentContext(userDataDir, { + channel: 'chromium', + args: [`--disable-extensions-except=${EXT}`, `--load-extension=${EXT}`, '--headless=new'], + }); + const page = await ctx.newPage(); + await page.goto(base + '/'); + let sw = ctx.serviceWorkers()[0]; + if (!sw) sw = await ctx.waitForEvent('serviceworker'); + const extId = new URL(sw.url()).host; + const popup = await ctx.newPage(); + await popup.goto(`chrome-extension://${extId}/popup.html`); + return { fixtures, base, ctx, page, popup }; +} + +test('Fase3 — el popup arma el config desde capture-config (fuente única) + B10 + default generic', async () => { + const { fixtures, ctx, popup } = await setup(); + try { + await popup.waitForTimeout(300); // dejar correr loadState (default + dropdown) + + // Dropdown poblado desde PRESETS; default = generic. + const opts = await popup.evaluate(() => + Array.from(document.querySelectorAll('#presetSelect option')).map((o) => o.value)); + expect(opts).toContain('generic'); + expect(opts).toContain('linkedin-voyager'); + expect(await popup.evaluate(() => document.getElementById('presetSelect').value)).toBe('generic'); + + // buildCaptureConfig usa el preset canónico de capture-config.js. + const cfg = await popup.evaluate(() => window.buildCaptureConfig('linkedin-voyager')); + const vals = cfg.patterns.map((p) => p.value); + expect(vals).toContain('/voyager/api/'); + expect(vals).toContain('/rsc-action/'); + expect(cfg.exclude.map((p) => p.value)).toContain('trackO11y'); + // B10: x-restli legible; csrf SÍ redactado. + const heads = cfg.redact.headers.map((h) => h.toLowerCase()); + expect(heads).not.toContain('x-restli-protocol-version'); + expect(heads).toContain('csrf-token'); + } finally { + await fixtures.close(); + await ctx.close(); + } +}); + +test('Fase3 — el filtro LinkedIn narrowea a datos y EXCLUYE el ruido (URLs relativas)', async () => { + const { fixtures, base, ctx, page, popup } = await setup(); + try { + await popup.waitForTimeout(200); + await popup.evaluate(async (urlBase) => { + const tabs = await chrome.tabs.query({ url: urlBase + '/*' }); + const captureConfig = window.buildCaptureConfig('linkedin-voyager'); + await chrome.runtime.sendMessage({ type: 'START', tabId: tabs[0] && tabs[0].id, captureConfig, outputFormat: 'jsonl' }); + }, base); + await page.waitForTimeout(900); + await page.evaluate(() => window.fireMixed()); + await page.waitForTimeout(900); + await popup.evaluate(() => chrome.runtime.sendMessage({ type: 'STOP' })); + const dl = await popup.evaluate(() => chrome.runtime.sendMessage({ type: 'DOWNLOAD', format: 'jsonl' })); + expect(dl.ok).toBe(true); + const jsonl = Buffer.from(dl.data, 'base64').toString('utf8'); + + // Datos capturados: + expect(jsonl, 'voyager/api/me capturado').toContain('/voyager/api/me'); + expect(jsonl, 'rsc-action capturado').toContain('/rsc-action/'); + // Ruido EXCLUIDO: + expect(jsonl, 'trackO11y NO debe estar').not.toContain('trackO11y'); + expect(jsonl, 'li/track NO debe estar').not.toContain('/li/track'); + expect(jsonl, 'static asset NO debe estar').not.toContain('/static/asset.js'); + } finally { + await fixtures.close(); + await ctx.close(); + } +}); + +test('Fase3 — Copy Cookies obtiene cookies httpOnly (li_at) vía chrome.cookies', async () => { + const { fixtures, base, ctx, popup } = await setup(); + try { + const res = await popup.evaluate((u) => chrome.runtime.sendMessage({ type: 'GET_COOKIES', url: u + '/' }), base); + expect(res.ok).toBe(true); + const names = res.cookies.map((c) => c.name); + expect(names, 'li_at httpOnly debe aparecer (fetch no puede leerlo)').toContain('li_at'); + const liat = res.cookies.find((c) => c.name === 'li_at'); + expect(liat.httpOnly).toBe(true); + expect(res.cookieHeader).toContain('li_at=AUTH_TOKEN_SECRET'); + } finally { + await fixtures.close(); + await ctx.close(); + } +}); diff --git a/test/e2e/fixtures-server.mjs b/test/e2e/fixtures-server.mjs new file mode 100644 index 0000000..2d6da3d --- /dev/null +++ b/test/e2e/fixtures-server.mjs @@ -0,0 +1,115 @@ +/** + * fixtures-server.mjs — deterministic local server for the e2e suite. + * + * Serves a page that fires fetch + XHR in the exact shapes the capture code + * has historically mishandled, plus Voyager-shaped endpoints — WITHOUT ever + * touching linkedin.com (no private data, fully replicable in CI): + * + * 1. fetch(new Request(url, {...})) → method/headers/body live on the Request (B8) + * 2. XHR with setRequestHeader → request headers (B7) + * 3. fetch with a big integer id → JSON precision + * 4. fetch on page-load → timing relative to START (B9) + * + * No dependencies; plain node:http. + */ +import { createServer } from 'node:http'; + +const PAGE_HTML = ` +ARE e2e fixture + +

    API Reverse Engineer — e2e fixture

    + + +`; + +function send(res, status, headers, body) { + res.writeHead(status, headers); + res.end(body); +} + +export function createFixturesServer() { + return createServer((req, res) => { + const url = (req.url || '').split('?')[0]; + + if (url === '/' || url === '/index.html') { + return send(res, 200, { + 'content-type': 'text/html; charset=utf-8', + // normal + httpOnly cookie (como li_at) para testear Copy Cookies. + 'set-cookie': ['sessionPref=light; Path=/', 'li_at=AUTH_TOKEN_SECRET; Path=/; HttpOnly'] + }, PAGE_HTML); + } + + if (url === '/voyager/api/me') { + return send(res, 200, { + 'content-type': 'application/vnd.linkedin.normalized+json+2.1', + 'x-restli-protocol-version': '2.0.0' + }, JSON.stringify({ data: { firstName: 'Test' }, included: [{ access_token: 'SHOULD_BE_REDACTED' }] })); + } + + if (url === '/voyager/api/messaging') { + return send(res, 200, { 'content-type': 'application/json' }, JSON.stringify({ ok: true })); + } + + // Data + noise endpoints para el test de filtro (todos 200). + if (url === '/flagship-web/rsc-action/actions/component') { + return send(res, 200, { 'content-type': 'application/json' }, JSON.stringify({ rsc: true })); + } + if (url === '/rest/trackO11yApi/trackO11y' || url === '/li/track' || url === '/static/asset.js') { + return send(res, 200, { 'content-type': 'application/json' }, JSON.stringify({ noise: true })); + } + + return send(res, 404, { 'content-type': 'text/plain' }, 'not found'); + }); +} + +/** Start the server on an ephemeral port. Returns { port, close() }. */ +export function startFixturesServer() { + const server = createFixturesServer(); + return new Promise((resolve) => { + server.listen(0, '127.0.0.1', () => { + const { port } = server.address(); + resolve({ + port, + close: () => new Promise((r) => { + // Chrome holds the connection keep-alive, so a plain server.close() + // hangs waiting for idle. Force-drop live sockets first (Node 18.2+). + server.closeAllConnections?.(); + server.close(() => r()); + }), + }); + }); + }); +} diff --git a/test/e2e/record-download.spec.mjs b/test/e2e/record-download.spec.mjs new file mode 100644 index 0000000..c5bf5c9 --- /dev/null +++ b/test/e2e/record-download.spec.mjs @@ -0,0 +1,98 @@ +/** + * record-download.spec.mjs — the browser-level proof. + * + * Loads the unpacked extension in a real Chromium (--headless=new), drives a + * full capture through the REAL extension contexts (popup → service worker → + * content script → injected MAIN-world interceptor → OPFS), and asserts the + * downloaded JSONL. This is the layer the node unit suite structurally cannot + * reach (real MV3 messaging, real injection, real OPFS), where most of the + * historical fix(...) bugs lived. + * + * Run: npm run test:e2e (pretest:e2e builds dist/unpacked first) + */ +import { test, expect, chromium } from '@playwright/test'; +import path from 'node:path'; +import os from 'node:os'; +import fs from 'node:fs/promises'; +import { fileURLToPath } from 'node:url'; +import { startFixturesServer } from './fixtures-server.mjs'; + +const REPO = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..'); +const EXT = path.join(REPO, 'dist', 'unpacked'); + +async function launch() { + const userDataDir = await fs.mkdtemp(path.join(os.tmpdir(), 'are-e2e-')); + // channel:'chromium' + --headless=new is the combination that actually loads + // MV3 extensions + starts the service worker in headless (the plain bundled + // build does not). See microsoft/playwright#33928. + const ctx = await chromium.launchPersistentContext(userDataDir, { + channel: 'chromium', + args: [ + `--disable-extensions-except=${EXT}`, + `--load-extension=${EXT}`, + '--headless=new', + ], + }); + return { ctx, userDataDir }; +} + +test('B1+B2 — la extensión real captura fetch+XHR y el JSONL descargado los contiene', async () => { + const fixtures = await startFixturesServer(); + const base = `http://127.0.0.1:${fixtures.port}`; + const { ctx } = await launch(); + + try { + // Abrir la página fixture PRIMERO — el SW MV3 arranca lazy, recién con la + // primera página (content.js se inyecta declarativamente en document_start). + const page = await ctx.newPage(); + await page.goto(base + '/'); + + // Ahora sí, el SW está vivo. + let sw = ctx.serviceWorkers()[0]; + if (!sw) sw = await ctx.waitForEvent('serviceworker'); + const extId = new URL(sw.url()).host; + + // B1: los buffers deben existir en el SW REAL (no pre-inyectados por un mock). + const buffersOk = await sw.evaluate(() => !!self.OpfsBuffer && !!self.MemoryBuffer); + expect(buffersOk, 'OpfsBuffer/MemoryBuffer deben cargar vía importScripts en el SW real').toBe(true); + + // Popup: contexto de extensión con chrome.runtime + chrome.tabs. + const popup = await ctx.newPage(); + await popup.goto(`chrome-extension://${extId}/popup.html`); + + // START sobre la tab fixture, vía popup → SW (mensajería MV3 real). + const started = await popup.evaluate(async (urlBase) => { + const tabs = await chrome.tabs.query({ url: urlBase + '/*' }); + const tabId = tabs[0] && tabs[0].id; + const captureConfig = { + preset: 'linkedin-voyager', + patterns: [{ type: 'literal', value: '/voyager/api/' }], + filterMode: 'OR', + redact: { enabled: true, headers: ['csrf-token'], body: ['access_token'] }, + }; + const res = await chrome.runtime.sendMessage({ type: 'START', tabId, captureConfig, outputFormat: 'jsonl' }); + return { tabId, res }; + }, base); + expect(started.tabId, 'el popup debe encontrar la tab fixture').toBeTruthy(); + + // Dar tiempo a la inyección del interceptor + registro del PING del content script. + await page.waitForTimeout(1000); + + // Disparar los requests DESPUÉS de START (inyección actual = al-grabar). + await page.evaluate(() => window.fireRequests()); + await page.waitForTimeout(1000); + + // STOP + DOWNLOAD vía popup → SW. + await popup.evaluate(() => chrome.runtime.sendMessage({ type: 'STOP' })); + const dl = await popup.evaluate(() => chrome.runtime.sendMessage({ type: 'DOWNLOAD', format: 'jsonl' })); + expect(dl.ok, 'DOWNLOAD no debe responder "No captures"').toBe(true); + + const jsonl = Buffer.from(dl.data, 'base64').toString('utf8'); + const lines = jsonl.trim().split('\n').filter(Boolean).map((l) => JSON.parse(l)); + const me = lines.find((l) => l.request && l.request.url && l.request.url.includes('/voyager/api/me')); + expect(me, 'el endpoint /voyager/api/me debe estar capturado (B1 cableado + B2 filtro)').toBeTruthy(); + } finally { + await fixtures.close(); + await ctx.close(); + } +}); diff --git a/test/e2e/sw-restart-resume.spec.mjs b/test/e2e/sw-restart-resume.spec.mjs new file mode 100644 index 0000000..5485090 --- /dev/null +++ b/test/e2e/sw-restart-resume.spec.mjs @@ -0,0 +1,90 @@ +/** + * sw-restart-resume.spec.mjs — Fase 2: durabilidad real (pausa/continuar). + * + * Prueba en Chromium REAL lo que el unit no puede simular de verdad: que una + * grabación sobreviva al teardown del service worker MV3. Forzamos el teardown + * con CDP (ServiceWorker.stopAllWorkers), lo despertamos con un mensaje, y + * verificamos que las capturas pre-restart sobreviven (restoreFromExisting + * re-abre el archivo OPFS y reconstruye el contador desde disco — B4). + */ +import { test, expect, chromium } from '@playwright/test'; +import path from 'node:path'; +import os from 'node:os'; +import fs from 'node:fs/promises'; +import { fileURLToPath } from 'node:url'; +import { startFixturesServer } from './fixtures-server.mjs'; + +const REPO = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..'); +const EXT = path.join(REPO, 'dist', 'unpacked'); + +async function launch() { + const userDataDir = await fs.mkdtemp(path.join(os.tmpdir(), 'are-restart-')); + const ctx = await chromium.launchPersistentContext(userDataDir, { + channel: 'chromium', + args: [`--disable-extensions-except=${EXT}`, `--load-extension=${EXT}`, '--headless=new'], + }); + return ctx; +} + +async function pollState(popup, predicate, tries = 30, delay = 200) { + let last; + for (let i = 0; i < tries; i++) { + last = await popup.evaluate(() => chrome.runtime.sendMessage({ type: 'GET_STATE' })); + if (predicate(last)) return last; + await popup.waitForTimeout(delay); + } + return last; +} + +test('Fase2 — la grabación sobrevive a un restart del service worker', async () => { + const fixtures = await startFixturesServer(); + const base = `http://127.0.0.1:${fixtures.port}`; + const ctx = await launch(); + + try { + const page = await ctx.newPage(); + await page.goto(base + '/'); + let sw = ctx.serviceWorkers()[0]; + if (!sw) sw = await ctx.waitForEvent('serviceworker'); + const extId = new URL(sw.url()).host; + + const popup = await ctx.newPage(); + await popup.goto(`chrome-extension://${extId}/popup.html`); + + // START + capturar requests. + await popup.evaluate(async (urlBase) => { + const tabs = await chrome.tabs.query({ url: urlBase + '/*' }); + await chrome.runtime.sendMessage({ + type: 'START', + tabId: tabs[0] && tabs[0].id, + captureConfig: { preset: 'generic', patterns: [{ type: 'literal', value: '/voyager/api/' }], filterMode: 'OR', redact: { enabled: false, headers: [], body: [] } }, + outputFormat: 'jsonl', + }); + }, base); + await page.waitForTimeout(1000); + await page.evaluate(() => window.fireRequests()); + await page.waitForTimeout(800); + + let state = await pollState(popup, (s) => s && s.total >= 1); + const before = state.total; + expect(before, 'debe haber capturas antes del restart').toBeGreaterThanOrEqual(1); + + // Forzar el teardown del SW (simula el sleep ~30s de MV3). + const cdp = await ctx.newCDPSession(page); + await cdp.send('ServiceWorker.enable'); + await cdp.send('ServiceWorker.stopAllWorkers'); + + // Despertar el SW con un mensaje desde el popup → corre el bloque restore. + state = await pollState(popup, (s) => s && s.total >= before); + expect(state.total, 'las capturas pre-restart deben sobrevivir (restoreFromExisting)').toBeGreaterThanOrEqual(before); + + // DOWNLOAD trae las capturas que sobrevivieron. + const dl = await popup.evaluate(() => chrome.runtime.sendMessage({ type: 'DOWNLOAD', format: 'jsonl' })); + expect(dl.ok, 'DOWNLOAD no debe abortar tras el restart').toBe(true); + const lines = Buffer.from(dl.data, 'base64').toString('utf8').trim().split('\n').filter(Boolean); + expect(lines.length, 'el JSONL debe contener las capturas que sobrevivieron').toBeGreaterThanOrEqual(before); + } finally { + await fixtures.close(); + await ctx.close(); + } +}); diff --git a/test/opfs-buffer.test.mjs b/test/opfs-buffer.test.mjs index 61c1f6e..9493b51 100644 --- a/test/opfs-buffer.test.mjs +++ b/test/opfs-buffer.test.mjs @@ -59,37 +59,35 @@ function makeOpfsMock() { return { name, size: data.byteLength, - async arrayBuffer() { return data.buffer.slice(data.byteOffset, data.byteOffset + data.byteLength); } + async arrayBuffer() { return data.buffer.slice(data.byteOffset, data.byteOffset + data.byteLength); }, + async text() { return new TextDecoder().decode(data); } }; }, - // OPFS FileSystemFileHandle.createSyncAccessHandle() - async createSyncAccessHandle() { + // OPFS FileSystemFileHandle.createWritable() — the async write API that + // actually exists in MV3 service workers (createSyncAccessHandle does NOT). + async createWritable(opts) { + const keep = !!(opts && opts.keepExistingData); + const existing = dir.get(name); + let data = (keep && existing && existing.data) ? existing.data.slice() : new Uint8Array(0); + let pos = 0; return { - write(buffer, opts) { - const at = (opts && opts.at !== undefined) ? opts.at : currentSize; - const existing = dir.get(name); - const old = (existing && existing.data) || new Uint8Array(0); - // Grow buffer if needed. - const newSize = Math.max(old.byteLength, at + buffer.byteLength); - const next = new Uint8Array(newSize); - next.set(old, 0); - next.set(new Uint8Array(buffer), at); - dir.set(name, { kind: 'file', data: next }); - currentSize = Math.max(currentSize, at + buffer.byteLength); - writes.push({ at, length: buffer.byteLength, content: buffer }); - }, - truncate(size) { - const existing = dir.get(name); - const old = (existing && existing.data) || new Uint8Array(0); - const next = old.slice(0, size); - dir.set(name, { kind: 'file', data: next }); - currentSize = size; - }, - getSize() { - const existing = dir.get(name); - return existing ? existing.data.byteLength : 0; + async seek(p) { pos = p; }, + async write(chunk) { + const bytes = typeof chunk === 'string' + ? new TextEncoder().encode(chunk) + : new Uint8Array(chunk.buffer || chunk); + const end = pos + bytes.byteLength; + if (end > data.byteLength) { + const next = new Uint8Array(end); + next.set(data, 0); + data = next; + } + data.set(bytes, pos); + pos = end; + writes.push({ at: end - bytes.byteLength, length: bytes.byteLength }); }, - close() { /* no-op */ } + async truncate(size) { data = data.slice(0, size); if (pos > size) pos = size; }, + async close() { dir.set(name, { kind: 'file', data }); currentSize = data.byteLength; } }; } }; diff --git a/test/pausa-resume.test.mjs b/test/pausa-resume.test.mjs new file mode 100644 index 0000000..27c9baf --- /dev/null +++ b/test/pausa-resume.test.mjs @@ -0,0 +1,148 @@ +/** + * pausa-resume.test.mjs — Fase 2 (pausa/continuar, durabilidad real). + * + * Estos tests son la red que faltaba para que una grabación sobreviva al + * sleep del service worker MV3 y para los verbos PAUSE/RESUME. Antes de la + * Fase 2, el SW al despertar perdía el buffer en memoria y dejaba el archivo + * OPFS huérfano (B4), y DOWNLOAD abortaba "No captures" aunque hubiera datos + * en disco (B5). + */ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { makeOpfsMock, loadBackgroundFresh } from './_chrome-mock.js'; + +function flushAsync() { return new Promise((r) => setImmediate(r)); } +async function flush(n = 6) { for (let i = 0; i < n; i++) await flushAsync(); } + +async function sendMessage(ctx, msg, sender) { + await flushAsync(); + return new Promise((resolve) => ctx.listener(msg, sender || { tab: { id: 1 } }, resolve)); +} + +const SENDER = { tab: { id: 1 } }; + +// Build an OPFS mock pre-populated with N raw-entry JSONL lines (the shape the +// SW writes to disk: {url, method, ...}). Simulates a captures.jsonl left on +// disk by a recording that the SW restart is about to resume. +function seedOpfs(n) { + const mock = makeOpfsMock(); + const lines = []; + for (let i = 0; i < n; i++) { + lines.push(JSON.stringify({ url: 'https://www.linkedin.com/voyager/api/feed/' + i, method: 'GET', status: 200, isNewEndpoint: true })); + } + const text = lines.join('\n') + (n ? '\n' : ''); + mock.dir.set('captures.jsonl', { kind: 'file', data: new TextEncoder().encode(text) }); + return mock; +} + +test('Fase2/B4 — restore tras SW restart reconstruye contador + dedup desde el archivo', async () => { + const opfs = seedOpfs(3); + const ctx = loadBackgroundFresh({ + navigator: opfs.navigator, + chrome: { storageSession: { isRecording: true, recordingTabId: 1, outputFormat: 'jsonl', filterMode: 'OR', sessionId: 's1' } }, + }); + await flush(); // correr restoreFromExisting + readAll + + let state = await sendMessage(ctx, { type: 'GET_STATE' }, SENDER); + assert.equal(state.total, 3, 'el contador debe reconstruirse a 3 desde disco (v1.4.2 daba 0)'); + assert.equal(state.unique, 3, 'el dedup debe reconstruirse a 3 endpoints'); + assert.equal(state.isRecording, true); + + // Un CAPTURE nuevo CONTINÚA appendeando (4), no reinicia la sesión. + await sendMessage(ctx, { + type: 'CAPTURE', + entry: { url: 'https://www.linkedin.com/voyager/api/feed/NEW', method: 'GET', status: 200, requestHeaders: {}, responseHeaders: {}, responseBody: {} }, + }, SENDER); + await flush(3); + state = await sendMessage(ctx, { type: 'GET_STATE' }, SENDER); + assert.equal(state.total, 4, 'tras un CAPTURE post-restart el total debe ser 4 (continúa, no resetea)'); + + // DOWNLOAD trae las 4 líneas (3 de disco + 1 nueva). + await sendMessage(ctx, { type: 'STOP' }, SENDER); + await flush(3); + const dl = await sendMessage(ctx, { type: 'DOWNLOAD' }, SENDER); + assert.equal(dl.ok, true, 'DOWNLOAD no debe abortar tras restart con datos en disco'); + assert.equal(dl.lineCount, 4, 'el JSONL descargado debe tener 4 líneas'); +}); + +test('Fase2 — restore sin archivo previo cae a memoria sin romper (no hay sesión que resumir)', async () => { + // storageSession dice isRecording pero NO hay captures.jsonl en disco. + const ctx = loadBackgroundFresh({ + chrome: { storageSession: { isRecording: true, recordingTabId: 1, outputFormat: 'jsonl', filterMode: 'OR' } }, + }); + await flush(); + const state = await sendMessage(ctx, { type: 'GET_STATE' }, SENDER); + assert.equal(state.total, 0, 'sin archivo previo, el contador arranca en 0'); + assert.equal(state.isRecording, true); +}); + +test('Fase2/B3 — PAUSE no trunca y RESUME continúa appendeando (no resetea)', async () => { + const opfs = makeOpfsMock(); + const ctx = loadBackgroundFresh({ navigator: opfs.navigator }); + await sendMessage(ctx, { type: 'START', tabId: 1, captureConfig: { patterns: [], filterMode: 'OR' } }, SENDER); + await flush(); + + const cap = (u) => sendMessage(ctx, { + type: 'CAPTURE', + entry: { url: u, method: 'GET', status: 200, requestHeaders: {}, responseHeaders: {}, responseBody: {} }, + }, SENDER); + + for (let i = 0; i < 3; i++) await cap('https://x/api/a/' + i); + await flush(); + let state = await sendMessage(ctx, { type: 'GET_STATE' }, SENDER); + assert.equal(state.total, 3); + + // PAUSE — conserva el archivo, no trunca. + await sendMessage(ctx, { type: 'PAUSE' }, SENDER); + await flush(); + state = await sendMessage(ctx, { type: 'GET_STATE' }, SENDER); + assert.equal(state.isRecording, false, 'PAUSE → isRecording false'); + assert.equal(state.paused, true, 'PAUSE → paused true'); + assert.equal(state.total, 3, 'PAUSE conserva las 3 capturas'); + + // Una captura DURANTE pausa debe descartarse. + await cap('https://x/api/IGNORADA'); + await flush(2); + state = await sendMessage(ctx, { type: 'GET_STATE' }, SENDER); + assert.equal(state.total, 3, 'una captura durante PAUSE NO cuenta'); + + // RESUME — continúa la misma sesión (append), no resetea. + await sendMessage(ctx, { type: 'RESUME' }, SENDER); + await flush(); + state = await sendMessage(ctx, { type: 'GET_STATE' }, SENDER); + assert.equal(state.isRecording, true, 'RESUME → isRecording true'); + assert.equal(state.paused, false, 'RESUME → paused false'); + assert.equal(state.total, 3, 'RESUME continúa con las 3 previas (no resetea)'); + + for (let i = 0; i < 2; i++) await cap('https://x/api/b/' + i); + await flush(2); + state = await sendMessage(ctx, { type: 'GET_STATE' }, SENDER); + assert.equal(state.total, 5, 'tras RESUME + 2 capturas, total = 5 (3+2)'); + + await sendMessage(ctx, { type: 'STOP' }, SENDER); + await flush(2); + const dl = await sendMessage(ctx, { type: 'DOWNLOAD' }, SENDER); + assert.equal(dl.ok, true); + assert.equal(dl.lineCount, 5, 'el JSONL final debe tener 5 líneas (3 pre-pausa + 2 post-resume)'); +}); + +test('Fase2 — START tras PAUSE SÍ trunca (sesión nueva, no continúa la pausada)', async () => { + const opfs = makeOpfsMock(); + const ctx = loadBackgroundFresh({ navigator: opfs.navigator }); + await sendMessage(ctx, { type: 'START', tabId: 1 }, SENDER); + await flush(); + for (let i = 0; i < 3; i++) { + await sendMessage(ctx, { type: 'CAPTURE', entry: { url: 'https://x/api/old/' + i, method: 'GET', status: 200, requestHeaders: {}, responseHeaders: {}, responseBody: {} } }, SENDER); + } + await flush(); + await sendMessage(ctx, { type: 'PAUSE' }, SENDER); + await flush(); + + // START = sesión nueva → trunca. NO debe arrastrar las 3 viejas. + await sendMessage(ctx, { type: 'START', tabId: 1 }, SENDER); + await flush(); + await sendMessage(ctx, { type: 'CAPTURE', entry: { url: 'https://x/api/new/0', method: 'GET', status: 200, requestHeaders: {}, responseHeaders: {}, responseBody: {} } }, SENDER); + await flush(); + const state = await sendMessage(ctx, { type: 'GET_STATE' }, SENDER); + assert.equal(state.total, 1, 'START trunca: solo la captura nueva, no las 3 de la sesión pausada'); +}); diff --git a/test/sw-wiring.test.mjs b/test/sw-wiring.test.mjs new file mode 100644 index 0000000..ab0f1de --- /dev/null +++ b/test/sw-wiring.test.mjs @@ -0,0 +1,86 @@ +/** + * sw-wiring.test.mjs — the HONEST test. + * + * Loads the service worker the way Chrome loads it (only src/background.js, + * via a real importScripts, WITHOUT pre-attaching globalThis.OpfsBuffer / + * MemoryBuffer). This is the test the 71-green-but-broken suite never had. + * + * Before the B1 fix: background.js never importScripts its deps, so + * self.OpfsBuffer / self.MemoryBuffer stay undefined → the SW captures + * nothing → these tests FAIL (red). That red is the first honest signal + * the project has ever produced: green now means "captures in real Chrome". + */ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { installChromeMock, makeOpfsMock } from './_chrome-mock.js'; +import { makeSwContext, loadServiceWorker } from './_sw-loader.mjs'; + +function setup() { + const ctx = installChromeMock(); + const opfs = makeOpfsMock(); + const sandbox = makeSwContext({ chrome: ctx.chrome, navigator: opfs.navigator }); + loadServiceWorker(sandbox); // loads ONLY background.js, like the manifest + return { ctx, opfs, sandbox }; +} + +// Drive a message through the SW's onMessage listener and await the response. +function send(listener, msg, sender = { tab: { id: 1 } }) { + return new Promise((resolve) => { + const returned = listener(msg, sender, resolve); + // Sync handlers call respond() before returning; async handlers return + // true and call respond() later. If a handler does neither, don't hang. + if (returned !== true) queueMicrotask(() => resolve(undefined)); + }); +} + +const tick = () => new Promise((r) => setImmediate(r)); + +test('B1 — el SW resuelve OpfsBuffer y MemoryBuffer cargado como en Chrome (sin pre-inyectar globals)', () => { + const { sandbox } = setup(); + assert.ok( + sandbox.OpfsBuffer && typeof sandbox.OpfsBuffer.createOpfsBuffer === 'function', + 'background.js debe cargar opfs-buffer.js vía importScripts y exponer self.OpfsBuffer' + ); + assert.ok( + sandbox.MemoryBuffer && typeof sandbox.MemoryBuffer.createMemoryBuffer === 'function', + 'background.js debe cargar memory-buffer.js vía importScripts y exponer self.MemoryBuffer' + ); +}); + +test('B1 — el SW registra el listener de onMessage', () => { + const { ctx } = setup(); + assert.ok(typeof ctx.getMessageListener() === 'function', 'el SW debe registrar chrome.runtime.onMessage'); +}); + +test('B1 — flujo real START → CAPTURE → DOWNLOAD captura ≥1 entry (sin globals pre-inyectados)', async () => { + const { ctx } = setup(); + const listener = ctx.getMessageListener(); + assert.ok(listener, 'el SW debe registrar onMessage'); + + await send(listener, { type: 'START', tabId: 1, outputFormat: 'jsonl' }); + await tick(); + await tick(); // dejar resolver el init async de OPFS + + await send(listener, { + type: 'CAPTURE', + entry: { + method: 'POST', + url: 'https://example.com/api/thing?x=1', + requestHeaders: { 'content-type': 'application/json' }, + requestBody: '{"a":1}', + status: 200, + responseHeaders: { 'content-type': 'application/json' }, + responseBody: '{"ok":true}', + timestamp: '2026-06-24T00:00:00.000Z', + duration: 10 + } + }); + + const state = await send(listener, { type: 'GET_STATE' }); + assert.equal(state.total, 1, 'GET_STATE.total debe ser 1 tras una captura (hoy es 0: buffers null)'); + + const dl = await send(listener, { type: 'DOWNLOAD', format: 'jsonl' }); + assert.equal(dl.ok, true, 'DOWNLOAD no debe responder "No captures"'); + const jsonl = Buffer.from(dl.data, 'base64').toString('utf8'); + assert.match(jsonl, /example\.com\/api\/thing/, 'el JSONL descargado debe contener el endpoint capturado'); +});