Add transform context for annotating compiler optimizations#216
Draft
gnidan wants to merge 21 commits into
Draft
Add transform context for annotating compiler optimizations#216gnidan wants to merge 21 commits into
transform context for annotating compiler optimizations#216gnidan wants to merge 21 commits into
Conversation
The TCO back-edge JUMP previously emitted a gather wrapper around its invoke and return contexts. Multiple discriminator keys can coexist on a single context object without gather wrapping, so the JUMP now carries a flat combined context with both `invoke` and `return` keys directly. Updates the countCallSites helper in optimizer-contexts.test to check invoke and return independently rather than as an either/or, so flat multi-discriminator contexts get counted in both buckets. The TCO-specific assertion now finds the back-edge JUMP by the presence of both discriminators rather than by a gather wrapper.
* format: make invoke.target optional for internal calls Internal calls via JUMP normally carry a code pointer to the callee's entry point. When the compiler inlines a function, the JUMP is elided — there is no physical call instruction and no code target to point at. The callee identity (identifier, declaration, type) remains meaningful, but the target pointer does not. Same pattern as #211 (making return.data optional). Unblocks inlining: bugc can emit invoke contexts on inlined first instructions without fabricating a target pointer. - Schema: drop target from InternalCall.required, expand description, add worked example for inlined case - TS types: mark target optional; guard relaxed - Spec page: document optionality and point at transform + gather for inlining annotation - bugc: guard target access in patchInvokeInContext; tests assert target defined before dereferencing * format: prefer flat form for invoke + transform composition Pair with #212's flat-form guidance: when an inlined body's first instruction carries both an invoke and a transform, those belong as sibling keys on a single context — gather isn't needed because `invoke` and `transform` don't collide.
* format: add transform context for compiler optimizations
Adds a new context type annotating instructions with the
compiler transformations that produced them. The value is an
array of short identifiers; the list may repeat the same
identifier when the transformation has been applied multiple
times (e.g., ["inline", "inline"] for doubly-inlined code).
Transform is *additional* annotation. The invoke/return contexts
for the logical call are still emitted at the call boundary so
debuggers see the source-level call stack; the transform context
tells debuggers how the call was physically realized. Consumers
that ignore transform contexts get a sound source-level view
from the semantic contexts alone.
v1 identifiers:
- "inline": marked instruction is part of an inlined function
body; surrounding invoke/return contexts name the inlined
callee.
- "tailcall": marked instruction is a tail-call-optimized
back-edge JUMP or continuation, where the call was realized
without pushing/popping a full activation.
The identifier set is extensible. Debuggers unfamiliar with a
given identifier should preserve it as an opaque label. Order
in the array is not semantically significant — the multiset is
what matters.
Unblocks the final shape of TCO back-edge annotations in
bugc (#210): a tail-call-optimized JUMP can now carry
`gather: [return, invoke, transform: ["tailcall"]]`.
Includes:
- schemas/program/context/transform.schema.yaml
- schemas/program/context.schema.yaml: wire into the if/$ref
union.
- packages/format/src/types/program/context.ts: Context.Transform
interface, isTransform guard, and Transform.Identifier union
preserving autocomplete for known values.
- packages/format/src/types/program/context.test.ts: register
Context.isTransform with the schema guard test harness.
- packages/web/spec/program/context/transform.mdx: spec page
covering role, v1 identifiers, repetition/composition, and
interaction with gather.
* format: expand transform v1 vocabulary with fold and coalesce
Adds two more identifiers to the v1 transform context
vocabulary, based on bugc optimizer's audit of transformations
the compiler currently performs or will perform:
- "fold" — compile-time constant folding. The marked
instruction carries the result (typically a PUSH) replacing
a compute sequence that appeared in source.
- "coalesce" — read-write merging. The marked instruction is
part of a SHL/OR sequence (or similar) introduced by the
compiler to combine adjacent source-level reads or writes,
such as packing narrower fields into a single storage slot.
Together with the previously-defined "inline" and "tailcall",
this covers the four transformations bugc emits today or will
emit in the near term (inline once a function inlining pass
lands). Propagate was considered for v1 and deferred as
borderline.
Updates:
- transform.schema.yaml: description enumerates the four v1
identifiers; examples include single-identifier cases for
each plus combinations ["inline", "fold"], ["coalesce",
"coalesce"].
- context.ts: Transform.Identifier union extended with "fold"
and "coalesce" (still keeps `string & {}` for extensibility
and autocomplete).
- transform.mdx: subsection for each identifier with a concrete
EVM-level example, updated repetition/composition section
with new combinations.
* format: prefer flat context composition, document gather scope
The context schema's discriminator keys combine via allOf of
if/then rules, so a single context object can carry multiple
keys at once (e.g., `invoke`, `return`, and `transform` all
side by side). Use gather only when two contexts would collide
on the same key.
- transform spec: switch the TCO back-edge example from gather
to the flat form; revise the tailcall bullet accordingly
- transform schema: note in the description that flat
composition is preferred; gather is for key collisions
- gather spec: add a "When to use" section flagging the flat
form as the default and listing the canonical collision
cases (multiple frames, multiple variables blocks)
Contributor
|
The TCO back-edge JUMP already carries a flat context with both invoke (the new iteration's call) and return (the previous iteration's return). Add a third sibling key, transform: ["tailcall"], marking the instruction as a tail-call-optimized back-edge. This is an additive annotation: it does not replace the invoke/return pair (which state the source-level facts) but tells debuggers the pair was realized as a TCO back-edge rather than a real frame push/pop, so they can avoid inventing a spurious frame. Consumers that ignore transform contexts still get a sound source-level view from invoke/return alone. Widens the emitted context type to Return & Invoke & Transform and extends the optimizer-contexts test to assert the back-edge JUMP carries transform containing "tailcall".
Add tailcall (transform context) support to the trace widgets: - extractTransformFromInstruction: gather/pick-aware collector for compiler transform identifiers (duck-typed until #212's guard lands) - extractCallInfoFromInstruction: attach isTailCall when a tailcall transform is present alongside the invoke/return - buildCallStack: a TCO back-edge carries both return and invoke on one instruction; replace the top frame in place (reuse) instead of popping to empty, and mark it isTailCall. Fixes a real call-stack correctness bug for tail-recursive loops. - CallStackDisplay: tail-call chip on the reused frame - CallInfoPanel: tail-call banner variant - Propagate isTailCall through ResolvedCallFrame / ResolvedCallInfo - CSS (+ web theme copies) for the transform/tailcall styling Tested: 9 new unit tests in mockTrace.test.ts covering extraction, the isTailCall flag, and frame replacement. Does not touch the docs TraceDrawer opt level or examples (held for product decisions).
Add two self-tail-recursive BUG programs (accumulator sum and factorial) and a Tail-call optimization section to the tracing page. Both programs fold under bugc's level-2 optimizer (verified: the recursive call terminator is eliminated and replaced with a loop trampoline), so they exercise the new tailcall transform context in the tracer widget. The section explains how a TCO back-edge JUMP composes return, invoke, and transform: ["tailcall"] as sibling keys on one context (the flat form), and how a debugger can reconcile that with the source-level call stack. Also register program/context/transform in the web schemaIndex; it was missing (gather was present), which broke the docs build for #212's transform spec page.
In transform.mdx and gather.mdx, the first reference to frame contexts now links to /spec/program/context/frame, matching the existing [`gather`](...) link precedent. Frame is the one composition concept a reader reaching these pages may not have met yet.
cloneFunction dropped the optional Ir.Function.loc and sourceId
fields, returning only { name, parameters, entry, blocks }. Since
the first optimization pass clones the module, every function lost
its declaration source info from optimization level 1 upward.
evmgen gates declaration emission on func.loc && func.sourceId, so
all invoke/return contexts lost their declaration source ranges at
optimized levels — measurably 3/3 declared at level 0, 0/3 at
levels 1-3 on the widget's runtimeInstructions path.
Copy loc/sourceId in cloneFunction's return so declarations survive
optimization. Adds a regression test asserting invoke/return
contexts still carry declaration at levels 1, 2, and 3 (with a
level-0 baseline).
…n panels (#222) * web: fix tracer-drawer opcodes/state panels not filling height The trace panels live in a flex:1 grid whose implicit row was auto- sized to content, so dragging the drawer taller left dead space below the panels instead of growing them. Give the grid an explicit 1fr row and min-height:0 (on the grid and its items) so both panels absorb the added vertical space and scroll internally. * web: add optimizer-level selector (O0/O2) to tracer drawer The drawer hardcoded optimizer level 0. Add an O0/O2 toggle in the drawer header that recompiles + retraces at the chosen level, so readers can flip to level 2 and watch optimizer transforms (e.g. the tailcall annotation on TCO back-edges) appear. compileAndTrace now takes the level explicitly; a ref mirrors the state so the example- load effect reads the current level without re-running on toggle. * programs-react: cover the flat (production) TCO back-edge shape bugc #217 emits the TCO back-edge as a single flat context object (return + invoke + transform keys together), but the existing tests only exercised the gather shape. Add flat-shape variants for transform extraction, the isTailCall flag, and frame replacement, plus a guard that stripping the marker (the #10 failure mode) drops tail-call handling. * web: dedupe call stack + tailcall render + right-column panels - Reuse the shared, tailcall-aware buildCallStack / extractCallInfo / extractTransform from @ethdebug/programs-react via a thin adapter (bugc .debug.context -> ethdebug format shape); drop the drawer's inline call-stack builder and local extractCallInfo. - Render the tail-call chip on the reused call-stack frame and a tail-call variant on the call-info banner. - Right column (gnidan's picks 1/2/4): resolved variable values (name: value via pointer resolution), gas remaining + per-step delta, and a transform annotations panel with per-tag glosses. Sections are now collapsible. * web: widen optimizer selector to O0/O1/O2/O3 gnidan's call: expose all four bugc optimizer levels (each distinct — L1 fold/prop/DCE, L2 +CSE/TCO/jump-opt, L3 +merging) rather than a two-state O0/O2 toggle, future-proofing for other transforms. The recompile+retrace already took the level; just widen the control from two buttons to four (mapped over OPT_LEVELS, per-level tooltips). The tailcall demo still lands on O2.
Update the tail-call-optimization walkthrough to reference the tracer drawer's Opt optimization-level selector (O0-O3), telling readers to compare O0 with O2 (TCO kicks in at level 2), instead of the generic "optimizer control" wording.
* bugc: emit fold transform on constant folding Marks constant-folded values with transform:["fold"] so debuggers can show that a value is a compile-time-evaluated constant rather than source the user wrote. Adds Ir.Utils.addTransform(debug, ...ids), which composes transform markers as a flat sibling key on a debug context (per the flat- composition convention) and appends to any existing transform array — so an instruction touched by multiple passes accumulates the multiset (e.g. ["fold","coalesce"]). ConstantFoldingStep applies it to folded binary and hash results. Although a folded const instruction is typically dissolved by constant propagation + DCE, ConstantPropagationStep already carries the folded const's operationDebug into the consuming instruction, so the marker survives to the emitted bytecode's runtimeInstructions (the tracer widget's path) at levels 1-3. Adds an end-to-end test asserting the fold marker reaches the bytecode at levels 1/2/3 and is absent at level 0. * bugc: route tailcall transform through addTransform helper Refactor buildTailCallJumpOptions to emit its transform:["tailcall"] marker via Ir.Utils.addTransform instead of a hand-written key, so all transform emission (fold/tailcall/coalesce/...) routes through the one helper. Behavior is identical — the back-edge JUMP still carries return + invoke + transform:["tailcall"] — but composition stays consistent (e.g. a folded-then-tailcall instruction would accumulate the multiset) and there's no divergent hand-rolled site. Existing #217 tailcall tests pass unchanged.
Marks the SHL/OR field-packing sequence produced by ReadWriteMerging (level 3) with transform:["coalesce"], so debuggers can show that a packed-storage write is compiler-synthesized rather than source the user wrote. Every instruction the merge produces (shifts, ORs, and the merged write) routes its debug through Ir.Utils.addTransform, which appends coalesce to any existing transform array — so a folded value packed into a word composes as ["fold","coalesce"]. Per #212, coalesce = read-write (SHL/OR) packing specifically; the CFG-merging passes (block-merging, return-merging) are left unmarked for v1. Adds an end-to-end test: coalesce marker present at level 3, absent at levels 0-2.
* format: spec the inlined-call virtual activation contract Document how an inlined internal call is represented and how a debugger reconstructs its virtual activation, per the inlining design (gnidan rulings: reuse jump:true, omit arg/return value pointers in v1, adopt DWARF "activation" naming). - invoke.mdx: add "Inlined internal calls" (jump:true = call kind not a literal JUMP; target omitted; transform:["inline"]) and a "Reconstructing activations" section defining the real-vs-virtual activation distinction, annotation-driven per-instruction membership (robust to non-contiguous inlined bodies), graceful identity degradation, and v1 value resolution (structure + locals; no arguments/data pointers). - transform.mdx: enrich the inline id (virtual activation, cross- link) and add an inlined-call-site worked example noting that a helper inlined at N sites yields N virtual activations. No schema change. Scope-agnostic re: the single- vs multi-site heuristic (bugc pass scope, decision #3, still open). * format: describe inlining eligibility qualitatively (decision #3) Reflect gnidan's #3 heuristic (leaf or small, non-recursive, internal, at all call sites) as compiler-typical, keeping the threshold qualitative — the format is invariant to the eligibility rule. * format: sharpen activation discriminator + membership wording Apply debugger review precision notes on #229: - Real-vs-virtual: the reliable discriminator is the inline transform marker, not target-absence — a real internal call may also omit target (#213), so no-target does NOT imply virtual. Reword the real/virtual clauses accordingly. - Membership: reword "iff it carries transform:["inline"]" (read as exact array equality) to "carries an inline identifier in its transform list" so composed markers like ["inline","fold"] still confer membership. * format: frame virtual-activation structure as compiler-typical Apply debugger note (a): "reliably carries ... identity" contradicted the graceful-degradation paragraph (identity is optional). Reword to "an inlining compiler typically preserves declaration and per-instruction source ranges; identity fields remain optional and degrade gracefully" — a compiler-behavior expectation, not a format guarantee.
Adds InliningStep (level 2, first — after L1 fold, before CSE/TCO/ JumpOpt) that replaces calls to eligible internal functions with a copy of the callee body spliced into the caller. Every inlined instruction is annotated transform:["inline"] via addTransform, and the body is bracketed by a virtual invoke (jump:true, identity, no target — the #213 optional-target signal) / virtual return, so a debugger can reconstruct a virtual activation. This lights up `inline` in the tracer, completing the transform set. v1 eligibility (correctness over coverage; follow-ups noted): - internal, non-recursive, single-return, LEAF callee; - applied at all call sites; callee deleted once fully inlined; - NOT inlined into self-recursive / TCO'd callers: inlining a helper into a tail-recursive call's arguments rewrites count(succ(n)) -> count(n+1), which the tail-call optimizer mishandles (pre-existing bug, tracked separately). Guarding this keeps the tailcall demos pristine. Return values use dest-substitution (not a continuation phi), which is robust to L3 block-merging; deep clones use structuredClone to preserve bigint const values. Updates optimizer-contexts tests: leaf helpers (add/dbl/addThree) now inline at L2+ (no real caller JUMP; inline activation instead), while recursive/multi-return functions (fact, isEven/isOdd, count) are untouched. Deferred to follow-ups: non-leaf/nested callees, multi-return, size-threshold (non-leaf) inlining.
…hip) (#233) Amend the invoke spec's "Reconstructing activations" section to resolve an ambiguity #230's inline pass exposed: an inlined body's invoke/return markers must bracket the body (invoke on the first instruction, return on the last), never duplicate across interior instructions. Duplicated boundary markers push/pop spurious activations. - State the push/pop display semantics: invoke opens inclusive of its instruction, return closes after its instruction. - Require bracketed emission; permit the single-instruction body (entry==exit) to carry both invoke and return, processed push-then-pop. - Separate the two concerns explicitly: push/pop determines an activation's lifetime; membership determines which open activation an instruction belongs to. An activation stays open across non-member (interleaved caller) instructions. No schema change.
…234) Inlined internal calls (level-2 `inline` transform) produced VIRTUAL activations that the call stack mis-rendered: because the extractor checks invoke before return, an inlined instruction's context (which carries both) always read as invoke, so buildCallStack pushed a frame per inline site and never popped — phantom `dbl > dbl` frames leaked to the end of the trace, indistinguishable from real calls. - CallInfo/CallFrame: add `isInline` (virtual activation tag). - extractCallInfoFromInstruction: set `isInline` from the `inline` transform marker (alongside the existing `isTailCall`). - buildCallStack: tag virtual frames; add a defensive per-instruction membership guard that tears down any trailing virtual frame once execution reaches an instruction without the `inline` marker, so a virtual activation can never leak into caller code. - Render an `⧉ inline` chip (dashed, italic) on the call-stack frame in both the docs TraceDrawer and the standalone CallStackDisplay, mirroring the tailcall chip — distinct from a real call. Verified end-to-end against real O2 bytecode: each inline site now shows one virtual `dbl` activation across its body and the stack returns to top level between/after sites (no phantom frames).
* bugc: bracket inlined invoke/return to boundary ops in evmgen Inlined virtual activations mis-rendered in the tracer: the call stack never popped (phantom frames). Root cause was in evmgen lowering, not the optimizer. An IR instruction lowers to N EVM micro-ops, and the generic lowering attaches that instruction's whole operationDebug to every op. That is right for source/variables/transform context (all N ops map to the instruction) but wrong for invoke/return, which are positional activation boundaries: invoke marks one push point, return one pop point. Broadcasting them across the whole op-run makes push/pop reconstruction see every op as both a push and a pop. The optimizer IR is already correctly bracketed (verified per-pass: invoke/return counts are stable through inlining + all L2/L3 passes); the smear appears only after lowering. Real calls are unaffected because their invoke/return ride single-op JUMP/JUMPDEST terminators — only inlined activations put them on multi-op compute instructions. Fix (bracket-activation.ts, wired into block generation): for the ops emitted by one instruction, keep invoke on only the first op and return on only the last op, stripped from the interior; transform markers and source/variables stay on all ops. General evmgen invariant, not inline-specific — a no-op for single-op terminators. Reaches invoke/return nested in pick/gather composites, not just flat leaves. Tests: new inline-bracket.test.ts asserts one push + one pop per site on dbl@2-sites (both=0), invoke-before-return ordering, entry vs exit placement on a multi-instruction body, and unchanged runtime behavior at O0-O3. Full suite green (428 passed). * bugc: add tailcall + real-call no-op guards to bracket test Architect's requested safety net for the general invoke/return bracketing invariant: assert it stays a no-op where invoke/return ride single-op carriers. - tailcall back-edge (tailRecursiveSum, O2): the combined invoke+return on the one back-edge JUMP survives (both>=1). - mutually recursive real calls (never inlined): no op carries both, and invoke/return are still present (real call tracing intact).
…on correlation) (#236) * format: define context `name` as a referenceable identifier Reframe the `name` context from a "disambiguation label" to its intended purpose: a machine-generated identifier for a context that other contexts reference by name, unique within a program. State the per-program uniqueness requirement. Apply it to activation reconstruction: an `invoke` context declares an activation's name; the matching `return` and the body's member instructions reference it. This pairs a call with its return order-independently — resolving the two cases push/pop + the inline marker alone cannot: adjacent same-function activations and reordered/interleaved bodies. When present, names are authoritative for activation structure; push/pop + marker-count are the name-less fallback. - name.schema.yaml: rewrite description (purpose + uniqueness); examples. - name.mdx: frame `name` as the referenceable-context-name primitive, then pick-selection and invoke/return correlation as applications; note gather carries distinct co-located names (tailcall). - invoke.mdx: add "Correlating with `name`" to Reconstructing activations (Layer 2 authoritative when present). No schema structure change. * format: sharpen name uniqueness wording (declared once, referenced freely) Per writer review of #236: 'a name must be unique' misread as 'appears once', but a name deliberately recurs — the invoke declares it, the return and every body instruction reference it. State the requirement on the declaration: each name is declared by exactly one context; other contexts reference it freely. Mirror the wording across name.schema.yaml, name.mdx, and invoke.mdx. Plus prose polish (marker-run phrasing, pick handle).
Per architect's rulings on #23, replace the pragmatic invoke-wins reconstruction with the locked contract: - extractCallEvents: expose ALL call events (ordered invoke-before- return) so a co-located invoke+return context is never swallowed as invoke-only. extractCallInfoFromInstruction now returns the first event (display banner unchanged). Parsing split into parseInvoke/parseReturn/parseRevert + collectCallInfos. - buildCallStack close-after semantics: an invoke opens a frame inclusive of its instruction; a return closes it AFTER (deferred until the viewed step advances past it), so a frame is visible while parked on its return-bearing instruction — uniformly for real and virtual frames, and correct for a single-instruction inlined body. - marker-keyed dedup: the caller-JUMP/callee-JUMPDEST collapse now also requires matching inline-ness, so a virtual invoke never merges with an adjacent real invoke of the same name. - generalized membership guard: force-pop trailing virtual frames whose depth exceeds the instruction's inline-marker count (handles nested inlining and residual smear). Tests: close-after lifetime, single-op body, adjacent-site split via the return, marker-keyed dedup, nested (double-marker) inlining, and a real-call close-after regression. 59 programs-react tests green. Verified end-to-end on real O2 (dbl@2 sites): one virtual frame per body, top level between/after, no phantom — holds even pre-de-smear.
…d shapes (#239) Add fixtures for both emission shapes the reconstruction must handle: - bracketed (post #235): invoke on the body's first op, return on the last, transform:["inline"] on all — frame visible across every body op INCLUDING the exit op (close-after), gone at the gap. - legacy smeared: every op carries invoke+return+inline — close-after still yields exactly one frame per body, no accumulation across gap-separated bodies. Verified end-to-end on real O2 (dbl@2 sites, #235 bracketed emission): one virtual frame per body incl. the exit op, top level between/after.
The `name` context has a schema (program/context/name) but was never
mirrored into the TypeScript types — no Context.Name interface, no
isName guard, and it was absent from the Context union and isContext.
Add Name (`{ name: string }`) and isName, mirroring isFrame, and wire
them into the union and isContext (schema-canonical order: name first).
Adds schema-guard test coverage for the name context.
Unblocks upcoming consumers (name as invoke/return correlation id, #26)
before compiler/UI start reading names. Behavior-preserving addition.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.