Scriba

A projection engine for Elixir event-sourced systems. Pairs with Commanded for the event-sourcing side; replaces the role commanded_ecto_projections used to play before it stopped being actively maintained.

defmodule MyApp.Projections.Orders do
  use Scriba.Projection,
    name: "orders",
    source: {Scriba.Source.Commanded, application: MyApp.CommandedApp},
    target: {Scriba.Target.Ecto, repo: MyApp.Repo},
    parallelism: 16

  def handle(%OrderPlaced{} = event, _meta) do
    {:insert, %OrderReadModel{id: event.order_id, status: "pending"}}
  end

  def handle(%OrderShipped{order_id: id}, _meta) do
    {:update, OrderReadModel, [id: id], set: [status: "shipped"]}
  end

  def handle(_, _), do: :skip
end

{:ok, _} = Scriba.start_projection(MyApp.Projections.Orders)

That's the API. The rest is operational scaffolding you get for free.

---

Status

v0.1. The architectural contract is frozen (SCRIBA_ARCHITECTURE.md) and the operational primitives — telemetry, dead-letter routing, retry policy, real pause/resume — are in place. Property tests cover per-stream ordering, position monotonicity, and effectively-once delivery under crash.

What's deliberately out of scope for v0.1:

• Lag and throughput metrics beyond raw telemetry events (v0.2 + LiveView dashboard).
• Online rebuild / shadow targets / atomic swap (v0.3).
• Sources other than Commanded; targets other than Ecto/Postgres (v0.4).
• Multi-target fan-out (v0.4).

See SCRIBA_ARCHITECTURE.md §2 for the full in-scope / out-of-scope split.

---

Why this exists

You're running Commanded. You need read models. The library you'd have reached for — commanded_ecto_projections — has been unmaintained for years and has known sharp edges around crash recovery, per-stream ordering across parallelism, and lag visibility.

Scriba is an opinionated rewrite of that role with three principles:

1. Correctness over throughput. Position update and read-model write commit atomically inside a single Ecto.Multi. There is no configuration that lets you turn this off, because you should not want to.
2. Per-stream ordering is preserved. All events for a given stream_id route to the same processor via consistent hash. Within a processor, events are serial. Across streams, events are parallel.
3. Sharp edges are documented, not hidden. Dead-lettered events advance the cursor (skip-and-continue, not block-the-projection). The handler return contract is six tagged tuples, not a DSL. Lag is a telemetry-consumer concern, not an engine feature.

---

Installation

Add to your mix.exs. The :scriba Hex package becomes available after the v0.1.0 Hex publication — until then, depend on the repo directly with {:scriba, github: "thatsme/scriba"}:

defp deps do
  [
    {:scriba, "~> 0.1.0"}, # available after Hex publication
    # Optional: only needed if using Scriba.Source.Commanded
    {:commanded, "~> 1.4"},
    # Optional: only needed if using Scriba.Target.Ecto
    {:ecto_sql, "~> 3.11"},
    {:postgrex, "~> 0.17"}
  ]
end

Add a migration to your repo for Scriba's tables:

defmodule MyApp.Repo.Migrations.AddScribaTables do
  use Ecto.Migration
  def up,   do: Scriba.Migrations.up()
  def down, do: Scriba.Migrations.down()
end

This creates two tables in your read-model database:

• scriba_positions — per-stream cursor (one row per {projection, version, stream_id}).
• scriba_dead_letters — failed events for inspection / manual replay.

Start projections during your application boot. The idiomatic pattern is a small Task in your supervision tree that calls Scriba.start_projection/1 once Ecto and Commanded are up:

# Bank.Application or equivalent
children = [
  MyApp.Repo,
  MyApp.CommandedApp,
  MyApp.Projections.Starter   # Task that calls Scriba.start_projection
]

See examples/bank/lib/bank/projections/starter.ex for the working pattern.

---

Core concepts

name and version

A projection's identity is (name, version). name is the stable logical label ("orders"). version is an integer (default 1) you bump when you want to run a new projection side-by-side with the old one during a cutover.

# orders v1 — the current production projection
defmodule MyApp.Projections.OrdersV1 do
  use Scriba.Projection, name: "orders", version: 1, ...
end

# orders v2 — running alongside, populating a new read model
defmodule MyApp.Projections.OrdersV2 do
  use Scriba.Projection, name: "orders", version: 2, ...
end

Do not encode the version in the name (name: "orders_v2"). The macro warns at compile time when it sees that pattern, because commanded_ecto_projections historically used it and it makes side-by-side versioning awkward. See architecture §5.

Handler return shapes

Your handle(event, meta) clauses must return one of these six values:

Return	Effect
:skip	Event acknowledged, no read-model write. Cursor still advances.
{:insert, schema_struct}	Ecto.Multi.insert/3
{:update, schema_module, filter_keyword, [set: keyword]}	Ecto.Multi.update_all/4 filtered by the keyword
{:delete, schema_module, filter_keyword}	Ecto.Multi.delete_all/3
{:multi, %Ecto.Multi{}}	Merged into the batch's Multi — escape hatch for :inc, complex queries, etc.
{:error, reason}	Routes to dead-letter (after retries, if enabled).

Raising an exception is also valid; it's converted to a dead-letter entry whose error_kind is the exception's module name. Per architecture §4.2.

The meta map

The second argument to handle/2:

%{
  id:          "event-uuid",
  stream_id:   "aggregate-uuid",
  type:        "OrderPlaced",
  position:    42,
  metadata:    %{correlation_id: "..."},
  occurred_at: ~U[2026-05-14 12:00:00Z]
}

Use :id for idempotency keys (globally unique). Use :position for ordering within Scriba's internal accounting; don't use it as a stable identifier across event-store rebuilds.

Per-stream ordering

Events with the same stream_id route to the same processor via consistent hashing — different events on the same stream are never processed concurrently. Events across streams are parallel, bounded by :parallelism.

This is the invariant that makes "balance += amount" projections correct without explicit locking.

Atomic position commit

Every successful batch commits the user's read-model writes AND the per-stream cursor advances in one Ecto.Multi transaction. There is no observable state where the read model advanced but the cursor didn't — or vice versa.

This is the property that makes crash recovery work: on Coordinator restart, the source is told to resume from the durably committed cursor, and Scriba's source-side dedup skips any events the upstream re-delivers below that cursor.

---

Operational features

Telemetry

Nine events fire — from the Pipeline, the Coordinator, and position-cache init. The full surface table is in Scriba.Telemetry's moduledoc and in architecture §6.3. Highlights:

[:scriba, :projection, :event, :start | :stop | :exception]
[:scriba, :projection, :batch, :stop]
[:scriba, :projection, :dead_letter]
[:scriba, :projection, :started | :paused | :resumed]
[:scriba, :projection, :cache_initialized]

:event :stop fires once per successful handler invocation — counting these gives you exact "events processed" without doing position-arithmetic across streams. :dead_letter fires once per dead-lettered event with {name, version, position, stream_id, event_type, error_kind} metadata for alerting.

Scriba.Telemetry.Handler is a placeholder GenServer slot in Scriba's supervision tree — v0.1 has no default attaches. You write your own with :telemetry.attach_many/4 in your application's start/2.

Dead-letter routing

A failing event — handler returned {:error, _} or raised, AFTER retry exhaustion — gets a row inserted into scriba_dead_letters atomically with the cursor advance. The projection does not block on bad events. This is a deliberate sharp edge, documented at architecture §9.2:

"When an event is dead-lettered, the position advances past it. The alternative — blocking the projection until the bad event is resolved — is the #2 complaint on ElixirForum after lag visibility."

If you want block-on-failure semantics for a specific projection, you build that on top: subscribe to [:scriba, :projection, :dead_letter] telemetry, page someone, manually replay from the dead-letter table once they've fixed the underlying issue.

Retry policy

Default: 3 attempts with exponential backoff (100ms, 1s, 10s) before dead-letter. Configurable per projection:

use Scriba.Projection,
  ...
  retry: [max_attempts: 5, backoff: [100, 500, 2000, 10_000]]

Or retry: false to opt out (one attempt, immediate dead-letter on failure).

The retry loop is in-handler Process.sleep — see architecture §9.1 and §9 for why this is the right primitive rather than Process.send_after (Broadway's processor model). Each retry attempt re-invokes :telemetry.span/3, so per-attempt :event :start / :event :stop / :event :exception events fire. Operators counting :event :start per event_id can see retry activity without a dedicated :retry event.

Pause / resume

Scriba.pause(MyApp.Projections.Orders) signals the source to stop yielding new events. The Pipeline tree stays alive; in-flight events finish their commit lifecycle. Scriba.resume(...) reverses the signal. Both fire telemetry; both are honest about asynchrony (the source-pause signal is send/2, so by the time pause/1 returns the signal is in the source's mailbox but the source's handle_info may not yet have run).

pause on :paused and resume on :running return {:error, {:invalid_state, _}} — they are deliberately not idempotent. Silent idempotency hides bugs. If you want "make sure this is paused" semantics, check Scriba.info/1 first or pattern-match the matching-state error case as success.

---

Failure modes worth knowing

Handler raises

Re-raise is caught by Scriba, the exception is logged as a :event :exception telemetry event, and the event enters the retry loop. After retry exhaustion, the original exception's struct and stacktrace are recorded in scriba_dead_letters with error_kind equal to the exception module name (e.g. "Elixir.ArgumentError").

Multi transaction fails

A database-level failure of the Ecto.Multi commit (constraint violation, dropped connection) marks the whole batch as failed via Broadway. The source's acknowledger does not advance, so the events are re-delivered the next time the projection's Pipeline runs (which in production usually means: when the database is back).

Multi failures do not retry through the per-event retry policy. The retry layer wraps the handler call, not the Multi commit. If your DB is intermittently failing, you want it to recover at the DB level — not for individual events to retry-then-dead-letter against a sick database.

Source redelivers events the projection has already committed

This happens after Pipeline restart — Commanded's subscription resumes from its acked position, which may be behind Scriba's durably committed position. Source-side dedup at the Pipeline's handle_message/3 (architecture §3.4) catches these: events whose position is at or below the committed cursor for their stream are returned as :skip without invoking the handler.

This is the property P3 — effectively-once under crash exists to verify. Its real-Postgres property test is test/property_db/pd3_cursor_resume_test.exs (PD3 — recovery resumes from the committed cursor, 100 iterations against real Postgres), with the integration-test side in test/scriba/projection/pipeline_test.exs.

---

Example app

examples/bank/ is a self-contained Mix project that demonstrates the full path: real Commanded (Commanded.EventStore.Adapters.InMemory for fast iteration), real Ecto, real read model. The projection module itself is ~50 lines including comments.

cd examples/bank
mix deps.get
mix bank.setup   # creates the database, runs migrations
mix bank.demo    # opens 3 accounts, dispatches 50 random ops, prints balances

The demo's wait-for-completion uses a telemetry counter on :event :stop — exactly the pattern you'd reach for in your own test code. See examples/bank/lib/mix/tasks/bank.demo.ex.

---

Running Scriba's own tests

Command	What runs	Requires
mix test.fast	Non-property tests	—
mix test.property_db	Real-Postgres property tests in test/property_db/	SCRIBA_TEST_DB_* env vars
mix test.all	Everything ExUnit will run with the current environment	— for always-runnable parts; env vars for property_db

mix test.all is the canonical full-suite command. Status reports should name the command: "fast suite: 99/0" rather than "99 tests, 0 failures." Naming the command in reports avoids the ambiguity that bit us during the diagnostic phase.

Real-Postgres property tests

Property tests in test/property_db/ (Phase B of the rebuild) need a live Postgres. Set all five of:

Variable	Example
SCRIBA_TEST_DB_HOST	localhost
SCRIBA_TEST_DB_PORT	5432
SCRIBA_TEST_DB_NAME	scriba_test
SCRIBA_TEST_DB_USER	postgres
SCRIBA_TEST_DB_PASS	postgres

Local convenience: copy .env.local.example to .env.local and fill in real values. config/test.exs auto-loads it; .env.local is gitignored. Shell environment variables override .env.local.

Policy is all-or-nothing-or-error: all five set → tests run; all five unset → tests excluded with a startup message; any subset partially set → config/test.exs raises at config load. Partial configuration is treated as a misconfiguration, not a graceful degrade.

The database must already exist; mix test does not create it. Migrations run in test_helper.exs against an existing connection.

---

Documentation

• SCRIBA_ARCHITECTURE.md — the architectural contract. Read this before opening a PR that changes engine behavior.
• examples/bank/README.md — example app walkthrough.
• Scriba.Telemetry moduledoc — full v0.1 telemetry event surface.

---

Versioning

Scriba follows semver. The v0.1.0 API surface (the seven functions in Scriba plus the macro at Scriba.Projection) is frozen — no breaking changes within 0.1.x. New optional features may land in 0.1.x patch releases.

Scriba.Target and Scriba.Source behaviours are not yet frozen — v0.4 will widen them when non-Commanded sources and non-Ecto targets land. Custom adapter authors should pin against a specific 0.1.x.

---

License

Apache-2.0. See LICENSE.

---

Contributing

Issues and PRs welcome at https://github.com/thatsme/scriba. Architecture-affecting changes should reference the relevant section of SCRIBA_ARCHITECTURE.md; deliberate departures from the contract require documentation in the PR explaining why.