Benchmarks and codebase metrics

Status: source candidate for the first slice (decision): pnpm metrics, pnpm metrics:compare and the CI metrics job on pushes to main. Rows marked planned in the catalog are not collected yet; the roadmap owns their order.

Every change to Maia should show what it did to size, speed and codebase health, better or worse, and a regression should ask for a decision instead of slipping in. The codebase measures itself: the catalog below is code (scripts/metrics/catalog/catalog.mjs), every snapshot is a JSON file, and the policy that fails or warns is the same locally and in CI.

Commands

pnpm metrics                                # size, health, startup -> .artifacts/metrics/<sha>.json
pnpm metrics --suite node,observation       # the existing Node benchmarks (minutes)
pnpm coverage                               # line and branch coverage per package (minutes)
pnpm metrics --suite all --build            # every suite but terminal, after building
pnpm metrics --suite terminal [--quick]     # tmux and GNU screen, Linux (10 minutes; --quick under one)
pnpm metrics:compare                        # HEAD against its merge base with origin/main, same machine
pnpm metrics:compare --base origin/main~3 --runs 20
pnpm metrics:compare --snapshot <file>      # against a stored snapshot, as CI does

pnpm metrics writes .artifacts/metrics/<sha>.json (<sha>-dirty.json for an uncommitted tree; ignored by Git) and prints a table. A later run of other suites on the same revision and machine adds to that snapshot. pnpm metrics:compare builds the base in a temporary worktree under the system temp directory (installed from the shared pnpm store, built through this checkout's Turborepo cache), measures both trees on this machine, prints the comparison, writes compare-<base>-<head>.json beside the snapshots, and removes the worktree, also on Ctrl-C (--keep-worktree keeps it). Each side's subprocesses get their own throwaway MAIA_HOME; nothing touches a running Node or ~/.maia. The command exits 1 when a metric fails its policy and 130 when interrupted.

Wall time on a busy machine is noisy: both commands say so when the load average passes 70% of the CPUs. Compare wall time with pnpm metrics:compare only, never across machines or runs.

What the large projects do, and what we take

Catalog

Owner names the code that owns the number. Noise: exact values repeat for the same source on one platform, so any change is real; per platform values (packs with native binaries, installed packages) compare only on the same platform; timing values are wall time; trend values come from model runs, compare only with a run of the same workload and are reported, never failed. Warn and fail thresholds count a change when it is worse by both amounts. Wall time warns only when the change is also significant (p < 0.01) on one machine, and its change never fails. terminal.node.base.bytes, the Node's idle memory, fails instead when a run measures more than 10% above the accepted value of node.idle.memory.bytes, whatever the base measured (D-177).

Suite size

Suite health

Suite startup

Suite node

Node and native-host changes (D-182) are judged on this suite:

• What counts: the TypeScript Node and its complete process-host/helper tree, never Pi workers. Active-workload comparisons include the hosts they launch, alongside the idle Node baseline.
  • Memory is PSS on Linux. macOS and Windows report their closest measure, named in the snapshot, as the terminal suite does.
  • Disk is write_bytes from /proc/<pid>/io over 3 runs.
• Throughput: 30 seconds after a 5-second warm-up, 5 runs. The clients share the host with the Node in both the baseline and candidate.
• Install: the unpacked bytes one platform's server install needs: the Node, its process hosts and helpers, and the Node.js runtime and Pi packages the worker needs.
• Startup: preserve the Node's startup phases, database included.
• Baseline and comparison:
  • TM0 adds node.idle.memory.bytes. The catalog (scripts/metrics/catalog/catalog.mjs) holds its accepted value, from the development host (Linux x64, 31 October 2026: three runs of 76.2–76.4 MiB on a warm compile cache; 99–102 MiB on 28 October, before the cache). Its warning needs --runs 3: with each side spread over a band about 1 MiB wide, a 10% growth reaches p ≈ 0.04 with two samples a side and p ≈ 0.002 with three.
  • RN0 adds the remaining planned metrics and records the TypeScript Node's values on the development host as the baseline.
  • RN1 and terminal-host changes compare the same TypeScript Node before and after the change, on one machine with the alternating rounds of Comparison. Publish the table and explain each metric past its warning threshold. The terminal suite's failure rules still apply; there is no Rust Node comparison or Node cutover gate.

Suite observation

S12a's measurement: 100 samples each, on an Apple M4 Pro (12 cores, Node 24.19.0) under parallel test load (load average 5–32):

The idle row is the median and range of six alternating runs each; the spread between runs is wider than the difference.

The latency bound belongs to this benchmark alone: wall time also counts the moments the machine gives the process no CPU, which a loaded machine stretches to seconds. pnpm test proves the wake itself instead: its live observation test freezes the Effect clock, so the heartbeat cannot fire, and requires every commit to reach an idle subscriber.

S12b's budgets live here too: 100 ms delta-to-line p95 (both harnesses, 1 and 100 Sessions, and beside a stalled subscriber) and 150 ms median catch-up CPU. test:observation-benchmark exits non-zero above them. pnpm metrics records the values whatever that exit status, and like every timing metric warns on a significant regression and never fails. pnpm test runs the same scenarios and keeps every count, ordering and resync check, but holds latency and CPU to ten times the budgets. That still catches a hang, an added wait, a viewer held back by the stalled one or a tenfold regression, and leaves a smaller regression to the metrics comparison. On the 12-core Linux development host, 11 quiet runs (load 3–7) measured p95 34–93 ms and catch-up CPU medians of 104–149 ms, so the catch-up budget has almost no headroom there.

Suite coverage

pnpm coverage builds the measured packages, then runs every package's own test script with coverage turned on, one package at a time, and records a snapshot. Each suite writes coverage/lcov.info: Node's built-in coverage (--experimental-test-coverage, source-mapped for the compiled suites) for the node:test packages, @vitest/coverage-v8 for protocol and the Node (decision). A suite that fails reports its package as unavailable, not as a lower number. No floor gates coverage yet, and CI does not run it: it costs one full test run per package (3 min 22 s on a 12-core laptop, build included), so it adds no CI minutes.

Suite dx

Suite lab

pnpm dev:lab scenario stress records this suite (dev lab); pnpm metrics does not collect it, because it needs running lab Nodes. Each run writes snapshot.json and manifest.json beside its results under .artifacts/lab/stress/<run>/. --baseline <snapshot.json> compares the run with an earlier one in stored mode, which reports timing metrics as trends and never fails them, and only when the manifest.json beside the baseline shows the same workload: Node count, concurrency, turns and injected failures. Another workload is refused. An unexpected outcome, an unresolved Session or create, a failed ps or lsof, or a leftover (a process or listening port the run did not start with) is not a metric: it fails the run itself.

Suite agent

Planned (CE1, D-173). Two kinds of number:

agent.context.fixed.bytes needs no model call: Codex reports its prompt through codex debug prompt-input, and the other executors' fixed context comes from their launch options. Token totals and reported model-call counts come from each turn's usage item, tool counts from the item trace and the pass rate from each task's checks, on runs of a fixed task set in the dev lab. A turn reports usage when it completes, so CE1b first qualifies the first-call cache measurement; a measure an executor does not report, such as Claude Code's separate reasoning tokens, is reported as unavailable. A run spends model usage, so the owner approves each one and CI never runs it; trend values compare only with a run of the same task set, models, reasoning settings and harness versions, recorded in its manifest, and never fail.

Suite terminal

Source candidate (TM0, D-175). It compares Maia's terminal multiplexer with other terminal multiplexers on one machine. The owner's target (D-177): in every scenario, each terminal and each pane costs less CPU, memory and disk than in tmux. Maia has no row yet: its rows and their failures against tmux come with its multiplexer (TM2b). Until then only the Node's base can fail.

The comparison rules:

• A terminal's cost: the scenario's total across every process of the tool. tmux, GNU screen and other tools run no Node, so their cost is their total. Maia's rows (TM2b) will subtract the same Node idle with no terminal open for as long as the scenario runs, measured for each scenario.
• A pane's cost: that cost divided by the scenario's panes, the terminal.pane.* rows. They count every pane the scenario opens: print-hidden counts its visible pane too.
• The whole server (MP0): after each tool's scenarios the runner prints, to stderr, one line per scenario with the tool's total across every process, servers and clients, and its cost per pane. Idle-N scenarios show what N panes cost at rest; the others show a loaded tool. It works the same for a tool loaded from outside the repository, adds no snapshot row and fails nothing.
• Against tmux: both costs must be below tmux's in the same run, and tmux counts all its processes. The check comes with Maia's rows (TM2b), and so does the rule that a run fails, whatever its numbers, if any terminal output reaches the disk (TM2a).
• The Node's base: its own line. A full run starts the checkout's built Node with a throwaway home and profile, leaves it idle with no Session or terminal for 3 minutes after it is ready and reads its PSS, as node.idle.memory.bytes does in the node suite; --quick reports it unavailable. A run fails when it measures more than 10% above the accepted value of node.idle.memory.bytes, whatever the base measured. pnpm metrics:compare --base measures each checkout's own Node and shows the two side by side, a line that never fails. Every report shows the totals and the base too.

Each row's subject is the tool and the scenario, such as tmux/idle-10 (tmux/quick/idle-10 with --quick). The scenarios, the same on every tool:

How it runs:

• Conditions: a 200x50 client, a scrollback limit of 100,000 lines, /bin/sh with no rc file and a minimal environment whose HOME is the scenario's fresh state directory. Each scenario starts its own server on a private socket in that directory, under ~/.cache/maia-os/terminal ($XDG_CACHE_HOME), and removes it afterwards; nothing touches another tmux server or screen session. The directory is on disk because writes to tmpfs never reach write_bytes.

• Client: one client is attached through every scenario, run by script(1), the PTY util-linux provides; the runner types into it and scans, then drops, its output for markers the workloads print.

• Processes: a tool's processes run its executable and carry the scenario's MAIA_TERMINAL_BENCH marker: servers and clients, never the shells and workloads in its panes or the PTY harness. CPU is, per process, the larger of the live threads' schedstat sum (exact) and the process total in stat (10 ms steps), which keeps the time of threads that exited; a process's delta never goes below zero. Memory is the peak of the summed PSS (smaps_rollup, sampled every 250 ms) and disk is write_bytes (io). A process that exits inside a window leaves its usage out. A thread that exits inside a window can still read up to one tick low (10 ms), which TM2b's Rust rows must account for.

• Workloads: scripts/metrics/terminal/workload.mjs writes the same bytes for every tool and needs no network. Each waits on a FIFO, so the window starts when the runner opens it, not when a slower tool gets to it.

• Tools: each multiplexer is a JSON file of command templates that one driver runs: scripts/metrics/terminal/tmux.json and screen.json are complete examples. Its keys:

  • name, and executables: paths or names on the tool's PATH, whose processes count;
  • environment and files: variables, and files written into the state directory, such as a configuration file;
  • server and ready, optional: a long-running server, and a command that succeeds once it answers;
  • open, visiblePane and hiddenPane: a command that adds a pane, whose ID comes from capture (a dotted path into the JSON it printed, or "" for its whole output) or from a pane template;
  • run, attach and stop: type a line in a pane, start a client, and the commands that end the session.

  Templates may use {state}, {session}, {columns}, {rows}, {scrollback}, {shell} and every captured value; run adds {pane} and {line}; pane steps add {index} (the panes added so far) and, for visiblePane, {target} and {direction} (right or down) of a breadth-first split. A placeholder is lowercase letters, so tmux's #{pane_id} stays as written.

• Another multiplexer: MAIA_TERMINAL_ADAPTER=<file> names such a file kept outside the repository. Its rows go to a file in MAIA_TERMINAL_RESULTS (default: maia-terminal-results in the system temp directory), which must be outside the repository; they never reach the snapshot.

• Node base: a started Node holds startup garbage, more than twice its settled size, until a collection about 100 s in: 245 MiB of PSS, then 99–102 MiB, or 76 MiB once the compile cache is warm, on the development host (28 and 31 October 2026). The runner starts the Node once in the same profile first, uncounted, to fill the cache, and prints that cold start (time to ready) on stderr without gating it. Hence the 3 idle minutes, counted from the ready record the Node prints, which each measurement and each round of a comparison adds. The Node gets the tools' minimal environment. A Node that exits early, or is not ready within 60 s, makes the row unavailable with the reason.

• Planned: Maia's rows (TM2b); echo over SSH.

• Platforms: Linux. macOS and Windows report the suite unavailable until they get their closest memory measure (physical footprint, private working set), named in the snapshot.

• Who runs it: pnpm metrics and --suite all do not collect it, because it starts other tools; a person or the dev lab names it. CI does not run it.

Screen engines (TM0)

Measured once, for the owner's pick before TM2b (D-175), on the development host (12 vCPU, Linux x64, rustc 1.99.1, tmux 3.6), 28–29 October 2026. A harness outside the repository links each candidate crate into a binary of its own, beside a baseline binary with no engine, and feeds each the same bytes: the suite's workloads with the terminal's newline translation applied. No crate joins a Cargo.toml until TM2b adds the one the owner picks. Each throughput run waited for a 1-minute load below 4.

• Conditions: a 200 × 50 screen with 10,000 lines of scrollback, D-175's default, fed in 16 KiB reads. Every transitive license is Apache-2.0, MIT, Zlib or Unlicense.
• Throughput: the median of 5 runs, a fresh terminal each run; mixed is the suite's 100 MiB print, redraw 600 full-screen frames on the alternate screen, lines 100,000 lines through the scrollback. Engine times are parse-only wall time with 10,000 lines of history: the print takes 1.9 s in alacritty_terminal, 3.3 s in avt and 4.2 s in vt100. For context only, tmux spent 8,169 ms of CPU on the same print in the suite's full run, the whole tool with 100,000 lines of history and a client attached. These figures do not establish comparative speed or D-177 compliance.
• Memory: PSS growth after filling the scrollback. For lines, the per-terminal figure rounded identically to 0.1 MiB with 1 and 10 engine instances in one process; mixed was measured with 1. Dividing by the 2,000,000 cells gives an estimated 24 (alacritty_terminal), 17 (avt) and 32 bytes (vt100) a cell; memory can vary with content, such as combining characters. tmux, measured separately as its server's PSS over an empty pane with no client attached, holds the same 10,000 lines in 10.8 MiB (lines) and 10.5 MiB (mixed), a third to a sixth of the candidates' figures.
• Snapshot: copying the 10,000 visible cells (character, colors and four flags) into a plain array, 1,000 times, after 30 frames. All three report changed rows: alacritty_terminal through damage(), avt through a dirty observer, vt100 by diffing against a copy of the previous screen.
• Binary growth: a stripped release binary (fat LTO, panic = "abort") minus the baseline's 343,864 bytes. alacritty_terminal's tree includes PTY and event-loop crates no feature turns off.
• Correctness: each fixture also ran in tmux (private socket, the same size and history), and its capture was compared row by row, with the tabs it keeps expanded to 8-column stops and trailing whitespace removed. All three engines match its final text rows on colored log lines with wraps (291 rows), on scrollback (1,001 rows) and on alternate-screen frames with synchronized output and 256 and true color (50 rows). Colors, attributes and synchronized presentation were not checked. On a fixture of edge cases (wide and combining characters, tabs, cursor moves, erase, insert and delete, saved cursor, a scroll region) they match every visible row but DEC line drawing: tmux keeps those cells in the line-drawing set, and alacritty_terminal and avt draw them as box characters; vt100 ignores the character set and draws the letters. Scrollback differs in one case: tmux also saves the lines that scroll out of a region whose top is below the screen's top, and the three engines save only lines leaving a region that starts at the top row.

Snapshot maia.metrics/v1

One measurement run of one revision on one machine, validated with Effect Schema (scripts/metrics/snapshot/snapshot.mjs). Metric metadata stays in the catalog, so a snapshot is small and today's catalog reads an old one. A measurement is exact (value), sampled (samples and their median as value) or unavailable (status and the reason, with the fix).

{
  "schema": "maia.metrics/v1",
  "createdAt": "2026-10-02T02:58:07.512Z",
  "source": { "revision": "<40 hex>", "dirty": false },
  "environment": {
    "platform": "darwin",
    "arch": "arm64",
    "node": "v24.22.1",
    "cpuModel": "Apple M2 Max",
    "cpuCount": 12,
    "memoryBytes": 34359738368,
    "loadAverage": [14.1, 16.2, 13.6],
    "ci": false
  },
  "suites": ["size", "health", "startup"],
  "measurements": [
    { "metric": "cli.standalone.bytes", "value": 259640 },
    { "metric": "tests.files", "subject": "@maiaos/cli", "value": 51 },
    { "metric": "cli.start.help.ms", "samples": [64.1, 66.0], "value": 65.05 },
    {
      "metric": "node.start.bootstrap.p95.ms",
      "status": "unavailable",
      "reason": "test:startup-budget:built wrote no evidence: ..."
    }
  ]
}

A new field or meaning needs maia.metrics/v2.

History

• Locally: .artifacts/metrics/, ignored by Git, one file per revision.
• In CI: each passing metrics job uploads its snapshot as the maia-metrics-baseline artifact (kept 90 days). The next run asks the Actions API for the newest such artifact whose commit is an ancestor of HEAD, downloads it and compares. The job checks out the full history (a few megabytes), so the baseline commit and the trailers since it are always present. A failed job uploads no baseline, so its regression stays in scope until a commit fixes or accepts it, as change-scope keeps a failed run's changes in scope. A failed lookup (the API, the token) fails the job; only "no baseline yet" records a snapshot without comparing.
• Why artifacts, not a branch or a service: exact metrics can be measured again from any commit, so Git already holds their full history; a snapshot only saves the rebuild. Wall time never compares across runs. Artifacts need no write permission, add nothing to the repository and no service. A durable metrics branch for long-term dashboards is an open decision.

Comparison

• Exact metrics: head minus base; relative is that change over the base. A worse change passes a threshold when it reaches both its absolute and its relative amount.
• Timing metrics, same machine (--base): one warm-up of each tree, then --runs rounds that alternate base-head and head-base, so drift in load falls on both sides. Welch's t-test compares the means; the report shows the means, the change, its 95% confidence interval relative to the base, the p-value and Node.js-style stars. A change counts when p < 0.01 and it passes the metric's warn threshold; it never fails a run. A suite that samples once by default samples twice per side here; --runs 1 gives insufficient.
• Timing metrics, stored snapshot (CI): trend, never a verdict.
• Ceilings: a metric with a ceiling in the catalog fails a run, in either mode, when the head measures past its margin over the accepted value, whatever the base measured. The row keeps the base beside it. terminal.node.base.bytes fails more than 10% above the accepted value of node.idle.memory.bytes (D-177). A Metrics-Accepted trailer accepts it for its own commits only: later runs compare with the same accepted value and fail again, so a lasting raise is a reviewed change of that value in the catalog.
• Per-platform metrics across platforms: not comparable.
• Lost and new measurements: an exact metric the base measured and the head cannot fails (no longer measured), so a broken collector or a missing artifact cannot pass and become the next baseline. A new package is graded from zero on the per-package exact metrics (dependencies, npm pack, tests), so adding a workspace package fails until a Metrics-Accepted trailer gives the reason, as any new dependency does. Publishing a private package grows its npm pack the same way. Pull requests do not run this job, so the trailer belongs in the squash commit that publishes it.
• Other verdicts: new (a metric or subject the base did not measure, such as a new protocol export), removed, unavailable (with both reasons) and trend for a metric the catalog no longer has.

Regression policy

Triage. A failing metrics job turns main red with an annotation per metric. The metric's owner (catalog) either fixes it or accepts it in a commit on main that explains why:

build(cli): bundle the coordinator MCP server

Metrics-Accepted: cli.standalone.bytes The MCP server loads on demand (S4c)
Metrics-Accepted: dependencies.production[@maiaos/cli] Approved in the decision log

A key without [subject] accepts every subject of that metric. The trailer counts in any paragraph of a commit message that holds nothing but Key: value lines, so a squash merge keeps it; a wrapped reason continues on an indented line. The next run compares with the last passing baseline, sees the trailer in the range and passes; the baseline then moves. A ceiling does not move: later runs fail again until a reviewed catalog commit changes its accepted value. Accepting a warning is optional. Thresholds change only in a reviewed catalog commit.

CI job and its cost

The metrics job runs after the scope job, in parallel with the quality job, on pushes to main whose change selects any package (change-scope's packages output), so documentation-only pushes skip it; pull requests do not run it yet. It installs, builds the measured packages (unchanged ones restore from the Turborepo cache), finds and downloads the baseline, runs size, health and startup, and uploads the report (30 days) and, when it passes, the next baseline (90 days).

Estimated from today's CI step timings (pnpm and Node setup about 30 s, install 40-100 s, cache restore a few seconds) plus about 40 s of measurement: 2.5-4 minutes of wall time, billed as 3-4 Linux minutes per run (GitHub rounds each job up to a minute). When the Turborepo cache lacks the client's export, building it adds 1-2 minutes. Re-running the job uploads a report of its own and replaces the baseline. The repository is private, so these count against the plan's minutes at the Linux rate; macOS minutes cost ten times as much, and this job never uses macOS. A full CI run today bills about 55 minutes (the macOS job alone 20), so the job adds about 6%. Running the same steps inside the quality job would bill about 1 minute instead, at the cost of delaying that job by that minute; a separate job keeps CI feedback as fast as before.

Limits

• Test counts are static: a test.each table counts once, a test built in a loop not at all. They trend; they never fail.
• Coverage counts only the files a package's tests load, so a module no test imports does not lower it. Node's test coverage is still experimental (stability 1); a Node upgrade can change its numbers.
• effect.gaps and tests.owned_mocks read the rule ledgers, which the lint rules keep exact; they cost no extra scan.
• test:startup-budget:built keeps its own wall-time budgets, so a loaded machine can fail it (authenticated readiness p95 above 200 ms) while it still writes the evidence the node suite reads. Its Node bundle ceiling is 3,264 KiB, the 3,037,358-byte bundle of 2 October 2026 plus about 10%.
• The port-lease and hold benchmarks have no package script; the runner calls the Node package's Vitest on their scopes (src/port-lease with the port lease acquire latency test name, and src/hold) until Node adds scripts for them.
• The observation benchmark builds the Node's dependencies itself and takes about four minutes.
• repository.install.bytes reads node_modules/.pnpm as it is. A long-lived checkout keeps packages an older lockfile installed (pnpm does not prune them on a frozen install), so compare it from fresh installs: CI always installs fresh, and so does the base side of pnpm metrics:compare.

Open decisions

These need the owner's approval (a new service, permission or dependency):

• Durable history: a metrics orphan branch that CI appends to, for dashboards past 90 days. Needs contents: write in CI.
• Pull-request comments with the comparison. Needs pull-requests: write and a workflow_run workflow for forks.
• Hosted services: CodSpeed (instruction counts in CI; a GitHub App and an npm plugin), Bencher (cloud, or a service to run), RelativeCI or bundlewatch (hosted bundle history).
• New dependencies: size-limit for bundle budgets; Vitest bench (already installed, but Vitest 5 changed its API) for micro-benchmarks.
• A dedicated benchmark runner (self-hosted, tuned like rustc-perf's) to gate wall time.
• A weekly triage owner who reads the trends, as rustc does.
• The screen engine (TM2b): one crate of the measured three, approved when TM2b adds it (D-175).