Reference benchmark
Colloq's first benchmark asks a deliberately narrow question: what does the current checked conversation runtime cost relative to a small hand-written protocol?
It is a reproducible microbenchmark, not evidence that Colloq is fast. It distinguishes compilation, plan-backed session creation, reference and compact checked transitions, cold whole-conversation execution, reference and compact warm execution, and a hand-written baseline.
Workload
All variants execute the same request/token/done exchange:
- Create a client worker and a server worker.
- Send one prompt from client to server.
- Send the configured number of ordered tokens from server to client, with a continue response after each token.
- Send a terminal done message.
For whole-exchange measurements, every variant includes thread creation, in-process channels, JSON encoding, and JSON decoding in every sample. The conventional baseline is a hand-written pair of Rust message loops. Reference Colloq additionally uses self-describing Colloq Wire envelopes and state/sequence checks. Compact Colloq performs the same checks while exchanging a transition ID, sequence, and optional payload. Cold Colloq includes plan compilation; both warm paths reuse one verified plan.
Run an optimized build on an otherwise idle machine:
cargo run --release --locked -- benchmark \
--iterations 500 --warmup 50 --tokens 3
The CLI emits one JSON report containing the build profile, target, workload parameters, nanosecond samples summarized as minimum/median/mean/p95/maximum, and median ratios. Warmups are excluded. Whole-exchange variants rotate order to reduce a simple first/last ordering bias. There are no timing thresholds in the test suite; tests assert only that every workload completes correctly.
Compact Colloq Wire result
The checked-in compact-wire report used an Apple M4, Darwin arm64, Rust 1.96.0, 500 measured iterations, 50 warmups, and three tokens:
| Measurement | Median | p95 |
|---|---|---|
| Colloq plan compilation | 51.6 µs | 66.8 µs |
| Two Colloq sessions from plan | 0.125 µs | 0.167 µs |
| Colloq reference checked transition | 2.417 µs | 2.500 µs |
| Colloq compact checked transition | 1.833 µs | 1.917 µs |
| Baseline JSON transition | 0.250 µs | 0.292 µs |
| Colloq cold whole exchange | 145.9 µs | 235.5 µs |
| Colloq reference warm exchange | 92.5 µs | 166.8 µs |
| Colloq compact warm exchange | 83.6 µs | 154.6 µs |
| Hand-written whole exchange | 55.2 µs | 112.5 µs |
Compact encoding made the isolated checked Colloq transition 1.32× faster than the reference envelope and the complete warm exchange 1.11× faster. Median overhead versus the hand-written whole exchange fell from 1.68× to 1.51×, still missing the provisional 1.25× target. The compact checked transition remained 7.33× the smaller baseline JSON transition, while creating two sessions took only 125 ns.
The earlier Plan v0 and pre-plan results remain checked in for history. OS scheduling makes isolated runs noisy and the benchmark implementation changed between them, so only reference and compact samples within the current report are treated as a controlled comparison.
Interpretation and next measurements
The checked-transition measurement creates fresh machines and prepares the compact codec before the timer, then times one client emit, envelope JSON encoding and decoding, and one server accept. It deliberately combines protocol checking and representation cost; it does not yet isolate them. The baseline transition constructs, encodes, decodes, and checks its smaller prompt message.
The benchmark still does not isolate channel transfer, transport latency, throughput, allocation, lookup, or model execution. Compact wire is still JSON and reconstructs owned semantic strings before checking, so this is not a lower bound for a binary or zero-copy implementation.
The memory benchmark does not include the network session preface. A TCP or QUIC session first exchanges one 281-byte JSON preface plus a four-byte length prefix and one acceptance byte in each direction. The acceptance phase waits for both peers to validate, adding one application-level round trip before frame zero. Persistent connection reuse and resumption are not implemented.
Useful next experiments are:
- measure the plan-bound preface over persistent and newly established TCP/QUIC connections;
- add mutual client authentication and replay-resistant session freshness;
- split state-machine checking, transition lookup, allocation, serialization, and channel transfer;
- compare the compact JSON experiment with a schema-driven binary payload codec;
- compare TCP and QUIC against conventional equivalents under controlled network faults;
- move to a representative AI workload before making any performance claim.
The benchmark succeeds as research even when Colloq loses: it gives optimization work a fixed workload and makes regressions visible.