The rules publish before
the numbers.
Telys is 0.1.0b4 — these are prototype-validated figures on a disclosed rig, not production numbers. A result only counts when it is measured under matched recall on matched hardware, with raw samples and the script that produced it. The rules below still bind every number: what we measure, against whom, and the ways we are required to lose.
Rules the harness enforces, not caveats we append.
Each rule below is encoded as a machine check in the benchmark harness. A run that violates one is not footnoted — it is marked invalid and excluded from every scorecard.
“Every multiplier is a hypothesis until measured under matched recall and matched hardware, with raw samples and a reproduction script published.”
Telys benchmark doctrineA benchmark we cannot lose is worthless
The generated scorecard carries a mandatory Where-we-lose section, auto-populated from every cell a comparator wins. If that section is empty, the report build fails — a clean sweep is treated as a broken harness, not a triumph.
Embedded vs embedded is the fair fight
We never headline an in-process call against a competitor answering over TCP. Against embedded libraries, in-process vs in-process is the honest transport. Against servers, transport is matched like for like, or the number is annotated and kept out of headlines.
Matched tiers, or no comparison
An in-memory run is never scored against a competitor that fsyncs every commit. Every result declares its durability tier — in_mem, wal_async, or wal_fsync — and the harness hard-fails any comparison across tiers.
We never imply we beat FAISS when we are calling FAISS
FAISS is the ANN core Telys embeds — a foundation, not an adversary. Every vector scorecard shows the raw-FAISS traversal cell first; where it is a tie, the tie is printed. Honest wins are claimed elsewhere: rerank kernels, filtered planning, and the engine around the index.
Each comparator isolates one claim.
A single leaderboard number confounds algorithm, transport, durability, and storage format. The set is chosen so each comparison isolates exactly one variable — with a same-node control under every headline.
| System | Shape | What the comparison isolates |
|---|---|---|
| FAISS (same node) | Control baseline | The floor, not an adversary. Telys embeds FAISS, so the raw-FAISS number is the baseline we are; the published delta is what the engine adds around it — at the same index string and the same measured recall. |
| Postgres + pgvector | Relational + vector | The filtered-hybrid query: whether pushing a scalar predicate into the candidate path beats an index that must over-fetch or post-filter, at equal recall across filter selectivities. |
| LanceDB / Chroma | Embedded library | The closest competitors, in-process vs in-process — the one place an in-process number is a fair headline. Same ANN floor; the comparison isolates the engine around it: unified row-id space, cold-open, footprint. |
| Qdrant / Milvus | Server | Algorithm quality — recall at fixed k — separated from network round-trips. Latency is reported with the server hop annotated and is never headlined against an in-process call. |
| Pinecone | Managed service | Recall at fixed k and throughput per unit of compute. Raw latency against an embedded engine is refused as a headline; the same-node control carries the quality claim instead. |
| DuckDB / ClickHouse | Analytical engine | Scan and aggregate over the same physical Parquet file, so storage-format advantage is separated from engine advantage. Vectorized analytical engines are excellent here; their wins publish. |
What gets measured.
The matrix defines measurements, not hypothesized wins. Every latency cell reports the full distribution — never just the mean — under open-loop offered load. Every speed cell is bound to a recall gate computed by the harness against shipped ground truth; a cell whose gate is unmet is emitted invalid and cannot enter a scorecard.
| Workload | Setup | What it measures |
|---|---|---|
| cold-open | Page cache dropped, fresh process start | Time from process start to first successful query — segment mmap and index-sidecar mapping against server boot and recovery. Measured separately from steady state, never folded into it. |
| exact FLAT | Brute-force exact search, f32 and int8 | The distance-kernel floor at recall 1.0 by construction, against the same-node exact baseline on the same buffers. Every int8 cell reports its recall delta versus f32, so quantization loss is never hidden. |
| filtered hybrid | Scalar predicate + top-k, swept across three selectivities | Latency at equal recall as the filter narrows, plus the planner's chosen strategy per cell — pre-filter then exact, pre-filter then IVF, or search then post-filter — so the adaptive decision is auditable. |
| full scan | Full-column scan and aggregate | Raw columnar throughput over shared Parquet, engine versus engine on identical bytes. |
| selective scan | Predicate scan at low match rates | Whether footer min-max and zone-map pruning actually skip work — reported as bytes read and rows touched, not wall clock alone. |
python -m bench.orchestrator.run \
--suite vector --rig m-laptop
# one command: pinned comparators, content-addressed
# datasets, warmup, steady-state measurement
# reviewer mode: the same run with Telys excluded# every run emits a machine-checked ResultEnvelope; # a run that cannot fill every field is invalid index_string = "IVF4096,PQ16" # same for Telys and the FAISS control omp_num_threads = 8 # FAISS threading pinned, disclosed transport = "in_process" # headlined only vs in-process durability_tier = "wal_fsync" # cross-tier comparison hard-fails recall_at_k = 1.000 # harness-measured, never self-reported samples_path = "bench/results/filtered_impact.txt"
Runs repeat at least five times across fresh process starts, on pinned hardware profiles, one system at a time. Datasets are content-addressed; query sets are frozen files, not generated at run time. The harness specification is committed today; the harness publishes with the results.
Measured, conditioned, reproducible.
Each row is a real run on the rig named above, at the recall gate printed beside it. The win surface is physical layout: on an identical contiguous subset our Mojo scan ties raw FAISS — the delta below is what the key-partitioned layout buys, not a faster distance kernel. These are prototype figures, re-run under the full fairness contract as the harness matures.
| Workload | Setup + recall gate | Measured result (M4 Max, single-thread, in-process) |
|---|---|---|
| Filtered slice · 0.1% sel | N=1,000,000 D=128 K=10, warm, recall 1.000 (exact) | partition-slice p50 0.013 ms — layout win vs full-scan ~373×, vs scatter-gather ~19×. Same-node FAISS Flat on the identical subset: 0.040 ms (tie — proves the win is layout, not kernel). |
| Filtered slice · 0.4% sel | N=1,000,000 D=128 K=10, warm, recall 1.000 (exact) | p50 0.032 ms — ~128× vs full-scan, ~10.8× vs scatter-gather. FAISS Flat control 0.035 ms (tie). |
| Filtered slice · 1.6% sel | N=1,000,000 D=128 K=10, warm, recall 1.000 (exact) | p50 0.112 ms — ~42× vs full-scan, ~10.3× vs scatter-gather. FAISS Flat control 0.120 ms (tie). |
| Giant-partition rescue | 247k rows in N=400,000 D=128, nprobe calibrated, recall 0.9915 (floor 0.98) | IVF-within-partition restores the layout win where an exact slice degrades: exact slice 1.71 ms → IVF 0.033 ms. Without IVF, a single giant partition holds only ~4–5×. |
| Governed auto-nprobe | N=1,000,000 D=128 Q=500 nlist=1000, recall held 0.998 (floor 0.98) | FIND 0.28–0.31 ms at baseline nprobe=32 → 0.081–0.088 ms at governed auto-nprobe=8. Recall stays above the floor — no silent recall-for-speed trade. |
| FIND vs FAISS control | Iso-recall, same transport, both in-process, N=1,000,000 D=128 | Telys 0.073–0.074 ms @0.998 vs raw FAISS 0.080 ms @0.999 — parity to modestly ahead. Scale-dependent: behind (0.74×) at N=200k, up to ~1.6× only at large memory-bound N. FAISS quality, Mojo-accelerated. |
| GIL-free concurrency | N=200,000 D=128, 12 perf cores, in-process Mojo IVF kernel | FIND throughput scales ×6.9 over 8 threads (15,128 → 104,756 qps). Throughput scaling, not per-query latency; not compared against server engines. |
| Insert-to-recall freshness | Base N=100,000, M=300 inserts, D=128, in-process | A just-inserted vector is top-1 on the very next query: recall@1-of-new = 1.000. Insert p50 0.3 µs. Correctness gate, not a throughput headline. |
Every cell above traces to a named script and a saved raw output in bench/results/ (filtered_impact.txt, partitioned.txt, partition_acceptance.txt, opt.txt, telys.txt, concurrency.txt, freshness.txt). The persistence reopen figure is held back pending a re-measured raw sample. Every multiplier is a hypothesis outside the conditions printed in each row.
Built for code — and scored on a code benchmark.
Telys retrieves code, so the quality bar is CoIR, not a general-text suite. LexicalCodeIndex is wrapped as an mteb SearchProtocol and scored by mteb.evaluate on the identical path as the bm25s baselines — no methodology gap, no self-scoring. Baselines are reproducible runs from 2026-07-21, not the published CoIR-paper figures (several of those do not reproduce). Metric: nDCG@10.
| CoIR task | multigram | bm25-ref | bm25-code | code-lexical · shipped | vs best baseline |
|---|---|---|---|---|---|
| CodeTransOceanContest | 0.160 | 0.478 | 0.600 | 0.718 | ✅ |
| CodeTransOceanDL | 0.340 | 0.344 | 0.351 | 0.366 | ✅ |
| CosQA | 0.048 | 0.188 | 0.213 | 0.218 | ✅ |
| SyntheticText2SQL | 0.265 | 0.249 | 0.372 | 0.441 | ✅ |
| CodeFeedbackMT | 0.288 | 0.592 | 0.591 | 0.674 | ✅ |
| CodeFeedbackST | 0.162 | 0.682 | 0.682 | 0.722 | ✅ |
| StackOverflowQA | 0.223 | 0.703 | 0.687 | 0.733 | ✅ |
| AppsRetrieval | 0.001 | 0.048 | 0.014 | 0.034 | ❌ |
| Average | 0.186 | 0.445 † | 0.488 | 7/8 | |
Seven of eight wins; average 0.488 vs 0.445 for the stronger baseline of each task († best of bm25-ref / bm25-code, per task) — and 2.6× the multigram average. The sole loss, AppsRetrieval, sits near zero for every lexical method: long problem statements against code is embedding territory.
Code-aware tokens + BM25
Identifiers split the way code is written — camelCase, snake_case, dotted paths — feeding plain IDF/BM25. This alone wins most tasks: CodeTransOceanContest moves from 0.478 to 0.600 before any tuning.
Stopwords out, lite stemming
English stopwords removed and a lite suffix-stemmer so natural-language asks line up with code tokens — the lever that flips SyntheticText2SQL from a loss to 0.441.
k1 = 1.8 · b = 1.0
The shipped defaults, Algenta-tuned on CoIR: heavier term-frequency saturation and full length normalization — the lever that flips StackOverflowQA.
Every cell reproduces from one script — bench/mteb_code_lexical.py — running both sides through the same mteb.evaluate call (runs of 2026-07-21). code-lexical is the index Telys ships, at its shipped defaults.
Wins, ties, and losses — in print.
The publication format is fixed before the first number exists, so the numbers cannot bend it.
Where we lose, in print
Every scorecard carries a Where-we-lose section and a Where-we-only-match-FAISS section, both auto-populated from the cells. The report fails to build if either is empty.
Reproduce the field without us
The harness runs with Telys excluded, so a skeptic can reproduce the comparator and control numbers alone and confirm each competitor was tuned per its own guide — configs committed and diffed against vendor defaults.
Every cell traces to samples
Each published cell links its raw per-operation samples and the exact script that produced them. Results are append-only, keyed to the engine revision and the bundled dependency versions it statically links.
These are prototype numbers, each carried with its conditions. Hold us to this contract — and to the figures as we re-run them under the full harness.