Evaluation · held-out speech retrieval
Measured on held-out speech.
Across 826 held-out queries with a known answer, Tierline placed a correct passage in its first five results 83.29% of the time (688 of 826), 10.77 points above frozen XLS-R embeddings on the same queries. A person reviewing results listens from the top: this is how often the answer is among the first five passages they hear.
55.69% Frozen XLS-R
72.52% Tierline
83.29%
- held-out queries
- 1,057
- with a reviewed reference passage
- 826
- with no correct answer
- 231
- held-out speakers
- 18
- recording sessions
- 52
- speech collections
- 3
01 · Ranking
Most of the time, the first result is the right one.
A person reviewing results listens from the top. So the question that matters is how far down the list the first correct passage sits. For 601 of 826 positive queries it was the first result.
Show the numbers as a table
| Measure | Queries | Share |
|---|---|---|
| Hit@1 | 601 of 826 | 72.76% |
| Hit@5 | 688 of 826 | 83.29% |
| Hit@10 | 735 of 826 | 88.98% |
| Hit@5, 95% speaker-cluster bootstrap interval | 79.4% – 86.7% |
02 · Baselines
Compared with two baselines
The same queries, the same splits, three ways of searching. Tierline’s Hit@5 is 10.77 points above the stronger of the two published baselines, frozen XLS-R embeddings.
What each method does
- MFCC-DTW
- Mel-frequency cepstral coefficients, a classic hand-designed description of a sound’s spectrum, compared by dynamic time warping. No learned model.
- Frozen XLS-R embeddings
- The pretrained XLS-R speech model’s representations, used without the rest of Tierline’s retrieval path.
- Tierline
- The configuration in the measurement record: compressed speech embeddings, an approximate index searched only within the partitions a person may use, and bounded reranking of the candidates. How it works describes the architecture.
Four baseline configurations were compared on the same splits; the two shown are MFCC-DTW and frozen XLS-R.
03 · Harder tests
A different speaker, or no correct answer
Another speaker said it
Finding the same speaker saying the same words again is the easy case. When the reference passage was spoken by someone else, Hit@5 was 72.30% (201 of 278).
The answer is not there
A good search says so when nothing matches. Of 231 queries with no correct passage, Tierline accepted a wrong one 8 times: a false-accept rate of 3.46%, at a threshold fixed before the test.
04 · Robustness
Where it is weaker.
Archival audio is rarely clean. The same held-out queries were run through six reproducible degradations. Compression costs least; dropped-out audio costs most, 16.58 points.
- clean audio
- under the condition
Show the numbers as a table
| Condition | Hit@5 | Difference from clean |
|---|---|---|
| Lossy compression | 78.69% | −4.60 points |
| Background noise | 74.70% | −8.59 points |
| Reverberation | 73.00% | −10.29 points |
| Time stretching | 71.91% | −11.38 points |
| Clipped audio | 69.85% | −13.44 points |
| Audio dropout | 66.71% | −16.58 points |
05 · The system behind the search
Speed, storage and repeatability
What it costs a community to run the search, and how quickly it answers.
Approximate search that keeps the right candidates
99.04%
Mean candidate-set Recall@500 against exact search
Comparing a query with every one of 3,271,840 indexed speech windows is slow. Tierline’s HNSW index proposes the 500 nearest windows approximately; exact search defines the right 500. On average the approximate set contained 99.04% of them.
The index searches only the partitions a person may use. Each candidate set is then reranked frame by frame.
Half of all answers within 0.92 s
Half of all queries returned within 0.92 s; 95% within 2.71 s; 99% within 3.89 s, on the GPU workstation described in Methods.
A quarter of the bytes, after compression
After compression, each window is described by 256 numbers. Stored as one byte each instead of four, the raw vectors take 75% less space; in a controlled comparison the complete SQ8 index used 63.4% less memory than the float32 index.
Two runs, matching outputs
Two benchmark executions from configuration records tied to the source code produced matching metric outputs.
06 · Known limits
Where retrieval is less reliable.
- Languages and collections
- The results combine three speech collections. Accuracy on a particular language or collection can differ, and is measured with a program on its own recordings.
- Overlapping speech
- Diarization and the language classifier both degrade where two people talk at once; neither is enabled by default.
- Unseen contrasts
- Sound contrasts rare in XLS-R’s pretraining may be poorly represented at layer 11.
- Short queries
- Near the 0.5-second floor a query carries few frames to align.
- Repeated material
- Acoustically similar repeated passages can crowd the results even after suppression.
- Metadata errors
- A wrong access assignment does more harm than any retrieval error: the search applies whatever permissions the records give it.
07 · The search in your browser
What runs when you search here
The search in Explore and the Workspace runs in your browser on the example recordings, with a smaller acoustic matcher (log-mel frames and subsequence DTW). The reference engine has also been run end to end on those recordings with the XLS-R weights, and reproduced; that run is published on Explore. The figures above are from the held-out evaluation.
08 · Methods and data
Scope, denominators and conditions.
Reported on 13 October 2026. What was measured, on what, and under which conditions:
- Metrics
- Hit@1, Hit@5 and Hit@10 (a correct passage within the first k results); false-accept rate on queries with no correct answer; candidate-set Recall@500 against exact search; query latency (median, 95th and 99th percentile).
- Queries
- 1,057 held-out queries: 826 with an independently reviewed reference passage, 231 with no correct answer. Hit@k is over the 826.
- Speech
- Three speech collections; 18 held-out speakers across 52 recording sessions; 3,271,840 indexed coarse speech windows.
- Configuration
- Compressed speech embeddings of 256 dimensions, stored as int8; an HNSW index (SQ8) searched only within the permission partitions a person may use, 500 candidates per query; bounded reranking of the candidates. The candidate count and the dimensions match Tierline’s architecture file (K = 500, PCA to 256).
- Baselines
- MFCC-DTW and frozen XLS-R embeddings, on the same queries and splits.
- Hardware
- Latency was measured on a GPU-equipped reference workstation. Tierline’s reference engine is built to run on a CPU.
- Runs
- Two executions from configuration records tied to the source code, with matching metric outputs.
- Reported
- 13 October 2026.
Held-out speech retrieval evaluation
- Scope
- 1,057 held-out queries across three evaluated speech collections: 826 positive queries with independently reviewed reference passages and 231 no-match queries. 18 held-out speakers across 52 recording sessions, with speaker separation where applicable.
- Conditions
- Hit@k is computed over the 826 positive queries. The no-match acceptance threshold was fixed before test evaluation. Collection-level results were evaluated alongside the combined score. Baselines were compared under identical evaluation splits.
- Source
- Tierline project record, 13 October 2026
Retrieval under degraded audio
- Scope
- Six simulated degradation conditions, applied with reproducible transformations and evaluated on the same held-out query set.
- Conditions
- Hit@5 per condition.
- Source
- Tierline project record, 13 October 2026
Approximate versus exact search
- Scope
- 3,271,840 indexed coarse speech windows across the evaluated collections; a 500-candidate configuration with permission-aware HNSW search and bounded reranking.
- Conditions
- Candidate-set Recall@500 relative to exact Flat search, under a separately defined eligible-index benchmark.
- Source
- Tierline project record, 13 October 2026
Query latency
- Scope
- End-to-end query latency of the retrieval configuration above.
- Conditions
- Measured on a declared GPU-equipped reference workstation. The browser search on this site is a separate, smaller matcher.
- Source
- Tierline project record, 13 October 2026
Storage and memory
- Scope
- 256-dimensional compressed speech embeddings.
- Conditions
- Raw vector payload: int8 against float32. Full-index memory: a controlled SQ8-versus-float32 HNSW comparison.
- Source
- Tierline project record, 13 October 2026
Reproducibility and validation
- Scope
- Benchmark executions, their configuration records and their review.
- Source
- Tierline project record, 13 October 2026
Public-data benchmark protocol
A further benchmark, separate from the evaluation on this page, is defined on public data (MLCommons Multilingual Spoken Words Corpus release 1.0.0) so that the retrieval mechanism can be checked without any community collection.[16] Its corpus, degradations and acceptance thresholds are fixed in configs/benchmark-v1.yaml, and its results will be published here when it runs.
Corpus, degradations and acceptance thresholds
Corpus
- Languages
- en, de, ca, fr, rw
- Hours per language
- 2.0
- Pseudo-recordings
- 60 × 120 s
- Distinct speakers per keyword
- ≥ 12
- Dev / test queries per keyword
- 1 / 1
- Seed
- 1729
Degradations
Low-pass at 3400 Hz; additive noise at 10 dB SNR; both combined; speed 0.90× and 1.10×; dropout up to 20 ms per span, 0.5% of duration. Seeded per clip.
Acceptance gates
| Property | Threshold |
|---|---|
| Offline execution | 0 connections attempted |
| Policy isolation | 0 unauthorized candidates |
| Export timing fidelity | ≤ 1.0 ms |
| Export annotation preservation | 100% |
| HNSW candidate recall vs. Flat | ≥ 98% |
| Clean Precision@10 | ≥ 0.60 |
| Clean Recall@50 | ≥ 0.80 |
| Recall within five minutes of listening to the results | ≥ 0.80 |
| Median listening time to reach 80% recall | ≤ 300 s |
| int8 ΔPrecision@10 | ≥ −0.03 |
| int8 ΔRecall@50 | ≥ −0.02 |
| p95 latency, 100 h index | ≤ 2.0 s |
| p95 latency, 500 h index | ≤ 5.0 s |
| Peak memory per query | ≤ 12 GB |
| Stored frame payload per hour of audio | ≤ 60 MB |