Repository navigation
[ENH] Hierarchical spann - #7890
Open
lyon-chroma wants to merge 49 commits into
Open
lyon-chroma wants to merge 49 commits into
lyon-chroma wants to merge 49 commits into
Conversation
Cherry-pick hierarchical SPANN index prototype from benchmarking branch: - hierarchical_centroid_profile benchmark driver - hierarchical_index module (k-means tree with 1-bit RaBitQ, per-level reranking, dynamic beam) - datasets module for loading HuggingFace datasets - Cargo.toml dev-dependencies for the benchmark Made-with: Cursor
This reverts commit 0ddf0e4.
## Why A reopened index keeps each leaf's existing posting list on disk. Adding one vector previously loaded the whole list before appending a row. A checkpoint that touched many leaves therefore filled writer memory with posting data and pinned blockfile pages during the add phase. ## What changes The writer keeps new posting rows in memory while each leaf's persisted rows remain on disk. The leaf tracks its total row count so balancing can detect a split or merge without loading an ordinary add target. An operation that must inspect existing rows loads and combines the two parts under the leaf lock. Commit reads and writes one dirty leaf's complete posting list at a time; a clean leaf inherits its persisted list. The index stores one authoritative version for each vector ID in scalar metadata. A posting row carries the version it had when written, which can be stale after reassignment. The writer now keeps only new or changed versions in its mutable map. It reads unchanged versions from scalar metadata in batches of at most 64 IDs and keeps at most 65,536 results in a bounded cache. The mutable map takes precedence if an update races with a read. The scalar reader releases its pinned blocks after each batch, so those blocks cannot accumulate across a checkpoint. Each new checkpoint stores its largest vector ID. On reopen, IDs above that value are known to be new and need no metadata read. Older checkpoints have no summary and use exact point reads; a writer opened from one does not infer an unsafe maximum from only the IDs it changes. Missing persisted postings and mismatched posting lengths fail explicitly. An optional benchmark validation checks that all expected valid IDs are reachable from the root before commit and after reopen. The validation materializes every posting list and is kept out of timing runs. ## Verification `CARGO_TARGET_DIR=/Users/chroma/workspace/chroma/target cargo test -p chroma-index --test hierarchical_posting_delta` passes eight tests. They cover complete postings after commit and reopen, existing-ID updates and tombstones, stale postings in another leaf, concurrent posting loads with additions and version updates, splitting after reopen, sparse IDs, old checkpoints without the maximum-ID summary, and cache eviction. The benchmark target compiles with `cargo check`; focused rustfmt and `git diff --check` pass. Earlier separate validation runs on Wikipedia and MS MARCO checked all 150,000 and 300,000 indexed IDs before commit and after reopen. Those runs precede the bounded version-cache change; the eight focused tests above verify the current code. ## Performance evidence and limits The table records earlier two-checkpoint runs on real 1,024-dimensional Wikipedia and MS MARCO vectors, with 150,000 vectors per checkpoint. Checkpoint 2 adds to a reopened index. The earlier PR code still used a dense version array or retained read-only versions in the mutable map. These results show the posting-list deferral's behavior at this scale, but they do not measure the current version lookup design or establish performance parity. Both arms used serial balancing and full-precision writer navigation and neighbor reassignment. | Dataset, checkpoint 2 | Add | Balance | Total checkpoint | Posting loads / entries | Peak process RSS | Cold R@100 | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | Wikipedia baseline | 1.22 s | 4.71 s | 8.03 s | 118 / 170.6k | 3.53 GiB | 73.9% | | Wikipedia earlier change | 1.17 s | 3.94 s | 7.25 s | 114 / 150.3k | 3.68 GiB | 79.4% | | MS MARCO baseline | 1.16 s | 5.93 s | 9.63 s | 149 / 208.1k | 4.17 GiB | 80.0% | | MS MARCO earlier change | 1.16 s | 6.08 s | 10.68 s | 108 / 147.2k | 3.89 GiB | 81.9% | The earlier change loaded fewer posting entries in both datasets but did not show a consistent total-time or memory improvement. The current bounded cache trades long-lived version memory for scalar metadata reads and possible cache churn. We have not measured that tradeoff at a representative 50M-vector scale. ## Existing deletion behavior Deletion marks the authoritative version as tombstoned. A leaf within its size bounds can retain a stale posting until a later scrub. The read-only scoring path does not consult authoritative versions, so deleted IDs in untouched leaves remain a separate correctness concern.
## Summary A leaf split may reassign vectors using full precision embeddings or quantized codes. The full precision path does not read the table that maps vector IDs to their old quantized-code slots, yet each split builds that table. This change constructs the table only for quantized reassignment, when recursive reassignment is enabled. Quantized reassignment continues to use the same table and code positions. The new leaf codes remain available to persistence and quantized readers. ## Verification - `cargo check -p chroma-index --bench hierarchical_spann_profile_quantized` passed. - `rustfmt --check rust/index/benches/hierarchical_index/writer/writer.rs` passed. - `git diff --check` passed. This change removes one allocation and hash-table construction from each full precision split. Performance benchmarks were not requested for this change.
## Summary The benchmark checks how many ground-truth vectors remain reachable after each search level. It now resolves authoritative versions only for those ground-truth vectors. It still walks the selected subtrees and checks every candidate posting ID for membership in the ground-truth set, but it skips checkpoint metadata lookups for postings that cannot affect the reported recall. After the writer reopens a checkpoint, resolving one unchanged vector version can read scalar metadata from the blockfile. The previous diagnostic resolved a version for every posting beneath the selected nodes on every query, even though the calculation only uses 100 ground-truth IDs. That repeated metadata work dominates the benchmark's recall phase. This change limits version resolution to relevant postings while preserving the authoritative check that rejects stale posting rows. The change is confined to the benchmark writer diagnostic in `rust/index/benches/hierarchical_index/writer/diagnostics.rs`. Reader search, index construction, and the reported recall definition are unchanged. ## Measurement Both arms used the Wikipedia 1,024-dimensional dataset, two 150,000-vector checkpoints, 10 brute-force ground-truth queries, one write thread, full-precision NPA, read tau 2.0, and rerank factor 1. Each arm rebuilt the tree independently, so exact search-recall percentages and leaf counts vary slightly. The control binary includes the earlier ground-truth version prefetch experiment; that prefetch alone does not solve the slow subtree scan. The change arm uses this branch without that prefetch experiment. | Arm | Recall phase duration | Search recall, cold pass | | --- | ---: | ---: | | Control with ground-truth prefetch | 38.65 s | 78.70% | | This branch | 64 ms | 79.70% | This measures the benchmark's recall phase, which includes diagnostics, and does not imply that production reader search or index building became faster. The benchmark remains linear in the posting rows it scans; this change removes the unrelated version lookups from that scan. A larger index would still benefit from a direct way to find the leaves containing ground-truth IDs. ## Checks - `cargo build --release -p chroma-index --bench hierarchical_spann_profile_quantized` passes. - `rustfmt --edition 2021 --check rust/index/benches/hierarchical_index/writer/diagnostics.rs` passes. - `git diff --check` passes. - The matched real-dataset benchmark completes and reports reachable ground-truth vectors at each level.
## Why Writer navigation chooses a beam width partly from the number of tree nodes at each level. Without a cached width, navigation walks the whole tree to count those nodes for each percentage beam floor. A 300,000-vector build performs hundreds of thousands of navigations, so the repeated walk touches node maps and centroids that the distance calculation does not need. ## What changes During deferred adds, the tree shape is stable, so the writer reuses one level-width snapshot throughout the add phase. Each balancing round replaces that snapshot with widths from its starting tree before its workers run. Balancing clears the snapshot when it finishes, so the next add phase computes widths again. One balancing worker follows the same loop as multiple workers. Tau-only schedules skip width counting because their beam has no percentage floor. A round snapshot can use a percentage floor from the start of the round after splits or merges change that level's width. A large rebalance can make the snapshot stale enough to set a beam floor that is too narrow and reduce recall for vectors placed in that round. The beam remains within the configured minimum and maximum. The benchmark measures recall as well as build time for this reason. Parallel balancing also needs each replacement subtree and posting to remain reachable. A split now attaches its children to the current root-reachable parent, even if the saved parent moved during another split; an internal orphan wraps the entire current root when it needs a new level. Removal likewise finds the current parent before deleting a child reference. Reassignment publishes a new vector version only while its first replacement posting is held in a destination leaf, so a concurrent scrub or split cannot discard both the old and new posting. These correctness repairs apply to all builds. The benchmark can count valid IDs and missing child references from the current root before commit and after reopen. It can set add and balance worker counts separately, samples brute-force ground truth with a fixed seed, and permits Wikipedia to run without a precomputed ground-truth file. Full-precision writer navigation emits no per-call warning, which keeps logging out of the measured hot path. ## Benchmark method These are one-checkpoint smoke benchmarks on real 1,024-dimensional Wikipedia EN and MS MARCO vectors, using 300,000 vectors per run on the same arm64 macOS host and the same optimized binary. The earlier cache-off/on comparison used identical settings and a separate fresh index directory for each run. The write path uses full-precision navigation and `--fp-npa`; the tree uses branching factor 100, split threshold 2048, merge threshold 512, and one replica. Recall is R@100 on ten deterministic sampled data queries against brute-force neighbors within the 300,000-vector slice. The reported total is the checkpoint time, including add, balance, load, commit, and reopen. The results below were measured before this snapshot-state simplification, with an earlier binary that provided a temporary cache-off comparison mode. The current binary always uses cached widths and the parallel balancing loop, even with one worker. The off rows are historical baselines rather than commands to reproduce with this PR; these timings do not measure the latest commit. | Dataset | Add / balance workers | Policy cache | Add | Balance | Checkpoint total | Warm R@100 | Valid IDs before / after reopen | Missing child references | | --- | ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | | MS MARCO | 64 / 4 | Off | 1.41s | 13.88s | 18.14s | 81.3% | 300,000 / 300,000 | 0 | | MS MARCO | 64 / 4 | On | 0.931s | 12.08s | 15.88s | 80.5% | 300,000 / 300,000 | 0 | | Wikipedia EN | 32 / 4 | Off | 1.40s | 9.68s | 13.85s | 74.0% | 300,000 / 300,000 | 0 | | Wikipedia EN | 32 / 4 | On | 0.913s | 8.59s | 12.26s | 75.9% | 300,000 / 300,000 | 0 | | MS MARCO | 64 / 64 | Off | 1.37s | 13.90s | 18.09s | 80.6% | 300,000 / 300,000 | 0 | | MS MARCO | 64 / 64 | On | 0.991s | 10.58s | 14.45s | 81.0% | 300,000 / 300,000 | 0 | The measured cache-on runs reduce add time by 28–35%. The single-run checkpoint totals favor the cache by 11–20%, but parallel scheduling changes the tree and balance time between runs. One final-code trial per setting does not estimate variance, so these results do not establish a stable whole-build speedup or a recall difference. A 300,000-vector tree has roughly 220 leaves; it also does not establish the effect at the 40M/113M scales of the recorded builds. ## Verification `CARGO_TARGET_DIR=/Users/chroma/workspace/chroma/target cargo test --release -p chroma-index --test hierarchical_policy_cache` passes all six focused tests for snapshot reuse and phase clearing, moved parents, orphaned internal subtrees, stale root identity, child removal, and version publication. `CARGO_TARGET_DIR=/Users/chroma/workspace/chroma/target cargo check -p chroma-index --bench hierarchical_spann_profile_quantized`, focused rustfmt, and `git diff --check` pass after removing the serial balancing switch. Every result in the table has all 300,000 valid IDs reachable from the current root and zero missing child references both before commit and after reopen.
## Summary A leaf split checks nearby leaves for vectors that belong closer to its new centroids. This change scans each neighboring leaf in place, queues the vector IDs selected for reassignment, and releases the leaf's map guard before moving those vectors. The quantized scan borrows IDs, versions, centroid, and packed codes. The full precision scan loads missing embeddings outside the leaf guard, then borrows IDs, versions, centroid, and cached embedding handles during distance checks. Both paths keep the current version test. Because candidate moves occur after the scan, each candidate also carries its posting version and is checked again immediately before reassignment. Candidates are processed in posting order; an earlier move that changes a later candidate's version causes the later move to be skipped. The change is confined to the neighboring-leaf reassignment path (`rust/index/benches/hierarchical_index/writer/writer.rs`). ## Copy volume At 1024 dimensions, the base quantized path copies 266 KiB of IDs, versions, and packed codes plus a 4 KiB centroid for a 2,048-entry leaf. The PR path queues only selected `(ID, version)` pairs, about 8 bytes per candidate. The base full precision path copies IDs, versions, a centroid, and `Arc` handles for cached embeddings; it does not copy the embedding payloads. The PR path allocates only IDs whose embeddings need loading and selected candidate pairs. In the paired Wikipedia runs, about 2.8–3.9% of evaluated neighbor vectors reached reassignment. ## Verification - `cargo check -p chroma-index --bench hierarchical_spann_profile_quantized`, an optimized benchmark build, `rustfmt --check`, `git diff --check`, and commit hooks pass. - The benchmark uses an exact `hierarchical-spann` base binary and a PR-head binary, each built in release mode. Both use the same 300,000 Wikipedia vectors at 1024 dimensions, two 150,000-vector checkpoints, eight add workers, eight balance workers, cached ground truth for 10 seeded queries, and one recall row at tau 1.5 with 1x reranking. | NPA distance mode | Base build | PR build | Base recall@100 | PR recall@100 | | --- | ---: | ---: | ---: | ---: | | Full precision | 20.94 s | 22.47 s | 80.80% | 80.10% | | Quantized | 19.99 s | 19.46 s | 79.70% | 79.20% | All four runs report zero orphaned vectors. This is a reachability check, not full posting validation. One run per mode cannot establish a stable speed or recall difference: parallel balancing produces different tree shapes, and the full precision run is slower at the PR head while the quantized run is slightly faster. The neighbor scan's aggregate timing also includes reassignment and nested balancing, so it does not isolate allocation time. The borrowed scan also holds a leaf read guard while it checks versions and distances. `scrub` already reads versions under that leaf guard, and the 65,536-entry version cache is normally warm immediately after scrub, but a cache miss during the scan can extend the guard across scalar blockfile I/O and delay parallel writes to that leaf. Further repeated runs and strict posting validation are needed before treating this as a measured performance gain.
tl;dr: when navigating the tree - read in place, don't copy then read - organize data (node children) so that it can be read contiguously, not as point reads ## What changes Adding a vector repeatedly searches a tree of centroids to choose a posting list. This branch keeps each internal node's child centroids together in one packed buffer, so an add worker reads one live parent entry to score its children. The child IDs remain in the same order as the packed centroids. The separate map and reference arrays built at the start of each add batch are gone. An opened writer packs the tree once before its first add. Structural changes mark affected parents. While parallel balancing splits and merges nodes, navigation uses live child lookups. After the balance workers join, the writer refreshes only marked parents, including a new root or a parent whose child centroid changed. Posting updates do not refresh navigation because they do not change child centroids. The packed buffer uses a shared allocation so a structural copy of the navigation object does not copy the centroid data. The packed buffer is a derived copy of child centroids, so it consumes extra memory and must be refreshed after structural changes. This branch keeps that copy inside each live parent instead of maintaining a second tree-wide navigation index. Balance refresh time is included in balance time; the benchmark's add-preparation line measures one-time initialization and reports packed centroid payload bytes. The measurements below show slower add work under the tested settings. ## Verification The nine focused `hierarchical_policy_cache` release tests pass. They cover navigation equivalence, a missing child, level-specific beam settings, reuse of the same packed buffer across add batches, a child-centroid change after balancing, and split/root repair cases. The benchmark target builds with `cargo bench -p chroma-index --bench hierarchical_spann_profile_quantized --no-run`. ## Benchmark at 1M vectors The comparison uses two repeats per revision and dataset on the same `r6id.16xlarge` instance. Each run indexes 1M real 1024-dimensional vectors in ten 100K-vector checkpoints. All use L2, full-precision writer navigation and neighbor reassignment, parallel balancing, 100 precomputed real-corpus queries, a 1 GiB block cache, and final-checkpoint recall at tau 1.5 with one rerank vector. Wikipedia uses 32 add and balance threads, write beam tau 0.5, level taus `_,_,0.30`, and RNG factor 0. MS MARCO uses 64 add and balance threads, write beam tau 1.5, and RNG factor 4. No run sets `--write-level-min-pcts` or `--verify-valid-postings`. `Add task` sums timed `add()` work across threads; it is not a wall-clock component. `Balance` sums checkpoint balance durations. `Build` sums all checkpoint durations, including loading, adding, balancing, committing, and reopening. Peak RSS comes from `/usr/bin/time -v`. Recall is warm L2 recall at 100. | Dataset | Revision | Add task mean | Balance mean | Build mean | Peak RSS mean | Recall mean | | --- | --- | ---: | ---: | ---: | ---: | ---: | | Wikipedia | Base | 129.11s | 231.32s | 260.50s | 7.85 GiB | 77.28% | | Wikipedia | #7829 | 99.22s | 240.13s | 268.14s | 7.85 GiB | 77.23% | | Wikipedia | #7836 | 106.41s | 213.88s | 242.00s | 7.96 GiB | 77.38% | | Wikipedia | #7838 | 144.02s | 204.24s | 233.07s | 7.99 GiB | 77.20% | | MS MARCO | Base | 240.18s | 310.01s | 404.18s | 7.48 GiB | 80.81% | | MS MARCO | #7829 | 184.49s | 313.07s | 403.56s | 7.47 GiB | 81.08% | | MS MARCO | #7836 | 191.74s | 308.42s | 399.68s | 7.37 GiB | 80.13% | | MS MARCO | #7838 | 252.52s | 300.66s | 398.51s | 7.66 GiB | 80.63% | PR #7838's Wikipedia runs took 256.93s and 209.20s to build, with 76.67% and 77.72% warm recall. Its MS MARCO runs took 404.23s and 392.79s, with 80.32% and 80.94% recall. All four commands exited successfully. The pinned revision is `f3a236b96e7b8364ff438f80c48180641711bf03`; exact per-run commands, logs, timing output, and the executable hash are on the benchmark instance under `/home/ubuntu/benchmark-data/navigation-1m-100k-pr7838/`. The saved control runs are under `/home/ubuntu/benchmark-data/navigation-1m-100k/`. PR #7838 increases mean add work by 11.5% on Wikipedia and 5.1% on MS MARCO compared with the base. Compared with #7836, it increases add work by 35.3% and 31.7%. Its lower mean build times come from lower measured balance time in these runs. Wikipedia build time differs by 47.73s between #7838 repeats, almost entirely tracking a 48.21s balance difference; that variation exceeds the mean build-time gap between #7838 and #7836. Two repeats do not establish an overall build-time winner or a recall change. The runs omit `--verify-valid-postings`, so the benchmark's `0 orphaned` summary does not prove that every indexed vector has a posting reachable from the root. ## Alternatives for review This is a standalone alternative to [PR #7829, which copies a compact navigation snapshot per add batch](#7829), and [PR #7836, which separates immutable node navigation from postings but builds a per-batch reference index](#7836). All three target `hierarchical-spann`; they are competing approaches to the same navigation improvement and are not a merge stack. The writer's navigation and balance logic is in `rust/index/benches/hierarchical_index/writer/writer.rs`. The shared node payload is in `rust/index/benches/hierarchical_index/common.rs`.
tl;dr: when navigating the tree - read in place without locks, don't lock, copy, then read - organize data (node children) so that it can be read contiguously, not as point reads ## What changes An add batch routes vectors through a stable tree, then writes posting rows to the selected leaves. With many workers, writing each row immediately makes workers wait for leaf locks while they are still routing. This change lets each worker collect posting rows in its own per-leaf buffers and joins all workers before it writes the rows. The flush groups buffers by destination and takes one write lock per leaf. Balancing starts only after the flush finishes. The batch method owns the full phase and requires exclusive access to the writer. Each buffer stores an ID, its assigned version, and its quantization code; the writer's embedding map retains the full vector. This also replaces a full embedding copy with an `Arc` clone when the input is already shared, so the measured gain includes that allocation change. The immediate `add()` method remains available for callers outside this benchmark path. The flush appends to in-memory rows even when older posting rows remain on disk. Version checks reject deleted IDs, repeated IDs get new versions, and a vector can appear in more than one leaf. A missing destination fails the batch instead of silently dropping a posting. The exclusive batch method keeps the tree stable from routing through flush. ## Validation The writer and posting persistence integration targets pass: 11 and 18 tests respectively. They cover parallel worker buffers, repeated IDs across workers with different embeddings, deleted IDs, replication to two leaves, aligned IDs/versions/codes, lazy persisted posting merge, commit, and reopen. The benchmark target passes `cargo check`. All four 1M-vector benchmark commands exited with status 0. ## Benchmark The branch starts at the pinned `ff2db33` control commit and includes the packed live-parent navigation change from #7838. It is an independent alternative targeting `hierarchical-spann`; the PR diff contains that navigation change plus local buffers. The target branch has a separate neighbor reassignment change after the pinned control commit, so merged performance needs reevaluation. Related alternatives are #7829 (navigation snapshot per batch), #7836 (immutable reference index), and #7838 (packed centroids on live parents). Each run indexed 1M 1024-dimensional vectors in ten 100K checkpoints, with two runs per dataset. Both used L2, full-precision writer navigation and neighbor reassignment, 100 precomputed queries, and final warm recall at tau 1.5 with one rerank vector. Wikipedia used 32 add and balance threads, write beam tau 0.5, level taus `_,_,0.30`, and RNG factor 0. MS MARCO used 64 threads, write beam tau 1.5, and RNG factor 4. All runs used a 1 GiB block cache. No run used `--write-level-min-pcts` or `--verify-valid-postings`. Add wall time includes packed navigation setup, worker collection, grouping, and flush. Build time sums checkpoint add, balance, load, commit, and reopen components. The two late MS MARCO run 2 balance durations are printed to one decimal minute, so its build estimate has about six seconds of rounding uncertainty. Worker task totals overlap across threads and are not CPU time. Peak RSS comes from `/usr/bin/time -v`; recall is warm L2 recall at 100. ### All revisions: two-run means The table combines the saved controls with this PR. Add task sums time measured inside worker add calls across threads; for this PR it measures delta collection and excludes the later flush. Add wall measures elapsed indexing across all ten checkpoints, including collection and flush, and is the comparable end-to-end add measure. Balance and build means use the printed add, balance, load, commit, and reopen components for every revision; long durations rounded to tenths of a minute make the MS MARCO build mean for this PR approximate (about three seconds of uncertainty). Peak RSS is the mean of the two per-run peaks, and recall is mean warm L2 recall@100. | Dataset | Revision | Add task mean (s) | Add wall mean (s) | Balance mean (s) | Build mean (s) | Peak RSS mean (GiB) | Recall mean (%) | | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | | Wikipedia | Base | 129.11 | 4.478 | 231.32 | 258.78 | 7.85 | 77.28 | | Wikipedia | #7829 | 99.22 | 3.490 | 240.13 | 266.64 | 7.85 | 77.23 | | Wikipedia | #7836 | 106.41 | 3.647 | 213.88 | 240.23 | 7.96 | 77.38 | | Wikipedia | #7838 | 144.02 | 5.938 | 204.24 | 233.08 | 7.99 | 77.20 | | Wikipedia | This PR (#7841) | 84.09 | 3.179 | 190.81 | 216.79 | 7.35 | 77.26 | | MS MARCO | Base | 240.18 | 4.128 | 310.01 | 402.94 | 7.48 | 80.81 | | MS MARCO | #7829 | 184.49 | 3.346 | 313.07 | 406.25 | 7.47 | 81.08 | | MS MARCO | #7836 | 191.74 | 3.483 | 308.42 | 401.99 | 7.37 | 80.13 | | MS MARCO | #7838 | 252.51 | 5.990 | 300.65 | 396.77 | 7.66 | 80.63 | | MS MARCO | This PR (#7841) | 146.98 | 2.688 | 417.69 | ≈510.18 | 6.91 | 80.72 | The per-run comparison between #7838 and this PR follows. | Dataset | Revision | Run | Add wall (s) | Collect (s) | Group + flush (s) | Balance (s) | Build (s) | Peak RSS (GiB) | Recall (%) | | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | Wikipedia | #7838 | 1 | 5.720 | — | — | 228.34 | 256.95 | 7.94 | 76.67 | | Wikipedia | #7838 | 2 | 6.155 | — | — | 180.13 | 209.21 | 8.04 | 77.72 | | Wikipedia | This PR | 1 | 3.142 | 3.042 | 0.094 | 191.67 | 217.64 | 7.40 | 77.16 | | Wikipedia | This PR | 2 | 3.215 | 3.116 | 0.096 | 189.95 | 215.94 | 7.29 | 77.35 | | MS MARCO | #7838 | 1 | 6.313 | — | — | 303.54 | 400.10 | 7.58 | 80.32 | | MS MARCO | #7838 | 2 | 5.666 | — | — | 297.77 | 393.44 | 7.73 | 80.94 | | MS MARCO | This PR | 1 | 2.675 | 2.562 | 0.110 | 305.14 | 397.45 | 7.06 | 80.81 | | MS MARCO | This PR | 2 | 2.701 | 2.593 | 0.105 | ≈530.25 | ≈622.91 | 6.75 | 80.62 | Mean add wall time falls 46.5% on Wikipedia (5.938 to 3.179 seconds) and 55.1% on MS MARCO (5.989 to 2.688 seconds) relative to #7838. The saved base, #7829, and #7836 mean add wall times are 4.478, 3.490, and 3.647 seconds on Wikipedia and 4.128, 3.346, and 3.483 seconds on MS MARCO. At checkpoint 10, reported leaf-lock wait amortized per registered posting falls from 18.7–30.3 microseconds to 48–206 nanoseconds on Wikipedia and from 67.3–75.3 microseconds to 200–506 nanoseconds on MS MARCO. A buffered flush takes one lock for many postings, so this is not the wait time of a single lock acquisition. **The MS MARCO full-build result is a material risk.** Candidate run 2 takes two balance rounds at checkpoints 9 and 10, about 150 and 168 seconds, versus one round and about 54 and 51 seconds in candidate run 1. Its summed timed work across concurrent scrub calls rises to 90.2 and 102.3 worker-minutes versus 1.2 and 1.1 in run 1; merge timing also rises to 1.7 and 1.8 worker-minutes versus milliseconds. Split counts, persisted posting loads, and final leaf counts remain similar. The logs identify extra rounds with scrub and merge latency, but do not establish the initiating cause. Buffering changes posting order and may change the later tree. The two-run sample does not support a full-build improvement claim; the MS MARCO candidate mean is about 510 seconds versus #7838's 397 seconds. Warm recall and peak RSS stay within the range of the saved runs. Each candidate summary reports 1M unique vectors and zero orphaned IDs, but that summary scans all nodes and does not prove that every valid posting is reachable from the root. The performance flags omit the reachable-posting check; focused integration tests cover the persisted posting path. The source checkout, pinned executable, runner, commands, exit codes, logs, `/usr/bin/time` records, and SHA-256 are on the EC2 host under `/home/ubuntu/chroma-benchmark-leaf-deltas` and `/home/ubuntu/benchmark-data/navigation-1m-100k-leaf-deltas`. The exact tested source is `c65f153b62157df19cc399a0e805e716804952dc`, and executable SHA-256 is `01a5417c4e9da47c2edfa5db28b1e2a1ca5204a89cf459a3e51a32b6b43b084b`. A detailed local summary is at `/private/tmp/lyon-navigation-leaf-deltas-1m-results.md`.
tl;dr cache versions to avoid blockfile reads ## Why this helps An opened index writer checks each vector's current version while it cleans stale entries from leaf posting lists. Unchanged versions live in scalar metadata. The previous 65,536-entry FIFO cache evicts versions during a 1M-vector balance, so workers repeatedly issue point reads through one metadata-reader lock. ## What changes The writer now retains unchanged checkpoint versions for its lifetime in pages covering 4,096 vector IDs. Each slot stores a version or one of two states: unread and absent. Pages are allocated only when an ID in that range is read. New or changed versions still take precedence in the mutable overlay. The memory report counts allocated page storage. Two existing integration tests now call the current one-worker balancing entry point. A dense 1M-ID checkpoint uses about 2 MiB of version slots; 10M IDs use about 19 MiB, plus the page map and allocator overhead. Sparse IDs allocate only touched pages. The cache is discarded when the writer closes at the next checkpoint. ## Benchmarks The controlled replay starts from the same saved 800k-vector MS MARCO tree and adds two 100k-vector checkpoints. Each configuration ran twice on the lyon `r6id.16xlarge` instance. The baseline is PR #7841 at `c65f153b`; the candidate is that exact commit plus this version-cache change. This PR applies the same cache change directly to the current `hierarchical-spann` branch, so the timing comparison isolates the cache on the leaf-delta stack rather than on this PR's exact base. | Cache | 900k balance | 1M balance | 900k total checkpoint | 1M total checkpoint | | --- | ---: | ---: | ---: | ---: | | 65,536-entry FIFO | 45.5–47.1 s | 48.2–48.7 s | about 60 s | about 65 s | | Compact pages | 3.8–4.0 s | 4.1–4.2 s | 18.5–18.6 s | 20.1–20.2 s | Full 1M-vector builds used ten 100k checkpoints, 1024D vectors, full-precision writer navigation and neighbor reassignment, parallel balancing, a 1 GiB block cache, and 100 precomputed recall queries. Each baseline and candidate dataset ran twice. Times are the benchmark's build summaries; recall is final cold R@100 at tau 1.5. | Dataset | Baseline build | Candidate build | Baseline R@100 | Candidate R@100 | | --- | ---: | ---: | ---: | ---: | | MS MARCO | 6.7 / 10.5 min | 2.3 / 2.4 min | 80.81 / 80.62% | 80.37 / 80.36% | | Wikipedia | 3.7 / 3.7 min | 1.2 / 1.2 min | 77.16 / 77.35% | 76.95 / 77.14% | Split seeds vary between full builds, so these recall differences cannot be attributed to the cache change from unpaired runs. Earlier builds with the same settings produced MS MARCO recall between 80.32% and 80.94%, and Wikipedia recall between 76.67% and 77.72%. The version and posting integrity tests pass. MS MARCO arguments: `--dataset ms-marco --metric l2 --checkpoint 10 --checkpoint-size 100000 --threads 64 --balance-threads 64 --write-navigation fp --fp-npa --write-beam-tau 1.5 --write-rng-factor 4 --num-queries 100 --recall-tau-values 1.5 --recall-rerank-vectors 1 --max-cache-bytes 1073741824 --compute-gt-clusters false`. Wikipedia arguments: `--dataset wikipedia-en --metric l2 --checkpoint 10 --checkpoint-size 100000 --threads 32 --balance-threads 32 --write-navigation fp --fp-npa --write-beam-tau 0.5 --write-level-taus _,_,0.30 --write-rng-factor 0 --num-queries 100 --recall-tau-values 1.5 --recall-rerank-vectors 1 --max-cache-bytes 1073741824 --compute-gt-clusters false`. ## Verification `cargo test -p chroma-index --test hierarchical_posting_delta --test hierarchical_policy_cache` passed all 23 tests on the feature-branch PR head. `cargo check -p chroma-index --bench hierarchical_spann_profile_quantized` passed. The four full dataset builds and four controlled replays exited successfully.
…7852) ## Summary Balancing scores a parent's children while moving vectors between leaves. A shared parent payload stores their IDs beside contiguous full-precision centroids, so scoring avoids fetching every child's node from the shared map. Search resolves candidates in distance order and counts only live nodes toward its candidate limit. Removed nodes establish neither the closest distance nor the limit that cuts off further candidates. The child IDs and centroids live in one private, validated payload. One method edits child lists and publishes matching centroids. A second method changes an internal node's centroid and refreshes its entry in the next parent. Searches retain a stable reference and release the parent map guard before scoring. The node map remains the source of actual leaves and internal nodes; the parent payload supplies their scoring data. The persisted index format is unchanged. The PR also fixes first reassignment after reopening an index. Registration imports an unchanged vector's persisted version into the writable version map before updating it and preserves the old version until a replacement posting exists. Both benchmark variants include this fix, so the comparison holds version handling constant. The implementation is in the prototype writer under `rust/index/benches/hierarchical_index/writer/`. A separate stacked cleanup, #7855, removes unused navigation-dirty tracking. ## EC2 results (provisional) The comparison measures control `049cc0eb1` and the exact PR head `22d32265e` on the lyon r6id.16xlarge instance. It uses 1,024-dimensional datasets, full-precision navigation and neighbor reassignment, parallel balancing, and 100k checkpoints. Repeated MS MARCO runs independently copy the same validated 800k checkpoint and add 200k vectors. The two control runs are reused from the preceding comparison on this instance; the two final-head candidate runs execute sequentially afterward. The final comparison therefore does not interleave control and candidate runs. | Comparison | Control balancing | Candidate balancing | Reduction | Control checkpoint total | Candidate checkpoint total | Reduction | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | MS MARCO 800k→1M mean, two runs each | 9.04 s | 4.83 s | 46.5% | 39.62 s | 35.84 s | 9.6% | | Wikipedia fresh 1M, one run each | 49.90 s | 32.75 s | 34.4% | 75.72 s | 58.61 s | 22.6% | | MS MARCO fresh 1M, one run each | 47.53 s | 34.50 s | 27.4% | 139.69 s | 127.87 s | 8.5% | Checkpoint totals include dataset loading, addition, balancing, commit, and reopen. They exclude final recall and separate correctness checks. The first 100k checkpoint takes longer in the candidate's full builds: Wikipedia balancing is 12.41 versus 13.60 seconds; MS MARCO is 11.67 versus 14.27 seconds. Later checkpoints improve. Single full-build comparisons do not establish why startup differs. Recall across 100 precomputed queries is 81.05% versus 81.15% for the replay means, 77.72% versus 77.88% for fresh Wikipedia, and 81.14% versus 80.61% for fresh MS MARCO. The MS MARCO full build therefore measures a 0.53 percentage-point recall reduction. These small samples and single full-build runs do not establish recall equivalence. Peak process memory is 7.27 versus 7.52 GiB for Wikipedia and 7.14 versus 7.34 GiB for MS MARCO. ## Validation Both local test harnesses pass 51 executions. Three regressions fail on the preceding PR head and pass on the current head: a removed nearest child consuming the candidate limit, a removed merge source consuming the limit, and reassignment missing a live sibling. Tests also cover 125 combinations of distance cutoffs and candidate limits, a child returning after being absent during publication, parallel splits, cascading internal splits, retained payloads, and persisted versions. Final-head correctness validation finds an intermittent failure. The first MS MARCO run fails before its 200k commit: 200,000 vectors are expected, but only 199,750 have reachable valid postings. The table includes every final-head validation run, including that failure. | Final-head run | Successful exact-ID checks | Outcome | | --- | ---: | --- | | final-validate-msmarco | 2 | before commit: 200000 expected ids, 199750 root-reachable valid ids; missing sample [132381, 128356, 133456, 136011, 121409, 128301, 120162, 129671], extra sample [] | | final-validate-msmarco-recheck | 20 | Passes through 1M before every commit and after every reopen. | | final-validate-wikipedia-1 | 20 | Passes through 1M before every commit and after every reopen. | | final-validate-wikipedia-2 | 20 | Passes through 1M before every commit and after every reopen. | A deterministic regression reproduces a vector-loss mechanism on both the control and final candidate. A merge removes its source posting before transferring every vector. If a concurrent internal split temporarily removes the root, reassignment can exhaust three immediate attempts without publishing a replacement. That preexisting defect remains unresolved; the reproduction does not prove the cause of the specific 250 missing vectors. The timing results are provisional because validation has not established that parallel builds consistently preserve every vector. An earlier Wikipedia validation also found a missing child reference at 800k. A separate deterministic regression reproduces that error on both revisions when a merge sees a replacement leaf before its parent pointer is assigned. This PR repairs that case by always finding and removing the reachable parent link. The regression passes with the fix, but the reproduction does not prove the cause of the specific missing child in that earlier run. Independent replay copies leave the validated source checkpoint's metadata checksum unchanged. ## Benchmark settings Common settings are `--metric l2 --checkpoint 10 --checkpoint-size 100000 --write-navigation fp --fp-npa --num-queries 100 --recall-tau-values 1.5 --recall-rerank-vectors 1 --max-cache-bytes 1073741824 --compute-gt-clusters false`. Wikipedia adds `--dataset wikipedia-en --threads 32 --balance-threads 32 --write-beam-tau 0.5 --write-level-taus _,_,0.30 --write-rng-factor 0`. MS MARCO adds `--dataset ms-marco --threads 64 --balance-threads 64 --write-beam-tau 1.5 --write-rng-factor 4`. Replays add `--resume --save-dir <independent-checkpoint-copy>`. Correctness checks add `--validate-postings` and are measured separately. The benchmark target is `hierarchical_spann_profile_quantized` in `chroma-index`. Logs, exact commands, resource measurements, revisions, and parsed results are saved on EC2 under `/home/ubuntu/benchmark-data/child-navigation-fix-7852`. The validated source checkpoint is preserved on EBS under `/mnt/benchmark-source/checkpoints/child-navigation-800k`.
## Summary The writer tracks changed nodes so saving an index persists their metadata. A separate set tracks parents that need navigation refresh, but no code reads that set. Navigation refresh rebuilds the packed child centroids for every live parent. This change removes the unused set and the map lookups that populate it. The first add after reopening an index calls the full navigation refresh directly. The persistence tracking remains in place. This PR lands the cleanup from #7855 on `hierarchical-spann`. That PR is merged into the child-navigation branch, while the feature branch contains the child-navigation implementation from #7852 without this cleanup. The diff contains only the three-file cleanup under `rust/index/benches/hierarchical_index/writer/`. ## Validation The patch matches #7855 exactly. Diff checks pass, and a source search finds no remaining references to the removed set or refresh wrapper. Both policy/navigation and posting-delta test harnesses pass: 51 test executions cover additions, navigation refresh, cache behavior, persistence, and reopening (`cargo test -p chroma-index --test hierarchical_policy_cache --test hierarchical_posting_delta`). GitHub's repository-wide lint job fails on formatting in files unchanged by this PR, including posting-list storage, dataset loaders, and instrumentation. Its Rust formatting output reports no failures in the three changed writer files. The feature branch has no branch protection; the local tests above validate this cleanup. Other repository-wide CI jobs are still running at merge time.
## Summary A merge moves vector postings, the rows that record which leaf contains a vector. A vector must keep a valid source posting until a destination accepts its replacement. This PR preserves that rule when parallel balancing temporarily removes the root or a destination disappears. The source leaf stays readable throughout transfer. Destination searches exclude that source before applying their candidate limit. A direct transfer publishes a posting with the existing version, then removes one matching source row and its code. This preserves replicas in other leaves. Reassignment publishes its new version before the old source row becomes stale. Failed transfers and concurrent additions remain in the source. Cleanup removes the source only when an atomic check finds it empty. Each merge reserves its chosen source and destination together. Another merge that needs either leaf defers its transfer, which prevents the reproduced reciprocal pair from swapping rows without shrinking the tree. A short mutex protects reservation and release; workers transfer disjoint pairs in parallel. The reservation is released before follow-up balancing. Per-vector reassignment can still reach other active leaves, so this change does not guarantee complete balancing convergence. This PR is stacked on #7852. It changes the prototype writer under `rust/index/benches/hierarchical_index/writer/` and keeps the persisted index format unchanged. The separate unused-navigation-tracking cleanup in #7855 is not included. ## Balancing limit Parallel balancing runs at most eight outer rounds per call. The loop warns with the remaining oversized and undersized leaf counts when it reaches the limit. The eight measured builds finish ordinary balances within four rounds; one balance reaches the previous 100-round cap. Eight rounds bound repeated work while allowing room beyond the observed ordinary balances. Reaching the cap can leave balancing unfinished. ## Validation Both local test harnesses pass 59 executions. A deterministic regression pauses the actual merge after destination selection, removes the root, then resumes transfer. It fails on the base with a missing vector and passes with this fix. Additional tests cover all destinations disappearing, a concurrent append to the source, a merge without a usable target, other replicas remaining valid, and an overlapping merge deferring without entering transfer. All four EC2 validation builds pass on the lyon instance: two fresh 1M-vector builds for each 1,024-dimensional dataset, with 100k checkpoints, full-precision navigation and neighbor reassignment, and parallel balancing. Every run checks the exact vector-ID set and every child reference before every commit and after reopening. All 80 checks pass. | Run | Final vector count | Before-commit checks | After-reopen checks | Result | | --- | ---: | ---: | ---: | --- | | msmarco-1 | 1,000,000 | 10 | 10 | Pass | | wikipedia-1 | 1,000,000 | 10 | 10 | Pass | | msmarco-2 | 1,000,000 | 10 | 10 | Pass | | wikipedia-2 | 1,000,000 | 10 | 10 | Pass | The tested runtime revision is `63be72d31`. The later commit `fb7a9dab9` clarifies one comment and changes no runtime code. These runs establish correctness for the exercised cases; validation materializes posting lists, so their timing is not an ordinary-build performance comparison. ## Tradeoffs A merge copies the small source's IDs and versions while retaining its posting buffer. Successful direct transfers acquire the source leaf lock and search for their accepted row. Removing an entire source this way requires a quadratic number of ID comparisons; the default 512-row merge threshold bounds the ordinary cost. A future optimization can remove accepted rows in one scan while preserving concurrent additions. The pair reservation mutex is held only for booking and releasing the two leaf IDs, not during vector transfer. Ordinary navigation performs no additional candidate-filtering pass. ## EC2 settings Common arguments are `--metric l2 --checkpoint 10 --checkpoint-size 100000 --write-navigation fp --fp-npa --num-queries 100 --recall-tau-values 1.5 --recall-rerank-vectors 1 --max-cache-bytes 1073741824 --compute-gt-clusters false --validate-postings --save`. Wikipedia adds `--dataset wikipedia-en --threads 32 --balance-threads 32 --write-beam-tau 0.5 --write-level-taus _,_,0.30 --write-rng-factor 0`. MS MARCO adds `--dataset ms-marco --threads 64 --balance-threads 64 --write-beam-tau 1.5 --write-rng-factor 4`. Each run saves to an independent checkpoint directory. Logs, exact commands, resource measurements, and the tested revision are saved under `/home/ubuntu/benchmark-data/merge-posting-preservation/final` on EC2. ## Performance comparison without posting validation The comparison below measures revision `63be72d31` with the 100-round cap. It does not measure the new eight-round cap. Eight sequential EC2 runs compare the PR's exact parent (`22d32265e`) with its final runtime revision (`63be72d31`): two fresh 1M-vector builds per version for Wikipedia and MS MARCO, with 100k checkpoints. Settings match the validation runs except that posting validation is disabled. Both versions persist each checkpoint to a fresh local NVMe directory. The first round runs baseline then candidate; the second reverses their order. All eight processes exit successfully. | Dataset | Measurement | Baseline mean | PR mean | Change | | --- | --- | ---: | ---: | ---: | | MS MARCO | Total build time | 131.8 s | 145.7 s | +10.5% | | MS MARCO | Balancing time | 35.13 s | 50.20 s | +42.9% | | MS MARCO | Add time | 2.55 s | 2.59 s | +1.5% | | Wikipedia | Total build time | 64.5 s | 62.2 s | -3.6% | | Wikipedia | Balancing time | 34.31 s | 32.09 s | -6.5% | | Wikipedia | Add time | 2.93 s | 3.11 s | +6.1% | The MS MARCO candidate build times are 135.0 and 156.3 seconds; baseline times are 134.1 and 129.6 seconds. The slower candidate reaches the 100-round balancing limit at 900k vectors and spends 24.48 seconds balancing that checkpoint. It exits balancing with zero oversized leaves and two undersized leaves remaining. Baseline runs require one round and spend 2.70 and 2.88 seconds there. Initial balancing also costs more on the candidate. This behavior needs investigation before treating the fix as having negligible performance cost. The direct instrumented merge calls total less than 0.2 worker-seconds per MS MARCO run. These timings omit follow-up balancing outside the merge body, so they do not explain the wall-clock increase. The logs identify repeated balancing work; they do not establish its exact cause. Both MS MARCO candidate runs report 1M valid postings, while the baselines report approximately 999.4k and 999.8k. The broken baseline can do less work. The timed runs do not perform exact posting validation; separate correctness runs pass all 80 checks. Mean recall at 100 neighbors is 81.17% versus 81.08% for MS MARCO and 77.63% versus 77.82% for Wikipedia. Two repeats do not establish a recall change or a stable performance estimate. Raw results are on EC2 at `/home/ubuntu/benchmark-data/pr7856-performance-20261006`. A report, logs, exact commands, hashes, resource measurements, and summary parser are saved locally under `/Users/chroma/Documents/ChatGPT/Hierarchical SPANN/pr7856-performance/`.
) ## Summary Each checkpoint needs a specific range of vectors from the dataset. The loaders currently decode every column in the overlapping Parquet file, then discard vectors preceding the checkpoint. A Parquet file stores independently readable sections called row groups. Selecting the required groups avoids decoding earlier vectors repeatedly. This change selects only the embedding column and row groups intersecting the requested range. It applies the offset and length within those groups while retaining the dataset’s physical row IDs and existing float conversion. Wikipedia and MS MARCO share the range-selection helper. Existing source caches and ground-truth files remain usable without conversion or new arguments. ## Performance On the existing EC2 benchmark node, three sequential paired runs load the first 1M vectors in ten 100k requests. The second pair reverses execution order. Both paths use the same optimized binary and jemalloc allocator. These are warm-OS-cache measurements from local NVMe; vector hashing occurs outside the loading timer. | Dataset | Baseline median | This PR median | Loading time reduction | |---|---:|---:|---:| | MS MARCO | 80.64 s | 15.78 s | 80.4% | | Wikipedia | 16.17 s | 15.85 s | 2.0% | MS MARCO stores roughly 1M rows per shard, so avoiding repeatedly decoded prefixes saves about 65 seconds per 1M-vector input load. Wikipedia shards are already roughly the checkpoint size; its small result does not establish a meaningful performance improvement. This measures dataset loading rather than index construction, balancing, or commit latency. Median loader-process peak RSS is 737 MiB versus 701 MiB for MS MARCO and 692 MiB versus 705 MiB for Wikipedia. The change adds no retained vector cache or additional persistent dataset files. ## Validation Both datasets pass exact comparisons over the first 1M IDs and every f32 component. All twelve timed runs produce matching per-dataset checksums. Two regression tests cover f32/f64 embeddings, a preceding text column, null rows, row-group boundaries, empty ranges, and a missing embedding column. The hierarchical benchmark target compiles on the current feature branch. The real EC2 harness also completes two 100k checkpoints on each dataset with full-precision writing, parallel balancing, saving, and reopening. These are integration smoke checks; recall is skipped at the 200k boundary. Changed-file formatting and diff checks pass. ## Scope The changes apply to benchmark dataset loaders under `rust/index/benches/datasets/`. The index algorithms and query/ground-truth loading code are unchanged. The baseline dataset sources are from `af6366cce6`; the candidate is `48b8c656e`. Raw measurements and reproduction sources are saved in this task’s `dataset-loading-optimization` report directory. Co-authored-by: lyon-chroma <lyon@trychroma.com>
## Summary Queries spend most of their time comparing compressed vector codes in posting lists. Each comparison counts the set bits in four intersections with the query's packed bits. A portable x86 build performs that calculation with ordinary instructions even when the CPU provides dedicated bit-count instructions. This change detects CPU capabilities once per process and stores the selected single-vector and posting-list scoring functions in a shared, thread-safe initializer (`OnceLock`). Both scoring entry points use that selection. Compatible x86 CPUs use vector population count, which counts set bits across several machine words at once. Other compatible x86 CPUs use scalar population count. Other processors use the existing portable calculation. CPU checks run during initialization before selecting any specialized function. Subsequent calls use the stored function pointers. The public scoring functions call the cached function pointers directly. Both single-vector and batch kernels use one shared arithmetic function, which is forced inline so the selected CPU features apply to its bit-count loop. Single-vector scoring calls the selected function directly; batch scoring performs one indirect call for the entire list and then inlines the arithmetic for each vector. Without inlining, the specialized outer loop calls a separately compiled portable inner function. The reader reuses one distance buffer across its leaves, then passes the same vector IDs and distances to the existing result selection. Its additional temporary memory is four bytes per vector in the largest selected leaf, per active query. The affected code is the hierarchical reader's posting scoring in `rust/index/benches/hierarchical_index/reader/reader.rs` and the one-bit distance kernel in `rust/index/src/quantization/single_bit.rs`. The stored format, distance formula, search thresholds, result selection, and writer algorithms stay the same. ## Additional kernel validation A short EC2 kernel comparison against PR revision `9cd5638956` scores 4,096 codes with six alternating single-vector/batch passes at each dimension. The medians below exclude the first warmup pass. At 1,024 dimensions, single-vector scoring improves from 66.98 to 20.72 ns/code; batch scoring remains approximately 15 ns/code (15.07 to 15.11). At 64 dimensions, the figures are 15.35 to 10.51 ns/code for single-vector scoring and 6.66 to 6.66 for batch scoring. At 1,536 dimensions, they are 94.18 to 20.89 and 17.37 to 17.40 respectively. Every pass checks exact score bits. These are kernel measurements; the full-reader measurements below describe the batch optimization and do not measure an additional query-latency gain from caching the selection. ## Measurements The comparison searches the exact same saved 1M-vector index for each real 1,024-dimensional dataset. Three paired repetitions alternate the original posting loop and the candidate. Each run executes the same 100 queries on 32 workers, with a read threshold of 1.5, 100 results, no full-precision reranking, and a 1 GiB block cache. | Dataset | Reader state | Baseline average query | Candidate average query | Reduction | Baseline scoring | Candidate scoring | | --- | --- | ---: | ---: | ---: | ---: | ---: | | Wikipedia | Warm | 12.13 ms | 4.90 ms | 59.6% | 10.67 ms | 3.40 ms | | MS MARCO | Warm | 11.03 ms | 4.50 ms | 59.2% | 9.70 ms | 3.17 ms | | Wikipedia | Cold reader state | 20.07 ms | 12.47 ms | 37.9% | 10.90 ms | 3.67 ms | | MS MARCO | Cold reader state | 17.67 ms | 10.77 ms | 39.1% | 9.83 ms | 3.33 ms | These are concurrent-query averages on an r6id.16xlarge with an Intel Xeon Platinum 8375C and local NVMe. Cold reader state clears posting data and block pins; it leaves the operating system's filesystem cache populated. It does not model cold object storage. No other benchmark or heavy compilation overlaps the measurements. A separate single-worker pair gives warm query means of 11.0 → 4.1 ms on Wikipedia and 10.3 → 3.7 ms on MS MARCO. Warm p99 falls from 15.61 → 6.19 ms and 16.20 → 6.15 ms respectively. These percentiles come from only 100 queries per pass and are coarse sample measurements, not production estimates. A kernel probe scores 4,096 generated codes. The original loop takes approximately 85.35 ns/code, scalar hardware population count takes 28.85 ns/code, and vector hardware population count takes 15.11 ns/code. Disassembly confirms `vpopcntq` in the vector loop. An Apple ARM probe keeps the portable batch fallback near the original 15–16 ns/code after warmup; that short measurement establishes no cross-platform speedup. ## Validation All 16 reader runs return identical top-100 IDs, ordering, and floating-point score bits in their cold and warm passes. This comparison covers 3,200 query invocations. Recall stays at 77.81% for Wikipedia and 81.01% for MS MARCO on these fixed indexes. The differential test compares both public scoring entry points and the available single-vector and batch kernels with the portable arithmetic, requiring identical score bits. It exercises Euclidean, cosine, and inner-product distance; dimensions 1, 7, 8, 63, 64, 65, 300, 1,024, and 1,536; padded final words; and empty and small lists. It compares the batch entry point, portable loop, and available specialized loops with individual-code scoring. All eight one-bit quantization tests pass in release mode on the x86 benchmark node, including the extended differential test for shared dispatch. All eight one-bit quantization tests and 59 existing policy/posting integration tests pass locally. The complete profiling benchmark compiles. Formatting and diff checks pass; compilation reports the existing unused `clear_child_scoring` warning. ## Reproduction The source base is `af6366cce6`, with Rust 1.92.0, the existing Cargo release profile, and no `RUSTFLAGS` override. Saved posting-preserving indexes from #7856 provide the same data to both reader kernels. The temporary paired harness preserves the original scalar posting loop, selects one or 32 query workers, and records result bits and percentiles outside timed search. These comparison switches and logging are excluded from this PR. Build the original feature revision and this branch separately, then run the existing profiling benchmark against the same checkpoint with `--resume --no-commit --checkpoint 10 --checkpoint-size 100000 --metric l2 --num-queries 100 --recall-tau-values 1.5 --recall-rerank-vectors 1 --max-cache-bytes 1073741824 --compute-gt-clusters false --save-dir <checkpoint>`. The saved checkpoint must already contain ten 100k additions, so the command adds no new vectors. Wikipedia uses `--dataset wikipedia-en --threads 32 --balance-threads 32 --write-navigation fp --fp-npa --write-beam-tau 0.5 --write-level-taus _,_,0.30 --write-rng-factor 0`. MS MARCO uses `--dataset ms-marco --threads 64 --balance-threads 64 --write-navigation fp --fp-npa --write-beam-tau 1.5 --write-rng-factor 4`. Raw logs, commands, comparison source, binary hashes, and the report are preserved locally under `read-scoring-performance` in this chat's workspace and remotely under `/mnt/data/read-scoring-results` on Lyon's benchmark node. The measured binaries are copied there so another build cannot overwrite them. The shared-dispatch implementation calls selected functions directly and keeps all distance arithmetic in one shared function. A final uncontended kernel check on commit `6208c4b78` measures 19.77 ns/code for single-vector scoring and 15.08 ns/code for batch scoring at 1,024 dimensions. Exact score comparisons and all eight x86 release quantization tests pass. --------- Co-authored-by: lyon-chroma <lyon@trychroma.com>
## Why this helps Balancing moves vectors between groups of vectors at the leaves of the tree. Before moving a vector, the writer checks its current version so an older posting cannot replace a newer update or deletion. Persisted scalar metadata stores many versions together in each storage block. A cold version lookup fetches a whole block but copies only the requested version into the compact cache. The writer releases decoded blocks after each batch of 64 requests. Scattered vector IDs repeatedly fetch the same blocks when the shared cache cannot retain them, making other balancing workers wait for serialized version loading. ## What this PR does A cold lookup copies every version from the requested vector's metadata block into the compact cache. Later lookups for any vector in that block use the compact values. Writer opening remains lazy. The scalar reader returns owned integer pairs and releases its temporary ownership of the decoded block before returning. The existing shared block cache can retain its own bounded copies; this read adds no per-reader pins. This keeps transient decoding memory bounded by one block and avoids keeping borrowed rows alive after their storage is released. The compact cache still stores two bytes per touched vector slot, with pages allocated only for the ID ranges it touches. The mutable version map remains authoritative for updates and deletions during the writer session. Copying checkpoint values cannot overwrite that map. Current-version lookup checks the mutable map again after loading. Missing IDs retain negative cache entries; sparse IDs and checkpoints without maximum-ID metadata retain their existing behavior. The reader change is in `rust/blockstore/src/arrow/blockfile.rs` and `rust/blockstore/src/types/reader.rs`. The writer uses it in `rust/index/benches/hierarchical_index/writer/persistence.rs`. ## Benchmark results The comparison uses a shared immutable 9M Wikipedia checkpoint and adds the next 1M vectors on the existing 64-vCPU, 512-GiB Intel Xeon EC2 node. Every run starts from an independent copy. Settings use 1024-dimensional vectors, L2 distance, 32 add workers, 32 balancing workers, full-precision navigation and neighbor reassignment, and a 1 GiB shared block cache. Runs and compilations are serialized. Two alternating samples per implementation skip persistence to preserve an identical starting checkpoint: | Measurement | Control | Metadata-block fill | |---|---:|---:| | Balance samples | 22.80 / 23.54 s | 14.70 / 14.16 s | | Mean balance | 23.17 s | 14.43 s | | Balance reduction | — | **37.7%** | | Initial open | 65 / 62 ms | 62 / 61 ms | A separate complete run per implementation commits and reopens the resulting 10M index: | Measurement | Control | Metadata-block fill | |---|---:|---:| | Balance | 23.75 s | 15.03 s | | Commit | 9.44 s | 9.56 s | | Initial open + checkpoint | 54.99 s | 46.55 s | | Reduction including open | — | **15.4%** | | Peak RSS | 63.06 GiB | 62.67 GiB | | Recall at tau 1.5, without full-precision reranking | 74.55% | 74.54% | | Valid unique vectors / orphans | 10M / 0 | 10M / 0 | One committed sample does not establish a recall or commit-time change. Parallel balancing can change work order and tree shape. Recall printed by the no-commit runs is excluded because the harness reads the persisted 9M index while comparing to 10M ground truth. Its resumed-run throughput is also excluded because its numerator includes the existing 9M vectors. A separate reproduction loads the same 200 leaves from independent copies of the final 40M metadata and postings. It measures the actual writer loading path, including opening, but performs no balancing, embedding loading, or persistence: | Measurement | Control | Metadata-block fill | |---|---:|---:| | Full process samples | 24.77 / 24.88 s | 3.06 / 3.05 s | | Mean process reduction | — | **87.7%** | | Approximate process read traffic | 40.1 / 40.8 GiB | 1.21 GiB | The selected leaves touch enough metadata blocks to fill all 40M versions into about 80 MB of compact pages. All cases have zero physical filesystem input; read traffic measures repeated copies from the operating system's file cache. The local storage backend hashes fetched files, so these percentages do not estimate production object-storage speedup. This is not a full 40M balancing checkpoint. ## Verification and reproducibility The posting/version and policy suites pass 60 test executions. They cover concurrent loads and updates, deletion versions, sparse IDs, legacy metadata, stale postings, reopening, and committing. The added owned-row test preserves sparse values and tombstone bytes while verifying that the reader pins zero blocks. The original 40M index remains unchanged: all 109,539 files match its before manifest for name, size, modification time, and inode. The original 9M source is read only. Only the investigation's explicitly released independent temporary copy is recycled between committed runs. The benchmark base is `22eb704e71490d95cf715bdb8901f7d6c8a3cf34`. The production benchmark SHA256 is `4cce0c907c925c6727c72426c942fa48c750afd4a0db6c152a308fd5368f2fc2`. Its saved implementation patch has the same production behavior as this commit; the final commit includes formatting and test changes. Component measurements use a separate saved instrumentation patch and binary. ```sh BINARY --dataset wikipedia-en --metric l2 --checkpoint 10 --checkpoint-size 1000000 --threads 32 --balance-threads 32 --write-navigation fp --fp-npa --write-beam-tau 0.5 --write-level-taus _,_,0.30 --write-rng-factor 0 --num-queries 100 --recall-tau-values 1.5 --recall-rerank-vectors 1 --compute-gt-clusters false --resume --save --save-dir INDEPENDENT_COPY --max-cache-bytes 1073741824 ``` Write-level minimum percentages are omitted. Short timing samples substitute `--no-commit` for `--save`. Raw logs, commands, patches, hashes, metrics, and preservation proof are saved at `/home/ubuntu/benchmark-data/bulk-version-cache-fixes` and in Lyon's local `bulk-version-cache-fixes/REPORT.md`. ## Full 21M→22M follow-up The larger test confirms the same bottleneck in a complete committed increment. Each implementation and control runs twice, in the order control, block fill, bulk scan, bulk scan, block fill, control. Every run starts from an independent copy of the same preserved 21M tree. The fixture extends the validated 10M output to 21M with the block-fill implementation; it does not reuse or overwrite the original 40M tree. Each increment commits, reopens, and evaluates 100 exact queries against saved 22M ground truth. | Measurement | Control | Metadata-block fill | Bounded bulk scan | |---|---:|---:|---:| | Balance samples | ~240 / ~216 s | 28.22 / 29.22 s | 27.50 / 22.20 s | | Mean balance | ~228 s | 28.72 s | 24.85 s | | Balance reduction | — | **~87%** | **~89%** | | Initial-open samples | 146 / 140 ms | 145 / 145 ms | 1.42 / 1.38 s | | Checkpoint samples as printed | 4.7 / 4.2 min | 1.1 / 1.1 min | 1.1 / 1.0 min | | Mean initial open + checkpoint | ~267 s | ~66 s | ~64 s | | Reduction including open | — | **~75%** | **~76%** | | Full-process samples | 586.98 / 423.62 s | 110.19 / 109.67 s | 111.10 / 106.12 s | | Mean full process | 505.30 s | 109.93 s | 108.61 s | | Peak RSS range | 125.98–126.41 GiB | 121.55–121.84 GiB | 121.64–121.88 GiB | | Recall range, tau 1.5 without full-precision reranking | 72.99–73.30% | 73.15–73.22% | 73.12–73.23% | | Valid unique vectors / orphans, every output | 22M / 0 | 22M / 0 | 22M / 0 | | Balance rounds | 3 / 2 | 8 / 8 | 6 / 3 | The CLI rounds durations longer than 60 seconds to 0.1 minutes. Control balancing and all checkpoint totals therefore have roughly ±3 seconds of display rounding; their averages and percentage reductions are approximate. Full-process times come from the precise external process timer. The full-process comparison includes expensive post-reopen diagnostics and recall, so it does not replace the separately reported write-checkpoint comparison. The amount of reassignment work remains similar while destination loading gets much faster. In the first sample, the control performs 454.7k reassignments and 772 splits; block fill performs 451.2k reassignments and 767 splits. Average destination registration within reassignment falls from 10.5ms to 31.5µs, and the average entire reassignment falls from 33.9ms to 2.7ms. These timers include waiting and nested work and cannot be added to obtain wall time. The two fixes have similar overall performance in this sample count. Bulk scan has a lower mean balancing time, while block fill preserves faster opening. Parallel work ordering changes the number of balance rounds and tree shape; two samples do not establish a small performance difference between the fixes or recall equivalence. Candidate recall ranges fall within the control's observed range. The peak-memory difference also includes changes in parallel work and reader pinning, so it is not attributed solely to cache filling. The shared 21M source remains at `/mnt/data/bulk-version-cache-fixes-source-21m` for future comparisons. All 57,510 files match its before/after manifest, totaling 93,212,784,474 bytes. Its checkpoint SHA256 is `bb1b21158c5f070c3a108fe7f2b9233b5e9477f827b423a016ee44e3e201095d`. The original 40M manifest and checkpoint hash also match. Existing global ground truth is restored after the comparison, the lock is released, and no benchmark is running. ## Alternative considered [The bulk-scan alternative (#7875)](#7875) fills all checkpoint versions during writer opening. It produces similar balancing and 40M component results but adds about half a second to opening and reopening 9M versions and about 2.3 seconds to opening 40M versions in the reproduction. Demand-driven metadata-block fill is the recommended default because it preserves fast opening and avoids scanning unused versions. ## Additional committed comparison: 25M to 26M Both published PR commits complete the larger increment in about 69 seconds including opening, adding, balancing, committing, and reopening. The whole process takes about two minutes when final index validation and recall are included. The metadata-block fill keeps opening fast; the bulk scan moves a little balancing work into opening and reopening. These single clean samples do not establish a meaningful overall advantage for the bulk scan. The common starting tree contains 25M Wikipedia vectors. It is generated by extending an independent copy of the preserved 21M fixture with the metadata-block implementation. Both timed runs begin with independently copied files from that same frozen 25M source. The source retains its exact file manifest after both runs. The original 40M index retains all 109,539 original files and the same checkpoint hash. The preserved 21M source is read only throughout preparation. No 39M build runs. | Measurement | Metadata-block fill, #7874 | Bulk scan, #7875 | |---|---:|---:| | Initial writer open | 0.170 s | 1.660 s | | Load next 1M source vectors | 14.00 s | 13.94 s | | Add 1M vectors | 10.26 s | 10.27 s | | Balance | 27.07 s, 2 rounds | 25.93 s, 1 round | | Commit | 16.10 s | 16.09 s | | Reopen | 0.155 s | 1.500 s | | Sum of initial open and checkpoint phases | 67.76 s | 69.39 s | | Full process including diagnostics and recall | 120.09 s | 121.29 s | | Peak process RSS | 142.91 GiB | 143.83 GiB | | Final recall at tau 1.5 without full-precision reranking | 72.81% | 72.70% | | Final valid unique vectors | 26M | 26M | | Orphan vectors | 0 | 0 | | Sampled process read traffic, including OS file cache | 117.85 GiB | 117.86 GiB | | Physical filesystem reads | 0.34 MiB | 23.72 MiB | The phase sum uses the CLI's individual second-resolution timings instead of its rounded total in minutes. Process read traffic is sampled every 50 ms from the benchmark process and includes the final whole-index diagnostics and recall. Most reads are served by the operating system's file cache; these figures do not measure a production network workload. Single-sample recall differences and different balancing round counts are not evidence of a correctness or performance regression. The first metadata-block run also completes writing and reports 26M valid unique vectors, zero orphans, and 72.82% recall. Its balancing takes 28.74 seconds. Its observer raises a permission error while reading the exiting process, so that attempt lacks a verified exit/resource summary and is excluded from the clean comparison. The observer now handles that exit race, and the repeated metadata-block run and the bulk run both exit successfully. The benchmark code remains unchanged. The benchmark executables are built directly from the published commits: `bc0faf9ecd8cff6287c95605a7501d73b4371d14` for #7874 and `b5c566540bb29309054e4ff69fbf858444de9e0d` for #7875. Their SHA256 hashes are `112429cfddb816dc5c5f2c507d9d982cdc175535f1385723b805207175385ceb` and `17f94a6cbe636c63794ee2db91a22c5e94c584eccc606e0d93e60a325db75e2e`. The exact command uses checkpoint 26, checkpoint size 1M, 32 add and balance threads, full-precision write navigation and reassignment, a 1 GiB cache, 100 precomputed exact queries, and the approved beam settings. No minimum-percentage level settings are supplied. The frozen source is `/mnt/data/version-cache-source-25m`. The clean saved outputs are `/mnt/data/version-cache-pr7874-26m-verified` and `/mnt/data/version-cache-pr7875-26m-verified`. Full CLI output, commands, hashes, resource summaries, sampled counters, checkpoint metadata, and preservation proof are saved in the local `25m26m` directory and remotely at `/home/ubuntu/benchmark-data/version-cache-25m26m`. There is no new unfixed 25M control run, so this comparison does not measure a percentage improvement against an unfixed tree of the same size. Co-authored-by: lyon-chroma <lyon@trychroma.com>
## Summary Preparing a Wikipedia checkpoint requires decoding its source files before vectors can enter the index. A million-vector checkpoint spans about ten files, but decoding them sequentially leaves independent work waiting. This change decodes up to four local files concurrently, limited further by the machine's available CPU count. The loader first resolves source paths and identifies the physical rows needed from each file. Downloads remain sequential. It decodes files in groups of at most four and combines their results in source order. A single-file group uses the calling thread. Every worker finishes before the loader returns, including when decoding fails. The output retains physical vector IDs, exact float values, null-row handling, and the existing conversion from 64-bit to 32-bit floats. The decoder reads only the requested embedding rows and column. Index algorithms, dataset selection, and benchmark arguments keep their existing behavior. ## Validation Two local correctness tests pass. They cover ordered output across multiple groups, partial-file requests crossing row groups, null rows, both float representations, preservation of negative zero, empty input, worker-count bounds, and decoding failures. The tests compare every output float by its bits between serial and parallel decoding. The complete dataset modules type-check locally using existing compiled dependencies. Changed-file formatting and whitespace checks pass. These checks use direct Rust compiler invocations; a full Cargo benchmark build has not been run. No benchmarks have been run, and the EC2 machine is untouched because another benchmark owns it. Performance improvement and peak-memory impact remain unmeasured. Concurrent decoding adds temporary decoded data from up to four files, and every file is decoded before its vectors are combined into the returned batch. The four-worker cap is a conservative starting point rather than a measured optimum. ## Code reference The Wikipedia loader plans file ranges in `rust/index/benches/datasets/wikipedia.rs`. Bounded decoding is in `rust/index/benches/datasets/wikipedia_shards.rs`. Correctness coverage is in `rust/index/tests/wikipedia_parallel_loading.rs`. Co-authored-by: lyon-chroma <lyon@trychroma.com>
#7876) ## What this changes This PR brings the hierarchical index feature branch up to date with the current main branch and repairs the checks that compile its experimental writer and benchmarks. It merges main into the feature history, so existing feature commits remain intact. The integration preserves both kinds of storage values: hierarchical tree nodes and vector postings from the feature branch, and sparse-vector posting blocks from main. The shared storage dispatch handles both. Dependency resolution starts from main’s lockfile and adds the benchmark dependencies required by the feature. ## CI repairs Main supplies the current test infrastructure, including creating the network configuration directory before starting the local Kubernetes cluster and pinning a working cluster version. A representative previous integration failure occurs during that setup, before any index test runs. The feature’s index harnesses now pass the new block-read concurrency argument to the storage provider. A value of zero retains the previous unlimited-read behavior. Cargo declares runnable benchmarks explicitly, so shared helper source files are not compiled as standalone benchmarks. The legacy quantized benchmark requires the existing optional USearch feature. The public SPANN configuration supports four-bit vector codes without the prototype’s centroid or data rerank options. The legacy nonhierarchical writer uses that public configuration and current code-size and quantization APIs. Its separate benchmark rejects unsupported options explicitly. The hierarchical benchmark’s navigation and rerank arguments remain available. The feature source follows the current formatter and strict lints. Large commit and flush futures are allocated on the heap to keep their caller’s async stack small. Iterator and comparison repairs preserve behavior, including rejecting incomparable floating-point distances. Narrow lint allowances cover intentionally shared helper code and explicit internal helper arguments; the workspace’s strict lint gate remains enabled. ## Validation - `cargo check -p chroma-index --all-targets` passes. - The full CI lint command passes locally with Rust 1.99.0, matching GitHub CI: `cargo clippy --workspace --all-targets --all-features --keep-going -- -D warnings -D clippy::large_futures -D clippy::all`. - The final hierarchical policy, posting-delta, and parallel-loader suites pass all 62 test executions, including the newly merged metadata-block and loader improvements. - All 148 all-feature index library tests pass. The typed-chunk quantization repair passes all eight single-bit quantization tests. - `cargo fmt --all -- --check` and `git diff --check` pass. - [The full GitHub CI run](https://github.com/chroma-core/chroma/actions/runs/37986194829) passes on `562b3de5e07d620ae60fe92f79de00f6b175ad05`, including `all-required-pr-checks-passed`. Linux lint, Rust feature/unit/doc/long-running/integration tests, the release compile gate, JavaScript tests, Go cluster tests, and Python bindings/single-node/distributed/Windows suites pass. ## Revisions and scope The merge integrates main `5f30478c11ffdcf8cba1f872993c29d8f84ce0d9` into hierarchical-spann `22eb704e71490d95cf715bdb8901f7d6c8a3cf34`. The feature sync also preserves its merged metadata-block version cache (#7874) and parallel Wikipedia loader (#7872) at feature head `f3ac04b1ae9df1025cf0a960d5a91d9b451f6d18`. Open performance PRs are excluded. The EC2 benchmark machine and saved Wikipedia checkpoints remain untouched. --------- Co-authored-by: Robert Escriva <robert@trychroma.com> Co-authored-by: Hammad Bashir <HammadB@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: tanujnay112 <tanujnay112@live.com> Co-authored-by: dbeglord <lyon@trychroma.com> Co-authored-by: Gabe Shahbazian <gabeshahbazian@gmail.com> Co-authored-by: Dave Dash <dave@trychroma.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Greyson LaLonde <greyson.r.lalonde@gmail.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: vortex-captain <75063846+vortex-captain@users.noreply.github.com> Co-authored-by: Yi Ren <reny@microsoft.com> Co-authored-by: Rolly Calma <115199279+Ghraven@users.noreply.github.com> Co-authored-by: Shakti <rkfshakti@gmail.com> Co-authored-by: Nimra Khalid <nimra3261@gmail.com> Co-authored-by: Domen Kožar <domen@cachix.org>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
Reviewer ChecklistPlease leverage this checklist to ensure your code review is thorough before approving Testing, Bugs, Errors, Logs, Documentation
System Compatibility
Quality
|
This comment has been minimized.
This comment has been minimized.
## Description of changes The incremental index benchmark reports a build duration that includes work outside its checkpoint phase timers. In the saved 40M Wikipedia run, the rounded phase totals add up to 35.98 minutes while the build summary reports 42.5 minutes. The remaining time needs measurement before it can be attributed to an index operation. This change measures elapsed time using one active phase in the benchmark driver. It reports separate durations for insertion, balancing, loading, commit fork and flush, reopening, checkpoint metadata, memory/statistics diagnostics, posting validation, reader cleanup, writer destruction, input destruction, garbage collection, snapshot retention, output, and outer-loop setup/bookkeeping. Final posting materialization and tree statistics have their own buckets. Every transition closes the previous interval before opening the next. The build summary checks that the durations sum exactly to the same elapsed-time boundary used by the reported build total. The new tables print seconds to six decimal places, so readers can reconcile them without relying on rounded minute values. Existing checkpoint phase logs remain available. Checkpoint timing tables are buffered in memory and printed after the build timer stops, keeping the added report output outside the measured build. The existing insertion and balancing logs use the same completed intervals as the exclusive wall-time table. Progress setup and driver bookkeeping remain separate from insertion. Recall and wall-time report formatting occur after the build boundary. Worker task durations remain concurrency diagnostics. Their timers overlap and nest, so summing navigation, split, and reassignment durations measures neither wall time nor CPU time. The patch also includes the optional snapshot hook used by the saved baseline's separate harness patch, allowing snapshot costs to be measured in this harness. Snapshot retention requires saving and rejects `--no-commit`. The indexing implementation and data formats are unchanged. ## Test plan - The benchmark compiles with `cargo check --offline -p chroma-index --bench hierarchical_spann_profile_quantized`. - Strict benchmark Clippy passes with `-D warnings -D clippy::large_futures -D clippy::all`. - Three accounting tests pass with `rustc --edition 2021 --test rust/index/benches/build_wall_time.rs`: interleaved insertion/balancing partitions elapsed time, checkpoint deltas exclude previous work, and checkpoint reports remain buffered until build completion. - A local synthetic smoke run with two 500-vector checkpoints, commits, GC, and snapshot retention exits successfully. Its reports appear after the build boundary, the insertion durations agree across both outputs, and wall-time reconciliation reports a zero difference. - Formatting and `git diff --check` pass. No production-size benchmark rerun is included. This PR provides instrumentation for the next run; it does not assign the saved run's roughly 6.5-minute remainder to a particular operation. ## Migration plan None. This change affects the benchmark harness only. --------- Co-authored-by: lyon-chroma <lyon@trychroma.com>
…l/smaller search space (#7897) Change from 1bit centroids & 4 bit query -> 4 bit centroids & fp query so that we can have greater navigation accuracy and thus less posting list scanning and reranking ## Summary The hierarchical benchmark reader uses representative vectors, called centroids, to choose branches and leaves. This change prepares four-bit centroid codes when the reader opens and scores them against the original floating-point query with the RaBitQ authors’ packed kernel. The reader builds its immutable scoring cache from saved original centroids, so existing checkpoints need no format migration. Missing centroids or incorrect dimensions return data-loss errors. CPU preparation runs in a blocking task after releasing original centroid block pins. Navigation retrieves child codes with a single map lookup. The packed-code constructor explicitly requires codes quantized relative to zero. The reader selects vector instructions once at open, with AVX-512, AVX2/FMA, and scalar implementations. Posting lists and original vectors for final reranking continue to load on demand. This PR contains the packed FP-query path. The bit-plane and quantized-query alternatives remain on the comparison branch. Persisting four-bit centroid codes is a separate format change. ## Benchmark results The measured comparison uses the saved 40M Wikipedia index, 100 exact L2 queries, 32 workers, a 1 GiB block cache, and three repetitions. Settings differ to compare similar final recall. | Navigation | Leaf beam | FP vectors reranked | Reachable recall | Recall@100 | Warm mean | Cold mean | Cold filesystem reads/query | |---|---:|---:|---:|---:|---:|---:|---:| | One-bit centroids / four-bit query | 256 | 800 | 94.57% | 94.42% | 28.54 ms | 950.69 ms | 724.70 MiB | | Packed four-bit centroids / FP query | 192 | 800 | 94.59% | 94.44% | 24.21 ms | 941.23 ms | 723.55 MiB | | Packed four-bit centroids / FP query | 240 | 400 | 95.40% | 94.51% | 25.28 ms | 528.38 ms | 441.24 MiB | The smaller beam scans 24.9% fewer posting codes and reduces warm latency by 15.2%. The reduced-reranking setting reduces warm latency by 11.4% and filesystem reads by 39.1%. These improvements include the beam and rerank settings above. Preparing the cache takes roughly 2–3 seconds and adds approximately 16 MiB of scoring-cache allocations in the measured comparison. Preparation is excluded from query timings. Cold runs clear reader posting data and block pins; provider and OS caches may retain data. Filesystem reads do not measure production network traffic. Figures come from [the measured experiment revision](https://github.com/chroma-core/chroma/tree/b635eee6f83e56ecf6c8e0c1eada25b7b66b4bd3). This focused PR retains its packed FP-query algorithm and SIMD loops; it has not received a separate EC2 benchmark run. Full results, limitations, and reproduction commands are in [the benchmark report](https://github.com/chroma-core/chroma/blob/codex/packed-fp-navigation/rust/index/benches/PACKED_CENTROID_NAVIGATION.md). ## Validation - Three kernel tests pass, covering reference-estimator agreement, padding, nibble values, constant queries, and cancellation against a double-precision reference. - All 28 reader integration and imported writer tests pass, including existing checkpoints, fixed and adaptive beams, lazy posting loading, and malformed checkpoint errors. - Clippy passes with all targets and features and warnings denied. - Formatting and diff checks pass. - The measured experiment verifies the native x86 SIMD paths on EC2. The port includes source attribution and the RaBitQ Library Apache-2.0 license. --------- Co-authored-by: lyon-chroma <lyon@trychroma.com>
## What changes Reader navigation scores a parent's children to choose which leaves to scan. This PR keeps each parent's child IDs, four-bit centroid codes, and distance correction values together in immutable buffers. Scoring reads one parent entry and walks its codes in child order. This applies the sibling locality approach from [#7838](#7838) to the reader's four-bit centroid / full-precision query path. Reader open prepares the four-bit codes from persisted full-precision centroids, then consumes those individual code allocations into contiguous sibling buffers. The temporary node-to-code map is released after preparation. Queries avoid a tree lookup to copy the parent's child IDs and a hash lookup for every child's code. The tree topology stays fixed after open, so the buffers remain valid when posting lists are loaded or cleared. Child IDs and codes are derived from the same list of pairs behind private fields, and callers use one scoring iterator. Opening rejects missing child nodes and duplicate child references with errors naming the affected child and parents. The scoring kernel, child order, beam selection, and persisted checkpoint format remain unchanged. The root's own prepared code is discarded because navigation scores only children. Memory diagnostics count the batched numeric payload and its copied child IDs, while excluding container and allocator overhead as before. Reader open performs an additional packing pass. The measurements below show lower warm navigation and query latency, with the same final recall on an identical checkpoint. ## Verification - All 29 tests in `cargo test -p chroma-index --test hierarchical_reader_packed_centroids` pass. They include reader navigation equivalence, lazy posting loading and clearing, reopen equivalence, missing or malformed centroid errors, and duplicate references within or across parents. - All four tests in `cargo test -p chroma-index --lib packed_centroid` pass. The new batch test verifies exact distance and ordering equivalence for all three distance metrics, empty and singleton batches, multiple siblings, and padded or full dimensions up to 1024. - `cargo clippy -p chroma-index --all-targets --all-features -- -D warnings` passes. - Formatting and `git diff --check` pass. ## Benchmark: Wikipedia 39M to 40M The comparison uses two sequential 39M-to-40M runs per revision on lyon (`r6id.16xlarge`), starting from independent writable copies of the saved 39M checkpoint. Both use 1024-dimensional Wikipedia vectors, 32 add/balance/recall threads, a 1 GiB block cache, and 100 exact L2 queries at 40M. The baseline is `083ef95d6d3474f76cd703246294ae28c5f50368`; the candidate is `faaea93a67536fcc5fbf6178164b1a743d047649`. The high-recall row uses tau 2, maximum beam 256, and rerank factor 8. | Metric, mean of two runs | Baseline | Contiguous reader | | --- | ---: | ---: | | Warm navigation | 5.30 ms | 2.20 ms (58.5% lower) | | Warm query | 29.30 ms | 25.85 ms (11.8% lower) | | Final recall@100 | 95.365% | 95.365% | | Cold query | 832.55 ms | 814.35 ms | | Reader open after indexing | 653 ms | 732.5 ms | | Full checkpoint wall time | 92.81 s | 93.75 s | A separate pair on the exact same final 40M checkpoint reports 95.37% recall and identical displayed scan counts in all 12 corresponding query rows. At tau 2 / rerank 8, warm navigation falls from 5.1 to 2.3 ms and total warm query latency falls from 28.4 to 25.5 ms. Reader open takes 276 and 272 ms in this separate pair, so the roughly 80 ms increase after indexing does not establish a consistent open-time regression. Cold-latency differences are smaller than the observed baseline variation; the evidence supports a warm CPU benefit. The beam, posting scans, and reranking work remain the same. The CLI's per-level coverage columns use the writer's one-bit diagnostic path and are excluded from conclusions about the four-bit reader. Cold clears reader postings and block pins, while provider and OS caches remain; these local NVMe measurements do not measure production network latency. The report with exact arguments, all warm rows, phase timings, and limitations is in [`rust/index/benches/READER_CONTIGUOUS_CENTROIDS.md`](rust/index/benches/READER_CONTIGUOUS_CENTROIDS.md). Full CLI logs, commands, binary hashes, and source-preservation checks are saved on the node under `/mnt/data/reader-contiguous-7903/results/`. The original 40M checkpoints are preserved. Both temporary benchmark source trees have an identical ground-truth path override so the node's shared query cache remains unchanged. ## Code references The sibling scoring buffer lives in `rust/index/src/quantization/packed_centroid.rs`. Reader preparation, navigation, and memory accounting live in `rust/index/benches/hierarchical_index/reader/`. --------- Co-authored-by: lyon-chroma <lyon@trychroma.com>
## What this changes Read performance can be measured directly from a saved hierarchical index. The benchmark opens its persisted tree metadata and runs the existing query sweep, without downloading the source corpus, inserting vectors, balancing, committing, collecting garbage, or scanning all postings for build diagnostics. The new `--query-only` mode requires `--save-dir` and `--ground-truth`. The ground-truth Parquet file must contain answers for the checkpoint's exact vector count, and its query dimensions must match the index. This prevents recall from being measured against a larger corpus. Write/resume options and build-only diagnostics are rejected. Query concurrency uses `--threads`; the existing read beam, tau, rerank, lazy recall, and cache flags remain available. The sweep reports end-to-end recall, cold/warm latency, phase timings, and scanned-vector counts. Per-level reachable recall and the optimal-leaf baseline require full posting scans, so query-only mode skips them. Their diagnostic columns and total-byte estimates are omitted or marked `-`. The normal build-and-query path retains those diagnostics. Usage and the input schema are documented in `rust/index/benches/docs/query-only.md`. A complete Wikipedia checkpoint invocation is included there. The metric must match the checkpoint's build metric, which its metadata does not record. The saved directory must remain stable while it is being queried. ## Validation - Four targeted tests pass under Rust 1.99. The saved-index test runs cold/warm queries with rerank factors 1 and 8, compares every checkpoint file byte-for-byte, and rejects mismatched vector counts and dimensions. Input tests cover exact checkpoint selection and CLI conflicts. - The benchmark and integration test pass strict Clippy with `-D warnings -D clippy::large_futures -D clippy::all` under Rust 1.99. - `cargo fmt --all -- --check` and `git diff --check` pass. - Validation runs locally against a 200-vector saved fixture; no existing EC2 benchmark or checkpoint is changed. Co-authored-by: lyon-chroma <lyon@trychroma.com>
Contributor
|
Found 2 test failures on Blacksmith runners: Failures
|
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light.svg)
## Summary The hierarchical index assigns each vector to one leaf. Replication does not justify its extra postings and selection machinery in our experiments, so this change makes single-leaf assignment the only behavior. - Adds, buffered adds, and reassignments choose the nearest candidate leaf. They no longer build lists of replica destinations or apply replication filters. - Reads select and sort scored vectors directly. The replica deduplication map is removed, and its timing field is named for sorting. - The benchmark removes `--max-replicas`, `--write-rng-epsilon`, and `--write-rng-factor`, along with their configuration fields, output, and cache-directory components. - Tree diagnostics retain valid-entry and orphan counts. They remove replica distributions, replica distances, and the per-vector maps used to compute them. Version checks and safe posting transfers remain necessary to keep vectors reachable during concurrent balancing. The changes are confined to the hierarchical prototype; the separate production SPANN implementations retain their configuration and behavior. ## Compatibility The checkpoint format stays the same. Existing single-copy checkpoints can be opened with an explicit `--save-dir`; the generated default directory name changes because it no longer includes replication settings. Existing commands must drop the three removed flags. Checkpoints containing multiple live copies of a vector must be rebuilt before use with this reader. This PR removes support for those indexes rather than migrating them. ## Validation The tests cover single-leaf assignment during direct and buffered adds, reassignment version publication, concurrent splits and merges, checkpoint persistence, and saved-index reads. All four test suites pass (135 test executions; the suites share some module tests): ```sh rustup run 1.99.0 cargo test -p chroma-index --test hierarchical_policy_cache --test hierarchical_posting_delta --test hierarchical_reader_packed_centroids --test hierarchical_query_only rustup run 1.99.0 cargo fmt --all -- --check git diff --check ``` No performance benchmark is run for this removal. The add and read algorithms retain the single-copy behavior selected by the previous default configuration. Co-authored-by: lyon-chroma <lyon@trychroma.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of changes
Summarize the changes made by this PR.
Test plan
How are these changes tested?
pytestfor python,yarn testfor js,cargo testfor rustMigration plan
Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?
Observability plan
What is the plan to instrument and monitor this change?
Documentation Changes
Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?