Any Store

A document-oriented embedded database for Go with a MongoDB-like query language, built on an in-tree btree/pager/WAL storage engine derived from SQLite (aligned with its design, not a literal port — the on-disk format is our own). Schema-less documents, rich indexes, full-text and vector search, ACID transactions, multi-process access, page-level integrity and optional encryption — pure Go, no CGO.

Features

• Mongo-style queries — $eq/$in/$gt/..., logical operators, dot-path fields, modifiers ($set, $inc, ...). Formal semantics in docs/query-filter-contract.md.
• Indexes — compound, unique, sparse, multikey (arrays), asc/desc per field; created and dropped at runtime.
• Cost-based planner — index selection driven by btree page statistics and per-prefix selectivity sketches; Explain() shows the chosen plan and its candidates.
• Full-text search — btree-resident inverted index with BM25 ranking via $text: phrases, prefixes, anchored and per-field matching for search-as-you-type. See docs/full-text-search.md.
• Vector search — btree-resident ANN indexes (IVF-SQ, HNSW, brute-force) via $knn. See docs/vector-search.md and docs/vector-engine.md.
• Aggregation — MongoDB-style pipelines: shaping ($match, $group, $sort, $unwind, $facet, ...), computed expressions (arithmetic, conditionals, comparisons, date math), field-to-field $expr predicates, primary-key $lookup joins, and materialization via $merge/$out — with planner pushdown. See docs/aggregation.md.
• ACID transactions — snapshot-isolated read transactions, single-writer write transactions.
• Multi-process — SQLite-like contract: any number of OS processes may open, read and write the same database file at any time (WAL + shared-memory index, busy handling, cross-process DDL reconciliation).
• Durability — idle auto-flush, explicit checkpointing, crash-safe WAL recovery, sentinel-triggered quick check after unclean shutdown.
• Integrity & encryption — per-page XXH3-128 checksums by default; optional AES-GCM or (X)ChaCha20-Poly1305 (AEAD doubles as integrity).
• Streaming iterators — low-memory scans with a cursor API; AnyEnc arena encoding minimizes GC churn.
• Cross-platform — pure Go, no CGO; runs anywhere Go runs, including js/wasm (see wasm/).
• CLI — inspection, import/export and interactive shell.

Quick start

Install

go get github.com/anyproto/any-store/v2

CLI (optional):

go install github.com/anyproto/any-store/cmd/any-store-cli2/v2@latest

Hello, Any Store

package main

import (
    "context"
    "fmt"
    "log"

    anystore "github.com/anyproto/any-store/v2"
    "github.com/anyproto/any-store/v2/anyenc"
)

func main() {
    ctx := context.Background()

    db, err := anystore.Open(ctx, "/tmp/demo.db", nil)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    users, _ := db.Collection(ctx, "users")

    _ = users.Insert(ctx,
        anyenc.MustParseJson(`{"id": 1, "name": "John"}`),
        anyenc.MustParseJson(`{"id": 2, "name": "Jane"}`),
    )

    res, _ := users.Find(`{"id": {"$in": [1,2]}}`).Sort("-name").Iter(ctx)
    for res.Next() {
        doc, _ := res.Doc()
        fmt.Println(doc.Value().String())
    }

    // Storage footprint: doc count, sizes, compression and per-index stats.
    st, _ := users.Stats(ctx)
    fmt.Printf("docs=%d total=%d bytes ratio=%.2fx\n",
        st.DocCount, st.TotalSizeBytes, st.CompressionRatio)
}

The full end-to-end example lives in example/ and the API docs.

Choosing document ids. Unlike SQLite there is no hidden rowid — the document id is the btree key. Increment-like ids (anyenc.NewObjectID(), timestamps, sequences) keep inserts append-ordered and reads cache-friendly; fully random ids (e.g. UUIDv4) scatter them. In practice this only matters for very hot write/read paths or huge collections.

Transactions

tx, _ := db.WriteTx(ctx)
_ = users.Insert(tx.Context(), anyenc.MustParseJson(`{"id": 3}`))
_, _ = users.Find(`{"id": 3}`).Update(tx.Context(), `{"$set": {"seen": true}}`)
if err := tx.Commit(); err != nil { // or tx.Rollback()
    log.Fatal(err)
}

Any operation run with tx.Context() joins the transaction. Read transactions (db.ReadTx) pin a consistent snapshot; writers never block readers. One write transaction is active at a time — across all processes.

A transaction belongs to one goroutine at a time: the calls made on it — operations run with its context, the methods of its savepoints and iterators, Commit and Rollback — must not run at once from two goroutines. Calls nest: from inside the operation that runs it, a query.Modifier may read through the transaction's context, open collections, and write other collections (a write made with the context opens a savepoint on the transaction; one made with a context that carries no transaction waits for the writer lock the transaction holds until the database closes). What that operation holds, the modifier must not pull away or layer over: a write made with the context to, or a schema change of, the collection whose modifier is running fails with ErrWriteInModifier, a Commit or Rollback of the transaction or of a savepoint fails with ErrTxEndInModifier, and a savepoint opened with the context (WriteTx) fails with ErrSavepointInModifier — a modifier that needs its work undone returns an error, which fails the operation and rolls back everything the operation and the modifier wrote. A call made while another goroutine's call is in progress panics with ErrTxConcurrentCalls, detected the way the runtime detects concurrent map writes: best effort — the overlap is seen, the goroutine is not, so a call that lands inside a running modifier passes for a nested one — and the transaction stays usable by the call that was in progress; Commit and Rollback wait that call out instead of refusing, so a deferred Rollback still ends the transaction. A query.Filter or query.Sort implementation and Config.OnIntegrityError run inside the call that evaluates them and must not use the transaction. Once the transaction has ended, an operation run with its context fails with ErrTxIsUsed, and so do the Next and Doc of an iterator still open on it; a second Commit or Rollback returns nil, as does the iterator's Close. The Rollback of a savepoint ends the iterators opened or moved while it was open — inside it, or inside a savepoint nested in it: their Next and Doc fail with ErrIterRolledBack, their Close returns nil, and the transaction goes on; an iterator that last moved before the savepoint opened continues. A write that fails rolls back a savepoint of its own and ends the iterators moved inside it the same way: one a query.Modifier advanced while the write ran. A write to the collection an iterator of the same transaction is scanning never corrupts the store. A scan the plan drives with a cursor — a full scan, an index scan — goes on from the key it stood on, as a cursor of the same SQLite connection does: the current document deleted, the scan goes on with its successor; a document inserted ahead of the position is visited, one inserted behind it is not; an update that moves a document within the scan's order — an indexed field under an index scan — can make the scan visit it again or miss it. A plan that collects its result before yielding it — a sort run in memory, a $knn probe (KnnProbe in Explain) — yields what it collected: a document inserted since is not visited, one deleted since fails its Doc with ErrDocNotFound, and an aggregation over such a plan ends there with that error. The $knn search over the vector index (KnnSearch) collects its candidates the same way and skips one deleted since: it yields the first k that remain, fewer once they run out. Doc returns the document as Next read it. To change every matched document once, use Query.Update and Query.Delete, which collect the ids before they write. Drop, DropIndex and CompactVectorIndex fail with ErrIterOpen while an iterator of the transaction is open on the collection, advanced or not, until its Close: they free the trees the iterator reads. SQLite refuses a DROP while any other statement of the connection is running; here an iterator on another collection does not count.

Indexes

_ = users.EnsureIndex(ctx,
    anystore.IndexInfo{Fields: []string{"name", "-createdDate"}},          // compound, mixed order
    anystore.IndexInfo{Fields: []string{"email"}, Unique: true},           // unique
    anystore.IndexInfo{Fields: []string{"nick"}, Sparse: true},            // skips docs missing the field
)

exp, _ := users.Find(`{"name": "Jane"}`).Explain(ctx)
fmt.Println(exp.Plan) // chosen index, cost breakdown, rejected candidates

Array fields index every element (multikey), and a dotted path through an array of objects ("items.sku" over {"items":[{"sku":1},{"sku":2}]}) indexes one entry per element — the same MongoDB field-path semantics filters and sorts use; $elemMatch binds several predicates to one element. The cost-based planner picks the cheapest index per query; IndexHint overrides it.

Every range index this release creates records the format it was built under. When a release changes how an index derives its entries, opening the database with that release rebuilds the indexes the change affects, one write transaction per index, before Open returns; unaffected indexes are left as they are. The rebuild takes as long as creating those indexes would and blocks Open for that time; a crash in between keeps the indexes already rebuilt. Once nothing is left to rebuild the file records it, and later opens skip the catalog check; a schema change committed by a release without this record makes the next open check again. An index the rebuild cannot produce from the data (a unique index whose documents the new format finds duplicate: explicit nulls under a sparse index, elements of an array under a dotted path; or a document this release rejects) is left as it was: Stats and Explain report it outdated, the planner never uses it, and writes keep maintaining it with entries of the new format, so a unique one checks new writes against a mixed set and can reject an insert or update it should accept. Drop and recreate it once the data allows; EnsureIndex with the same definition leaves it alone.

Once a release has opened a file, older releases must not write documents to it: they do not stamp the entries they write, so entries in the older format would sit under a current stamp that nothing re-checks. An index an older release creates is rebuilt at the next open. An older release opening the file reads it (the stamp is ignored) and, if its own format differs, rebuilds every range index in its own format at open.

Full-text search

_ = docs.EnsureIndex(ctx, anystore.IndexInfo{
    Kind:   anystore.IndexKindFulltext,
    Fields: []string{"title", "body"},
})

res, _ := docs.Find(`{"$text": {"$search": "wal checkpoint"}}`).Limit(10).Iter(ctx)

BM25-ranked, transactional (a $text write is visible to queries in the same tx), combinable with any other filter, sort and limit.

Vector search

_ = docs.EnsureIndex(ctx, anystore.IndexInfo{
    Kind: anystore.IndexKindVector,
    Vector: &anystore.VectorParams{
        Field:  "embedding",
        Dim:    768,
        Metric: anystore.VectorCosine, // or VectorL2 (VectorDot: HNSW/brute-force modes only)
        Mode:   anystore.VectorModeIVFSQ, // or BTree/Hybrid (HNSW), BruteForce
    },
})

res, _ := docs.Find(`{"embedding": {"$knn": {"$query": [0.1, 0.2, ...], "$k": 10}}}`).Iter(ctx)

ANN results stream in distance order and compose with filters; $ef tunes recall per query, int8 quantization and IVF cell counts are per-index options. Large deployments run IVF-SQ; the API default is VectorModeBTree.

Aggregation

res, _ := orders.Aggregate(`[
    {"$match": {"status": "paid"}},
    {"$group": {"_id": "$region", "total": {"$sum": "$amount"}}},
    {"$sort": {"total": -1}}
]`).Iter(ctx)

$match/$sort/$limit prefixes are pushed down into the index planner; the rest streams through the pipeline. Expressions compute derived values in $project/$addFields/$group (arithmetic, $cond/$switch, comparisons, $dateDiff-family date math over the first-class dateTime type); $expr brings field-to-field predicates into $match; $lookup resolves object-id relations as primary-key point lookups; $facet runs N sub-pipelines over one scan; $merge/$out materialize results into a collection where they are indexable like any other data. Operator reference and Mongo divergences: docs/aggregation.md.

Multi-process

The database file behaves like SQLite in WAL mode: any process may open it at any moment and read or write concurrently. Coordination runs through file locks and a shared-memory WAL index; snapshot reads are never blocked, writers serialize, and busy situations back off through a configurable handler. DDL from a peer process (created/dropped collections and indexes) is reconciled automatically on the next transaction.

Durability

Any Store performs WAL checkpoints and fsync after idle periods.

db, _ := anystore.Open(ctx, "data.db", &anystore.Config{
    Durability: anystore.DurabilityConfig{
        AutoFlush: true,
        IdleAfter: 20 * time.Second,  // flush after 20s of inactivity
        FlushMode: anystore.FlushModeCheckpointPassive, // ...Full, ...Restart, ...Truncate
        Sentinel:  true,  // track dirty state, quick-check on next open
    },
})

// Manual flush, e.g. before app suspension (waits up to 100ms for pending writes to settle).
db.Flush(ctx, 100*time.Millisecond, anystore.FlushModeCheckpointPassive)

Sentinel: when enabled, a .lock file marks not-explicitly-persisted writes so an unclean shutdown triggers an integrity quick check on open.

Lifetime: the idle flush runs until Close. It is not bound to the context passed to Open, which only bounds the open itself; a DB that is never closed keeps the flush goroutine alive.

Integrity

Every non-encrypted database carries an XXH3-128 page-trailer checksum (16 bytes/page) by default — corruption is caught on read. There is no opt-out; the cost is <1% on writes and effectively zero on reads. Encrypted databases derive integrity from the cipher's AEAD tag instead. File state is authoritative on reopen — existing plain databases stay plain, existing checksum databases auto-install the codec regardless of caller config.

Conceptually mirrors SQLite's cksumvfs, generalized to also surface AEAD failures via the same API.

db, _ := anystore.Open(ctx, "data.db", &anystore.Config{
    // Wire monitoring at Open time so failures during the first page-1
    // read (inside Open) are observable.
    OnIntegrityError: func(e anystore.IntegrityError) {
        log.Printf("integrity: page %d %v: %v", e.PageNo, e.Kind, e.Inner)
    },
    // Default false: corrupt pages fail reads. True is for forensic dumps.
    ContinueOnIntegrityError: false,
})

// Walk every page and report mismatches (works in encrypted mode too).
rep, _ := db.VerifyIntegrity(ctx)
fmt.Printf("scanned %d pages, %d errors\n", rep.Pages, len(rep.Errors))

The page-1 DB header (first 100 bytes) is outside the per-page hash; its invariants are validated separately at open. Full design: docs/btree/specs/integrity.md.

Encryption

db, _ := anystore.Open(ctx, "data.db", &anystore.Config{
    Encryption: anystore.EncryptionConfig{
        Passphrase: []byte("correct horse battery staple"),
        CipherType: anystore.CipherXChaCha20Poly1305, // or CipherAES256GCM, CipherChaCha20Poly1305
    },
})

Whole-file page-level AEAD; bring-your-own key management via EncryptionConfig.Codec.

An independent Go implementation following design conventions from SQLCipher (BSD-3-Clause) — reserved-tail page overhead, the page-1 plaintext-prefix rule, PBKDF2-HMAC-SHA256 key derivation. No SQLCipher code is included; the cryptography is Go-native and uses AEAD rather than SQLCipher's encrypt-then-MAC construction. Not SQLCipher-compatible on disk. See NOTICE.md.

Documentation

• API reference — pkg.go.dev/github.com/anyproto/any-store/v2
• Query semantics — docs/query-filter-contract.md
• Full-text search — docs/full-text-search.md
• Vector search — docs/vector-search.md, docs/vector-engine.md
• Aggregation — docs/aggregation.md
• CLI manual — any-store-cli2 --help

Contributing

1. Fork & clone
2. make test — the whole suite; make cover for library coverage
3. Create your feature branch
4. Open a PR and sign the CLA

Unit tests live beside the code they cover (db.go + db_test.go). The behaviour suites that drive the library through its exported API — query semantics, planner, index lifecycle, vector and full-text — live in test/. Long-running crash, multiprocess and fuzz harnesses run in a separate internal repo; a pull request does not need those green locally.

Please read our Code of Conduct before contributing.

⚖️ License

Any Store is released under the MIT License — see LICENSE.md.

The storage engine derives from SQLite (public domain) and follows page-encryption design conventions from SQLCipher (BSD-3-Clause). Neither project's source is included. Attribution and scope: NOTICE.md.