Repository navigation
Conversation
…UUIDs and scores, and refuse invalid inputs (MemMachine#1733) * Look up the segments that derivatives belong to in the segment store Event memory resolves a search hit, a derivative's vector record, to the derivative's segment through a `_segment_uuid` property it copies onto every vector record. The segment store already holds that mapping, on the derivative's own row. `SegmentStorePartition.get_segment_uuids_by_derivative_uuids` reads it. The SQLAlchemy partition answers with one query on the derivative table, served by its primary key, and checks the partition's liveness in the same statement, as its other reads do. A derivative the partition does not hold is left out of the result. The in-memory partition the event memory tests use implements it too. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Resolve event memory's search hits through the segment store A derivative's vector record carried its segment's UUID as the `_segment_uuid` property, and a search read it back from each match. That copy is only as fresh as the vector store's reads, and the segment store holds the same mapping on the derivative's row. A search now asks the vector store for no properties and maps the matched derivatives to their segments with one `get_segment_uuids_by_derivative_uuids` call. A match whose derivative the segment store no longer holds, because its segment was deleted and its vector outlived it, is dropped. Vector records no longer carry `_segment_uuid`, and the schema event memory expects of its collection no longer declares it; a collection created with it keeps declaring it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Resolve semantic search hits through a vector_uuid column on the feature row A feature's vector record was keyed by a UUID derived from the feature's id, and carried eleven properties copied from the feature's row, among them `feature_id`, which a search read back to resolve a hit to its row. The caller's feature metadata was merged into the same properties. - `update_feature` read the stored vector back through `VectorStoreCollection.get` to write the record again with fresh properties, since an upsert replaces a whole record. The value read is written back, so a stale read becomes a lasting wrong write: of two concurrent updates of one feature, one with a new embedding and one without, the second can write the old embedding over the new one; and a read that misses the record fails the update after its row was committed (MemMachine#1721). - Nothing filtered on the copied properties; the row is their authority. The feature row now carries `vector_uuid`, a fresh UUID that keys its vector record, and the record carries no properties. A search resolves its hits through that column, in the order the search returned them, dropping a hit whose row is gone. `update_feature` writes the vector store only when given a new embedding, and reads nothing back. The delete paths take the UUIDs of exactly the rows they delete with RETURNING. The collection declares no indexed properties. Breaking: a feature table created before this has no `vector_uuid` column, and the table is created with `create_all`, which adds none; and a collection created with the old declared properties no longer matches the configuration `open_or_create_collection` asks for. No migration is included. Fixes MemMachine#1721. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Answer a vector store query with record UUIDs and scores; remove get and close_collection A query returned each match's record: its properties, and its vector on request. They were copies of what the callers' own stores hold, only as fresh as the vector store's reads, and since the previous two commits no caller reads them. `get` has had no caller since semantic memory stopped reading its vectors back, and a `get` safe to write from would need every backend to reflect every write that returned before it, from any process. Every store's `close_collection` did nothing, and nothing called it. - `QueryMatch` carries a score and a `record_uuid`, and `query` loses `return_vector` and `return_properties`. Properties are still stored and filtered on. - `VectorStoreCollection.get` and `VectorStore.close_collection` go, with what only served them: the stores' record parsing, `VectorSearchEngine.get_vectors`, and sqlite-vec's vector decoding. - `Record` is input-only, so its vector is required and its properties default to `{}`. The model rejects a missing vector at construction, so the stores' checks for one and their coalescing of `None` properties go. Mechanical: event memory and semantic memory read `match.record_uuid` in place of `match.record.uuid` and stop passing the flags, and the in-memory test collection follows the interface. Tests of the return flags, of `get`, and of values read back through a query go. Tests that check what a store holds read the backend past the store: the SQLite stores' records tables, a Qdrant scroll of the collection's partition, a Milvus `get` on the client. The tests of a store refusing a `None` vector become tests that the model refuses a missing one, plus one that properties default to `{}` and one that a record without properties is stored. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Refuse vector store inputs no collection can hold or search Each store refuses, with a ValueError and before any backend call: - a record whose declared property holds a value of another type, since the collection indexes and filters the property as its declared type (`require_declared_types`: the value's type is the declared type, and a bool is not an int); - a record vector or a query vector whose width is not the collection's dimensions; - a query vector with a coordinate that is not finite, or a score threshold that is not finite. The model checks a record's own vector for finiteness: `Record.vector` is `list[FiniteFloat]`, so pydantic refuses a NaN or infinite coordinate when the record is built, in the pass that already validates each coordinate. A query vector is a plain sequence, so the stores check it. An embedding endpoint can return NaN whatever the caller does, so both kinds of vector are checked. Qdrant and Milvus check a filter's property keys with the other inputs, before the early return for no query vectors. `validate_identifier` uses `fullmatch`: `$` also matches before a trailing newline, so `"name\n"` passed as a namespace, a collection name or a property key. The ABC states these refusals in `upsert` and `query`. Tests: each store refuses each input, and a record refuses a coordinate that is not finite. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * State the vector store's consistency in its contract The contract said nothing about what a query sees of earlier writes. `VectorStoreCollection` now states what every store keeps: An `upsert` or `delete` is durable once it returns; queries may not reflect it right away. A store that guarantees more states it. Neither event memory nor semantic memory reads its own writes back through the vector store: a search's hits resolve through the segment store or the feature row, which hold the mapping. Milvus reads at the consistency level its collections are configured with, and a replicated Qdrant may answer a query from a replica that has not applied a write yet, so a stronger promise would bind every store to read settings with costs of their own. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Name the derivative-to-segment map segments_by_derivatives Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Drop a None check no property reaches, and a filter branch no caller takes - The Qdrant store's payload skipped a None property, but a Record's properties are PropertyValues, which exclude None, so the model refuses one before the store sees it. - Semantic storage's _apply_feature_filter took a Select or a Delete, but only selects reach it: delete_feature_set filters its DELETE ... RETURNING itself. It takes and answers a Select, and the cast at its caller goes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Tidy the tests the review found out of place - The SQLite stores' four input-validation tests sat inside TestFilters, under a "Delete" banner meant for TestDelete; they move into TestInputValidation, and the banner moves above TestDelete. - Two event memory schemas still declared `_segment_uuid`, which event memory no longer reserves: the context test declares `_timestamp` alone, and the missing-base-field test declares nothing. - The 40,000-feature test's comment says which SQLite the bind limit is the feature store's. Tests only; the same tests run and pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Fold the declared-type check into the vector store utilities declared_properties.py held one function beside utils.py, which holds the other input checks every store runs (dimensions, query vector, score threshold, identifiers). It imports only common.data_types, which imports nothing back, so it moves into utils.py as is. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Resolve semantic search hits in one statement Since queries answer record UUIDs alone, a semantic search mapped its hits to feature ids in one select, then loaded the features by id with the filter in another: two statements in two sessions, and the first ran even when the search found nothing. One select on the vector_uuid column with the filter now loads the features, ordered as the hits were, and no select runs for a search with no hits. Results are unchanged: the same features, in the same order, under the same filter. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Refuse a query limit that is not positive The contract said nothing of a limit at or below zero, and the stores disagreed: the SQLite stores and Milvus answered empty results, and Qdrant passed the limit on for the server to reject. A limit that is not positive is now a ValueError in every store, stated in the query contract beside the other refused inputs, and checked whether or not there are query vectors. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Refuse a search for no memories before it does any work A search whose top_k is not positive asks for nothing, yet it reached the vector store, which refuses a limit that is not positive, only after event memory had embedded the query: a request for nothing still cost an embedding call. SearchMemoriesSpec.top_k is now positive (gt=0), so request validation refuses it with a 422 naming top_k, the MCP tool included, and LongTermMemory.search_scored refuses a num_episodes_limit that is not positive before the query is embedded, for callers of the Python API. The event backend's window test no longer runs a limit of 0 on the in-memory vector fake, which does not check limits; its lower bound is now exercised with a negative expand_context. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Show the positive top_k in the OpenAPI document docs/openapi.json, as docs/tools/generate_openapi.py writes it for the search specification's top_k: exclusiveMinimum 0 and the description. The generator's other differences under the locked FastAPI are left to the regeneration that carries them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Rebased onto main. Main refuses a top_k that is not positive since MemMachine#1740, with ge=1 and its own tests, so this commit keeps that bound and those tests in place of the gt=0 bound and the test described above, and docs/openapi.json shows minimum 1. vector_store_semantic_storage.py keeps importing datetime, which the history methods MemMachine#1707 added use.
…SQL registry, with an incarnation per collection life and a purge (MemMachine#1734) * State a collection's lifetime in the vector store contract The contract had one process manage a collection, with the consumer sharding names across processes, and said nothing of what deleting a collection means for the handles and writes that outlive it. It now states what a store guarantees across a deletion, and leaves which processes may share a collection to each store: - A collection created again under a deleted one's (namespace, name) starts empty, and reclaiming the deleted one's storage leaves it untouched. - A handle whose collection was deleted may raise `VectorStoreCollectionHandleStaleError`, or, on a store that cannot tell, act on a collection created again under the same name. - `delete_collection` makes the collection unreachable when it returns. A store that reclaims storage later does so in `purge_deleted_collections`, which does a bounded amount of work per call, returns whether it ran a round, and is safe to call from several processes at once. - Creating a collection that is being created raises the already-exists error; opening one whose creation has not completed raises `VectorStoreCollectionPendingError`; a store that gives up after repeated attempts that made no progress raises `VectorStoreAttemptsExhaustedError`. - A record's UUID names it in its collection only. Every store keeps reclaiming storage in `delete_collection`, so `purge_deleted_collections` returns False on each, and none raises the pending or stale-handle error yet. Each store states which processes may share a collection: the SQLite store's engine lives in the process that opened the collection, sqlite-vec's database file in one host, and the Qdrant and Milvus stores serialize creating and deleting a collection within a process only. On each, a handle used after its collection is deleted acts on a collection created again under its name. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Add a SQL collection registry for vector stores whose backend cannot arbitrate one Qdrant and Milvus hold only points: the stores keep their catalogs of logical collections as points and entities, written without a transaction, and serialize creating and deleting a collection with locks in one process. Two processes creating or deleting one collection race, and a write in flight when its collection is deleted can land after the deletion, in a collection created again under the same name. `VectorStoreCollectionRegistry` is that catalog, arbitrated across every process that shares it: - `register` mints a fresh incarnation, the value a collection's records carry, under a (namespace, name) no live or pending collection holds. The collection is pending until `mark_live` marks its storage prepared. - `unregister` makes a collection unreachable when it returns and queues its incarnation as a tombstone; `unregister_incarnation` does the same for a caller holding the incarnation, and leaves a collection registered since under the same name. - `claim_purgeable_incarnation` hands one due tombstone to a purge round. A tombstone comes due once a retention, longer than any write can be in flight, has passed since the deletion, and is removed when a round finds no records. Its incarnation is not minted again before then, so a collection created again starts empty and no purge reclaims its records. `SQLAlchemyVectorStoreCollectionRegistry` keeps it in PostgreSQL or SQLite (3.35 or later, for RETURNING): a table of collections keyed by vector store, namespace and name, and a queue of tombstones claimed in the order they come due on the database clock. The primary key arbitrates registration, a conditional update marks a collection live, unregistration is one transaction, and on PostgreSQL a claim is a row lock. A round that raises backs its tombstone off, doubling up to a bound; after ten consecutive failed rounds the tombstone is dead-lettered: kept, its incarnation reserved, skipped by claims, and reported in an error log. No store uses it yet. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Add RegistryBackedVectorStore, a base for stores whose collections a registry arbitrates A store on the collection registry keeps the same lifecycle whatever its backend, so the lifecycle lives in a base and a subclass supplies the backend calls. `RegistryBackedVectorStore` registers a collection pending, prepares its storage (`_prepare_storage`), and marks it live; only a live collection opens, and opening a pending one raises `VectorStoreCollectionPendingError`. A collection deleted while its storage is prepared is not marked live, so its creation counts as one followed by a deletion. A preparation that raises or is cancelled unregisters the incarnation, shielded so that a cancelled creation still frees the name; a collection the registry cannot unregister stays pending until it is deleted. `open_or_create_collection` retries a bounded number of times, a second apart: it opens a pending collection once another creator marks it live, creates again when it loses the name or the mark, and raises the pending error or `VectorStoreAttemptsExhaustedError` when the attempts run out. `delete_collection` unregisters, and each `purge_deleted_collections` call runs one round (`_purge_round`) on a due tombstone. `RegistryBackedVectorStoreCollection` is a handle bound to one incarnation. Each operation checks its inputs, then that the collection registered under its name still carries that incarnation: `upsert` before and after its backend call, `query` before, and `delete` after. A write that raced the deletion raises `VectorStoreCollectionHandleStaleError` instead of reporting success, and the purge reclaims whatever it landed. A subclass implements `_upsert`, `_query` and `_delete`. `require_identifiers` checks a namespace and a name together. The tests drive the lifecycle through a store whose storage preparation each test controls, on a SQLite registry. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Run each vector store's purge from the resource manager A store that reclaims a deleted collection's storage later does so only when `purge_deleted_collections` is called. The resource manager starts one sweeper per vector store the first time it hands the store out, as it does for segment stores. A sweeper calls the purge again after a short pause while rounds keep running, after the idle interval once nothing is due, and logs a call that raises and retries it on the next tick. Sweepers in other processes may run the same store's purge at the same time, which the contract allows. Closing the manager cancels its sweepers, and a sweeper holds no reference to the manager. Every store returns False from the purge as of this commit, so a sweeper idles. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Open the event backend's collection a racing creator won The event backend's service locator opens a session's collection, or creates it when there is none and opens it after. Two callers building one session's memory at once can both find none: the loser's create raises the already-exists error, and an open while the winner is still preparing the collection raises the pending error. Either failed the build. The locator now retries open-or-create on either error, up to ten attempts a second apart: it opens the winner's collection once it is live, and creates the collection itself if the winner's creation was undone. A collection that stays pending through every attempt fails the build with a RuntimeError naming the partition. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Let the segment store's callers decide to mint again, not the insert The collection registry follows the segment store's incarnation design, with its retry decision in `register`, which counts the attempts. The segment store had that decision in its callee: `_insert_partition_row` logged "re-minting" and "retrying with a fresh incarnation", and its error was documented as "retry with a fresh incarnation". The insert now raises the error saying what happened (the incarnation awaits purge, or the insert failed with no row under the key), and `create_partition` and `_open_or_create_partition`, which count the attempts, log the rejection and mint another. No behavior changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Document the vector store's horizontal scaling design Five shared documents, made with every backend in mind: - horizontal scaling: the problem, the goals, a collection's lifecycle, and what each store guarantees; - the collection registry: its tables, operations, creation races, handles and their fencing; - purge: tombstones, the retention, the claim, backoff, dead-lettering and the sweeper, with the claim's measured cost; - consistency: what a query sees of earlier writes, and why the contract has no `get`; - isolation: between collections, and record UUIDs and their reuse. They describe the design the Qdrant and Milvus stores complete when they move onto the registry; each store's own document comes with that move. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Put "pending" before the noun in the registry's docstrings "Mark the collection registered, pending, under an incarnation live" set "pending" off in commas, which made the sentence hard to parse. It and its kind now read "the pending collection with the given incarnation", "a new pending collection" and "registered as pending", in the registry ABC, the registry-backed base and the horizontal scaling design. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * State what mark_live does with an incarnation that is not pending The contract said what a successful call does, but not what happens when no collection carries the incarnation or the one that does is already live. In both cases nothing changes and the call returns False, which is what the SQLAlchemy registry's conditional UPDATE does; the return value now reads as whether this call marked a collection live. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Raise from mark_live when the collection is no longer pending, not a bool mark_live answered whether it marked the collection live. Its one caller that read the answer, open-or-create, used False to mean that a deletion had removed the collection while its storage was prepared, and create_collection ignored it, so a creation a deletion undid reported success. A method named for an action now succeeds or raises, as the registry's other methods do. mark_live takes the collection's (namespace, name) beside its incarnation, matches on all three, and raises VectorStoreCollectionDeletedError, naming the collection, when no pending collection carries the incarnation. create_collection lets it propagate and states it in the VectorStore contract; open-or-create and the event backend's service locator catch it and create again. Tests: a creation whose collection is deleted during preparation raises; a second mark of one incarnation raises; the service locator creates again after a deletion undid its creation (fails against the locator before this commit). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Answer registrations from the collection registry: pending from register, live from resolve The registry had two addressing schemes. register, get and unregister took a (namespace, name); mark_live and unregister_incarnation took the incarnation register had minted, which every caller carried back by hand, and get answered a record whose `live` field went stale as soon as the creator's own mark_live ran. Any incarnation fit any call. The registry is now addressed by name alone: register, resolve, unregister and claim_purgeable_incarnation. What acts on one life of a collection is a registration, a handle bound to its incarnation, split by state: - register answers a PendingRegistration, whose creator marks it live (mark_live, which answers the LiveRegistration) or abandons it (unregister, this life only); - resolve answers a LiveRegistration, or None when no collection holds the name, and raises VectorStoreCollectionPendingError when the collection is pending; a LiveRegistration's require_current raises VectorStoreCollectionHandleStaleError once the collection is deleted. A registration's fields (namespace, name, configuration, incarnation) never change; the state is conveyed by which outcome a call takes, so an open is still one read and the fence one. A live collection offers no mark_live and no unregister: it is deleted by name only. The pending error carries the collection's configuration, so open-or-create still refuses a pending collection of another configuration without waiting for it. A store's collection handle is built from its live registration, and its fence is `registration.require_current()`. The SQLAlchemy registry supplies both registration types; the name-keyed and incarnation-keyed unregistrations share one transaction. The registry design document records why handles, why two types, and why no state fields. Tests: the registry's tests drive registrations (SQLite and PostgreSQL), with a new test that a live registration is current until its collection is deleted; the base's tests patch the pending registration's unregister where they patched the registry's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Name the registry's handles for their holders: a reservation and a registration PendingRegistration and LiveRegistration named the collection's state when the registry answered them, which a deletion by anyone falsifies: a "live" registration may have been deleted since. What does not change is what each handle's holder may do, so the handles are named for that, in the words of two known patterns: Try-Confirm/Cancel for the creator, and the stale handle for everyone else. - `register` is `reserve`, and answers a `Reservation`: the creator's hold on the name while it prepares the collection's storage. - `PendingRegistration.mark_live` is `Reservation.confirm`, which marks the collection live and answers its `Registration`. - `PendingRegistration.unregister` is `Reservation.cancel`. - `LiveRegistration` is `Registration`; `resolve`, `require_current` and `unregister(namespace, name)` keep their names. - The fields both share sit in a private base, `_RegistryEntry`. "Pending" stays the word for the collection's state, in the store contract and VectorStoreCollectionPendingError. The base store's creation flow, its task set and log text, the tests and the design documents follow; the registry design records why the names are roles. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Leave a confirmed collection alone when its reservation is cancelled Reservation.cancel() deleted the row carrying the reservation's incarnation whatever its state, while confirm() acted only on a pending row. A cancel issued after a confirmation that committed but whose answer was lost would have tombstoned a live collection outside the deletion by name, the one path that ends a live collection. cancel() now acts only while the row is pending, as confirm() does; once the reservation is confirmed it does nothing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Refuse a query limit that is not positive in registry-backed handles The handle answered empty results for a limit at or below zero, where the query contract now refuses it. It raises ValueError with the other input checks, before the liveness check and whether or not there are query vectors, so _query is called with a positive limit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * State how long the purge claim holds its transaction The claim's SELECT ... FOR UPDATE SKIP LOCKED transaction stays open, idle on PostgreSQL, through the backend's deletion, so a server-side idle_in_transaction_session_timeout shorter than a round fails the round, and one that keeps doing so dead-letters the tombstone. The purge document states the requirement beside the measured round durations. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Chain the locator's give-up error to the last error it retried When a session's collection stayed pending, or kept losing races, the retry loop gave up with a RuntimeError that carried no cause, so the log lost the pending error's registered_at, the time an operator needs to tell an abandoned creation from a slow one. The RuntimeError is now raised from the last error the loop caught; its type and message are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Never shift the purge backoff by a negative count The claim computes 1 << (failed_rounds - 1) for each candidate row, rows at 0 failures included, where the shift count is -1: defined on SQLite, undefined behavior inside PostgreSQL's int4shl. The row's first disjunct made it claimable regardless, so no claim went wrong, but the expression was undefined there. The count is now clamped at 0 with a CASE, which both dialects evaluate alike; backoffs after a failure are unchanged, and SQLite's plan for the claim is the same index range on (vector_store_name, enqueued_at). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Chain open-or-create's give-up error to the race it last lost When every attempt lost the reservation or the confirmation to another creator or a deleter, open_or_create_collection gave up with a VectorStoreAttemptsExhaustedError that carried no cause. It is now raised from the last lost race, as the event backend's locator chains its own give-up error; its type and message are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Drop the limit of 0 from the registry doc's no-op calls A query whose limit is not positive is refused with the other invalid inputs, before the handle checks its liveness, so it is no longer an example of a call with nothing to send that still checks. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Report a dead-lettered tombstone at or past the bound The error log fired only when a failed round brought the count to exactly the bound. The claim takes only tombstones under the bound, so a tombstone is dead-lettered once its count reaches the bound or goes past it, as when two purgers fail one tombstone at once; the log now fires in either case. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Run a purge round inside the registry instead of handing out a claim claim_purgeable_incarnation() was an async context manager, which cannot see what its body returns, so a round reported whether it found records through PurgeClaim.any_records_found: a field that started None and a guard that raised when a round left it unset. run_purge_round() takes the round as a callable instead. The registry claims the tombstone that came due first, calls the round with its namespace, configuration and incarnation, and records what the round returns under the claim. A round that raises still counts as a failed round. PurgeClaim, its None state and the guard go; the vector store's _purge_round hook is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Claim a purge round on SQLite with a write, as the segment store does On SQLite the claim was a SELECT whose FOR UPDATE SKIP LOCKED the dialect drops, and the driver defers BEGIN until a write, so the claim ran outside any transaction: two processes sharing a SQLite registry could claim one tombstone at once. The round recorded afterwards then decided the failed-round reset from what it read at claim time, so a round that succeeded could skip the reset after a racing round failed. On SQLite the claim is now the segment store's: an UPDATE ... RETURNING on the oldest eligible tombstone, which opens the write transaction, so purgers serialize at the claim and rounds run one at a time. PostgreSQL keeps FOR UPDATE SKIP LOCKED. The cost on SQLite is that its single write lock is held across the round's remote deletion: the registry's other writers wait for it, and past the driver's busy timeout they fail with a locked-database error. The purge document states it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Report a failed reservation cancel from the cancel's own task A creation whose storage preparation failed or was cancelled starts the reservation's cancel as a shielded task and logged its failure around the await. A creation cancelled again stops awaiting it, so a cancel that then failed went unlogged in the store, surfacing only as asyncio's "Task exception was never retrieved" without the collection's name. The task's done-callback now reports the failure, whoever is still awaiting it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Check identifiers with the shared helper in every store utils.require_identifiers checked the namespace and name of the registry-backed base's lifecycle calls, while the four stores kept the check inline: Qdrant and Milvus with the helper's own body, the SQLite stores with one combined check whose message, "Invalid namespace ... or name ...", named neither the rule nor which identifier broke it. Every store's lifecycle calls now use the helper, so an invalid identifier is reported the same way everywhere, with the rule it breaks. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Track open_collection like the other lifecycle calls The registry-backed store's create, open-or-create, delete and purge ran under the store's operation tracker, and open_collection, which resolves the name in the registry and is the first call the event backend's locator makes on every open, did not. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Say to drop the native collections before upgrading The deployment note said existing Qdrant and Milvus data is orphaned. The native collections keep their names, so the upgrade's new data lands beside the old: a Qdrant collection kept through it holds old points no search sees and no purge reclaims, and dropping it afterwards drops the new data too; an existing Milvus collection has the earlier schema, which the store cannot prepare. The note now says to drop the collections before upgrading, and why. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Cancel the reservation when confirming a creation fails or is cancelled A creation cancelled its reservation when preparing the collection's storage raised or was cancelled, so the name was free for the next attempt, but not when the confirmation that follows did: a failure or a cancellation while confirming left the collection pending, its name taken until someone deleted it. Preparation and confirmation now share one cleanup, the same shielded, logged cancel. The cancel acts only on a pending collection, so a confirmation that committed before its failure or cancellation was observed stands. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Say which writers wait on SQLite's lock during a purge round The purge document said the registry's other writers wait while a SQLite round holds the write lock. The lock covers the whole database file, so every store writing to it waits, and under the configuration wizard's defaults the episode store, session manager, segment store and configuration database share the registry's file. The round holds the lock across remote calls bounded by the store's request timeout, and past SQLite's default 5 s busy timeout, which the server does not change, those writers fail with a locked-database error. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Wrap the registry doc's creation-cleanup paragraph Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Lease the registry's purge claims instead of holding a transaction across the round The purge claim held its transaction for the whole round. On SQLite, whose write lock covers the whole database file, every writer to the database waited out the round's remote calls to Qdrant or Milvus, each bounded by the request timeout, and failed past SQLite's 5 s busy timeout with "database is locked"; with the registry in one SQLite file beside the other stores, as the configuration wizard sets up, that reached every store. On PostgreSQL the session sat idle in a transaction for the round, holding back vacuum. The claim is now a lease. One UPDATE ... RETURNING, committed at once, takes the oldest due tombstone that no unexpired claim holds (picked FOR UPDATE SKIP LOCKED on PostgreSQL; SQLite drops the clause and serializes the write), stamps its claimed_at with the database's now() and increments its claim_generation. The round runs with no transaction open. Its outcome is recorded in a short transaction: a round that found nothing removes the tombstone under any claim; one that found records or raised ends its claim, clearing or counting its failed rounds, only while claim_generation is still its claim's. A round that outlasted its lease therefore neither ends the claim taken after it nor counts a failure against it, and logs a warning. The lease lets purgers split a backlog; correctness rests, as before, on rounds being safe to repeat and to run on two purgers at once. A claim holds until its round ends or purge_lease_seconds (a new registry parameter, default 300) has passed since claimed_at, applied when a claim is decided, on the database clock, as the retention and the backoff are. Behavior changes: - SQLite purgers run rounds on different tombstones at once, where they ran one at a time, and the database stays writable during a round. - A round that found records always writes, to end its claim. - A cancelled or crashed round's tombstone is claimed again once its lease has passed, rather than at once. - An error recording a round's outcome is not counted as a failed round. collection_registry_gc gains claimed_at and claim_generation. The test that pinned SQLite's one-at-a-time rounds is replaced. The new tests: SQLite stays writable during a round to a connection with a 0.1 s busy timeout (the old claim made it raise "database is locked"); no PostgreSQL backend is idle in a transaction during a round; a claimed tombstone is skipped until its round ends or its lease passes; a round that outlasted its lease leaves the claim after it; purgers on separate engines claim each tombstone once; and the lease runs by the database clock. Removing any one lease mechanism fails at least one of them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Name the purge backoff's first delay base_purge_retry_backoff_seconds purge_retry_backoff_seconds read as the backoff itself, though it is the delay after the first failure, doubled for each further one. The base_ prefix says so and mirrors max_purge_retry_backoff_seconds, the cap. No configuration sets it: the stores construct the registry with its default. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Count a purge round that never ended as failed, and end a cancelled round's claim A round whose purger died wrote nothing after its claim, so it was never counted: a tombstone whose round killed its purger every time was claimed again each time its lease passed and never dead-lettered, and nothing tied the crash to the tombstone. The claim now counts such a round, as job queues count attempts. A claim that finds the oldest due tombstone's previous claim still open, its lease passed, runs no round: it counts that round as failed, as of when it was claimed (failed_rounds + 1, last_failed_at = the old claimed_at), ends the claim, bumps the generation so the old round's writes are fenced, and logs a warning naming the incarnation and the claim time, plus the dead-letter error when the count reaches the bound. One UPDATE handles both cases: its SET reads the row as it was, on PostgreSQL and SQLite alike, and leaves claimed_at null for an open claim, which RETURNING reports. No column is added. A cancelled round now ends its claim uncounted, in a write shielded from the cancellation, as a cancelled creation cancels its reservation, so its tombstone is claimable at once and a shutdown does not look like a crash. A release that fails is logged by its own task, and the round is then counted once its lease passes. Behavior changes: - A round whose purger died counts as a failed round once its lease passes, and its tombstone backs off from the time the round was claimed. - A round whose outcome could not be written counts the same way. - A cancelled round's tombstone is claimable at once, rather than after the lease. Tests, each failing when the behavior it pins is removed: a cancelled round ends its claim uncounted and logs nothing; a round that never ended keeps its tombstone until the lease passes, then is counted, reported and backed off from its claim time, and a lost release is reported; rounds that never end dead-letter the tombstone; purgers on four engines that find one unended claim count it once; and the overrun tests now expect the stale round counted once. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Test the registry's wiring, atomicity, isolation and behavior under concurrency New tests, run on SQLite and PostgreSQL; each fails, on both, with the behavior it pins removed: - A changed lease applies to claims already held, with a one-minute lease against a five-minute claim on both sides of the boundary (fails when the lease parameter is ignored; the other tests use the default). - The first purge backoff comes from its parameter, and doubles (fails when the parameter is ignored). - A deletion whose tombstone cannot be queued changes nothing: the collection stays registered and nothing is queued (fails when the delete and the queue insert commit separately, which would leave a deleted incarnation without a tombstone). - A reservation whose purge-queue check fails leaves the name free (fails when the insert commits before the check). - A round whose outcome cannot be written propagates the error uncounted, holds its claim until the lease passes, and is then counted once (fails when a write error counts as a failed round at once). - The same name in two namespaces is two collections that resolve, delete and purge apart (fails when the namespace is left out of a lookup). - A reservation whose minted incarnation collides with a tombstone being purged mints again at once (fails with the claim held across the round: SQLite's write lock, or PostgreSQL's row lock under the queue check). - Creators, deleters, readers and purgers on four engines race over shared names: only documented refusals, no database error or deadlock, no overlapping rounds, no incarnation both registered and queued, and the purge drains (fails without the lease). - Seeded sequences of every operation agree with a model of the contract step by step (fails when a cancel ends a live collection, a failed tombstone is claimed during its backoff, or a round that found records removes its tombstone). The fault-injection tests fail one statement on the engine by its prefix, naming the registry's tables. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Name the purge cutoffs and the consecutive count, and shorten comments - `_now_less(period, key)` is now `_database_time_ago(seconds)`. It takes a number of seconds or a SQL expression, so the retention, the lease, and the per-tombstone backoff cutoffs go through one helper, with anonymous bound parameters in place of the named ones. The registry keeps its durations as seconds. - The purge queue's `failed_rounds` is now `consecutive_failed_rounds`, the count it is: a round that finds records resets it. The dead-letter log names the new column. - The purge code's comments are shorter, and the design documents and docstrings this stack adds use serial commas. The claim's database cost is unchanged within run-to-run variation (no-op rounds, 500 per case, alternated 8 times, AC power): SQLite 1.52 vs 1.54 ms and 1.66 vs 1.60 ms, PostgreSQL 2.23 vs 2.45 ms and 2.14 vs 2.16 ms, for rounds that found records and found nothing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Split the purge round into named steps, and return a typed tombstone claim `run_purge_round` now reads as its sequence: claim the oldest due tombstone, report a round that never ended, run the round, then record its outcome, count its failure, or end its claim after a cancellation. Each step is a method named for what it acts on: - `_claim_oldest_due_tombstone` builds the claim and returns a `_TombstoneClaim` (what the round needs, and the claim's generation), an `_UnendedPurgeRound` (an expired claim, counted as failed), or None, in place of one result row whose fields meant different things in each case. - `_has_elapsed(seconds, since=column)` states each condition the claim checks (the retention since the deletion, the backoff since the last failure, the lease since the claim), with `_purge_retry_backoff_seconds` computing each tombstone's backoff. It replaces `_database_time_ago` and `_backoff_cutoff`; the SQL is unchanged. - `_report_unended_purge_round`, `_count_failed_purge_round`, `_report_dead_lettered_tombstone`, `_end_tombstone_claim_after_cancellation`, `_end_tombstone_claim`, and `_record_purge_round` take the claim. - `_insert` is `_insert_pending_collection`; the registry's and the store's task sets are `_tombstone_claim_endings` and `_reservation_cancellations`. The namespace test's purge loop is bounded, so a claim that never stops fails it instead of hanging it. Breaking the cancellation path, the unended-round path, or the elapsed check each fails registry tests. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Count each purge attempt in its claim A purge round whose purger dies writes nothing after its claim. The claim that later found such a claim past its lease counted the round, ran no round of its own, and reported it. That took a claim statement writing every column one of two ways, a second kind of claim result (`_UnendedPurgeRound`), and `last_failed_at` holding a claim time in that case so that `RETURNING` could report it. Each claim now counts its attempt, as job queues do when work is taken: `consecutive_failed_rounds` is `consecutive_attempts`, incremented by the claim's plain `UPDATE`. A round that finds records resets it, a round that raises ends its claim and sets `last_failed_at`, a cancelled round takes its attempt back, and a round whose purger died writes nothing. Each column has one meaning: `claimed_at` is the open claim's time, and `last_failed_at` is when a round last raised. - A tombstone whose round never ended is claimed again once its lease, and then the backoff, have passed. The backoff runs from `last_failed_at` after a raise and from the lease's end after a round that never ended. - A claim of a second or later attempt logs a warning naming the incarnation and the attempt ("attempt 3 of 10"). A raised round's error carries a note with both, so the sweeper's log of it names the tombstone. Nothing is logged later about an attempt that never ended. - After 10 attempts, claims skip the tombstone. A last attempt that raises logs the dead-letter error; one whose purger died is not reported, since nothing runs on the tombstone after it. New tests: a round that never ended is retried as the next attempt only after its lease and the backoff, with the retry warning; racing purgers retry it once; a cancelled retry of it leaves the tombstone claimable; resetting a dead-lettered tombstone's attempts returns it at once; and a raised round's error names the tombstone. The test of a count past the dead-letter bound is gone, since claims stop at the bound. Breaking the count, the backoff from the lease's end, the retry warning, the error's note, the cancel's take-back, the reset, a raise's date or end, or the dead-letter report or bound each fails a registry test. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Name the purge count attempts_without_progress, and tighten the purge docstrings `consecutive_attempts` did not say what resets it. The column counts the purge rounds claimed since a round last made progress, finding records and deleting them, the open one included and a cancelled round's attempt taken back: it is `attempts_without_progress`, and the dead-letter bound is `_MAX_PURGE_ATTEMPTS_WITHOUT_PROGRESS`. The dead-letter error and the design documents name the new column. The base backoff's docstring and description state the rule alone: seconds before a tombstone is claimed again after a failed round, doubled for each further failed round in a row. Where the wait is measured from stays in the purge design document. A raise on the last attempt is described as reported, the claims skipping the tombstone from then on, and the column and claim comments are shorter. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Replace the purge claim's measured costs with figures from the current claim The purge document's backoff and interference figures timed earlier forms of the claim, some of them the query alone. They are now measurements from 2026-10-06, each naming the commit whose code was measured: the backoff from commit 96a8b5b's claim, as whole calls, and the interference from commit 503687c's lease, whose rounds cost the same database time as commit 96a8b5b's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…n registry (MemMachine#1735) * Test the Qdrant store against Qdrant 1.19.1 The integration tests ran Qdrant 1.17.0. The Qdrant store's design, which follows in this stack, was measured against 1.19.1: its one-shot purge of an incarnation by filter, and the per-tenant index layout. 1.17.0 predates the filter-resolution fence that 1.19.0 added to filter deletes (qdrant#9678), whose extra cost 1.19.1 no longer shows, so the tests could not see the behavior the store is tuned for. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Bound every Qdrant request by a configured timeout The Qdrant client was built with qdrant-client's default timeout, so how long a write to Qdrant can be in flight was nothing the configuration stated. The purge that follows in this stack waits out a retention longer than any write can be in flight, and the request timeout is the part of that time the store controls. `QdrantConf.request_timeout_seconds`, a positive whole number of seconds defaulting to 30, is passed to the client. The sample configurations and the configuration docs show it. The tests' Qdrant clients are built by one fixture with the configuration's default, so they run with the timeout a configured store has. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Move the Qdrant store onto the collection registry The Qdrant store kept its catalog in a `__registry` collection per namespace and serialized creating and deleting a collection with locks in one process. A collection's name was the tenant discriminator on its points, so a handle kept writing into a collection deleted and created again under its name (MemMachine#1563), and a write in flight during a deletion outlived it. `QdrantVectorStore` is now a `RegistryBackedVectorStore`, and any number of processes sharing its registry may serve its collections: - Its catalog is the collection registry in the relational database `QdrantConf.collection_registry` names. The `__registry` collections, `registry_replication_factor` and the process-local locks go. The database manager builds and starts the registry before it opens the client, so a registry database it cannot resolve leaves no client open. - Every point carries its collection's incarnation in `sys-incarnation`, in place of the name, under the same tenant index, and every search filters on it. - A point's id is `uuid5(incarnation, record UUID)`, and the record UUID is kept in the payload as `sys-record_uuid`, which a search returns. The logical collections sharing a native collection share its id space: with the record UUID as the id, an upsert of a UUID another collection held replaced that collection's point. - Preparing a collection's storage creates its native collection and payload indexes, each under its own already-exists guard, so a creation that failed part way is completed by the next. - A purge round looks for one point under the incarnation and, finding one, deletes the incarnation's points with one filter-delete. `QdrantConf.tombstone_retention_seconds`, a day by default, is the retention, and the configuration refuses one below 10 x `request_timeout_seconds` + 300 seconds. - An upsert is halved only when Qdrant or a proxy refuses it as sent, with a 400 or a 413. Any other error raises at once: a timed-out upsert may still be applied, and sending it again adds load to a server already too slow. The collection lifecycle contract (`collection_lifecycle_contract.py`), which a store's tests mix in with hooks that read the backend directly, runs on Qdrant: stale handles, a collection created again starting empty, creation races and their outcomes, failed preparations, the purge, a write landing under a dead incarnation, and the registry lookups each operation makes. The store's own tests check what Qdrant holds by scrolling the incarnation past the store. Breaking: existing Qdrant data is orphaned, since its points carry names and its catalog is in the `__registry` collections; no migration is included. Every Qdrant store needs a relational database for its registry, which the samples, the Helm chart, the configuration wizard and the configuration docs name. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Document how the Qdrant store meets the shared contracts `design/qdrant_vector_store.md` records the Qdrant store's layout, its derived point ids and why they are one-way, filtered-search correctness, the purge by filter, and its consistency on one node and replicated, with the measurements behind each choice. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Build Qdrant handles from live registrations The collection registry now answers registrations: a PendingRegistration from register, which its creator marks live, and a LiveRegistration from resolve and mark_live, whose require_current is a handle's fence. The Qdrant handle takes its live registration in place of the namespace, name, incarnation, configuration and registry lookup, and _build_collection_handle builds it from one. The collection lifecycle contract follows: - the tests that fail a check or count checks patch the registration type's require_current, where they replaced the handle's lookup; - the racing winners register and mark live through a pending registration, and the winner is found with resolve; - churn counts VectorStoreCollectionDeletedError, which a creation undone by a concurrent deletion now raises, among the domain's outcomes. The Qdrant tests that build a handle on a mocked client give it a live registration that stays current. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Use the registry's reservations and registrations in the Qdrant store and the contract MemMachine#1734 names the registry's handles for their holders: `reserve` answers a Reservation, whose `confirm` answers a Registration, and whose `cancel` gives the name back. The Qdrant handle takes a Registration, and the collection lifecycle contract's racing winners reserve, then confirm. Names only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Drop the lifecycle contract's stale check of a query with limit 0 A limit at or below zero is now refused as invalid input, checked before the handle's liveness, so it no longer stands for a query with nothing to send to the backend; the query with no vectors still does. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Say to drop the Qdrant collections before upgrading, and list the registry The user docs did not say what happens to existing Qdrant data. The store keeps its native collections' names, so points left in one stay, invisible and never purged, and dropping the collection after the upgrade drops the new data too; databases.mdx now says to drop the collections before upgrading. The Helm README's configuration summary also lists collection_registry, which the configmap template sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Run every Qdrant store test against a real Qdrant server Qdrant's local mode answers differently from a server: it ignores payload indexes and raises its own exceptions where a server answers 404 or 409 over REST or NOT_FOUND or ALREADY_EXISTS over gRPC. The store fixture's client is now a testcontainers Qdrant over REST or gRPC, marked integration, so every store test exercises what a deployment runs. The tests that build a handle on a mocked client stay in the default suite. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Give the wiring tests' collection registry a file database The Qdrant wiring tests gave the collection registry an in-memory SQLite database, which aiosqlite serves from one shared connection; the segment store and the SQLite vector stores refuse such an engine, and the registry is to do the same. Each test now puts the registry in a SQLite file under its own tmp_path. What each test asserts is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Refuse engines that share one connection or hold SQLite in memory The registry's arbitration rests on each operation running in a transaction of its own: a reservation's insert and queue check, a deletion's row and tombstone, a purge claim. On an engine whose pool is StaticPool, every session shares one connection, so concurrent operations would run inside one another's transactions. On in-memory SQLite each connection gets a separate database, so the registry's state would not be shared even within one process. A SQLite configuration whose path is ":memory:" produces the former (aiosqlite uses StaticPool for it). The segment store and both SQLite vector stores already refuse both; the registry now does too, with the same messages. The dialect and SQLite-version refusal tests now use file engines, so each checks only its own refusal. The new tests fail with either check removed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Pin the Qdrant client's timeout wiring with a non-default value The wiring test asserted that the client got a 30 s timeout, which is also QdrantConf's default, so a manager that ignored request_timeout_seconds and passed 30 passed it. It now configures 7 s and asserts the client is built with 7; hardcoding 30, or dropping the timeout from the client's arguments, fails it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Bound the lifecycle contract's drain and settle before empty checks The contract drained purges with an unbounded while loop, so a store whose purge round kept reporting records hung the suite. The drain now fails the test once 1,000 rounds have each found something, far more than the few deleted collections a contract test leaves; a Qdrant round that always reports records now fails the purge tests instead of hanging them. The two tests that check that a collection created again under a deleted one's name starts empty now call the store's settle hook first. On a store whose reads lag its writes, a query could miss the old life's record for that reason alone, and the check would pass even if the new life could reach it; after settling it fails only when the new life cannot. Dropping the incarnation filter from Qdrant's search fails both. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Check a stale upsert by what it stores, not by its registry reads The contract counted require_current calls: two per upsert, one per query or delete. That pinned how a handle checks, not what the check guarantees, and a store that checked differently but kept the guarantee failed it. The test that replaces it pins the state: an upsert through a handle whose collection is already deleted raises StaleError, and the dead life's records, read past the store, are exactly those it held before. Removing the handle's liveness check before its write lets the record land under the dead incarnation and fails it. The rest of what the counts stood for is pinned by state elsewhere in the contract: a stale query and a stale delete raise, and an upsert whose collection is deleted between its check and its write raises. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Test that filtered queries and namespaces keep Qdrant collections apart Two collections of one namespace and configuration share a native collection, and the existing isolation tests either queried without a filter or used a filter only the querying collection's records met, so a search whose property filter displaced the incarnation filter, or sat beside it under `should`, passed them all. - A filtered query returns only its own collection's records: both collections hold records meeting every filter (a comparison, In, Or, Not(IsNull) on an undeclared property, And), and each query must return exactly its own. Replacing the incarnation filter with the property filter, or putting both under `should`, fails it. - Collections of two namespaces keep their records in separate storage, as VectorStore states: an upsert into one namespace's collection adds to that namespace's storage and leaves the other's count unchanged, under one name and configuration. Building the native collection's name without the namespace fails it. The lifecycle contract's count_stored hook now uses the module's _count_stored helper, which the namespace test also reads. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Test Qdrant purge rounds on missing storage and on a write after a round - A tombstone whose storage was never made is retired: a creation whose storage preparation fails before touching Qdrant cancels its reservation, queuing a tombstone for a native collection that does not exist. Its round must find nothing and retire it, raising nothing: the first purge call runs a round and the next finds none due. On Qdrant 1.19.1 the round's scroll of the missing collection raises UnexpectedResponse 404 over REST and AioRpcError NOT_FOUND over gRPC (a filter-delete raises the same). Removing the not-found handling around the scroll fails it on both transports; classifying only REST's 400, or gRPC's INVALID_ARGUMENT, as not found fails it on that transport. - A write landing after a purge round is reclaimed by the next: after a round deletes a deleted collection's records, a write through the stale handle's backend path, past its liveness checks, lands under the dead incarnation, as a write in flight across the deletion does. Draining must leave nothing under the incarnation. A round that reports nothing found after deleting retires the tombstone on its first round and leaves the late write in place, failing it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Test that a Qdrant upsert raises when a point is refused on its own An upsert refused with a 400 or 413 is halved until its halves fit or a single point is refused, and the store's docstring says the single point's refusal raises. The halving test refused only batches over a size, so every point was eventually accepted. The new test refuses one point whenever it is sent: the upsert must raise the refusal, with its status, rather than return as if the point were accepted. Returning, instead of raising, once halving reaches a single point fails it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Test ranking, scores and threshold direction for every similarity metric The suite queried cosine collections only, so a metric mapped to the wrong Qdrant distance, or a threshold kept on the wrong side for a distance, went unseen. For each of COSINE, DOT, EUCLIDEAN and MANHATTAN, on a real Qdrant over REST and gRPC, five vectors that the four metrics rank in four different orders, with no ties, are queried against one vector: - matches come best first, highest for a similarity and lowest for a distance; - each score is what QueryMatch defines: cosine similarity, dot product, Euclidean distance or Manhattan distance; - a threshold halfway between the second and third best scores keeps exactly the best two: scores above it for a similarity, below it for a distance. Mapping EUCLIDEAN to Qdrant's Manhattan distance or back, COSINE to dot product or back, dropping the threshold, negating it for distances, or applying it as a similarity for every metric fails it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Test a seeded Qdrant operation sequence against a model Two collections of one namespace and configuration, so sharing a native collection, take 200 seeded steps over a pool of 12 sorted uuid4s: upserts and re-upserts that change or drop properties (one of them undeclared), deletes of a collection's own UUIDs, the other collection's and absent ones, filtered queries (comparisons, In, IsNull, Not, And, Or) with a limit of the whole pool, deletion and re-creation under the same name, and purge drains. After every step, each collection's stored record UUIDs, read past the store, and an unfiltered query's matches (each record once, scored by its latest vector, best first) agree with a model of each collection's records; a query step's filtered matches agree with the model's filter; a drain leaves exactly the model's records in the native collection. Merging an upsert into the stored point's payload, so that a dropped or changed property keeps its old value, fails it; so does deriving point ids from the record UUID alone, searching without the incarnation filter, and a purge round that never deletes what it finds. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Test concurrent churn across two Qdrant stores sharing one registry Two stores, one on a REST and one on a gRPC client, each with its own registry object and engine on one registry database (a SQLite file, and PostgreSQL), serve eight seeded workers. Each worker owns four record UUIDs, so its records in each collection life follow from its own operations, and takes 50 steps over three collection names: upserts and deletes of its records, queries filtered on its own records, and deletion of a collection, half the time followed by its creation. A purger per store drains whenever a collection is deleted. - Every operation returns or raises a documented domain error (stale handle, already exists, pending, deleted, attempts exhausted); any backend or database error fails the test, and a deadline turns a deadlock into a failure. - A query whose collection was live throughout it, which a delete of nothing confirms afterwards, returns exactly its worker's records; any query returns none of the worker's other lives' records. - After quiescing and draining, each live collection's stored records and an unfiltered query equal what its workers wrote and kept, and a deleted life keeps no record but one whose upsert raised as stale: a write in flight across a deletion can land after a round found the incarnation empty, the race the tombstone retention closes, and the test runs with no retention. Searching without the incarnation filter, deriving point ids from the record UUID alone, misclassifying REST's 409 on an existing native collection, a purge round that never deletes, and a purge round that never returns each fail it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Make the Qdrant creation race tests race, and drop a dead index check The two-workers test checked that the native collection ended up indexed, but only the reservation's winner prepares storage, on a native collection no one else touches, so that check could not fail the way its docstring said; it is gone, and the docstring says what the test does. Its creators now reserve the name together, behind a barrier, so the loser always loses at the reservation rather than finding the winner already pending or live. It also writes a record through one handle and reads it through the other, so agreeing on one collection is shown by state. An open-or-create that raises on losing the reservation, and a strict create that treats a taken name as created, each fail it. A new test races what the old one meant to: two workers create two collections of one namespace and configuration at once, both preparations held at a barrier until both start, so both create the shared native collection together and one finds it already there. Both creations must succeed and each collection must hold its own records, over REST and over gRPC. Checking whether the native collection exists before creating it, without guarding the create, fails it on both transports while every sequential test passes; so does misclassifying the transport's already-exists answer. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Keep a Qdrant match that scores exactly the score threshold Qdrant rounds a query's score_threshold to single precision and keeps only scores strictly better than it, on every metric and transport, so a match scoring exactly the threshold was dropped, even a perfect cosine match under a threshold of 1.0. The SQLite stores and Milvus keep it, and the contract did not say which. The contract now says a match scoring exactly the threshold is returned. The Qdrant store sends the adjacent single-precision value on the worse side (numpy's nextafter in float32), so Qdrant keeps a score equal to the threshold, and checks the caller's threshold on the returned scores, so a score Qdrant's rounding let through but that falls short of the threshold as returned is dropped. A threshold beyond single precision is not sent; the store's check applies it. Sending the threshold, rather than only checking it in Python, keeps Qdrant from returning matches the threshold cuts. Measured against Qdrant 1.19.1 (4 collections of 5,000 768-dimension points sharing one native collection, 300 queries per cell with the two variants alternated, a threshold passing a quarter of the limit; AC power), p50 per query, threshold sent vs checked in Python only: REST 2.89 vs 3.04 ms at limit 10, 2.85 vs 2.96 at 40, 3.13 vs 3.42 at 100; gRPC 2.86 vs 2.93 ms at 10, 3.36 vs 3.80 at 40, 3.33 vs 4.23 at 100. Both returned identical matches. The new test, on every metric and on REST and gRPC, sets the threshold to a match's reported score (the match is kept) and one step better (it is dropped). It fails with the threshold sent unchanged, stepped the wrong way, without the store's check, or with that check strict. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Test that the SQLite stores keep a match scoring exactly the threshold The contract says a match scoring exactly the score threshold is returned. Both SQLite stores already keep it; each now has the test the Qdrant store has, on every metric it supports (cosine, dot and Euclidean on the USearch store; cosine and Euclidean on sqlite-vec): a threshold equal to a match's reported score keeps the match, and one a step better drops it. Each fails with its store's threshold check made strict. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Use serial commas in the Qdrant document, and shorten the threshold comment Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ich Qdrant operations filter on the incarnation (MemMachine#1775) * Accept an attempts-exhausted error in the lifecycle churn contract create_collection and open_or_create_collection document VectorStoreAttemptsExhaustedError as the outcome of creation races that make no progress, and the Qdrant store's own churn test already accepts it. The contract's churn test refused it, so it could fail on a documented outcome. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Say which Qdrant operations filter on the incarnation The design said every search, scroll, and delete of a handle filters on the incarnation. A handle's deletes name its points' derived ids, with no filter; the scrolls and filtered deletes are the purge's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The already-exists and not-found checks also matched a ValueError by its message text. Only qdrant-client's local mode raises those, and the store no longer runs in local mode: against a server, the client raises UnexpectedResponse with an HTTP status over REST and AioRpcError with a gRPC status code over gRPC. The message match and the ValueError the preparation and purge caught for it go. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…n registry, against a Milvus server (MemMachine#1736) * Move the Milvus store onto the collection registry, against a Milvus server The Milvus store kept its catalog in a `memmachine_<namespace>__registry` collection per namespace, whose `insert` does not enforce primary-key uniqueness, so two creators of one name both succeeded, and it serialized creating and deleting a collection with locks in one process. A collection's name was the partition key on its entities, so a handle kept writing into a collection deleted and created again under its name, and a write in flight during a deletion outlived it. Each call ran the sync client on a thread of the event loop's default executor, so calls waiting on Milvus queued every other `to_thread` call of the process behind them. `MilvusVectorStore` is now a `RegistryBackedVectorStore`, and any number of processes sharing its registry may serve its collections. The store was rewritten rather than adapted; what changes: - The catalog is the collection registry in the relational database `MilvusConf.collection_registry` names, built and started by the database manager before it opens the client. The registry collections and the process-local locks go. - Every entity carries its collection's incarnation as the partition key, with partition-key isolation, and in its primary key, `"{incarnation}:{record_uuid}"`. - Declared properties are typed, indexed fields (`_p_<name>`, a datetime as TIMESTAMPTZ with its UTC offset beside it in `_tz_<name>`), and undeclared ones live in the JSON field, still filterable. Negation is the complement, as on Qdrant: a negated condition holds where the property has no value. - The vector index is HNSW_SQ with 4-bit codes and FP16 refinement, what AUTOINDEX builds on CPU from Milvus 2.6.10, named so every server builds the same; a search rescores `limit x 8` candidates. Scores are the server's. - Reads run at Milvus's default consistency level, Bounded, which the store states: a query reflects every write made at least the server's `common.gracefulTime` before it. `MilvusConf.consistency_level` goes. - The store calls Milvus through pymilvus's `AsyncMilvusClient`, and bounds every request by `MilvusConf.request_timeout_seconds`. - A purge round lists a batch of the incarnation's primary keys, at most `MilvusConf.purge_batch_size`, and deletes them. `MilvusConf.tombstone_retention_seconds` is the retention, refused below 10 x `request_timeout_seconds` + 300 seconds as for Qdrant. A declared string's VARCHAR length is `MilvusConf.max_varchar_length`. Limits the server configures stay the server's. - A delete raises unless Milvus accepted every key sent. Milvus Lite is dropped. It is a separate embedded engine that scores, indexes and enforces collection properties differently, so a store tested against it is not tested against what production runs; MemMachine's local, single-node backend is the SQLite vector store. `MilvusConf.uri` defaults to `http://localhost:19530`, a URI with no scheme, which pymilvus reads as a Lite file, is refused, and the milvus extra no longer installs milvus-lite (the lock drops it with the packages only it required). The store's tests, including the collection lifecycle contract, run against a Milvus 2.6.24 server container as integration tests, and read past the store at Strong to check what Milvus holds. Breaking: existing Milvus data is orphaned, and an existing native Milvus collection has to be dropped; no migration is included. Every Milvus store needs a relational database for its registry, which the samples, the configuration wizard and the configuration docs name. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Document how the Milvus store meets the shared contracts `design/milvus_vector_store.md` records the Milvus store's layout: the shared native collection and partition-key tenancy, the composite key, the index and why it was chosen, declared properties as typed fields, the purge in batches, consistency levels, and the async client, with the measurements behind each choice. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Build Milvus handles from live registrations As for Qdrant: the Milvus handle takes its live registration in place of the namespace, name, incarnation, configuration and registry lookup, and _build_collection_handle builds it from one. The test that builds a handle on a mocked client gives it a live registration that stays current. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Take a Registration in the Milvus handle MemMachine#1734 renames the registry's LiveRegistration to Registration. Names only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Say to drop the Milvus collections before upgrading The Milvus store keeps the native collection's name from main, so on an upgraded server it finds main's collection, whose schema it cannot prepare, and the first request of every new session fails. The user docs' upgrade note now names Milvus beside Qdrant. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Test that Milvus queries and deletes stay within their collection Two collections of one namespace and configuration share a native collection. test_a_query_returns_only_its_own_collections_records gives both records that match the same filters and checks that each answers with its own records alone, unfiltered, on a declared property, on an undeclared one, and under a negation. It fails when _query lets the property filter replace the incarnation filter, ORs the two, or drops the incarnation filter: Milvus 2.6.24 refuses each of those searches under partition key isolation, and with isolation off as well the query returns the other collection's records. test_same_uuid_can_exist_in_different_logical_collections now also deletes the shared UUID through one collection and checks that the other still holds and returns its record. It fails when _delete deletes by a record_uuid filter in place of the composite key. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Test the Milvus score threshold and ranking for every metric test_a_threshold_keeps_the_matches_within_it_best_first upserts three records at increasing distance from the query, out of order, and queries with a threshold between the middle and the farthest: the two nearest match, best first, with the metric's scores. For Euclidean the distances are 1, 2 and 5 and the threshold 3, between the middle distance and its square; dot product and cosine take their own thresholds. It fails when the threshold is applied to Milvus's squared Euclidean distance before the square root, when every metric keeps scores at or above the threshold, when every metric keeps scores at or below it, when the matches are sorted in reverse, and when every metric's score is the square root of Milvus's distance. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Test that the Milvus store reads at Bounded or stronger test_the_store_reads_at_bounded_or_stronger creates a native collection, writes, queries, deletes the collection and drains the purge, then checks that describe_collection reports Bounded and that no search, query, get or hybrid search the store made names a level other than Bounded or Strong. The purge's listing and the tombstone retention rely on reads lagging writes by at most common.gracefulTime. It fails when the native collection is created at Session or Eventually, when the store's search names Eventually, and when the purge round's listing names Session. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Check random Milvus filters against the in-memory evaluator test_random_filters_agree_with_the_model writes 48 records whose properties cover every type, declared and undeclared, with datetimes at offsets from -12:00 to +14:00, instants a microsecond and a second apart, strings with quotes, backslashes and non-ASCII letters, and each property missing a quarter of the time. 200 seeded filter trees of Comparison, In, IsNull, And, Or and Not, with values of their properties' types, must select the UUIDs evaluate_filter selects, after one settle. The trees compare with != only through Not(=): on a missing property the store's != holds, as the complement of =, while evaluate_filter's does not. It fails when a declared datetime is written without its offset, when a negated condition no longer holds on a missing property, when negation is not pushed through And and Or by De Morgan's laws, and when undeclared datetimes are encoded at their own offset in place of UTC. Dropping the store's own UTC normalization of datetime filter literals is an equivalent mutant: Comparison and In normalize datetime values to UTC when they are built, so the store's call never changes a value. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Check a random sequence of Milvus operations against a model test_a_random_sequence_of_operations_agrees_with_a_model runs 60 seeded steps on two collections sharing a native collection: upserts of new and replacing records, deletes of present and absent records, deletion and recreation of a collection, which then reuses its UUIDs, and purge drains. Twelve UUIDs serve both collections and every life of each. After every step each collection holds exactly the model's records, once each, read at Strong; its settled queries, unfiltered and under random filters, select what evaluate_filter selects; a replaced record's old property values no longer select it; and after a drain no deleted life keeps a record. It fails when _upsert inserts in place of upserting, when the primary key leaves out the incarnation, and, by timing out its bounded drain, when the purge round never deletes. test_upsert_removes_stale_filter_fields now also checks that a filter on the replaced value finds nothing, which inserting in place of upserting fails. test_upsert_calls_native_upsert compared the entity the store built with itself; the model test checks the state an upsert leaves instead, so it is removed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Test Milvus purge rounds on a dropped collection and on two purgers The store fixture now builds its store with _started_store over a per-test registry database, and other_store builds one as another process would beside it: its own Milvus client and registry connections, over the same Milvus and registry database. test_a_purge_round_on_a_dropped_native_collection_retires_the_tombstone deletes a collection, drops its native collection, and drains the purge: the round raises nothing, and the next call finds nothing due. It fails when _purge_round no longer checks that the native collection exists, since Milvus refuses the listing. test_purgers_on_two_stores_reclaim_the_deleted_collections_alone deletes four collections of seven records each beside a live one in the same native collection, and drains both stores' purges at once in batches of 3. Every deleted collection ends empty, the live one keeps its seven records, and neither store finds anything due. It fails when a round reports finding nothing after deleting its batch, and when the listing leaves out the incarnation filter. test_a_purge_round_milvus_does_not_accept_in_full_raises gives a store a mocked client whose delete accepts one of the two keys the round listed: the purge raises. It fails when _purge_round no longer checks the count. test_a_delete_milvus_does_not_accept_in_full_raises now has Milvus accept one of two keys, so a check for zero alone fails it, and matches the exception's type rather than its wording. The drain in these tests is bounded, so a purge that never ends fails the test instead of hanging it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Test concurrent preparations of one Milvus native collection Milvus 2.6.24 answers two or more identical create requests for one collection, sent at once from separate clients, with success every time (four clients, five trials), and identical concurrent index requests likewise; a create whose schema differs gets code 1100, "create duplicate collection with different parameters". So a real server cannot make a racing creation take the store's already-exists branch. test_a_creation_that_loses_the_native_collection_to_another_completes_it covers that branch with a mocked client: the native collection is missing when checked and its create is refused as already existing. The creation indexes and loads the native collection, and the collection opens. It fails when the already-exists tolerance is dropped, and when a creation that loses the create returns without indexing and loading. test_a_creation_finding_the_native_collection_mid_creation_completes_it runs two stores on separate clients and registry connections. The first creates the native collection and is held before its indexes until the second's creation of another collection, which finds the native collection present, has returned and been written to and queried. Both collections end usable with every index. It fails when a creation that finds the native collection present skips its index and load steps. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Test Milvus settings with values other than their defaults The configuration, database manager and store tests set the request timeout, the declared-string length and the purge batch size to values other than their defaults, so a default passed in place of the configured value fails them. - test_every_request_carries_the_timeout runs at a 17-second timeout. It fails when the store or the handles it builds use 30 in place of the configured timeout. - test_milvus_client_kwargs_forwarded expects the configured 7-second timeout on the client, and test_milvus_creates_vector_store expects 7, 2048 and 500 in the store's parameters. Each fails when async_get_milvus_client passes the default for that setting. - test_parse_valid_storage_dict parses max_varchar_length 2048 and purge_batch_size 500 from the configuration. It fails when either field reads another key, so the configured value is ignored. test_milvus_conf_rejects_a_length_or_batch_size_that_is_not_positive and test_a_setting_that_is_not_positive_is_refused check that MilvusConf and MilvusVectorStoreParams refuse 0 for each of these settings. Each fails when that field's positivity constraint is dropped. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Test concurrent churn across two Milvus stores with purgers running test_churn_across_two_stores_with_purgers_keeps_every_collection_exact runs six workers, three on each of two stores with their own clients and registry connections over one registry database, for 40 seeded steps each on three collections sharing a native collection: upserts and deletes of records the worker alone owns, owner-filtered queries, and deletion and recreation of the shared collections. Each store's purger drains, in batches of 2, after every collection deletion, and each purge round begins at a settled read, as the tombstone retention would give it. A ledger records, by the incarnation each operation reached, the records upserts and deletes that returned leave, the records sent, and the upserts that raced their collection's deletion. Only the documented outcomes of a lost race may be raised, and the whole churn is bounded, so a deadlock fails it. During the churn, a query returns only records its owner sent to its incarnation. Once the churn is quiet, settled and drained, each live collection holds exactly its ledger's records, once each, with their latest values, and its settled owner and tag filters select them; the deleted collections keep only records of upserts that raced their deletion. In three runs the churn made 16 to 18 collection lives, 7 to 10 upserts raced a deletion, and queries, deletes and recreations met stale handles, existing names and deleted reservations. It fails when _upsert inserts in place of upserting, when the primary key leaves out the incarnation, when the purge listing leaves out the incarnation filter, when a round reports finding nothing after its batch, and, by timing out, when the purge round never deletes. The two-purgers test now builds its batched stores with the same _with_purge_batch_size helper. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Give the Milvus wiring tests' collection registry a file database The Milvus wiring tests built their collection registry on the module's in-memory SQLite configuration, which the Qdrant wiring tests' change below replaced with a file per test, since the registry refuses an in-memory or single-connection engine. Each Milvus wiring test now puts the registry in a SQLite file under its own tmp_path, as the Qdrant ones do. What each test asserts is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Test that the Milvus store keeps a match scoring exactly the threshold The contract says a match scoring exactly the score threshold is returned, and the Milvus store already keeps it. It now has the test the Qdrant and SQLite stores have, on cosine, dot and Euclidean: a threshold equal to a match's reported score keeps the match, and one a step better drops it. It fails with the store's threshold check made strict. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Use serial commas in the Milvus document and the store's comments Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Name the Milvus filter helpers for what they produce The module's expression helpers had names that did not say what they make: `_expr_string`, `_literal`, `_declared_literal`, `_fits`, `_absent`, `_condition`, and `_milvus_filter` are now `_expression_string_literal`, `_property_value_literal`, `_declared_property_value_literal`, `_is_comparable_with_declared_type`, `_property_absent_expression`, `_condition_expression`, and `_filter_expression`; the handle's `_score` is `_score_from_distance`. No behavior changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Send Milvus filter strings as UTF-8, not ASCII escapes json.dumps escapes a character outside the Basic Multilingual Plane as a UTF-16 surrogate pair, which Milvus's expression parser refuses, so a filter on a string such as "café 😀" failed to parse on a declared and an undeclared property alike. The literal now keeps its characters as UTF-8. The new test filters on such a string on both kinds of property; it fails with the ASCII escaping. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Compare an undeclared Milvus property only with values of a comparable type An undeclared property's condition compared only its stored value, so a string equal to an undeclared datetime's stored text matched it, where the declared path, the other stores' JSON properties, and the in-memory evaluator match nothing. The condition now also requires the stored type tag to be one the value compares with, by the rule the declared path already uses: a bool with a bool, an int or float with either, and any other value with its own type. The new test filters an undeclared datetime by its stored text, and an undeclared float by an int; it fails without the type check. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Index and load a part-made Milvus collection before a purge round lists it A creation that failed after creating the native collection, before its indexes or its load, leaves it unindexed and unloaded, and its cancelled collection's tombstone is due for purge. The round's listing query then raised "collection not loaded" on every claim, so the tombstone was dead-lettered unless another creation of the same namespace and configuration completed the collection first. A round now creates the missing indexes and loads the collection, the steps preparation already takes, before it lists. The new test fails a creation at its index step, then drains the purge: it fails with "collection not loaded" without the change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Drop the Milvus store's already-exists guard, which no supported server reaches Preparation caught a create refused as "already exist" by matching the message text. Milvus answers a create of an existing collection with the same schema with success, and refuses one with another schema as "create duplicate collection with different parameters" (code 1100, Milvus 2.6.24), which the text did not match, so the branch ran only under the mocked client of its own test. The guard, its message match, and that test go; the concurrent preparation tests cover racing creators against a real server. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Refuse a metric Milvus does not support before reserving the collection The store checked the metric while preparing storage, after the registry had reserved the name, so a refused configuration cost a reservation and its cancellation, and stayed pending until deleted if the cancellation failed. create_collection and open_or_create_collection now check it first. The test, for both operations, finds nothing to purge after the refusal; it fails with the check back in preparation. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Bound the Milvus connectivity check by the request timeout validate_milvus_client listed collections with no timeout, and the client's own timeout bounds only connecting, so a Milvus that accepts the connection and never answers hung validation, the one request request_timeout_seconds did not bound. The check now passes the configured timeout, on a store's first build and in build_all's validation. The new test checks both paths pass it; it fails with the timeout dropped. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Require testcontainers 4.15.0, the first with the community modules The shared conftest imports MilvusContainer from testcontainers.community, which 4.14.2, the declared floor, does not have, so an install at the floor could not import the conftest or collect any server test. The lock already pins 4.15.0; only the floor moves. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Dispose a mocked-client purge test's registry engine when it fails The test disposed the registry engine it opened only after its last assertion, so a failure left the engine and its aiosqlite thread open for the rest of the session. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Drop the Milvus store's re-sort of search results The store sorted each query's matches by score, though Milvus returns hits best first and the one transform the store applies, the square root of a Euclidean distance, keeps their order. On Milvus 2.6.24, 3,000 random vectors in each metric gave 450 of 450 result lists in order, filtered and not, at limit 100. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Say that a Milvus native collection keeps its VARCHAR length The store sizes a declared string's VARCHAR field when it creates the native collection, and a later preparation that finds the collection there leaves it as it is, so a changed max_varchar_length applies only to native collections created afterward. The setting's description, the configuration docs, and the design document now say so. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Match no declared Milvus int property with a float filter value The declared path counted an int and a float as comparable either way, so a float filter value on a declared int property reached Milvus as a float literal against an INT64 field, which Milvus refuses to parse ("cannot cast value to Int64", code 1100). A float now compares only with a float property, and matches no int one, as a value of another type matches nothing; an int still compares with a float property by value. The new test filters a declared float property with an int and a declared int property with a float; it fails with the float counted as comparable with an int property. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Create Milvus native collections at Bounded by name The store's reads run at the collection's consistency level, and the purge's listing and the tombstone retention rely on that level being Bounded or stronger. The store named no level when it created a native collection, so the level was whatever pymilvus defaults to, which a pymilvus release could change without the store noticing. The store now creates each native collection at Bounded. The test checks the create request names Bounded; it fails with the keyword removed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Report Milvus store operation latencies to a metrics factory The database manager built the Milvus store's parameters without a metrics factory, and MilvusConf had no metrics_factory_id, so a deployment could not configure one and every timed Milvus operation recorded nothing, where the Qdrant and Neo4j stores report theirs. MilvusConf takes metrics_factory_id, and the manager passes the factory it names, as it does for Qdrant. The wiring test checks the factory reaches the store's parameters; it fails with the factory not passed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Name Milvus's cap on a record's undeclared properties A record's undeclared properties are stored in one JSON field, and Milvus refuses an upsert whose JSON exceeds its common.JSONMaxLength setting, 65,536 bytes by default (internal/proxy/validate_util.go and pkg/util/paramtable/component_param.go at v2.6.24). The design document's list of server-configured limits and the database docs now name it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Refuse a Milvus Lite URI by its .db suffix, as pymilvus selects Lite The configuration refused a URI with no "://" as a Milvus Lite file. pymilvus serves a URI ending in .db with Lite and passes a unix: URI to the server as a socket address, so a file:// URI ending in .db passed the check and reached Lite, and a unix: socket URI was refused as a Lite file. The check now refuses a URI ending in .db, and its message names the Lite file. The tests refuse ./milvus.db, milvus.db, and a file:// URI ending in .db, and accept http, https, and unix: URIs; the file:// and unix: cases fail with the "://" check. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Write a declared Milvus datetime as its UTC instant The entity builder wrote a declared datetime's text with its own offset, while both filter literals convert to UTC first. Milvus refuses a TIMESTAMPTZ string whose offset has a seconds component ("invalid timezone name", code 1100 on 2.6.24), as local mean time and other historical offsets have, so such a record could not be upserted. The builder now writes the UTC instant; the offset field already keeps the offset written. The new test upserts a datetime at +00:19:32, reads back its instant and offset, and matches it by its instant; it fails with the original offset written, at Milvus's refusal. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Halve a Milvus upsert refused as too large, as the Qdrant store does Milvus's proxy refuses a request over proxy.grpc.serverMaxRecvSize (64 MiB unless configured) with gRPC's RESOURCE_EXHAUSTED status, before writing any of it, and pymilvus raises that status as it is: on 2.6.24 a 72 MB upsert was refused in 0.2 s with nothing stored. So a batch the Qdrant store accepts by halving it failed outright on Milvus. The store now halves an upsert refused with RESOURCE_EXHAUSTED until its halves fit or a single entity is refused. Any other error raises at once, since a timed-out request may still be applied. The store imports grpc for the status, so the milvus extra declares grpcio: 1.59.0 is the first release with Python 3.12 wheels, and grpc.aio.AioRpcError and grpc.StatusCode are older. The integration test upserts 1,200 records with a 60,000-character property, about 72 MB, and fails with RESOURCE_EXHAUSTED without the halving. The mocked-client tests pin the halving, which its test fails without, a single refused entity raising, and a timeout or another refusal sent once. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Name Milvus's datetime offset fields for their unit A declared datetime's UTC offset was stored in an INT32 field named _tz_<key>, which left its unit to the code. The field is now _tz_offset_seconds_<key>, under the prefix _TZ_OFFSET_SECONDS_FIELD_PREFIX; its value is unchanged, the offset in seconds. The schema test and the two datetime round-trip tests read the new names. A property key is at most 32 bytes, so a declared property's field names are at most 51 characters, within Milvus's proxy.maxNameLength (255 unless configured). A new test declares a datetime under a 32-byte key, writes it, reads both fields back, and matches it by its instant; the design document states the bound. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Keep a declared Milvus datetime as its UTC instant alone A declared datetime was stored as its UTC instant in a TIMESTAMPTZ field and its UTC offset in an INT32 field beside it. A filter compares only instants, and a search answers with record UUIDs and scores, so nothing reads the offset back; the store the vector records are derived from keeps it. The offset fields go: every declared property, a datetime included, is one field, `_p_<key>`, of those proxy.maxFieldNum allows. Undeclared properties keep their type-tagged JSON, offset included. The schema test asserts the native collection's exact field set, the datetime round-trip tests check that the stored instant is the written one, in UTC, and the longest-key test checks the one field. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Create Qdrant collections with strict mode off A filter may name a property that has no payload index, and the Qdrant store expects the server to scan for it. Qdrant Cloud turns strict mode on for new collections, with unindexed_filtering_retrieve false, so it refuses such a filter with 400 "Index required but not found" and the search fails (MemMachine#1776). The store now creates its collections with strict_mode_config enabled=False, which overrides the server's default. A collection created before this change keeps its strict mode; the collection update API can turn it off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Set the Qdrant collection's strict mode in-line The setting has one use, at collection creation, so it is written there with its reason beside it instead of as a class constant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test strict mode against a server that defaults it on The Qdrant test container now starts with Qdrant Cloud's strict-mode defaults for new collections, so the store's tests run where a collection that leaves strict mode on refuses a filter on an unindexed key. The strict-mode test checks that the server default is on, that the store's collection reads back with it off, and that the collection serves a filter on an unindexed property. It fails without the previous commits. The test against qdrant-client's in-process local mode is removed: local mode neither records nor enforces strict mode, so it could only check the argument passed to create_collection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Two versions of one record in one batch are ambiguous, and the stores disagreed on them: SQLiteVectorStore and Qdrant kept the last, SQLiteVecVectorStore failed partway with a raw sqlite3 UNIQUE constraint error, and Milvus refused the batch. Every store now raises ValueError naming the UUID before anything is sent to the backend, from one helper beside the other upsert input checks; the check costs one pass over the batch. The VectorStoreCollection.upsert docstring says so, and the registry-backed _upsert hook may rely on distinct UUIDs. Each store's test upserts a batch repeating a UUID after another record, expects the error, and finds nothing stored. It fails with the check removed: SQLiteVectorStore and Qdrant raise nothing, SQLiteVecVectorStore raises sqlite3's OperationalError, and Milvus 2.6.24 raises its MilvusException. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The stores disagreed on nan, inf, and -inf as property values: Qdrant over REST stores them as null, so the property reads back absent, and over gRPC refuses them with a raw error; Milvus 2.6.24 refuses one in a declared float field, takes one in an undeclared property, and cannot parse a filter on it; and the SQLite stores store them. Every store now raises ValueError naming the property key before anything is sent to the backend, for declared and undeclared properties alike, as the query path already refuses a non-finite query vector coordinate or score threshold. The VectorStoreCollection.upsert docstring says so, and the registry-backed _upsert hook may rely on finite values. Each store's test upserts a finite record and one with a non-finite value of each kind, on a declared and an undeclared property, expects the error, and finds nothing stored. It fails with the check removed: the SQLite stores, Qdrant over REST, and Milvus on an undeclared property raise nothing; Qdrant over gRPC raises grpc's AioRpcError; and Milvus on a declared property raises its MilvusException. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A filter compares datetime values by their instant, for equality and ordering alike, and a store keeps a datetime property's UTC offset as written, taking a naive datetime as UTC. Every store behaves so: the SQLite stores keep the offset beside the UTC instant in their type-tagged JSON, Qdrant keeps the RFC 3339 text with its offset in the payload, and Milvus keeps a declared datetime's offset in a field beside its TIMESTAMPTZ and an undeclared one's in its properties JSON. The VectorStoreCollection docstring says so. Each store's tests write a datetime at +05:30, declared and undeclared, read its offset back from the backend, and filter on it at other offsets: the same instant at -08:00 matches =, <=, and >=, a later instant at an earlier wall-clock time matches <, and an earlier instant at a later wall-clock time matches >. The read-back tests fail with the offset dropped at write (the type-tagged JSON's offset written as 0, Qdrant's payload converted to UTC, or Milvus's offset field written as 0). The filter tests fail with the filter node reading a value's wall-clock time as UTC, and on the SQLite stores also with the wall-clock text stored in place of the UTC instant. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A vector store's properties exist for filters, which compare a datetime by its instant, and a search answers with record UUIDs and scores, so no caller reads a stored offset back. The VectorStoreCollection docstring now says a store keeps a datetime property's instant, taking a naive datetime as UTC, and may drop its UTC offset; a filter compares datetime values by their instant, as before. Qdrant's read-back test checks the stored instant, not its offset. The tests of the type-tagged JSON, which keeps the offset (the SQLite stores and Milvus's undeclared properties), are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The event backend wrote every property of an event into its vector record and mapped the caller's whole filter onto the vector store, so a user key that a deployment never declared was both stored and filtered there: on Qdrant and Milvus as unindexed payload a filtered query scans for. The segment store already holds every property and already receives the whole filter for the context windows, so the vector side only duplicated work the segment store does anyway. The vector record now carries the keys the collection declares: EventMemory's reserved timestamp, the `_`-prefixed system properties an adapter stamps on the event, and the keys a project's `properties_schema` declares. The vector store is queried with the conjuncts of the filter that name only such fields; a conjunct is dropped whole when any field under it is undeclared, so dropping only ever widens the vector search, and the segment store narrows it back on the windows. An undeclared key never reaches the vector store, so a tenant's undeclared properties cannot shape what it stores or scans. `filter_fields` joins the filter parser: every field name a tree addresses. Rebased onto MemMachine#1631, where the vector record no longer carries the segment uuid (the segment store maps a derivative to its segment). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
A datetime written to Qdrant keeps its UTC offset only to the minute, over REST and gRPC alike: on 1.19.1, one written at -00:44:30 reads back at -00:44. The store wrote a datetime property at its own offset, so one at an offset with a seconds component, as local mean time had until 1972, was stored up to 59 seconds off its instant, and a filter on that instant missed it: the store did not keep the instant the collection's contract promises. The store now converts the datetime to UTC before writing it, which applies the whole offset to the instant, so nothing is left for Qdrant to cut. The read-back test checks that the payload holds the written instant in UTC. The new filter test writes a datetime at -00:44:30, under a declared key and an undeclared one, over REST and gRPC, and filters on its instant with =, <, and >. Without the change, all 8 of their cases fail. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
edwinyyyu
force-pushed
the
integration/vector-store-final-state
branch
from
October 10, 2026 00:31
587dde0 to
3e82239
Compare
The vector store got the conjuncts of a filter's top-level AND that name only declared fields, so an OR or a NOT with an undeclared field anywhere under it was dropped whole, and the vector search ran wider than it had to. It now gets the part of the whole tree on declared fields, the way a database planner uses an index for part of a condition: an AND keeps every operand that has such a part, an OR only when every operand has one, and a NOT applies the opposite rule to its operand. So `(a = 1 AND u = 1) OR (a = 2 AND u = 2)` gives `a = 1 OR a = 2`, and `NOT (u = 1 OR a = 1)` gives `NOT a = 1`. Every record that matches the filter still matches the part, and the segment store, which gets the whole filter, narrows the result back as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Taking the part of a filter that names only some fields depends on nothing but the filter and the fields, so it moves next to `filter_fields` and `map_filter_fields` as `extract_filter`, the name PostgreSQL, DataFusion, and Spark give the same operation. `EventMemory._vector_store_filter` had one caller, so `query` now maps the filter's fields and extracts its part on the declared ones itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hout it Rewriting a filter is checked the ways database engines check a planner's rewrites, against the in-memory stores' filter evaluator: - A table of filters and their expected parts covers every node kind under both parities of negation, with each operand kept or dropped, and alone reaches every line and branch of `extract_filter`. - Random filters up to four levels deep, over allowed and other fields and every kind of leaf: every record a filter matches, its part matches, and the part names only allowed fields; a filter on allowed fields alone is its own part; and when each other field appears once, a record matches the part exactly when some values of the other fields would make it match the filter. - Random filters through EventMemory, with the vector store getting the part and with it getting no filter: when the search reaches every record, both answer the filter exactly, and when it reaches fewer, the part only ever adds hits. The random filters and records come from a shared test module. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… seeds A vector search returns at most `vector_search_limit` seeds, and a condition on a field the vector store does not declare is applied to those seeds afterward, so a selective one can leave fewer seeds than the limit, or none, while matching segments exist further down the ranking. That bounds a query's cost by the limit. `query`'s docstring now says so, and a test pins it with an embedder that ranks the matching event below the limit: the condition on an undeclared field finds nothing, the same condition finds the event when the search reaches every seed, and a condition on a declared field finds it within the limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…agree EventMemory leaves to the segment store every condition the vector store cannot evaluate, so it relies on `get_segment_contexts` leaving out a seed that fails the filter. The SQL store does, but the contract only said the filter applies to the segments, and the in-memory fake kept a rejected seed whenever a neighbor in its window matched. The contract now says a seed that does not match has no entry, the fake checks the seed, and a test pins it through EventMemory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`query` said only conditions on undeclared fields wait for the seeds, but a declared condition in an OR with an undeclared one, or in a negated AND with one, waits too: the vector search applies only the part of the filter that names declared fields alone. And the expected collection schema said declaring a property makes it efficiently filterable, when a property left undeclared is not on the vector record at all, so the search cannot filter on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Building a vector record and extracting the vector store's part of a filter both read the collection's declared schema again, once per derivative for the record. `_schema_fields` already holds those keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixture's collection declares only system keys, so no test had a user key reaching the vector store; a wrong mapping of `m.<key>` would have dropped its condition from the search with every test passing. A test now declares `color` and checks that the record carries it and that `m.color` reaches the search as `color`. The three tests that spied on the vector store's queries share one helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ck) (MemMachine#1606) * Regenerate the OpenAPI document under the locked FastAPI `docs/openapi.json` predates the FastAPI release in `uv.lock` (0.141.1), whose `ValidationError` component carries `input` and `ctx`; regenerating the document with `docs/tools/generate_openapi.py` adds the two fields and changes nothing else. Separate from the API changes above it so their diffs of this file show only what they change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn * Remove per-project filterable properties A project could declare `properties_schema`, a set of caller property keys with types, on its long-term memory configuration; the event backend merged it into the vector store collection's indexed schema and rejected filters on any other `m.<key>`. That let a tenant create database resources (indexes, columns) by naming them in a request, which is what forced per-collection native resources named by a hash of their schema on the backends that limit them. The option is removed from the server configuration, the project API and the memory-configuration API, the Python SDK, the sample configurations, the configuration docs and the OpenAPI document. A filter may name any `m.<key>`; the stores evaluate it on the properties they hold. What a store indexes is decided by the deployment, not per project. A breaking API change on `speedkick`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn Rebased onto MemMachine#1631: the per-project schema also leaves MemMachine#1631's service locator, which creates the session's collection in a retry loop, and the commented option goes from the event sample configuration MemMachine#1698 added. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit a8322a7)
…res' lookup to get_partition
A vector store's logical collection becomes a partition, the segment
store's word for the same thing, and both stores' lookup is get_partition,
answering None like a Python get. Identifiers only, produced by the script
below; the (namespace, name) identity, the per-partition config and every
docstring are as they were, and the next change gives them their meaning.
The native clients' create_collection and delete_collection keep their
names.
The vocabulary also reaches what sits under the rename: the registry
package, its modules, classes and tables, the `collection_registry` option
of a Qdrant or Milvus backend and the store parameter and attribute that
carry it (`partition_registry`), the stale-handle and pending errors, the
registry-backed base and its handle lookup, the purge method
(`purge_deleted_partitions`, the segment store's name for it) and
open-or-create.
```sh
set -e
cd "$(git rev-parse --show-toplevel)"
git mv packages/server/server_tests/memmachine_server/common/vector_store/in_memory_vector_store_collection.py \
packages/server/server_tests/memmachine_server/common/vector_store/in_memory_vector_store_partition.py
git mv packages/server/server_tests/memmachine_server/common/vector_store/collection_lifecycle_contract.py \
packages/server/server_tests/memmachine_server/common/vector_store/partition_lifecycle_contract.py
git mv packages/server/src/memmachine_server/common/vector_store/collection_registry \
packages/server/src/memmachine_server/common/vector_store/partition_registry
git mv packages/server/src/memmachine_server/common/vector_store/partition_registry/collection_registry.py \
packages/server/src/memmachine_server/common/vector_store/partition_registry/partition_registry.py
git mv packages/server/src/memmachine_server/common/vector_store/partition_registry/sqlalchemy_collection_registry.py \
packages/server/src/memmachine_server/common/vector_store/partition_registry/sqlalchemy_partition_registry.py
git mv packages/server/server_tests/memmachine_server/common/vector_store/collection_registry \
packages/server/server_tests/memmachine_server/common/vector_store/partition_registry
git mv packages/server/server_tests/memmachine_server/common/vector_store/partition_registry/test_sqlalchemy_collection_registry.py \
packages/server/server_tests/memmachine_server/common/vector_store/partition_registry/test_sqlalchemy_partition_registry.py
git ls-files -z 'packages/server/*.py' 'docs/*.mdx' 'sample_configs/*.sample' 'sample_configs/*.yml' 'deployments/helm/templates/*.yaml' | xargs -0 perl -0pi -e '
s/VectorStoreCollection(?!Config)/VectorStorePartition/g;
s/in_memory_vector_store_collection/in_memory_vector_store_partition/g;
s/collection_lifecycle_contract/partition_lifecycle_contract/g;
s/CollectionLifecycleContract/PartitionLifecycleContract/g;
s/collection_registry/partition_registry/g;
s/SQLAlchemyCollectionRegistry/SQLAlchemyPartitionRegistry/g;
s/CollectionRegistry/PartitionRegistry/g;
s/RegisteredCollection/RegisteredPartition/g;
s/get_registered_collection/get_registered_partition/g;
s/_build_collection_handle/_build_partition_handle/g;
s/\bCollectionT\b/PartitionT/g;
s/register_collection\b/register_partition/g;
s/\b_Collection\b/_Partition/g;
s/CollectionRow/PartitionRow/g;
s/vector_store_collection(?!_schema|_namespace)/vector_store_partition/g;
s/open_or_create_collection/open_or_create_partition/g;
s/open_collection/get_partition/g;
s/_purge_deleted_collections_forever/_purge_deleted_vector_store_partitions_forever/g;
s/purge_deleted_collections/purge_deleted_partitions/g;
s/def create_collection\(/def create_partition(/g;
s/def delete_collection\(/def delete_partition(/g;
s/\.create_collection\((\s*namespace=)/.create_partition($1/g;
s/\.delete_collection\((\s*namespace=)/.delete_partition($1/g;
s/\.create_collection(?=\s*=\s*AsyncMock|\.assert_|\.side_effect|\.await_count)/.create_partition/g;
s/\.delete_collection(?=\s*=\s*AsyncMock|\.assert_|\.side_effect|\.await_count)/.delete_partition/g;
s/"create_collection"/"create_partition"/g;
s/"delete_collection"/"delete_partition"/g;
s/"open_or_create_collection"/"open_or_create_partition"/g;
s/only delete_collection is invoked/only delete_partition is invoked/g;
s/test_delete_collection_/test_delete_partition_/g;
s/open_partition/get_partition/g;
s/(vector_store_partition \(VectorStorePartition\):\n\s+)Vector store collection\./$1Vector store partition./g;
'
uv run ruff check --fix --quiet packages/server
uv run ruff format --quiet packages/server
```
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…uilt by the composition root A vector store was a factory of logical collections, each identified by a (namespace, name) pair and created with its own dimensions, metric and schema; Qdrant and Milvus shared one native collection among logical collections of equal configuration, under a name derived from a hash of that configuration, and the registry mapped (namespace, name) to it. A store is now one collection: `VectorStore(vector_store_name, vector_dimensions, similarity_metric, indexed_properties)` names its one native collection (or its tables and index files) at construction, every partition of it shares the store's dimensions, metric and schema, and `provision()` creates the store's durable resources idempotently, before `startup`. `create_partition(key)`, `open_or_create_partition(key)`, `get_partition(key)` and `delete_partition(key)` take a string key; a partition is the records carrying its incarnation in the store's native collection (Qdrant, Milvus) or a pair of tables (the SQLite stores), and the registry records what each partition was created under, so a store built with other dimensions, another metric or another schema raises `VectorStorePartitionSchemaMismatchError` instead of reading columns and vectors that are not there. "Collection" names only the native Qdrant or Milvus collection. Vector store names match `[a-z0-9_]+` and are at most 32 bytes, the rule partition and property keys follow, so every name works on every backend: the Qdrant store names its collection by the vector store name, and the Milvus store by `sys_` followed by it, since Milvus requires a leading letter or underscore and names its own internals with a leading underscore. The hash-derived native names go, and with them the per-partition config. The partition registry is keyed by partition key within one store: its tables, `partition_registry_pt` and `partition_registry_gc`, are shared by every store on a database and keyed by vector store name, so the registries of several stores share one database and nothing else. `mark_live` and `unregister_incarnation` take the incarnation alone. On the registry-backed base, the storage a store's partitions share is prepared by `provision()` (`_prepare_storage()`), and the storage a partition keeps of its own by `_prepare_partition_storage(key, incarnation)`, between its registration and its mark; the Qdrant and Milvus stores keep nothing per partition. A purge round works on one incarnation in the store's own collection. `DatabaseManager.get_vector_store(backend, vector_store_name=, vector_dimensions=, similarity_metric=, indexed_properties=)` builds and caches one store per (backend, vector store name), provisioning its registry and then the store; asking for a name again with other dimensions, another metric or other keys is a `VectorStoreConfigurationError`. The event backend's store is named by a UUIDv5 of its embedder id in the UUIDv5 of its backend's key in a namespace fixed for the event backend, written as 32 hexadecimal digits, so neither the key nor the id is constrained; stores on different backends never share a name, so their registries stay apart when the backends share a registry database, since a store name identifies one store among all the stores whose registries share it. Semantic memory's one store is named `semantic_memory` whatever its embedder and holds every org's features in one partition, `semantic_memory`, as main's one collection does. The event backend opens a session's partition with `open_or_create_partition`, which waits on one another worker is still creating. The SQLite stores change shape only: their registry tables become `vector_store_sqlite_pt` and `vector_store_sqlite_vec_pt`, keyed by vector store name and partition key, the pending-operation log is keyed the same way, and per-partition table names embed the vector store name. No migration is provided; existing SQLite vector data is orphaned. Rebuilt as one change on MemMachine#1631: this PR's earlier history carried copies of MemMachine#1631's commits as of `38684b385`, its own commits on them, and mirrors of MemMachine#1631's later commits in its terms. Its code is that history's final tree, which was verified, merged with main and without MemMachine#1624's commits. The design documents are still MemMachine#1631's and describe collections. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…up, not a separate provision A vector store and its partition registry each had a provision() that created their durable resources, run by the composition root before startup(). The split between provisioning a store and starting it is MemMachine#1570's to make for every store at once, so here startup() does both again, as on main: a store's startup prepares the storage its partitions share (the native collection and its indexes, or the SQLite tables), and the registry's startup creates its tables, as MemMachine#1631's does. The database manager starts the registry, then the store. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…er, live from resolve The registry is addressed by partition key only. `register(key, schema)` answers a PendingRegistration, whose `mark_live()` answers the same life as a LiveRegistration or raises VectorStorePartitionDeletedError when the partition was deleted meanwhile, and whose `unregister()` abandons that life alone. `resolve(key)` answers the LiveRegistration of the live partition, None when there is none, and raises VectorStorePartitionPendingError, which now carries the pending partition's schema, when it is pending. A LiveRegistration's `require_current()` raises VectorStorePartitionHandleStaleError once its partition is deleted. `get`, `mark_live(incarnation)`, `unregister_incarnation` and RegisteredPartition are gone, so deletion is by key or by a pending registration, never through a live one. The SQLAlchemy registry supplies frozen-dataclass registrations holding the engine and the vector store name, and a module function runs the unregistration transaction. The base handle takes its live registration, fences on `require_current()`, and `_partition_handle` builds a handle from one. `create_partition` lets VectorStorePartitionDeletedError propagate, and the VectorStore interface names it; open-or-create catches it and creates again, refuses a pending partition of another schema at once from the pending error's schema, and re-raises the last pending error. A `get_partition` of a pending partition of another schema still reports the mismatch first. The Qdrant and Milvus handles take the registration in place of the key, the incarnation and the registry lookup. Tests follow: the registry's tests answer registrations and gain one for `require_current`; the base's tests patch the pending registration type's `unregister` and expect the deleted error from a creation a deletion undid; the lifecycle contract patches the live registration type's `require_current`, registers its racing winners through pending registrations, and counts the deleted error among churn's outcomes; the Qdrant and Milvus tests that build a handle on a mocked client give it a registration that stays current. The same change as MemMachine#1734's 735c649, f88aaa8, e25be60 and b2bb2d8 and the handle halves of MemMachine#1735's 9096edd and MemMachine#1736's 5582cf4, for this PR's partition registry. The event-backend locator change has no counterpart: the locator here opens its partition with open-or-create, which creates again itself. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
MemMachine#1736's f56686f halves a Milvus upsert refused with RESOURCE_EXHAUSTED, and this PR's base carries the halving into the partition handle's `_upsert`. Its tests come here in partition terms: the integration test upserts 1,200 records with a 60,000-character property, about 72 MB, and fails with RESOURCE_EXHAUSTED without the halving; the mocked-client tests pin the halving, a single refused entity raising, and a timeout or another refusal sent once, on a handle `_partition_on` builds, which the mocked delete test now uses too. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…nt alone MemMachine#1736's 49b9ec2 drops the offset field beside a declared Milvus datetime, keeping its UTC instant alone, and this PR's base carries that into the store. The tests follow in partition terms: the native collection's fields are exactly the fixed ones and one per declared property; a declared datetime is stored as its instant in UTC; and a store whose declared datetime has the longest property key, 32 bytes, writes its one field, reads it back, and matches the record by its instant, the test MemMachine#1736's b015229 adds. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
The test of MemMachine#1813, merged into this PR's base, in partition terms: the server defaults new collections to strict mode, the store's collection reads back with it off, and a filter on a partition's unindexed property is served. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…quest The event backend created a session's vector store collection and segment store partition on the first request that opened the session, so a search or a write for an unknown session created storage as a side effect, and the service locator was the only place that knew both stores' create paths. The owner is the session. Every path that creates a session row runs through EpisodicMemoryManager._create_session, which inserts the row and, when the row is new, creates the session's partitions in its segment store and its vector store (create_episodic_memory_storage); an equivalent re-create accepts the row and leaves the storage as it is. The request path binds handles with the stores' lookups and raises SessionPartitionMissingError when a partition is absent: a session without its storage is broken, not new. Deleting a session with no open instance deletes its partitions by key, so a session whose storage was never fully created can still be deleted. MemMachine.create_session goes through the manager for the same reason. The semantic manager owns its one collection and creates it, once, at the storage's first use. With that, nothing calls the stores' open-or-create. The API is unchanged: the manager's open-or-create still creates a session a memory request names, now through the same path. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
Nothing calls them since a session's storage is created with the session: `open_or_create_partition` leaves the vector store interface, its registry-backed base and the two SQLite stores; `open_or_create_partition` and `close_partition` leave the segment store interface and its implementation, and SegmentStorePartitionConfigMismatchError, which only the segment store's open-or-create raised, goes with them. A store creates on `create_partition`, strictly, and looks up on `get_partition`, answering None; create-if-absent is the owner's, where the key's provenance is known. With open-or-create go its bounded wait on a partition another caller is still creating, where `get_partition` raises VectorStorePartitionPendingError for one, as before, and its retry after a creation a deletion undid, which `create_partition` reports with VectorStorePartitionDeletedError. Source changes are deletions only. The tests that exercised open-or-create as a fixture use a test-side create-if-absent instead, and the tests of its own semantics go; the refusal of a pending partition of another schema is tested through `get_partition`, which refuses it the same way. The partition registry design document drops open-or-create too. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
A partition stored every property of a record and filtered on any key, which made a caller's arbitrary keys part of the store's schema: the SQLite stores kept them in a JSON column and filtered with json_extract, Milvus the keys its schema did not declare in a JSON field, and a filter on a key the store never indexed scanned. Since EventMemory routes a filter on an undeclared key to the segment store, the vector store need not hold undeclared keys at all. A partition now stores the properties its store declares and no others. `upsert` raises UndeclaredPropertyKeyError before anything is sent for a record naming an undeclared key, and PropertyTypeMismatchError for a value of another type than its key declares; `query` raises UndeclaredPropertyKeyError for a filter naming an undeclared key and UnsupportedFilterError for a node outside the partition's `supported_filter_nodes`. Both SQLite stores keep one typed, indexed, nullable column per declared key on the records table (sql_columns.py); sqlite-vec 0.1.9 rejects NULL in a vec0 metadata column and a declared key is optional per record, so that store keeps the columns on the records table and hands the KNN a `rowid IN (SELECT ...)` allowlist, evaluating the filter during the search instead of after it. Qdrant drops the JSON copy and keeps a payload field per declared key; Milvus drops its JSON field and keeps its typed field per declared key. Datetimes are stored as microseconds since the epoch where a backend has no datetime type. Since every key a filter may name is now indexed, the Qdrant store creates its collection in strict mode (`unindexed_filtering_retrieve` and `_update` false, Qdrant Cloud's default): a filter on an unindexed key is refused by the server instead of scanned for. A leaf whose value is of another type than its key declares matches nothing, as on the SQL stores; the Qdrant compiler answers it with a filter no point satisfies, since the server would refuse the condition for the field's index, and Milvus no longer compares an int with a float key. Local mode does not record the setting, so a unit test checks the request and integration tests the server's answer. `declared_schema_contract.py` states the contract every backend's test module runs: which records a filtered search admits, over fixtures small enough that every backend searches them exactly, checked after each upsert so an approximate index fails on recall, by name, and not on the filter. On the registry-backed stores, the checks are the base handle's: `upsert` runs require_declared_properties in place of the type check it ran, and `query` runs require_supported_filter against the subclass's `supported_filter_nodes`, which each subclass now implements. A datetime column on the SQLite stores has a `tz_<key>` column beside it holding the UTC offset in seconds, written with the value and read by no filter, so a stored datetime is the value written, as on Milvus, Qdrant and the segment store; the microseconds column alone would keep only the instant. The Qdrant and Milvus design documents describe the declared-only properties and Qdrant's strict mode. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
A declared datetime on the SQLite stores had a `tz_<key>` column beside its microseconds column, holding the UTC offset in seconds that no filter reads. The vector stores keep a datetime property's instant only, as MemMachine#1736 now does for Milvus and MemMachine#1788 for Qdrant, and the segment store keeps the offset, so the column, `offset_column_name`, and the value written to it go: each declared property is one column, and a datetime stays microseconds since the epoch. The tests read a stored datetime back as its instant in UTC, and the roundtrip test checks that the stored instant equals the written one. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
Every embedder MemMachine ships produces vectors meant to be compared by cosine -- OpenAI hard-coded it, Bedrock defaulted to it, SentenceTransformer only reported what the model declared -- while every layer that touched a score paid for the other three metrics in direction flags, threshold directions, and per-backend tables mapping the enum onto native metric names. `SimilarityMetric` is gone; scores are cosine similarities in [-1, 1] and the names say so: `QueryMatch.score` and `SearchMatch.score` become `cosine_similarity`, and `query(score_threshold=)` becomes `query(min_cosine_similarity=)`, which no longer needs a direction to be meaningful and is refused when not finite, as the threshold was. A store and its partitions no longer have a `similarity_metric`, the schema a partition is registered under no longer records one, and a search engine factory takes the dimensions alone. The Bedrock embedder's `similarity_metric` config key and semantic memory's `vector_similarity_metric` go with it, and the install and configuration docs drop them. The vector graph stores carried a metric per stored embedding, as a companion property beside every vector; that is gone and `Node.embeddings` holds plain vectors. NebulaGraph's `cosine()` cannot take `APPROXIMATE` and its vector indexes offer only L2 and IP. Cosine similarity between unit vectors is their inner product, so embeddings are normalized on the way in and compared with `inner_product()` against an IP index. The cosine half of MemMachine#1598 (`abf92a3a4` on speedkick), re-derived on the one-collection store: the rest of MemMachine#1598 (queries answer record UUIDs and scores, no `get`, semantic memory's `vector_uuid`) and all of MemMachine#1603 are in registry-backed base and its partition handle lose the metric, the stores' own threshold checks become `require_valid_min_cosine_similarity`, and the design documents describe cosine scoring and a schema without a metric. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…kend The filter language is a closed union: Equals, Ordering, In, IsNull, n-ary And and Or, and Not. Every compiler is an exhaustive match. The parser maps `!=` and `<>` to Not(Equals), `NOT IN` to Not(In) and `IS NOT NULL` to Not(IsNull), so a negation means one thing. A leaf on a field holding no value is false, IsNull on it is true, and Not is the ordinary complement over the record set. The SQL compilers render Not as `NOT COALESCE(x, FALSE)`; they compiled `~inner`, which is three-valued and dropped rows holding no value. Milvus already pushes Not to the leaves by De Morgan, since its own `not` excludes entities lacking the field, and Qdrant's must_not already admits them. A predicate matches only a value of the compared type. The graph stores and the Neo4j semantic storage get the union only; their NOT over a comparison on a missing property stays three-valued, since no backend here can test a change. Where the compiled output changes: common/filter/sql_filter_util.py and common/vector_store/sql_columns.py (COALESCE under Not), and the parser's mapping of `!=`. Where behavior changes: the in-memory test collection's evaluator matches values by type, the declared-schema contract gains the complement law and the typed-match test, and two semantic-storage tests that ordered string values go, since ordering strings is not expressible in the union. An `In` over no values admits nothing: every compiler renders it as false, so EventMemory's empty id lists (session, source, block kind) keep meaning "nothing", and its complement admits everything. Every other file in this commit changes node names, import paths and dispatch shape only, with the same output as before. The tests the rebuilt base adds use the closed union too, and the segment store's random context-read test models the new semantics: a condition on a property with no value is false and a negation is the complement, where the model had followed SQL's three-valued logic. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
A leaf whose value was of another type than its key declares matched nothing on every store, each compiler rendering it as a filter no record satisfies, so a caller who compared a key with the wrong type got an empty result and no sign of the mistake; and `score > 3` on a float key matched nothing, since the parser reads `3` as an int. A NaN or an infinity reached the backends too: it matched every held value on Qdrant's REST transport, a NaN failed on its gRPC transport, and either failed on Milvus. `bind_filter` replaces `require_supported_filter` at every store's query and returns the filter the store compiles in place of the caller's. It still refuses an undeclared key and a node outside the store's `supported_filter_nodes`, and in the filter it returns every value is of its key's declared type: an int for a float key becomes the float it equals, and a membership test of ints on a float key the disjunction of those floats' equalities, since a membership test holds ints or strs. Any other value of another type than its key declares, a bool for a number, and a float that is not finite raise PropertyTypeMismatchError, which `upsert` already raises for a record; the VectorStorePartition query documents it. The compilers' branches that rendered a mistyped leaf as no match are unreachable and go: the Qdrant store's no-match filter, the Milvus store's filtering of values by type, and the SQL columns compiler's false branches, keeping only an `In` over no values as false. None of the three needs the declared schema any more. The Qdrant and Milvus design documents describe the binding. The declared-schema contract replaces its no-match test with tests every store runs: a value of another type and a float that is not finite each raise from `query`, and an int compares with a float key as the float it equals, each filter and its complement. The first two fail with the check in `_require_value` removed, the non-finite cases also with its finiteness clause removed, and the third fails on Qdrant, whose float index refuses an int match, with the int-to-float conversion removed from `_bound_value` or from the membership branch of `_bound_filter`. The Milvus and Qdrant modules' own no-match tests go, the Qdrant one with the strict-mode reason it stated. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A record could not hold an int for a float key: upsert refused it as a type mismatch, though a filter may compare a float key with an int. A float property that is not finite reached the backends instead, and each kept it its own way: the SQLite stores held a NaN as no value and an infinity as written, Qdrant's REST transport held either as no value, and Qdrant's gRPC transport and Milvus refused the upsert. `bind_record` replaces `require_declared_properties` at every store's upsert and returns the record the store writes in place of the caller's: its keys are declared and every value is of its key's declared type, an int for a float key becoming the float it equals. Any other value of another type, a bool for a number, and a float that is not finite raise PropertyTypeMismatchError, by the same rule the bound filter follows. The in-memory test partition binds its records too, and the VectorStorePartition upsert and the error document the rule. The declared-schema contract checks the rule on every store: an int written to a float key is stored as the float it equals, read past the store through a new `stored_value` hook each store's test module supplies, and matched by a float filter, an int one, and a membership test of ints; and each mistyped or non-finite value raises from `upsert`. The per-store tests of refused values lose their int-for-float case. The first test fails with the int-for-float clause removed from `_require_value`, and on Qdrant, which keeps an int payload as an int, with `bind_record` returning the caller's record; the second fails with the finiteness clause removed, and its bool cases with a bool counted as an int. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every backend holds an int in 64 signed bits, and each answered a wider one its own way. A record holding one failed to write on the SQLite stores, on Qdrant's gRPC transport, and on Milvus, and Qdrant's REST transport stored it as the largest int64, which an equality filter on that value then matched. A filter holding one failed on the SQLite stores and on Milvus, failed an equality on Qdrant, and answered an ordering on Qdrant with its bound clamped, so `>= 2**63` matched the largest int64. The binding both `bind_record` and `bind_filter` apply now refuses an int outside -2**63 to 2**63 - 1 for any key with PropertyTypeMismatchError, before anything is sent; the bounds are MIN_INT_VALUE and MAX_INT_VALUE beside the error. An int for a float key is held to the same range, so its float always exists. The declared-schema contract adds out-of-range ints to its mistyped records and filters, and a test that every store compares the int64 extremes with filters at those extremes. The mistyped cases fail with the range clause removed from `_require_value`, and the extremes test with the range made exclusive. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The declared-schema contract gains a test that every store matches strings holding accented letters, an emoji, quotes, a backslash, and control characters, by equality and by membership. It fails on Milvus with `ensure_ascii=False` removed from `_expression_string_literal`, the UTF-8 literal [vector store scale-out 6/6] ports from MemMachine#1736's 1a7fffa. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…emMachine#1588) * Atomically swap vector search engine index files on save SQLiteVectorStore persists each collection's index by calling the search engine's save(), which wrote directly to the final path. A crash mid-write left a truncated/corrupt file. Because index_saved=True makes the on-disk index a durable contract (missing/corrupt is a hard IndexLoadError, not a silent empty rebuild), an interrupted save could render a collection unrecoverable. Write the index to a sibling temp file and swap it into place with os.replace (atomic on POSIX and Windows on the same filesystem), so a reader sees either the old or new index, never a partial write; a failed save leaves the previous index intact. Leftover temp files are cleared on load so a crash does not leak them across restarts. Implemented in the engines (shared index_persistence helper) rather than in SQLiteVectorStore/SQLiteVectorStoreCollection, since the index save location and number of files written differ across engine implementations. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Make the index swap durable, not only atomic The swap protects a reader from a torn index, but the vector store also trims its pending-operation log once `save` returns -- and that log is the only other copy of those vectors, since the records table stores no vector column. So the swap reaching disk is load-bearing rather than a bonus: - fsync the parent directory after the replace, since POSIX `rename(2)` leaves the new directory entry in the page cache. Best-effort and ignored on failure, matching SQLite's `unixSync`; a no-op on Windows, which has no equivalent operation. - stop swallowing a failed fsync of the temp file. SQLite draws the same line -- a file fsync failure raises SQLITE_IOERR_FSYNC while a directory fsync failure is ignored -- and `EIO` means the writeback already failed and the dirty pages were dropped, which is exactly when the save must not be reported as committed. The existing cleanup then leaves the previous index in place with the log untrimmed, so the next save retries. - use F_FULLFSYNC on macOS, where plain `fsync` leaves the data in the drive's volatile write cache, falling back when a filesystem refuses it. State the resulting obligation on `VectorSearchEngine.save` itself, since that is what the store now relies on: replace atomically, then make the replacement as durable as the platform allows. An engine whose backend already implements the whole protocol can delegate to it and skip these helpers; the rest use `atomic_index_write`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Let the engine own index durability, not the vector store The pending log holds the only durable copy of a vector between checkpoints -- the records table has no vector column -- so trimming it is safe only at an instant when the index provably holds those vectors. The temp-write + rename protocol this PR shipped could not provide that instant. A rename changes a directory entry, and Windows exposes no way to flush one: os.fsync is _commit, which is FlushFileBuffers, which is for file data, and you cannot open a directory to fsync it. The decisive evidence is SQLite's own -- it threads a directory-sync flag through every commit-relevant directory operation, honors it in unixDelete, and declares it /* Not used on win32 */ in winDelete. So os.replace could return, _save_collection_index could commit its trim durably behind it, and a power cut could still roll the rename back: records forward, index back, no copy of the difference left. MOVEFILE_WRITE_THROUGH is not a fix; its documented guarantee covers copy-and-delete (cross-volume) moves, not same-volume renames. Take SQLite's answer, which was not to harden the directory operation but to stop using one as a commit point (PERSIST commits by zeroing a header, TRUNCATE by truncating, WAL by appending frames). A base path now expands into two index slots plus a generation record each, created once and thereafter only overwritten. A checkpoint writes the index over the inactive slot and flushes it, then writes that slot's generation record and flushes that. The record is the commit, and it is a write into a file that already exists. It holds the generation and its bitwise complement, so a torn write reads as absent rather than as some other generation -- all or nothing without needing single-sector atomicity from the hardware. load takes the highest believable generation, and deliberately does not fall back to the older slot when the published index will not parse: the log was trimmed against the newer one, so the older is stale by exactly the ops that can no longer be replayed. Both backends already write straight to the path they are given, which is what this protocol wants -- verified that repeated saves preserve the inode and leave no stray files -- so no engine gains a temp file, a buffer, or a rename. Durability is entirely the engine's, including which artifact is live. The store keeps no slot pointer, manifest, or generation, so no schema change and no migration: what remains is one rule, never trim past what save says is durable, and _save_collection_index already had that order. index_path becomes index_base_path since it no longer names a file, and discarding a collection asks the engine layer which files that covers. BREAKING CHANGE: an index written by the previous protocol is not published under the new one, so a collection with index_saved=True raises IndexLoadError until its index directory is cleared and the records re-ingested. Anomaly tests walk every crash point in the publish sequence by constructing the on-disk state each would leave, plus one that pins the ordering itself (a failed index write must publish nothing) since state-based tests cannot observe it. Verified against three deliberate breaks -- dropping the complement check, writing the record first, and reusing one slot instead of alternating -- each caught. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Publish the index atomically, and stop promising durability The two-slot generation-record protocol bought a guarantee we have decided not to make: that a save survives a power failure. Every engine would have to implement and maintain that protocol, and the failure it buys out is bounded -- search recall for the records applied since the last checkpoint, repaired by re-ingesting them. The direction that actually costs, a published index that will not parse, is closed by the atomic swap on its own. So this returns to the temp-file-plus-rename publication and spends the difference on stating the contract instead of strengthening it: `save` publishes atomically, never durably; the store trims the pending log behind a publication a power failure can revert; a record whose vector is lost that way still resolves by uuid, is absent from search until it is upserted again, and nothing here detects the gap for the caller. Reverts the durability and engine-owned-publication commits, keeps the atomic swap, and adds a store-level test that reconstructs a reverted publication deterministically -- restore the previous index bytes after the trim -- to pin the direction it fails in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Report a lost embedding as lost, not as a missing feature `update_feature` reads the stored embedding back when a caller updates a feature without supplying one, and that is the only place in the server that depends on the index still holding a vector. With publication now atomic rather than durable, a power failure can leave a feature whose row is intact and whose vector is not -- a state this path reported as "Vector record not found", which points the caller at the wrong thing and hides the repair. Split the two cases. A record that is genuinely absent keeps the old message; a record whose embedding the index no longer holds says so and names the fix, which is to pass a fresh embedding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Let a failed fsync fail the save, and cut the essay around it `_flush_to_disk` wrapped its fsync in `contextlib.suppress(OSError)` and called itself best-effort. A failed fsync is exactly the evidence that the bytes are not safe to publish -- on Linux an EIO from fsync means writeback failed, reported once and then cleared -- so swallowing it and renaming anyway published a file we had positive evidence was bad. The safeguard cost something and, in the one case it existed for, guaranteed nothing. Nothing tested it either. Let it propagate. `atomic_index_write` already unlinks the temp and re-raises, so a failed flush now leaves the previously published index standing, which is the correct outcome. A test pins that. The fsync is not best-effort, and the docstring should not have said so: it rules out a class rather than narrowing a window. Because the flush completes before the rename is issued, and a durable write does not un-happen, the new name can never appear over incomplete bytes. What the missing directory fsync costs is the other direction -- the rename may not survive, so the publish reverts -- and that is the benign one this store already accepts. The module docstring was 76 lines against 49 of everything else, most of it argument rather than documentation: a walk through SQLite's `unixDelete` / `winDelete` sync-flag handling, and a rejected two-slot commit protocol. That is the PR's case for the design, not something to re-read every time someone opens a 20-line module, and the PR body carries it. What a reader here needs is the guarantee, the non-guarantee, and the cost. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL * Say why the swap is a rename, and what the reopened fd cannot see Two things the module was silent on. Why a rename at all. The stronger answer is to put the commit point inside the file, where an fsync reaches it portably -- SQLite never renames, and commits by truncating or zeroing its rollback journal, or in WAL mode by appending frames whose checksums make a torn tail self-identifying. Both need the writer to own the file format. A search engine owns its own and exposes `save(path)`, so above that call a rename is the only atomicity primitive left, and an engine whose format already commits that way needs none of this. Worth saying, because "why not do the better thing" is the first question the module invites. What the reopened descriptor cannot see. Flushing is fine on a fresh fd -- dirty pages belong to the file, not to the descriptor that dirtied them -- but error reporting is not: Linux hands a writeback error to descriptors open when it was recorded, so one recorded between the engine's close and this open is never reported and the save proceeds on bytes already known bad. Same shape as the 2018 PostgreSQL fsync report. It cannot be closed from here: the engine writes through its own descriptor and closes it before returning, and closing the window needs an engine that writes through a handle the caller supplies. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL * Fsync the index on a descriptor that predates the write The fsync was on a descriptor opened after the engine had written and closed its own, which flushes correctly -- dirty pages belong to the file, not to the descriptor that dirtied them -- but reports nothing useful. Linux samples the writeback error sequence when a file is opened, so a descriptor opened after an error was recorded never learns of it: the fsync returns success and the save publishes bytes already known bad. Same shape as the 2018 PostgreSQL fsync report. Open the temp before yielding it and hold it across the caller's write, so the descriptor predates the bytes and any error from writing them is reported here, where it fails the save. That assumes the caller writes in place. Both engines do -- verified: the inode is unchanged across `save_index` and `save`, and the held descriptor sees the written size -- but it is their behaviour, not their contract. An engine that built a file of its own and renamed it over the temp would leave this descriptor on an orphaned inode, and the fsync would report on a file nobody is about to publish. So it is checked before the fsync, and a mismatch fails the save rather than passing it quietly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL * State the in-place rule where the caller reads it The descriptor held across the body only flushes what the body wrote if the body writes the yielded path in place, and that requirement was recorded in `_flush_to_disk` -- a private function nobody writing an engine opens. It belongs on `atomic_index_write`, which is the API they use, alongside what happens when it is broken: an `OSError` and no publication, so the mistake surfaces at the first save rather than at a power cut. `_flush_to_disk` keeps the mechanism -- why the descriptor has to predate the write -- and now points at the rule instead of restating it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL * Ask the drive to flush on macOS, where fsync does not `os.fsync` is not the same guarantee on all three platforms this ships to. On Linux it flushes to the device, and on Windows `FlushFileBuffers` does the same. Darwin's `fsync` explicitly does not: it returns once the data reaches the drive, which may hold it in a volatile write cache. So on macOS the ordering this module is built on -- data durable before the rename is issued -- did not hold at the device, which is exactly the case it claims to rule out. `F_FULLFSYNC` asks the drive to flush that cache. Filesystems that cannot refuse it, and there `fsync` is the most that can be asked, so a refusal falls back; any other error is a write failure and propagates, as before. The flush-failure test patched `os.fsync`, which Darwin no longer reaches. It patches the module's own `_fsync` instead, which every platform does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL * Stop guessing which errnos mean "this drive cannot do that" The fallback from `F_FULLFSYNC` to `fsync` was gated on an errno allowlist -- ENOTSUP, EOPNOTSUPP, EINVAL -- so that a genuine write failure would propagate rather than be quietly downgraded. Probing the actual returns on Darwin shows the list is both incomplete and partly invented: ENOTSUP=45 EOPNOTSUPP=102 distinct here, so both are needed /dev/null F_FULLFSYNC -> ENODEV(19), while fsync succeeds pipe/socket F_FULLFSYNC -> EBADF(9) EINVAL never came from F_FULLFSYNC at all; it came from fsync So ENODEV -- a real refusal, on a path anyone can reproduce -- would have raised instead of falling back, and EINVAL was in the list by analogy rather than evidence. What a network mount answers is not knowable from here, which makes the whole list a guess that fails closed on whatever it missed. This codebase does not classify driver errors by guessing, and this was that. Fall through on any failure instead. It is not a suppression: `fsync` runs on the same descriptor and raises in its turn, so a flush that cannot happen still fails the save. What the fallback gives up is the drive-cache flush -- the guarantee this had before `F_FULLFSYNC` was asked for at all. That is also what SQLite does with this same call, for the same reason. The test drives it through `/dev/null`, which refuses with ENODEV and accepts `fsync`; it fails against the allowlist and passes without it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL * Fail the save when the drive cannot be told to flush `F_FULLFSYNC` failing fell through to `fsync`, on the reasoning that a refusal is a statement about the filesystem rather than a write failure. That reasoning does not survive asking what `fsync` alone actually buys on Darwin. Against a process or kernel crash it is enough: the data has left the OS for the drive before the rename is issued, so the rename dies in the page cache and the old index stands. Against power loss it is not. The data sits in the drive's volatile cache, the rename's metadata joins it moments later, and nothing orders them -- and the rename is a few bytes against an index of megabytes, so a drive flushing as it pleases can easily put the new name on media while the bytes behind it are still queued. That is the torn publication this module exists to prevent, in precisely the scenario its docstring is about. So the fallback answered a request for ordering with a flush that does not provide it, and said nothing. A filesystem that cannot order data ahead of a rename is not one to publish an index onto; raise, and let the operator point `index_directory` at storage that can. This is also the simpler code. Refusal and failure now take the same path, so no errno is inspected -- there is no line to draw and no list to get wrong, which is what the previous two revisions kept getting wrong in opposite directions. The `/dev/null` test went with the fallback it pinned. The earlier defence of falling back rested on network and FUSE mounts being a realistic home for an index directory. They are not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 21105d6)
Never reuse a row id in SQLiteVectorStore The records table used a plain INTEGER PRIMARY KEY, which is SQLite's rowid, assigned as max(rowid) + 1: deleting the highest row freed its id for the very next insert. query() scores keys in the search engine and resolves them to rows in a second step, holding nothing in between, so a reused id let a record that was never scored come back wearing the score of the record that was. Nothing about that result looks wrong: the record exists and the score is in range. Declare the table with sqlite_autoincrement=True so ids are never reused. A stale engine key then matches no row and is dropped. Both tests fail without the flag: one pins the id policy directly, the other parks a query between scoring and row lookup, retires the scored record, inserts another, and asserts the query returns nothing. First half of MemMachine#1468. Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…(speedkick) (MemMachine#1612) Own the search engine's concurrency in the store, not in each engine Each engine wrapped its five methods in its own read-write lock and the abstract class promised concurrent use. The store is the only caller, and it already knows which calls must exclude which: a search shares the engine with other searches, and everything else runs alone. Holding that lock in the store puts every exclusion in the one file that reads the engine, and lets a rule the engines could not express hold: a rewrite's remove and add are one step to a reader, where before a search could run between them and see neither version. Engines drop their lock and keep only the index calls; the abstract class now states that the owner serializes. The store keeps one read-write lock per collection beside the engine, kept for the store's lifetime like the engine's other per-collection state, and takes it at every engine call: searches on the read side; mutations, loads, and the index save on the write side. A rewrite's remove and add sit under one hold. The save's trim runs after the lock is released, so readers wait for the file write and never for SQL. The row-id regression test from MemMachine#1589 parked inside a wrapper engine's search, outside the real engine's lock; under the store's lock that parks the read side, and the writes it then awaits cannot proceed. It now parks where it meant to, between the engine search and the row lookup. One new test pins the one-step rewrite; it fails against the engines' own locks. The turbovec engine in flight carries the same lock and needs the same subtraction. Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…edkick) (MemMachine#1607) Serialize a collection's writes so the engine sees them in order A write commits to SQLite and only then applies to the search engine, so two writers to one uuid could reach the engine in the opposite order to the one they committed in: an upsert overtaking a delete re-adds a vector for a record that is gone, and two upserts inverting leave the engine serving the older vector. Never reusing a row id does not cover this, because an upsert of an existing uuid keeps its row id. A save in that window costs a write outright: it publishes the index and trims every applied log row, and a write that applied after the index was written is then in neither. A per-collection asyncio.Lock now spans a write from SQL commit through engine apply, mark-applied, and any save it triggers; shutdown's save takes it too. The lock belongs to the store, not to a collection handle: a handle is constructed per open_collection call, so several can address one collection, and only a shared lock serializes them. Readers are untouched. Three tests fail without the lock, each interleaving made deterministic by gating the engine: an upsert overtaking a delete of its uuid, a save trimming a write it did not publish, and that overtake across two handles, which a per-handle lock passes. The rest pin behavior the lock must preserve: an upsert surviving a delete of another record, disjoint concurrent upserts and deletes, writes racing a checkpoint, and batches that name one uuid twice. Fixes MemMachine#1468. Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…edkick) (MemMachine#1608) Refuse a pending row replay cannot honor, instead of dropping it Replay matched `operation_type == "upsert" and vector is not None` and let everything else fall through. An upsert row with no vector, a vector that is not a whole number of float32s, or an unknown operation_type was skipped, and a decodable vector of the wrong width reached the engine, which refused it with its own error at startup. The skipped rows were the worse case: and because the skipped row reached neither the engine remove set nor the mark-applied update it survived the restart to be skipped again on the next one. Between a write returning and the next index save the log holds the only copy of the vector, so the outcome was a record that exists in SQLite and can never be found by search. That is damage to a durable record, not a state to heal. Replay now raises PendingOperationCorruptError for all four, naming the collection, the row and the fault, and leaves the log intact for whoever repairs it. Both error types' docstrings now say what a caller should do: read the cause of an IndexLoadError before choosing a remedy, and never clear the log to get past a PendingOperationCorruptError. Four tests corrupt a log row each way and assert the restart refuses. The rest of TestPendingLogStates pins what replay guarantees for intact rows: a rewritten uuid replays its last write, an upsert then delete stays deleted, a failed save leaves the write replayable, and the save threshold counts log rows rather than writes. Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…tization `QdrantConf` gains `hnsw_config`, `optimizers_config` and `quantization_config`, plain mappings mirroring qdrant-client's `HnswConfigDiff`, `OptimizersConfigDiff` and `QuantizationConfig`, so qdrant-client stays optional for configuration parsing; the store's params validate them against qdrant's own models. They apply to the store's one collection, created at startup. `m` must be 0 or unset: the collection is multi-tenant and disables the global graph in favor of per-partition payload indexing, so a deployment tunes `payload_m`, which defaults to 16 as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
The sample configurations gain a commented block with the three keys, checked against qdrant-client's models. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…MemMachine#1609) Take SQLite's write lock at BEGIN, not at the first write `delete` resolves row ids and then writes, in one transaction. Under SQLite's default deferred BEGIN the write lock is taken at the first write, so another writer can take it between that read and the write, and the transaction that read first is the one that loses: its own write cannot upgrade, and SQLite reports that immediately rather than waiting out busy_timeout, because waiting could only deadlock. Reproduced in both journal modes: journal_mode BEGIN other writer our write delete deferred shut out fails delete IMMEDIATE shut out ok wal deferred commits fails wal IMMEDIATE shut out ok The store now emits BEGIN explicitly and lets a transaction ask for BEGIN IMMEDIATE, which every write path does. The mode is chosen per transaction, not per engine: a hook that asked for IMMEDIATE unconditionally would make every read take the write lock, and two readers would then serialize against each other. Two tests fail without it. One holds a competing lock across a read-then-write transaction and asserts that transaction completes; asserting instead that the other writer is excluded passes either way, because a deferred BEGIN shuts it out too, later and by a different lock. The other races four creates of one name: under a deferred BEGIN the losers read no stored config, go on to CREATE TABLE, and fail there with "table already exists" instead of VectorStoreCollectionAlreadyExistsError. Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…ches `MilvusConf` gains `vector_index`, a plain mapping of `index_type`, `params` (build parameters) and `search_params` (parameters every search passes), so pymilvus stays optional for configuration parsing; the store's params coerce it into `MilvusVectorIndex`. pymilvus has no models for index or search parameters, so the unit is the whole spec: build and search parameters come from one author for one index type. The index applies when startup creates the collection's vector index, and the search parameters to every query. Unset selects HNSW_SQ searched with refine_k 8, as before. `params` may not set `index_type` or `metric_type`, which pymilvus would let override the spec's own index type and the store's cosine metric. Milvus checks the parameters the index type takes when it creates the index and when a search reaches an indexed segment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k) (MemMachine#1610) Give every write a fresh row id, so a key names one version An upsert of an existing uuid kept its row id, so one engine key spanned every version of a record. A query reads the engine, then each candidate's properties, then its uuid, at three instants with nothing held between them, and a rewrite of the very record being returned could land between those reads: the score of one version paired with the filter verdict of another, or, when the overfetch loop rescored a key whose verdict was already cached, the other way round. No lock covers this, because reads deliberately take none. Every write now deletes the previous row and inserts a new one in the same transaction, stages a delete for the old key beside the upsert for the new, and the engine removes the old key and adds the new. Rows are immutable, so a key names one version: the score computed under it, the filter verdict for it, and the uuid it resolves to belong to that version, and a key whose version has been rewritten resolves to no row and is dropped. AUTOINCREMENT remains what keeps a retired key from being reissued, and the write lock what keeps two rewrites of one record in order. A batch that names a uuid twice is collapsed to its last record before the insert, which the on-conflict update used to do implicitly. The save threshold counts log rows and a rewrite now adds two, so a rewrite-heavy workload checkpoints about twice as often; that test's expectation changes accordingly. Measured on this machine, medians of three runs, records per second, 64-dimensional vectors, file-backed store with an index directory: batch save threshold insert before / after rewrite before / after 500 1000 20477 / 19290 16450 / 12902 500 none 21228 / 22486 19013 / 15899 1 none 341 / 345 316 / 274 Inserts move within run-to-run noise, in both directions. Rewrites cost 13-22% more, the upper end where the doubled log rows double the saves. Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The sample configurations gain a commented `vector_index` block under the Milvus alternative, checked against `MilvusConf` and `MilvusVectorIndex`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he core final state Open-or-create leaves the vector store interface; the service locator's tests build the cosine-only search engine of MemMachine#1663, and the SQLite stores' tests keep the cosine-only metric tests of MemMachine#1663. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every store refuses a batch naming a record UUID twice. A non-finite float is already refused by the declared schema's type check, so its separate check and tests go, and the datetime tests run on the declared key alone, since an undeclared key is refused. The store keeps the strict mode of MemMachine#1628 over the one MemMachine#1813 merged into the base, and MemMachine#1813's test of it off stays out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…chine#1672 to MemMachine#1676) into the final state Their upsert keeps the delete and fresh insert, writing the declared columns, and refuses a repeated UUID as MemMachine#1788 does; open-or-create stays out, as MemMachine#1625 removes it. Their tests create their partition as a session does and build filters with the closed union of MemMachine#1616. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…achine#1741) into the final state A Qdrant collection is created with the deployment's HNSW, optimizer and quantization options and with the strict mode of MemMachine#1628. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
edwinyyyu
force-pushed
the
integration/vector-store-final-state
branch
from
October 14, 2026 00:23
3e82239 to
5e638c3
Compare
xiongzubiao
force-pushed
the
feat/horizontal-scaling
branch
from
October 15, 2026 15:31
101182c to
cf7ae15
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose of the change
One branch holding every open PR of the vector store tree, merged and consistent, so the end state can be read and tested as a whole. Not for merge. The tree's PRs merge on their own. This branch is the reference they are split from and checked against: a split is consistent when its last PR's tree equals this one.
Included, at their heads on 2026-10-13: #1788, #1702, #1670, #1627, #1622, #1625, #1663, #1460, #1469, #1672, #1673, #1674, #1675, #1676, #1618, #1741, #1628, and #1616.
Not included: the event memory stack (#1684 and above), the PRs on
main(#1624, #1786, and #1792), and operator-declared indexed properties, which have no PR.How the conflicts were resolved
The branch starts at #1616's head, which carries #1702, #1670, #1627, #1663, and #1628, and merges each other leaf with a merge commit. Every PR in it is at its head restacked onto
feat/horizontal-scalingat15f2c0686, where #1813 is merged. Where two PRs meet, the final state takes the later design:feat/horizontal-scaling) with [declared schema 1/2] Make a vector store filter only on the properties it declares #1628. [vector store scale-out 6/6] Make a vector store one collection, with string-keyed partitions #1627 carries Create Qdrant collections with strict mode off #1813's strict mode off into its one collection and ports Create Qdrant collections with strict mode off #1813's test to partitions; [declared schema 1/2] Make a vector store filter only on the properties it declares #1628 creates collections in strict mode, since every key a filter may name is declared and indexed there, and drops the ported test for its own. The Qdrant test container keeps Create Qdrant collections with strict mode off #1813's Cloud-like strict-mode defaults.Verification
Rebuilt on 2026-10-13 from the PR heads rebuilt onto #1702's routing at any depth (
6631b5c77): #1616ad48f4407, #16259d3d89bec, #17888257b8bda, #1676d79a7da24, and #17412d0794bdd, with each merge's earlier resolution carried over. The head5e638c3ebdiffers from the previous head3e822399donly in #1702's routing files. There,ruff checkandruff format --checkare clean,ty checkis clean as CI runs it,uv lock --checkis clean, and the episodic memory, filter, and vector store unit tests pass, 975 passed and 6 skipped. The results below ran at earlier heads.Rebuilt on 2026-10-09 from the PR heads restacked onto
15f2c0686, #1813's merge: #161613d104245, #1625cd75d4cb9, #17888257b8bda, #1676a070372c4, and #17412c78deb61. The tree at the head3e822399dis identical to the previous head587dde06c, so the results below hold for it. At the restacked leaves, ruff, ruff format, and ty are clean, and their store suites pass, unit and integration, against Qdrant 1.19.1 with Cloud's strict-mode defaults and Milvus 2.6.24.At
a40e31a86, on 2026-10-09:ruff checkandruff format --checkclean;ty checkclean as CI runs it (uv run --frozen --all-extras ty check --project packages/server);uv lock --checkclean; the server and client suites without integration tests, 2535 passed; the integration tests of the vector stores, the resource manager, episodic memory, and semantic storage, 826 passed, 1 failed, and 13 skipped, against PostgreSQL 16 (pgvector), Neo4j 5.23, Qdrant 1.19.1, and Milvus 2.6.24 in containers. The failure was a Milvus test that the upsert validation merge had left without its asyncio mark. The head3fab491a3restores the mark and changes nothing else; there,ruff checkis clean and the Milvus store's test file, unit and integration, 143 passed. At the head587dde06c, after the re-merge of #1813, the Qdrant store's and the database manager's tests, unit and integration, 365 passed against the Qdrant container defaulting new collections to strict mode.Stack
Directly on
feat/horizontal-scaling, as a reference beside the vector store tree.🤖 Generated with Claude Code