Skip to content

ADR 0051: Typed batch upserts

Date: 2026-10-23. Status: Accepted on 2026-10-23, after the verification below passed (evidence), except the guarded benchmark, which follows separately. Extends ADR 0038 (bounded bulk mutations) and uses ADR 0050's write counters. At the owner's direction on 23 October 2026, implementation went ahead without the separate approval step planned for this ADR; the ADR and the code are reviewed together.

Context

Neo4jClient's write API centers on Merge(...).OnCreate().Set(...) and Unwind, and ingestion pipelines such as GraphRAG need "insert or update by key" for thousands of rows. GraphBridge offers, as checked on 23 October 2026:

  • SaveChanges: inserts need to know the row is new, and updates need it loaded and tracked. Upserting 10,000 rows means 10,000 lookups or a hand-written query.
  • ExecuteUpdateAsync and ExecuteDeleteAsync (ADR 0038): set-based and untracked, at most 10,000 rows per atomic call, one parameterized UNWIND, schema checked first. No insert.
  • Native Cypher: UNWIND $rows AS row MERGE (n:Label {key: row.key}) ... works, but the caller converts values, names properties and checks constraints by hand.

EF Core 10 has no upsert API (dotnet/efcore#4526 is still open), so the shape is GraphBridge's to choose. PL-34 settled that upserts stay outside Fluent Cypher, so EF remains the only owner of entity state.

Decision

1. API

GraphUpsertResult result = await db.People.ExecuteUpsertAsync(people);                                  // update every mapped property on match
await db.People.ExecuteUpsertAsync(people, updateProperties: p => new { p.Name, p.Email });            // update only these on match
await db.People.ExecuteUpsertAsync(people, updateProperties: p => new { });                            // insert missing rows, leave existing ones unchanged
  • ExecuteUpsertAsync<TEntity>(this DbSet<TEntity> set, IEnumerable<TEntity> entities, Expression<Func<TEntity, object?>>? updateProperties = null, CancellationToken cancellationToken = default), in namespace Microsoft.EntityFrameworkCore, beside GraphBridge's other EF extensions.
  • The Execute prefix says what ExecuteUpdateAsync says: the statement runs now, set-based, and bypasses the change tracker. It sits on DbSet<T> rather than IQueryable<T> because its input is entities, not a query. If EF Core later adds an upsert with the same name, GraphBridge adopts EF's API in a later preview.
  • On create, every stored mapped property is written, as SaveChanges writes an added entity.
  • On match, the properties named by updateProperties are copied from the incoming entity. null (the default) means every stored non-key property; an empty selector means none. A selector is used instead of ExecuteUpdate's setter builder because an upsert copies values from the incoming entity, while setters compute values from the stored row. The selector accepts the same member access and anonymous-type shapes as EF's HasKey, and anything else is refused.
  • Result: GraphUpsertResult(int Created, int Matched, GraphWriteCounters Counters). Created is the summary's NodesCreated; Matched is the row count minus Created. No extra round trip (ADR 0050).
  • Asynchronous only, like the rest of GraphBridge's write paths.

2. Key and constraint

  • The key is the EF primary key, mapped to its stored property names, with client-supplied values. Temporary or store-generated key values are refused, as SaveChanges refuses them.
  • A uniqueness constraint must cover the key, checked with SHOW CONSTRAINTS before any write, as SaveChanges and bulk mutations already check. Without it, MERGE is not safe against concurrent writers, so the call is refused before sending.
  • Composite keys follow SaveChanges: supported when the same constraint check accepts them.

3. Rows in one call

  • At most 10,000 entities per call (ADR 0038's cap), counted while enumerating; the 10,001st is refused before anything is sent.
  • Duplicate keys in one call are refused before sending. Keys are compared by their stored values, after value conversion, because that is what the graph compares. "Last one wins" was rejected: it hides a likely bug and makes the result depend on input order.
  • One statement, one transaction:
UNWIND $__gb_upsert_rows AS row
MERGE (n:`Person` {`person_key`: row.key.`person_key`})
ON CREATE SET n += row.create
ON MATCH SET n += row.update
RETURN count(n) AS affected

Each row carries its key map, its create map and its update map, built with the same GraphPropertyBinding conversions as SaveChanges. A null in the update map removes the property, as an EF null does on update. The call fails unless affected equals the row count. - Transactions: outside an application transaction, the call owns one transaction, with the context's command timeout, and commits it. Inside one, it runs in the caller's transaction, and the caller commits, as with bulk mutations.

4. Concurrency and retries

  • Concurrent writers. With the uniqueness constraint, Neo4j locks the key's index entry for a MERGE, so two calls that create the same new key don't create two nodes: the second waits for the first and then matches its node. This is the expected behavior on Neo4j 2026.09, and the live proof below must confirm it. If the server instead reports a constraint violation or a deadlock for the losing call, that error propagates unchanged, with error.type on the activity, and the call can be repeated.
  • Retries. GraphBridge doesn't retry automatically. A repeated call with the same entities produces the same graph, because an upsert writes the incoming values; only Created and Matched differ. That makes a repeat safe after a transient error or an uncertain commit.
  • Concurrency tokens: entity types with a concurrency token are refused in this slice. Checking a token on match needs its own design (FUT-21).

5. What stays out

Refused with a clear message, not silently partial:

  • native-reference dependents, whose foreign keys are stored as relationships (association and identified edges are supported since the amendment in section 6);
  • shadow properties, because the entities passed in carry no value for them;
  • server-generated keys and concurrency tokens.

Owned types and inheritance are refused earlier, when the model is built, for every GraphBridge operation, so an upsert never sees them.

Tracked instances are not refreshed, as with ExecuteUpdateAsync. The entities passed in are value carriers: their tracking state is neither read nor changed. Tracked upserts through SaveChanges and edge upserts between existing endpoints are later work.

6. Relationship upserts (amendment, 24 October 2026, PL-47)

Proposed on 24 October 2026 with its code; accepted on 25 October 2026 after the review below. GraphRAG's mentions and extracted relationships need idempotent writes, and Neo4jClient users MERGE relationships.

The same call takes relationship sets: db.Mentions.ExecuteUpsertAsync(mentions). The API, the result and the transaction rules don't change.

  • Identified edges match on their key, between the endpoints the row's foreign keys name:
UNWIND $__gb_upsert_rows AS row
MATCH (s:`Person` {`person_key`: row.source.`person_key`})
MATCH (t:`Chunk` {`Id`: row.target.`Id`})
MERGE (s)-[r:`MENTIONED_IN` {`Id`: row.key.`Id`}]->(t)
ON CREATE SET r += row.create
ON MATCH SET r += row.update
RETURN count(r) AS affected
  • Association edges match on their endpoint pair: MERGE (s)-[r:TYPE]->(t), with the payload in the create and update maps.
  • Rows carry {position, source, target, key, create, update}. Endpoint keys are the foreign key values converted with the principal key's bindings, which model validation already requires to share a converter.
  • A preflight runs first, in the same transaction. It looks up both endpoints of every row, takes their write locks, and counts existing relationships of the type that would conflict. It refuses the whole call, naming zero-based positions and never values, when:
  • an endpoint is missing;
  • for an identified edge, its key already belongs to a relationship between other endpoints, which a MERGE would try to create again and the uniqueness constraint would reject. An upsert never moves a relationship to other endpoints;
  • for an association edge, the pair already has more than one relationship of the type, which SaveChanges also refuses.
  • Locks make it safe. The endpoint locks are the statement SaveChanges uses (SET n.key = n.key). They serialize concurrent upserts of the same pair or key, so the second waits and then matches. Identified edges also need their relationship uniqueness constraint, checked before any write as SaveChanges checks it; endpoint labels need their node constraints.
  • Two calls that lock the same nodes in a different order can deadlock. Neo4j aborts one with a transient error; nothing of it is written, and repeating it is safe.
  • Two concurrent calls that give one identified key to different endpoints lock different nodes, so neither preflight sees the other. The second fails on the uniqueness constraint and writes nothing.
  • In one call, duplicate identified keys and duplicate association pairs are refused before sending. The MERGE must report exactly one relationship per row; otherwise the transaction aborts.
  • Result: Created is the summary's RelationshipsCreated.
  • Vectors: a refused vector in a relationship's payload names the entity's position, as for nodes (ADR 0054).
  • Still refused: concurrency tokens, shadow properties (including shadow foreign keys of implicit many-to-many join types), sentinel identified keys, and an updateProperties selector that names the key or an endpoint foreign key. Native-reference dependents stay refused: their link is written with the dependent node, not as an edge entity.

Verification for the amendment: - Unit: rendered preflight and MERGE for both edge kinds and their parameters; converted endpoint keys; the refusals for duplicates, sentinel keys, shadow foreign keys, concurrency tokens and selectors naming keys or endpoints; refusal messages built from preflight rows, with positions only; the result from the counters. - Live, Neo4j 2026.09: create then match for both kinds, with payload updates and an empty selector; a missing endpoint, a key bound to other endpoints and a duplicated pair are refused with nothing written; two concurrent calls creating the same relationships leave one relationship per key or pair; a missing relationship constraint is refused before any write.

Review (25 October 2026). Checked against the code, SaveChanges and the tests: - The rendered statements match the text above. The preflight's lock is SaveChanges's lock statement, and its conflict lookup uses the relationship constraint's index. - Every refusal names positions only; values and keys never reach a message or a log. - The exactly-one-per-row check after the MERGE is the backstop: if an endpoint lookup matched more than one node, the call would abort rather than write twice. - Changes from the review: the deadlock and concurrent-key cases above are now documented; a refused vector in a relationship payload names its position; unit tests added for the concurrency-token and missing-relationship-constraint refusals, and a live test for the missing constraint; the public API's summary and the Neo4jClient guide now mention relationships.

Not adopted

  • A Fluent Cypher Merge clause. Fluent Cypher stays read-only (PL-34).
  • Upsert state inside SaveChanges. EF has no entity state for "insert or update"; adding one would change what Added and Modified mean.
  • Batches committed separately within one call. A partial upsert would leave the caller unable to tell what was written. Callers who need more than 10,000 rows make several calls, each atomic.

Verification

  • Unit: the rendered Cypher and parameters; conversions through value converters; the updateProperties shapes and refusals; the refusals for duplicates, the 10,001st row, missing constraints, temporary keys, concurrency tokens, association and identified-edge entity types, native-reference dependents and shadow properties; owned and borrowed transactions; the result built from the counters. The model-build refusals of owned types and inheritance are covered by the model validation tests.
  • Live, Neo4j 2026.09:
  • created and matched counts equal an independent readback, and every stored value equals the same entity saved with SaveChanges, across the mapped property types;
  • updateProperties changes only the named properties, and an empty selector leaves existing nodes unchanged;
  • a missing constraint is refused before any write;
  • a failure inside an application transaction rolls back everything, including earlier statements in that transaction;
  • two concurrent calls creating the same keys leave exactly one node per key;
  • a cancelled call writes nothing.
  • Benchmark: 1,000 and 10,000 rows against SaveChanges and against a hand-written UNWIND … MERGE, recorded with the FS-20 bulk baseline. Changed on acceptance: the guarded benchmark harness's entity carries a row version, which upserts refuse, so an upsert mode needs its own entity, constraint, cleanup and runner checks; that is a separate task. An indicative measurement on the owned fixture (five samples after one warm-up, Release build) found no pathological cost: for 10,000 rows, median 260 ms for ExecuteUpsertAsync creating nodes, 238 ms matching them and 297 ms for the hand-written statement, against 14.3 s for SaveChanges inserts. No performance claim is published from it.

Result, 2026-10-23: 844 provider units (seven UpsertTests) and the six live ProviderUpsertTests pass, with the whole Integration suite (305). Concurrent writers: in three runs, two simultaneous calls over the same 200 new keys left exactly one node per key and neither failed; one call created all 200 nodes and the other matched them. The test does not prove the two statements overlapped on the server.

Correction, 2026-10-24: the unit list above named the relationship, native-reference and shadow-property refusals, but the suite had no test for them, and it listed owned types and inheritance as upsert refusals, which the model build makes unreachable. A unit test now covers the three refusals, and the list says where the other two are tested. The refusals themselves were unchanged.

Consequences

  • Ingestion: GraphRAG ingestion (PL-32) and Neo4jClient users' MERGE code get an EF-shaped equivalent with converters and constraint checks.
  • Scope: nodes, and since the 24 October amendment association and identified edges between existing endpoints.
  • Cost: one more public API to keep stable through the preview, and one more statement shape to keep in the compatibility probes.