A stored regional total can make dashboards faster, but it introduces a lifecycle problem: what happens when the source changes or a region disappears? $merge writes rows emitted by an aggregation; it does not automatically discover and delete every old target row the new aggregation no longer emits. This MongoDB 8.0 reference goes beyond the introductory stage example into complete refresh policy. The main mongosh fixture uses a serialized full refresh with a generation marker and stale-row cleanup. The worked exercise stores separate generations and publishes a pointer only after a complete build. Both operate solely in disposable teaching collections.
Define summary identity and replacement ownership
Choose the fields that identify one summary row. A regional report can use region, while a daily report might require both region and day. The on fields used by merge need the documented supporting unique index and must obey field-value constraints for the pinned server version. Create that index deliberately before the refresh. An ordinary nonunique index does not establish the required identity contract. Project out an inherited source _id when it is not the summary's intended identifier.
whenMatched: replace treats the new row as the owned summary representation, while the default merge behavior combines fields and can preserve old fields the refreshed output no longer supplies. Choose according to ownership. A target holding both calculated fields and manually edited annotations needs a different policy from a purely generated summary collection. whenNotMatched: insert creates newly appearing groups. None of these choices makes the entire multi-row refresh atomic, and a write error can leave earlier target changes completed.
Handle groups absent from the new result
An aggregation groups only the source rows it receives. If all region-B source rows are deleted, no B summary is emitted. Merge therefore has no incoming B row to replace or delete. The old target B row can remain indefinitely unless the refresh policy handles absence. Do not assume replace means replace the whole collection; it replaces matched documents individually. This stale-row issue is the main fixture's deliberate failure boundary.
A serialized full refresh can mark every emitted row with a refresh identifier, then remove rows in the same report scope carrying older identifiers after the build succeeds. That cleanup must know the complete intended scope. A partial-region refresh must not delete other regions. A failed or incomplete build must not be followed by broad stale cleanup. Concurrent runs can race over markers and deletion, so this simple policy requires serialization or stronger coordination. It is appropriate for the controlled fixture, not a universal concurrent dashboard updater.
Separate building from publishing when readers need coherence
Readers can observe mixed old and new rows during an in-place multi-row refresh. A generation design writes all rows under a new generation key, then updates a small active-generation pointer after the build completes. Readers obtain the active generation and query only its rows. Rows absent from the new generation naturally disappear from the active report without deleting old rows during the build. A unique index includes both generation and summary key.
Publication still needs a defined concurrency policy. A conditional pointer update can prevent a stale builder from replacing a newer published generation. Keep old generations long enough for readers that already obtained their pointer, then garbage-collect under a retention rule. A failed generation should never become active. The exercise demonstrates this architecture with one serialized builder and a conditional pointer change. It does not claim that reading the pointer and rows automatically supplies every consistency guarantee under arbitrary deletion, replication, or concurrent publishing.
Specify refresh inputs, retries, and operational limits
A repeatable refresh needs a defined source population and read-consistency contract. A generation tag controls output identity but does not freeze source data while a long aggregation runs. Decide whether a live approximate dashboard is acceptable or a point-in-time report needs stronger reads. Bound the report period, monitor duration, and provide enough job metadata to identify completed versus failed builds. Keep a clear last-success timestamp so users understand staleness.
Rerunning deterministic replace or desired-value updates can be safer than applying incremental totals twice, but inspect the actual merge policy. An incrementing whenMatched pipeline can double-count on a retry. Source corrections and deletions also complicate incremental maintenance. Rebuilds, change-driven updates, and generation publication have different cost and correctness tradeoffs. The examples intentionally recompute small totals from the full controlled source. For a large system, measure indexes and aggregation costs, coordinate builders, and test absent groups and partial failure before presenting materialization as merely a faster read query.
MongoDB documentation and version notes
MongoDB Materialized Summary Refresh Examples
Each example shows the code, the output it produces and what each part does.
Refreshing totals with $merge and stale-row cleanup
Run the complete mongosh fixture on MongoDB 8.0. The first build stores A and B totals under refresh r1. Source B is then removed and one A amount changes. The r2 build replaces A but emits no B row, leaving one stale target document. Only after the successful serialized build does the fixture remove older markers in this entire demo report. The final output contains A with its corrected total. This exposes the policy required for deletion rather than hiding it inside an assumed merge behavior. The collection is wholly generated, so replace does not erase user-owned annotations.
const demo = db.getSiblingDB('codevra_ref_refresh');
const orders = demo.getCollection('orders');
const totals = demo.getCollection('totals');
orders.deleteMany({}); totals.deleteMany({});
totals.createIndex({ region: 1 }, { unique: true });
orders.insertMany([{ _id: 1, region: 'A', amount: 10 },
{ _id: 2, region: 'A', amount: 20 }, { _id: 3, region: 'B', amount: 7 }]);
function refresh(refreshId) {
orders.aggregate([
{ $group: { _id: '$region', total: { $sum: '$amount' }, orders: { $sum: 1 } } },
{ $project: { _id: 0, region: '$_id', total: 1, orders: 1 } },
{ $set: { refreshId } },
{ $merge: { into: 'totals', on: 'region', whenMatched: 'replace', whenNotMatched: 'insert' } }
]).toArray();
}
function report() { return totals.find().sort({ region: 1 }).toArray().map(x => ({ region: x.region, total: x.total, orders: x.orders })); }
refresh('r1'); print(JSON.stringify(report()));
orders.deleteMany({ region: 'B' });
orders.updateOne({ _id: 1 }, { $set: { amount: 15 } });
refresh('r2');
print('stale ' + totals.countDocuments({ refreshId: { $ne: 'r2' } }));
totals.deleteMany({ refreshId: { $ne: 'r2' } });
print(JSON.stringify(report()));[{"region":"A","total":30,"orders":2},{"region":"B","total":7,"orders":1}]
stale 1
[{"region":"A","total":35,"orders":2}]MongoDB Materialized Summary Refresh in Real Projects
Publish a report generation with a pointer swap
Build a new generation without changing the active report, then publish it with a conditional pointer update. The solution retains generation g1 while g2 is written, so the first printed active report still contains both original regions. After publication, active readers see only A's new total. Old rows remain available for a separate retention policy. The pointer condition expects g1, preventing this demonstrated builder from blindly replacing a pointer that another publisher already moved. A production builder must also record successful completion and coordinate cleanup. The practice's static finite source keeps those lifecycle concepts visible without claiming to solve every concurrent consistency scenario.
const demo = db.getSiblingDB('codevra_ref_refresh');
const source = demo.getCollection('generation_source');
const rows = demo.getCollection('generation_rows');
const head = demo.getCollection('generation_head');
source.deleteMany({}); rows.deleteMany({}); head.deleteMany({});
rows.createIndex({ generation: 1, region: 1 }, { unique: true });
source.insertMany([{ _id: 1, region: 'A', amount: 10 }, { _id: 2, region: 'B', amount: 5 }]);
function build(generation) {
source.aggregate([{ $group: { _id: '$region', total: { $sum: '$amount' } } },
{ $project: { _id: 0, region: '$_id', total: 1 } }, { $set: { generation } },
{ $merge: { into: 'generation_rows', on: ['generation', 'region'], whenMatched: 'replace', whenNotMatched: 'insert' } }
]).toArray();
}
function active() {
const generation = head.findOne({ _id: 'active' }).generation;
return rows.find({ generation }).sort({ region: 1 }).toArray().map(x => [x.region, x.total]);
}
build('g1'); head.insertOne({ _id: 'active', generation: 'g1' });
source.deleteOne({ _id: 2 }); source.updateOne({ _id: 1 }, { $set: { amount: 20 } });
build('g2'); print(JSON.stringify(active()));
print(head.updateOne({ _id: 'active', generation: 'g1' }, { $set: { generation: 'g2' } }).matchedCount);
print(JSON.stringify(active()));[["A",10],["B",5]]
1
[["A",20]]MongoDB Materialized Summary Refresh FAQ
Does whenMatched replace remove every old summary row?
No. It replaces a matched document when the pipeline emits a row with that identity. A group absent from the new output has no incoming row, so its old target can remain. The main fixture exposes B as stale after refresh. Define deletion or generation policy explicitly, including scope and successful-completion checks. Replace is a per-document merge action, not whole-collection replacement.
Why is a unique index required for the chosen on key?
It establishes the target identity under the documented merge requirements. A region summary should have one row per region in the in-place design, while the generation design permits one row per generation and region. Choose the key from the actual report shape and create the supporting index. Do not reuse a source _id accidentally when it no longer represents one summary row.
Can I clean stale markers after a failed build?
Not safely under the demonstrated policy. A failed build may have updated only part of the target, and broad cleanup could delete valid rows that were never refreshed. Cleanup belongs after verified complete success and within the intended scope. Concurrent builders also need coordination. A separate-generation design keeps incomplete output inactive and can make failure recovery clearer, but still needs publication and retention rules.
Does a generation make the source a snapshot?
No. It organizes output and publication, not the consistency of input reads. A long build can still observe source behavior under its selected read semantics. Define a point-in-time strategy if the report requires one. Keep old generations available long enough for readers using an earlier pointer, and do not delete them immediately without considering that lifecycle. Output coherence and source consistency are related but distinct guarantees.
MongoDB Materialized Summary Refresh Exercises
Answer each one and check it instantly. If you get stuck, open the hint before the answer.
What active report is visible after building g2 but before publishing its pointer?
MongoDB Materialized Summary Refresh Quiz
2 questions to check what you learned.
Why can a deleted source group leave a stale target summary?
- No row for that group reaches merge to replace or remove it
- Merge always restores deleted source documents
- A unique index prevents all source deletes
Why use replace for a wholly generated row?
- It removes obsolete generated fields instead of preserving them by field merge
- It makes all report rows refresh in one atomic write
- It automatically deletes every absent group