We Recovered a Remote Catalog Orphan After April Content Update Receipt Passed - 2026 Case Study

This is a synthesized case study—a field pattern from April–June 2027 content-update weeks where teams filed april_content_update_warm_catalog_receipt_v1.json with warm_catalog_ok: true, watched A1–A6 go GREEN, and still saw Summer Sale browsers request orphan bundle hashes the new catalog no longer named. Players downloaded ghost bundles: old edge objects, leftover origin keys, or stale catalog edges that proved “warm” for the smoke label while other groups still resolved dead URLs. It is not a named studio turnaround. There are no invented download counts, no fake Crashlytics rates, and no fabricated refund percentages.
What follows is the failure signature, a D1–D3 recovery timeline, remote_catalog_orphan_recovery_receipt_v1.json with gates R1–R6, a player-visible checklist, and a fail-closed verify script teams run when warm-path receipts pass but CDN still serves orphan hashes.
Pair April warm catalog evening, Addressables remote catalog drift loop, Invalid Key audit challenge, Friday March capstone, Help #49 forward, forward blog #8 jq weekly catalog row diff + #10 numbered PATCH exception opinion.
Non-repetition note: Warm catalog evening owns A1–A6 candidate warm-path proof. Drift validation loop owns patch-week ongoing observation. Invalid Key challenge owns address/key audit before promotion. This URL owns orphan hash recovery after warm_catalog_ok already passed—ghost CDN keys, not another warm-path tutorial.
Why this matters now (April 2027 → Summer Sale traffic)
- Warm smoke ≠ whole catalog — A1–A6 often prove one remote label and one content hash; orphan groups can still 404.
- Summer Sale amplifies ghosts — Higher concurrent downloads surface edge keys the smoke player never touched.
- GREEN boolean theater — Producers cite
warm_catalog_okwhile support screenshotsRemoteProviderExceptionon a different key. - Partial uploads — New catalog uploaded; old
.bundleobjects left public without being referenced—or worse, still referenced by a stale edge catalog. - Forward weekly diffs — Blog #8 jq weekly catalog row diff expects orphan recovery before trusting week-over-week GREEN rows.
Direct answer: Recovery required orphan inventory from CDN vs ServerData, purge or unpublish dead keys, re-smoke non-smoke labels, file recovery receipt with R1–R6, and remote_catalog_orphan_recovery_ok: true before clearing Summer Sale content-update incidents as “warm path already passed.”
Beginner quick start — what recovered means here
Recovered in this pattern means:
- Admitted that
warm_catalog_okdid not cover every public remote key still reachable. - Listed orphan hashes (
orphan_inventory_v1.json) comparing CDN listing to active catalog references. - Removed or blocked orphans from public load paths (purge + origin delete + cache invalidation evidence).
- Re-proved at least two non-smoke labels plus the original smoke label on an installed player.
- Filed
remote_catalog_orphan_recovery_receipt_v1.jsonwith R1–R6 GREEN and set BUILD_RECEIPTremote_catalog_orphan_recovery_ok: true.
Success check: A clean-account player loads the sale-week content labels without requesting any hash listed in the orphan inventory; curl against purged URLs returns 404/410 (or host equivalent), not a silent 200 with old bytes.
The player-facing failure (typical signals)
| Channel | Typical wording |
|---|---|
| Player / Steam discussion | "Update downloaded then characters missing / pink textures" |
| Support | Screenshots of Unity Addressables remote load errors on keys that “worked in CI” |
Internal #content-update |
"Warm catalog GREEN—why are Sale players still failing?" |
| CDN metrics | Requests for hashes absent from current catalog JSON |
| Producer | "We already ran the warm evening—do not rollback" |
Internal language before fix: "Players are on bad cache" — when root cause was orphan objects still public or stale catalog at a second edge, not player malice.
Starting state (what was wrong)
| Layer | Symptom |
|---|---|
| Warm receipt | warm_catalog_ok: true, A1–A6 GREEN |
| BUILD_RECEIPT | april_content_update_allowed: true, warm row latched |
| Smoke label | april-warm-smoke loads on installed build |
| CDN origin | Previous content-build .bundle hashes still public |
| Edge | Or a second catalog path still serving catalog hash N−1 |
| Summer Sale players | Loads labels outside the smoke set → orphan URL 200/404 chaos |
The March capstone and warm evening may have been legitimately correct for their scope. The failure is orphan surface area, not necessarily a lie in A3/A4 evidence.
Root cause pattern (synthesized)
| Mechanic | What drifted |
|---|---|
| Narrow smoke label | Warm evening loaded one safe label; sale build loads five groups |
| Overwrite without GC | New ServerData uploaded; old objects never deleted from bucket |
| Dual catalog URLs | Profile RemoteLoadPath pointed at path A; leftover marketing/doc still mentioned path B |
| Cache longer than catalog | Edge kept catalog hash N−1 after origin updated to N |
| Partial multi-region sync | Smoke region warm; Sale traffic hit colder region still serving orphans |
| Invalid keys deferred | Renamed addresses left old hashed files reachable—pair Invalid Key challenge after recovery |
None require malice—only one-evening scope plus Summer Sale concurrency exposing the rest of the remote graph.
Timeline D1–D3 (recovery pattern)
| Day | Focus | Output |
|---|---|---|
| D1 | Admit + inventory | orphan_inventory_v1.json + freeze new content publishes |
| D2 | Purge orphans + re-warm broader labels | CDN purge evidence + player multi-label smoke |
| D3 | Recovery receipt + BUILD_RECEIPT latch | remote_catalog_orphan_recovery_receipt_v1.json |
Teams with the drift loop already running caught orphans in HEAD checks; teams that stopped at warm evening learned on Sale traffic.
D1 — inventory discipline (beginner path)
Goal: Stop arguing about warm GREEN. List every public remote object that is not referenced by the active catalog.
Player-visible checklist (copy)
# Remote catalog orphan — D1 player-visible checklist
build_label: {same string as warm receipt}
active_catalog_hash: {sha256 from warm receipt}
smoke_label: april-warm-smoke
sale_labels: {list of labels Summer Sale players hit}
cdn_base: {RemoteLoadPath}
orphan_freeze: no new Addressables content publish until R5
[ ] Screenshot player error (redact PII)
[ ] Note exact Addressables key / location that failed
[ ] Confirm warm smoke label still loads (yes/no)
[ ] Export current catalog from CDN (not only ServerData)
[ ] Start orphan_inventory_v1.json
Orphan inventory schema
{
"schema": "orphan_inventory_v1",
"captured_at": "2027-04-12T18:00:00Z",
"active_catalog_hash": "sha256:aaa111…",
"cdn_base": "https://cdn.example.com/addressables/april-2027/",
"referenced_hashes": ["bbb222…", "ccc333…"],
"public_object_hashes": ["bbb222…", "ccc333…", "ddd444…", "eee555…"],
"orphans": [
{
"hash": "ddd444…",
"url": "https://cdn.example.com/addressables/april-2027/ddd444….bundle",
"http_status_before_purge": 200,
"reason": "not_in_active_catalog"
},
{
"hash": "eee555…",
"url": "https://cdn.example.com/addressables/april-2027/eee555….bundle",
"http_status_before_purge": 200,
"reason": "referenced_only_by_stale_catalog_edge"
}
]
}
Beginner method without fancy CDN APIs:
- Save the active catalog JSON from the public URL (same path the warm evening used).
- Extract every bundle hash / filename the catalog names.
- From your upload log or bucket listing of the same prefix, list objects still public.
- Diff → orphans.
R1 pass: Inventory filed with ≥0 orphans; if zero orphans but players fail, escalate to stale catalog edge hypothesis (orphan inventory notes stale_catalog_url).
D2 — purge and broader smoke (developer path)
Goal: Make orphans unreachable; prove labels Sale players use—not only april-warm-smoke.
| Step | Action | Gate |
|---|---|---|
| 1 | Delete or unpublish orphan objects at origin | R2 |
| 2 | Invalidate CDN/cache for orphan URLs + catalog URL | R2 |
| 3 | curl -I orphan URLs → expect miss/404/410 |
R3 |
| 4 | Re-download active catalog; confirm hash still matches warm receipt or bump with documented content republish | R3 |
| 5 | Installed-player smoke: original label + ≥2 sale labels | R4 |
| 6 | Optional: run one Invalid Key spot-check on renamed addresses | cousin |
curl proofs (sketch)
# Orphan must not serve body after purge
curl -sI "$ORPHAN_URL" | tee release-evidence/april-content/orphan/d2_orphan_head.txt
# Active catalog still reaches intended hash
curl -sL "$CATALOG_URL" | sha256sum | tee release-evidence/april-content/orphan/d2_catalog_sha256.txt
Honest limits: Host providers differ on 404 vs soft 200 empty objects. Capture status + body hash either way—do not mark R3 GREEN on a 200 that still returns old bytes.
If you must republish content: Treat it as a new warm-path candidate—re-run A1–A6 on the new catalog hash, then continue orphan recovery against the new active set. Do not mutate the old warm receipt in place; add receipt_version notes.
D3 — recovery receipt and latch
Goal: File evidence that survives the next partner or Sale-week review.
| Step | Action | Gate |
|---|---|---|
| 1 | Attach inventory + purge proofs | R5 |
| 2 | Confirm player multi-label smoke artifacts | R5 |
| 3 | Write remote_catalog_orphan_recovery_receipt_v1.json |
R5 |
| 4 | Set BUILD_RECEIPT remote_catalog_orphan_recovery_ok: true |
R6 |
| 5 | Explicitly do not unset warm_catalog_ok unless warm evidence was false—recovery is additive |
R6 |
| 6 | Schedule forward #8 weekly catalog diff so orphans cannot reappear silently | forward |
Developer path — gates R1–R6
| Gate | Pass criterion |
|---|---|
| R1 | orphan_inventory_v1.json committed; active_catalog_hash matches current public catalog (or documented republish) |
| R2 | Every orphan has purge/unpublish evidence path |
| R3 | Post-purge HEAD/GET shows orphans unreachable or non-matching body hash |
| R4 | Installed player loads smoke label + ≥2 additional remote labels without orphan URLs |
| R5 | remote_catalog_orphan_recovery_receipt_v1.json under release-evidence/april-content/orphan/ |
| R6 | BUILD_RECEIPT remote_catalog_orphan_recovery_ok: true matches receipt |
remote_catalog_orphan_recovery_receipt_v1.json (template)
{
"schema": "remote_catalog_orphan_recovery_receipt_v1",
"content_window": "2027-04-content-update",
"generated_at": "2027-04-14T21:00:00Z",
"warm_cousin": "release-evidence/april-content/april_content_update_warm_catalog_receipt_v1.json",
"inventory_path": "release-evidence/april-content/orphan/orphan_inventory_v1.json",
"active_catalog_hash": "sha256:aaa111…",
"orphans_purged": ["ddd444…", "eee555…"],
"player_labels_smoked": ["april-warm-smoke", "april-characters", "april-ui"],
"gates": {
"R1": "GREEN",
"R2": "GREEN",
"R3": "GREEN",
"R4": "GREEN",
"R5": "GREEN",
"R6": "GREEN"
},
"cousin_receipts": {
"warm_catalog": "release-evidence/april-content/april_content_update_warm_catalog_receipt_v1.json",
"march_capstone": "release-evidence/post-fest/sprint-2027-W13/march_capstone_closeout_receipt_v1.json"
},
"remote_catalog_orphan_recovery_ok": true,
"notes": "warm_catalog_ok remains true; smoke scope was too narrow for Sale labels"
}
BUILD_RECEIPT root rows (D3 commit)
{
"warm_catalog_ok": true,
"remote_catalog_orphan_recovery_ok": true,
"remote_catalog_orphan_recovery_receipt": "release-evidence/april-content/orphan/remote_catalog_orphan_recovery_receipt_v1.json"
}
verify_remote_catalog_orphan_recovery.sh
#!/usr/bin/env bash
set -euo pipefail
REC="${1:-release-evidence/april-content/orphan/remote_catalog_orphan_recovery_receipt_v1.json}"
INV="${2:-release-evidence/april-content/orphan/orphan_inventory_v1.json}"
jq -e '.schema == "remote_catalog_orphan_recovery_receipt_v1"' "$REC" || exit 1
jq -e '.schema == "orphan_inventory_v1"' "$INV" || exit 2
jq -e '(.orphans_purged | length) >= 0' "$REC" || exit 3
for g in R1 R2 R3 R4 R5 R6; do
jq -e --arg g "$g" '.gates[$g] == "GREEN"' "$REC" || exit 4
done
jq -e '.remote_catalog_orphan_recovery_ok == true' "$REC" || exit 6
jq -e '(.player_labels_smoked | length) >= 3' "$REC" || exit 5
echo "remote_catalog_orphan_recovery verify: OK"
Fail-closed: content-update promotion branches that declare Sale readiness should require R6 when support already opened an orphan incident—not only warm_catalog_ok.
RED / YELLOW / GREEN routing
| Signal | Route |
|---|---|
| R1 RED | Cannot list CDN objects—stop Sale claims; get bucket listing access first |
| R2 RED | Orphans still public after “purge”—verify wrong prefix / dual base URL |
| R3 YELLOW | 200 with empty body still suspicious—capture body sha256 |
| R4 RED | Sale labels fail after purge—republish missing referenced bundles, then re-warm |
| Invalid Key errors | After orphans cleared, run Invalid Key challenge K gates |
| Drift loop RED mid-Sale | Keep orphan receipt; open drift loop ticket separately |
Worked example — D1 through D3 (synthetic)
| Day | Finding | Action |
|---|---|---|
| D1 | Catalog references 12 hashes; bucket has 14 public | Two orphans inventoried |
| D1 | april-warm-smoke still GREEN |
Freeze new publishes |
| D2 | Purge two orphans; invalidation IDs filed | R2–R3 |
| D2 | april-characters failed once—missing referenced bundle |
Re-upload missing referenced object (not orphan) |
| D3 | Three labels GREEN; receipt latched | R6; Sale incident closed as orphan—not “players lying” |
Stand-up note:
## Orphan recovery D3
- warm_catalog_ok: still true (narrow smoke was honest)
- orphans_purged: 2
- player_labels_smoked: 3
- remote_catalog_orphan_recovery_ok: true
- next: #8 weekly catalog row diff
How this differs from cousins
| Artifact | Owns |
|---|---|
| Warm catalog evening | Candidate warm path A1–A6 |
| Drift loop | Ongoing patch-week HEAD + player smokes |
| Invalid Key challenge | Address/key integrity before promotion |
| This case study | Orphan hash recovery after warm GREEN |
| Forward #8 jq weekly | Week-over-week catalog row diff |
| Forward #10 PATCH opinion | Numbered PATCH exception discipline |
Order after failure: Admit → inventory (D1) → purge + multi-label smoke (D2) → recovery receipt (D3) → optional Invalid Key spot-check → resume weekly diffs.
Prevention when warm evenings go well
- Smoke ≥3 remote labels on every warm receipt—not one.
- Diff bucket listing vs catalog before setting
warm_catalog_okif the bucket is small enough to list. - Document single
RemoteLoadPath; kill dual URL folklore. - After every content build, schedule origin GC for superseded hashes within 24 hours.
- Keep drift loop running through Sale week even if warm evening passed.
- Forward #8 weekly jq diff catches silent reintroduction of orphans.
Common mistakes
Blaming players for “bad cache” — Always inventory public hashes first.
Unsetting warm_catalog_ok reflexively — Narrow smoke can be honest; recovery is additive.
Deleting referenced bundles — Only purge orphans; missing referenced hashes need re-upload + re-warm.
Editor-only retest — R4 requires installed player.
Skipping invalidation — Origin delete without edge purge recreates the ghost.
Merging orphan OK with Invalid Key RED — Different receipts; clear orphans first, then rename keys carefully.
No freeze on D1 — New publishes mid-recovery recreate the inventory race.
Format ladder (April content-update cluster)
| Artifact | Role |
|---|---|
| Friday March capstone | april_content_update_allowed |
| Warm catalog #1 | Candidate warm path |
| This case study | Orphan recovery after warm GREEN |
| Drift loop | Patch-week observation |
| Forward #8 jq weekly | Catalog row diffs |
| Forward #10 PATCH opinion | Numbered PATCH exceptions |
| Help #49 forward | Symptom-first orphan / load failure fix |
Outbound references
- Unity Addressables documentation
- Addressables remote content
- Host CDN invalidation docs for your provider (AWS CloudFront, Cloudflare, etc.)—file invalidation IDs in R2 evidence
Key takeaways
warm_catalog_okcan be true while orphans remain public—scope of smoke matters.- Summer Sale traffic exposes groups the warm label never touched.
- Gates R1–R6 turn orphan panic into inventory → purge → multi-label smoke → latch.
orphan_inventory_v1.jsonis the first honest artifact.- Purge orphans only—re-upload missing referenced hashes and re-warm.
- Keep warm and orphan receipts separate; recovery is additive.
- Pair warm catalog evening for prevention width.
- Pair drift loop during Sale week.
- Pair Invalid Key challenge after ghosts clear if renames remain.
- Forward #8 weekly catalog diff prevents silent re-orphaning.
- Forward #10 numbered PATCH opinion covers exception discipline mid-Sale.
- Help #49 forward owns symptom-first player fix page.
verify_remote_catalog_orphan_recovery.shfail-closes false GREEN.- D1 freeze stops inventory races.
FAQ
Does this mean the warm catalog evening was useless?
No—it proved a candidate path. Orphans are a coverage problem: smoke one label, ship many.
Should we roll back the April content update?
Only if R4 fails after purge and referenced assets are missing. Prefer purge + multi-label smoke + re-warm over blind rollback.
Is an orphan the same as an Invalid Key?
No. Orphans are extra public hashes (or stale catalogs). Invalid Key is wrong address/key mapping in Unity. You can have either or both.
Can Editor Play Mode replace D2 player smoke?
No—R4 requires an installed build against the public CDN.
What if the bucket listing is unavailable?
R1 RED until you can list or reconstruct from upload logs; do not claim R2 from memory.
How does Help #49 differ?
Help = fastest safe player-facing fix steps; this case study = recovery receipt + gates after warm GREEN.
When do we run jq weekly catalog diffs?
After R6—forward blog #8—so orphans cannot return between Sale weekends.
Related GamineAI reads
- Your First April Content Update Addressables Warm Catalog Receipt
- Addressables Remote Catalog Drift - Unity 6 Production Validation Loop
- 7-Day Unity Addressables Invalid Key Audit Challenge
- Friday March Capstone Closeout Before April Content Update
- Build Content Hash Lockfiles - Unity Addressables CI
- Your First BUILD_RECEIPT JSON and Upload Log
- Unity Addressables Invalid Key at Runtime (help)
- jq Gate Sketch for april_content_update Weekly Catalog Row Diff — week-over-week catalog hashes after R6
- Stop Shipping Content Updates Without a Numbered PATCH Exception List — PATCH-00N before CDN / catalog publish
- Forward: Help #49
Closing
Warm GREEN is not orphan GREEN—when Sale players still fetch ghosts, inventory the public hashes, purge what the active catalog no longer names, re-smoke the labels browsers actually hit, and file remote_catalog_orphan_recovery_receipt_v1.json. Keep the warm catalog evening as the daily candidate proof, keep weekly catalog jq for Monday identity drift, authorize ships with the PATCH exception opinion, and keep this recovery pattern ready for the week the smoke label was honest but too small.