Back up and restore
Almost nothing in a PATINA deployment is precious, because almost all of it is derived from the blockchain. Knowing which part is which saves you from protecting the wrong thing.
- Which four items in a deployment are precious, and why the index itself is not one of them.
- What metadata has to travel with a snapshot for anyone to trust it after a restore.
- The restore order: check the metadata, restore into a store that is not serving, verify the root at the recorded height, catch up, cross check, then take traffic.
- The cheapest recovery path for six situations, including the one where restoring destroys the evidence.
- Why no backup can restore an artifact, undo a spend, or bring back a relic.
Precious or derived
Eight things live in a deployment. Only the first four cannot be recovered from the chain, and none of them is large enough to be a problem. If you protect nothing else, protect those.
| Item | Class | If you lose it |
|---|---|---|
| Deployment record | Precious | You do not know which window heights you were indexing. Keep it in version control. |
Specification and spec_sha256 | Precious | You cannot prove what rules you ran. Keep it in version control. |
| Configuration | Precious | Rebuildable by hand, slowly and with mistakes. Keep it in version control. |
| Verification log | Precious | You lose the last known good height, which is what makes an investigation finite. |
| The index itself | Derived | Nothing, except the time to rebuild it |
| Cached API responses | Derived | Nothing |
| Metrics history | Useful, not precious | You lose trend context for alerts |
| Bitcoin node data | Derived, expensive | Nothing, except a long resync |
The index contains no unique information. A snapshot exists so that recovery takes minutes instead of a rebuild, and for no other reason. If a snapshot is ever inconvenient or of doubtful integrity, throw it away and reindex. Rebuilding is always correct.
Snapshot procedure
-
Record the height and the root
Note
indexed_heightand the state root at that height. A snapshot without them cannot be verified after restore. -
Take a consistent copy
Stop writes, or use whatever consistent snapshot mechanism your storage engine provides. A copy taken while blocks are being applied is not a snapshot, it is a mixture.
-
Store the metadata with it
Network,
spec_sha256,parser_version,indexer_version,indexed_height, the state root, and the timestamp. -
Checksum the archive
So a corrupt restore is detected before it is served rather than after.
Restore procedure
-
Check the metadata first
Same network, same
spec_sha256. A snapshot from a different specification must not be restored, because it was produced by different rules. -
Restore into a store that is not serving
Never restore over a live index.
-
Verify the root at the recorded height
It must equal the root recorded when the snapshot was taken. If it does not, the snapshot is corrupt. Discard it and reindex.
-
Catch up
Let it index forward to the current tip with
/readyfailing throughout. -
Cross check before serving
Compare the state root against a second implementation at a common height. Only then take traffic.
Recovery paths, by situation
Find your row first, then act. One row tells you not to restore anything at all, because a restore would erase the evidence you need to explain what happened.
| Situation | Path |
|---|---|
| Process crashed, storage intact | Restart. Confirm it resumes from its last height. |
| Unclean shutdown, storage integrity unknown | Restore a snapshot, or reindex. Do not guess. |
| Storage lost | Restore a snapshot and catch up, or reindex. |
| Snapshot fails root verification | Reindex. Report the snapshot as bad so nobody else restores it. |
| State root disagrees with another implementation | Do not restore anything. Investigate first. Restoring hides the evidence. |
| Whole host lost | New host, node from scratch or from its own snapshot, then index restore or rebuild. |
Drill it
A restore procedure that has never been run is a hypothesis. Run it on a schedule:
- Restore the most recent snapshot onto a clean host.
- Verify the root at the recorded height.
- Catch up to tip.
- Compare against the live indexer at a common height.
- Write down how long the whole thing took. That number is your real recovery time, and it is the one to quote when someone asks.
What a backup cannot do
- It cannot restore an artifact. Artifacts live on the blockchain, not in your database.
- It cannot undo a spend, restore depth, or bring back a relic.
- It cannot recover anything for a user, because the indexer holds nothing that belongs to them.
That is the point of the architecture. The worst operational disaster in this stack costs time, not anyone's property. The public site puts the same case to holders, including what survives if this operation stops entirely, on every way this can let you down.