Incident actions
Know before the incident which levers exist. Most of the ones people reach for do not exist here, by design.
- The eight levers you actually have, and what each one costs for as long as it is pulled.
- The nine things nobody can do, including freezing an artifact and restoring depth after a reset.
- The first action for nine incident types, and the single incident where pulling a lever prevents real user loss.
- What to publish while it is still happening, and which sentence would be a lie.
- What to record afterwards, and when the cause deserves a new golden vector.
What can be paused
Eight levers. Failing /ready is the one to reach for first whenever you cannot stand behind
the answers you would serve: it takes this instance out of rotation and costs nothing on chain.
| Action | Effect | Cost |
|---|---|---|
Fail /ready | Load balancers stop sending traffic to this instance | Reads fail while it is off. Nothing on chain is affected. |
| Stop the indexer | No new blocks are processed | State goes stale. It catches up when restarted, with nothing lost. |
Set PATINA_ENABLED to false | The backend module unloads and the routes stop existing | The product surface disappears. Artifacts are unaffected. |
Set VITE_PATINA_ENABLED to false | The workspace is hidden in the frontend | People cannot use the interface. They can still use any indexer directly. |
| Disable the build helpers | No new unsigned plans are produced | Minting through our tooling stops. Anyone building transactions themselves is unaffected. |
| Rate limit or block traffic at the edge | Protects the instance | Legitimate readers are slowed too |
| Roll back to a previous build | Reverts serving behaviour | May require a reindex if the parser changed |
| Publish a notice | People learn what is happening | None. Do this early rather than last. |
What cannot be paused, by anyone
None of these has a lever, an override, or a key held by anybody. If an incident plan quietly depends on one of them, the plan is wrong, and the time to find that out is now rather than during the incident.
| Not possible | Why |
|---|---|
| Stopping mints | A mint is two Bitcoin transactions. Nothing in this stack is required for them. |
| Freezing an artifact | The artifact is an unspent output controlled by its holder's key |
| Reversing a spend | The spend is confirmed on Bitcoin |
| Restoring depth after a reset | Depth is derived from the carrier's height. There is nothing to edit. |
| Un-relicking an artifact | Relic is terminal in the state model |
| Extending the founding window | The heights are in the deployment record and the chain has already passed them |
| Changing a reason code after the fact | It is a function of a confirmed transaction |
| Hiding an artifact from the protocol | Anyone can index the same chain and see it |
| Editing rings | Rings are derived. Editing a database changes your copy, not the truth, and would show up as a state root mismatch. |
The only supported way to change indexed state is to change the blocks it comes from, which nobody can do, or to reindex. An operator who edits records has created a private fork of reality that will disagree with every other implementation at the next state root comparison.
The public site states the same limits to holders, switch by switch, in what the switches reach, and what they do not. Those two lists have to keep matching, so if you find a lever here that is missing there, one of the pages is wrong.
Incident playbooks
Find the row that matches what you are seeing and take the first action before you understand the cause. Understanding belongs in the third column, after the bleeding has stopped.
| Incident | First action | Then |
|---|---|---|
| Indexer far behind the tip | Fail /ready |
Find out whether the node or the indexer is stuck. Do not serve stale data as current. |
| Bitcoin node unreachable | Fail /ready |
Fix the node. The indexer must not guess in the meantime. |
| State root disagrees with another implementation | Publish the disagreement | Bisect to the first differing height, reduce to a vector, follow Report a disagreement. Do not change code by inspection. |
| Deep reorg beyond rollback history | Fail /ready |
Reindex from the deployment start height. |
| Spike in one invalid reason code | Read the transactions | A tool is probably broken. Publish guidance. This is not an outage. |
| Storage corruption | Take the instance out of rotation | Restore a verified snapshot, or reindex. See Back up and restore. |
| Traffic overload | Rate limit at the edge | Scale reads. Never shed load by serving cached numbers without their height. |
| A bug in the planner produces bad transactions | Disable the build helpers | This is the one case where pulling a lever prevents real user loss. Do it immediately, then explain. |
| Someone reports a lost artifact | Look up the artifact and the spend | Explain exactly what happened, honestly. There is nothing to restore, and saying otherwise would be a lie. |
Communication rules
- Say what is broken, what still works, and what you do not yet know.
- Say which heights are affected. Heights are checkable and adjectives are not.
- Never imply that artifacts are at risk when only your service is degraded. Those are different things.
- Never promise a recovery that the protocol cannot deliver.
- Publish the resolution, including what the cause was.
After every incident
- Record the last verified height and root from before the incident.
- Re-run verification against a second implementation before declaring it over.
- Add a golden vector if the cause was a parsing or state question.
- Write down which lever you wished you had, and whether it can exist without breaking the promise that nobody can freeze anything.