Verify and monitor

Verification proves the state is right. Monitoring tells you when that stops being true. You need both.

What you will know after this page

Which two checks can actually catch a wrong state, and which four pass happily while it is wrong.
How to compare state roots with a second indexer, and why comparing at the tip wastes a night.
The nine signals to watch, and why a rise in 503 during sync is the correct behaviour.
Which four reason code spikes mean a broken tool in the wild rather than a broken indexer.
What to write in the verification log so the next investigation is a bisection rather than a search.

Verification, in order of strength

Strength increases down the table, and so does cost. Start at the bottom when you have a real doubt, and keep the top rows for the things that run continuously.

Ways to check an indexer, weakest first.
Check	Proves	Run it
The process answers	Almost nothing	Continuously
Sync gap is small	The data is current, not that it is correct	Continuously
Golden vectors pass	The parser handles every known case correctly	Every build, before deploy
Spot checks against a block explorer	Individual artifacts look right	After any incident
State root matches a second implementation at the same height	The whole state agrees with another codebase	Daily, and after every reindex
A rebuild from scratch reproduces the same root	Your incremental path did not drift	Periodically, and after any rollback problem

The last two are the ones that matter. Everything above them can pass while the state is wrong.

Daily verification

Four steps and about a minute. Read both status responses, pick a height they have both passed, and compare the state roots there. The public site walks the same check by hand on take nobody's word for it.

# pick a height both have passed, then compare state roots at that height
curl -s https://mine.example.org/patina/status
curl -s https://other.example.org/patina/status

Confirm both report the same network and spec_sha256.
Pick a height at or below both indexed_height values.
Compare state roots at that height.
If they differ, stop and follow Report a disagreement.

Comparing at the tip is a mistake. One indexer will be a block ahead and you will chase a difference that is not one.

Signals to monitor

Nine signals. The middle column is what a working indexer looks like, and the last column describes the shape of a change rather than a number, because the number depends on your hardware.

What to watch, and what a change means.
Signal	Healthy	Investigate when
Sync gap, `tip_height - indexed_height`	Zero or one	It grows for several blocks in a row, or stops changing
Time since last processed block	Roughly block interval	Much longer than the node's own gap. Blocks are irregular, so alert on the difference from the node, not on absolute time.
Readiness	Ready, except during sync and replay	Ready while `indexed_height` is far behind, which means readiness is lying
Reorg events	Rare and shallow	Deep reorgs, frequent reorgs, or a rollback that fails to reach a fork point
Node connectivity	Connected	Any RPC failure. The indexer should fail closed, not guess.
Invalid event rate	Low and mostly grammar errors during a launch	A sudden spike, especially of one code, which usually means a broken tool in the wild
Error rate by status	Mostly 200	A rise in 500. A rise in 503 during sync is expected and correct.
Latency of artifact lookups	Stable	Growth with artifact count, which means a missing index
Storage growth	Slow and steady	Sudden growth without matching chain activity

Thresholds are yours to set

This page deliberately does not publish alert numbers. The right threshold depends on your hardware, your node, and your traffic. Measure your own baseline for a week, then alert on deviation from it. Copying someone else's numbers produces alerts that fire for no reason and silence when it matters.

Invalid events are a product signal

A spike in one reason code usually means a wallet or a tool is building transactions wrongly, not that the indexer is broken.

SEED_COMMIT_TOO_YOUNG spiking means people are miscounting the wait, or a tool is.
SEED_CARRIER_BELOW_MIN spiking means fees are being taken out of the carrier somewhere.
KEEP_ENTRY_BELOW_MIN spiking means artifacts are landing somewhere their owners did not intend.
SEED_BAD_GRAMMAR spiking means a builder is producing malformed markers.

Each of those is a reason to publish guidance, not to change the protocol. Route these alerts to whoever writes the documentation as well as to whoever runs the servers.

What not to alert on

An artifact resetting to depth zero. That is a holder spending their own coins, which is normal.
A relic appearing. Also normal.
A slow block. Bitcoin does that.
A drop in mint rate. It is not an outage and treating it as one leads to dishonest urgency.

Keep a verification log

Record, for each verification run: the date, the height compared, both state roots, both spec_sha256 values, and the result. When something does eventually go wrong, this log tells you the last height at which you know the state was right, which turns an open ended investigation into a bisection.