Tools
Paste the message, or type a word from it. Every entry below corresponds to a string or a behaviour that exists in this repository, so if something is not here it is worth checking whether it came from somewhere else in your stack.
TANDEM_NAMESPACE does not match the configured INIT tuple
Boot failure
The namespace you supplied differs from the one derived from your network, INIT txid, and spec hash. Almost always a typo in the txid or the spec hash rather than a wrong namespace.
Recompute it with the configuration builder. Remember the INIT txid is supplied in display order and reversed internally, while the spec hash is a raw digest and is never reversed.
TANDEM_CLOSE_HEIGHT must equal open height plus 4320
Boot failure
The founding window is a fixed 4,320 blocks, so the close height is derived from the open height rather than chosen independently.
Set close height to open height plus 4320 exactly.
TANDEM_INIT_HEIGHT must precede the open height by at least 1008 blocks
Boot failure
The INIT lead rule requires at least 1,008 blocks between the INIT confirming and the first height at which a CREATE can be valid.
Either lower the init height or raise the open height, and remember the close height moves with it.
<NAME> is required
Boot failure
One of the fourteen required variables is missing or empty. Values are trimmed, so a variable set to only whitespace counts as missing.
The message names the exact variable. Check the environment reference for its format.
<NAME> must be 32-byte lowercase hex
Boot failure
A hash-shaped variable is not 64 lowercase hexadecimal characters. Uppercase hex is rejected even though the bytes would be identical.
Applies to the INIT txid, the spec hash, the namespace, and the two binary digests.
<NAME> must be valid JSON
Boot failure
A trusted key map is not parseable. In a shell or compose file the braces and quotes often need escaping.
The empty value is the two characters {} and that is a valid, closed configuration.
<NAME> must be an HTTP URL without credentials, query, or fragment
Boot failure
The pipeline B base URL carries a username, password, query string, or fragment, or uses a scheme other than http or https.
Supply the origin and path prefix only. Trailing slashes are stripped for you.
AGREEMENT_KEY_ID is required when an agreement key is configured
Boot failure
A private or public agreement key was supplied with no key id. A verifier selects the trusted public key by key id, so a key without one cannot be used.
canonical_tip_missing
Readiness stays false
No row exists in the canonical block table, so there is no tip to report.
Expected with only this service running. Nothing in the repository ingests blocks or writes a canonical row, so a deployment has to supply that driver.
checkpoint_incomplete
Readiness stays false
There is no checkpoint at exactly the canonical height. The gate requires equality, not merely a checkpoint somewhere behind the tip.
Also expected on a fresh install. No code path inserts a checkpoint row, so populating that table is an operator responsibility.
agreement_signer_unavailable
Readiness stays false
The signing boundary is incomplete. It needs a key id, a 32-byte private key, and all four release identity values before it counts as configured.
If a public key is also supplied it must equal the one derived from the private key, otherwise the whole boundary is treated as unconfigured with no separate error.
bitcoin_network_mismatch
Readiness stays false
The chain reported by getblockchaininfo does not equal the value derived from your configured network. The comparison is exact.
The mainnet network maps to the chain name main, not mainnet. Every other network uses its own name.
bitcoin_core_unavailable
Readiness stays false
The RPC call failed or timed out. There is exactly one attempt with no retry or backoff, so a transient failure shows immediately.
Inside a container a loopback RPC URL points at the container itself. Use a reachable host address.
bitcoin_core_initial_block_download
Readiness stays false
The node is still performing initial block download and its view of the chain is not final.
Wait. Indexing a node that is still syncing would produce state you would have to discard.
canonical_tip_stale
Readiness stays false
The node height exceeds the canonical height by more than the configured lag, which defaults to 2 blocks.
Raising the lag hides the symptom rather than fixing it. Look at why indexing is behind.
database_unavailable
Readiness stays false
The probe's SELECT 1 failed. Any database error is collapsed into this single reason, so the detail is in the service log rather than in the response.
verification_unavailable
Every verified route returns 503
Agreement did not hold. The body is identical for every cause, deliberately, so a caller cannot probe which check failed.
The specific reason is in the service log as a warning from the gateway. Check readiness first, then the pipeline B endpoint, then the trust maps, then the compared fields.
pipeline B endpoint is not configured
Verified routes closed
PIPELINE_B_BASE_URL is unset, so there is no second opinion to request. This is the default state.
pipeline B key is not trusted
Verified routes closed
The envelope's key id is absent from the configured trust map for that pipeline. The signature is never checked, because there is no key selected to check it with.
During a rotation, add the new key id before the other pipeline starts using it.
agreement mismatch at <field>
Verified routes closed
One of the nine compared fields differs between the two pipelines. The message names the first field that differed, in the order the code checks them.
A mismatch at height usually means one pipeline is behind. A mismatch at block_hash means they are on different chains. A mismatch at a root means a genuine implementation difference and should be investigated rather than routed around.
verified agreement changed during data read
Verified routes closed
Agreement held before the query and differed after it. The gateway verifies twice around every read and discards the response if anything moved.
Occasional occurrences near a chain reorganization are normal. Sustained occurrences suggest an unstable tip.
verified mainnet responses are disabled
Verified routes closed
The network is mainnet and TANDEM_VERIFIED_MAINNET_ENABLED is not exactly the string true. The gateway refuses before it asks this pipeline to sign anything.
This is a deliberate gate and not a bug. Read the mainnet gates page before changing it.
pipeline B response is too large
Verified routes closed
The second pipeline returned more than 65,536 bytes. The limit is enforced both on the declared content length and while streaming, so a lying header does not get through.
An agreement envelope is small. A response near that size usually means the endpoint returned something other than an envelope, such as an HTML error page.
Container is healthy but traffic fails
Operational surprise
The image health check probes liveness only. It reports healthy as long as the process answers, regardless of whether any dependency is available.
Gate traffic on readiness instead. Do not interpret container health as agreement availability.
Prometheus gauges look stale
Operational surprise
The ready and canonical height gauges are updated as a side effect of a request to the readiness endpoint rather than on a timer.
Scrape readiness as well as metrics, or the values will only move when something else happens to ask.
HTTP 500 from the agreement endpoint
Operational surprise
A checkpoint exists at the requested height but the signing boundary is not configured, so the service cannot produce an envelope.
Configure the key id, private key, and all four release identity values.
ZMQ is configured but nothing happens
Operational surprise
The service subscribes to the configured topics and emits internal events, and no listener is registered for any of them, so notifications are received and discarded.
Treat ZMQ as intended design rather than working delivery. It does not currently trigger polling, because there is no polling loop to trigger.
Address lookup returns nothing
API surprise
Carrier addresses are matched against a database-generated column computed from the stored keys, so a lookup only finds carriers the indexer has actually recorded.
Also check the prefix. The address must use the human readable part for the deployment network, and a mainnet address is rejected outright on a signet deployment.
limit must be between 1 and 200
API surprise
Every list route caps its row count. There are no cursors, offsets, page tokens, or total counts anywhere in the API.
Nothing matches that. Try fewer words.
Some of the most alarming looking states are the system working exactly as designed.
Every verified route returns 503 on a fresh install. The gateway probes readiness before
anything else, and pipeline A cannot report ready at a canonical height here, so the log reads
pipeline A is not ready at a canonical height. Behind that, the pipeline B endpoint is unset and
both trust maps are empty, so the surface would still be closed even once readiness passed. It is
closed because it has nothing to stand behind, which is the entire point of it being fail closed.
Readiness reports canonical_tip_missing and checkpoint_incomplete forever. Nothing in this
repository ingests blocks or writes a checkpoint. Those two gates cannot clear until a deployment
supplies the drivers for them. This is the honest state of the project rather than a
misconfiguration.
The container is healthy while readiness is failing. The image health check probes liveness only. Both statements are true at once, and gating traffic on the wrong one is the mistake.
The gateway writes a warning naming the specific reason every time it withholds a response, while the caller only ever sees the generic body. So when a client reports 503 and you want the actual cause, read the service log rather than the response.
Configuration problems are always boot failures and always name the offending variable, so a process that is running has an internally consistent binding.
For runbooks organised by incident rather than by message, read incident handling. For the readiness gates in evaluation order, read readiness.