Arbitrum chain upgrade runbook

This runbook is the operational wrapper around an Arbitrum chain upgrade: what to validate before you start, what order to execute in, how to confirm the upgrade succeeded, and what you can undo if it doesn't.

It complements ArbOS upgrade, which remains the canonical reference for how each individual step works. This page does not restate those mechanics—it links to them at each step and focuses on the coordination, verification, and failure-recovery work around them.

An upgrade touches five things that must stay mutually compatible:

1. The Nitro version running on every node and validator.
2. The nitro-contracts deployed on the parent chain.
3. The WASM module root set in the Rollup contract.
4. The WAVM machines on each validator's disk—one per root the validator must validate (current and pending). A new Nitro version alone doesn't guarantee they are present.
5. The ArbOS version active on the child chain.

Most failed upgrades are a mismatch between two of these five, and the majority are mismatches between the onchain root (3) and the machines on validators (4). Troubleshoot WASM module root failures covers those in detail.

ArbOS upgrades are one-way

Once an ArbOS upgrade activates, you cannot downgrade the ArbOS version. Plan on rolling forward, not back. Read Rollback options before you schedule an upgrade, so you know which parts of the process are reversible and which aren't.

The Nitro version and the ArbOS version are independent: Nitro is per-node software you can deploy and roll back; the ArbOS version is chain-wide state, moved only by scheduleArbOSUpgrade. Upgrading Nitro does not change the chain's ArbOS version.

Record your target versions

Every version-specific value comes from the reference page for the ArbOS release you're targeting. Record all five before you begin, because later steps consume them as inputs:

Input	Where to get it
Target ArbOS version	The list of available ArbOS releases
Minimum Nitro version	The requirements section of the targeted ArbOS release page
nitro-contracts version	The targeted ArbOS release page. Not every ArbOS upgrade requires a contracts upgrade
WASM module root	The targeted ArbOS release page
Consensus tag	The consensus version paired with that WASM module root (for example, consensus-v51). Cross-check the pairing against the Dockerfile in the Nitro repository

Why the consensus tag matters

The consensus tag and the WASM module root are two names for the same artifact, and scripts/download-machine.sh requires both as arguments. Release pages sometimes reference point releases (for example, consensus-v51.1) that carry a different module root than the base tag. Record the exact pair you intend to run, and confirm it against the Dockerfile in the Nitro repository, which is the source of truth for which tag maps to which root.

Pre-upgrade validation

Complete every check in this section before scheduling anything onchain.

Confirm your current ArbOS version

Call ArbSys.ArbOSVersion() on your chain. This function returns the ArbOS version plus 55—a chain running ArbOS 40 returns 95. When you later schedule the upgrade, you pass the real version number, not the offset one. See Obtaining the current ArbOS version for details.

Confirm your current contracts version and upgrade path

Determine which nitro-contracts version your chain currently runs and whether a direct upgrade path to the target exists. Follow Check version and upgrade path in the chain-actions repository, substituting your chain's inbox address and network name.

Contract upgrades are not always a single hop. Confirm the full path now rather than discovering a required intermediate version mid-upgrade.

Confirm chain owner access

Every administrative action in this runbook is executed by the upgrade executor contract, not directly by your chain owner account. Before you begin, verify that:

• You control an account with the executor role on the parent chain upgrade executor.
• That account is funded on the parent chain.
• You can reach the child chain upgrade executor, which owns the ArbOwner precompile actions.

See Ownership and access to confirm which account holds which role.

Confirm the target machine is present on every validator

This is the check that most often gets skipped, and it's the one that causes the failures in Troubleshoot WASM module root failures. Verify that each validator has the machine for the target WASM module root on disk before you set that root onchain—not after the upgrade activates.

Machines are discovered by directory layout, so inspect the directory directly:

ls -l /home/user/.arbitrum/machines/
cat /home/user/.arbitrum/machines/latest/module-root.txt

A correctly populated machines directory looks like this:

machines/
├── latest -> 0x8a7513bf...          # symlink to the newest machine directory
└── 0x8a7513bf.../                   # directory name must equal the module root
    ├── module-root.txt              # contains 0x8a7513bf...
    ├── machine.v2.wavm.br
    └── replay.wasm                  # only present if the release publishes it

Staging the machine is only half the check. The block validator reads its machine list and resolves its pending root once, at startup, so restart each validator after staging and confirm the startup log before you set the root onchain:

INFO BlockValidator initialized  current=0x8a7513bf... pending=0xc2c02df5...

pending must equal your target root. If it doesn't, the validator will refuse the onchain root change in step 3. Don't defer this to post-upgrade verification—by then the root is already set.

WASM module roots are backward compatible

You can stage the machines on your validators and set the new WASM module root well before the ArbOS upgrade activates. Doing so does not disrupt the chain, so there's no reason to leave it until the last moment.

Order matters within that window, though: stage the machine on each validator and restart it first, then set the root onchain. A running validator only follows an onchain root change if its pending root already resolves to the new root, and it resolves that root at startup.

Determine whether you need download-machine.sh

scripts/download-machine.sh in the Nitro repository fetches a prebuilt WAVM machine for a given consensus tag. Whether you need to run it depends entirely on how you obtain your node binary:

Your setup	Do you need download-machine.sh?
Official Docker image, no STF customization	No. Machines are already baked into the image
Building the Docker image from source	Usually yes. The Dockerfile only downloads the machines on its uncommented RUN ./download-machine.sh lines
Custom STF build	Yes, and you must also rebuild the replay binary so the resulting module root matches what you set onchain

The Dockerfile only downloads recent machines

In the Nitro repository's Dockerfile, the RUN ./download-machine.sh lines for older consensus versions are commented out—only the most recent few are active. If your chain's Rollup contract still points at an older module root, a from-source build produces an image with no machine for that root, and your validator fails to start.

This is the single most common cause of the startup failures described below. Check the Dockerfile for a line matching your consensus version before assuming the image contains it.

To fetch a machine manually, run the script from your machines directory with both the consensus tag and its module root:

cd /home/user/.arbitrum/machines
./download-machine.sh consensus-v51.1 0xc2c02df561d4afaf9a1d6785f70098ec3874765c638e3cb6dbe8d3c83333e14c

The script creates a directory named after the module root, writes module-root.txt inside it, repoints the latest symlink at it, and downloads machine.v2.wavm.br (plus replay.wasm, if that release published one).

If you maintain a custom STF, downloading a published machine is not sufficient—see Customize your chain's behavior to learn how to build and register your own.

Execution order

Order matters. Each step assumes the previous one completed successfully.

Step	Action	Notes
1	Update Nitro on nodes and validators	Validators first, then remaining nodes. Must complete before the ArbOS upgrade deadline
2	Stage the target WAVM machine on every validator, then restart the validator	Must precede step 3. The block validator resolves its pending root at startup, so stage the machine before the step 1 restart and one restart covers both
3	Set the new WASM module root	Safe to do early, as long as every validator has been restarted with the target machine and its startup log shows pending=<target root>
4	Upgrade nitro-contracts, if the release requires it	Follow Nitro contracts upgrades
5	Schedule the ArbOS version upgrade	Pass the real version number and a UNIX timestamp. A timestamp of 0 upgrades immediately
6	Enable release-specific configuration or feature flags	Not always required; check the release page

Stage machines and restart before you set the root onchain

A running validator adopts a new onchain root only if that root equals its pending root; anything else is refused with unexpected wasmModuleRoot! cannot validate!. The legacy staker then stops acting entirely, while BoLD only warns. Both the pending root and the machine list are read once, at startup, so staging a machine under a running validator does nothing—the validator must be restarted before step 3. See Understand which roots your validator is required to have.

Upgrade nodes before the activation timestamp

A node that is too old to process the newly active ArbOS version stops with a fatal please upgrade to the latest version of the node software error. Because the ArbOS upgrade activates in consensus at the scheduled timestamp, every node must already be running a compatible Nitro version when that timestamp passes. Leave yourself margin.

Post-upgrade verification

Work through all four checks. A chain can look healthy on the surface while its validator is quietly unable to bond.

Verify the active ArbOS version

Call ArbSys.ArbOSVersion() again and confirm it advanced to your target, remembering the +55 offset. If the scheduled timestamp has passed and the version has not changed, the upgrade did not activate—check that you scheduled the correct version and that nodes are actually producing blocks past the timestamp.

Verify the WASM module root matches everywhere

Compare the root set in the Rollup contract against the root your validator resolved. Read the onchain value from the parent chain:

cast call <rollup-address> "wasmModuleRoot()(bytes32)" --rpc-url <parent-chain-rpc>

Then confirm your validator agrees. On startup, the block validator logs both roots it will validate against:

INFO BlockValidator initialized  current=0x8a7513bf... pending=0x8a7513bf...
INFO validator chosen            WasmModuleRoot=0x8a7513bf... chosen=jit maxWorkers=8

The current value should match the onchain wasmModuleRoot(). A validator chosen line appears for every root listed in BlockValidator initialized; if a machine for any root is missing, startup fails with cannot validate WasmModuleRoot.

Verify the validator is bonding

Confirm the validator is still advancing assertions and posting bonds after the upgrade. A validator that started cleanly can still be unable to act if the Rollup contract's root doesn't match what the validator actually validated with—that produces the runtime mismatch described in The validator starts but can't advance its bond.

If your chain has fast withdrawals enabled, re-check them specifically after the upgrade—see Fast withdrawals.

Verify the batch poster is still posting

Confirm the batch poster is submitting batches to the sequencer inbox at its usual cadence, and that no backlog is accumulating. See Monitoring tools and considerations for what to watch.

Batch poster log noise after an upgrade

New Nitro versions probe for contract features that older nitro-contracts deployments don't implement, so a version-skewed window between steps 1 and 4 can produce eth_call errors in the batch poster log that are benign. Confirm batches are still landing onchain before treating any of them as an incident. To diagnose messages that persist after the upgrade completes, see Batch poster troubleshooting and Batch poster recovery.

Troubleshoot WASM module root failures

These failures share a single root cause—the validator does not have, or hasn't loaded, the machine for a module root it's required to handle—but they surface at three different severities: fatal at startup, non-fatal at startup, and at runtime. Identify which one you have before changing any configuration.

Error message	Node starts?	What it means
latestWasmModuleRoot not set	No	No machine directory was found at all
cannot validate WasmModuleRoot <root>	No	Machines were found, but none provides this specific required root
unable to find validator machine directory for the on-chain WASM module root	Yes	Startup compatibility pre-check failed. Logged at ERROR, but does not stop the node
on-chain WASM module root did not match with any of the allowed WASM module roots	Yes	Same pre-check, when --validation.wasm.allowed-wasm-module-roots is set and nothing matched
wasmroot doesn't match rollup : <root>, valid: [<roots>]	Yes	Runtime mismatch. The validator runs but cannot advance its bond
unexpected wasmModuleRoot! cannot validate! found <onchain>, current <x>, pending <y>	Yes	The onchain root changed to a root that is neither the validator's current nor its pending root, so the validator refuses to adopt it. The legacy staker then stops acting entirely; BoLD logs a warning and continues

Operators upgrading Nitro have reported the two fatal cases wrapped in a failed to create node startup error (failed to create consensus node on newer versions); the specific message quoted in that error tells you which one you're looking at.

The node won't start

For either fatal case, work through these in order:

1. Check the Dockerfile for your consensus version. If you built the image from source, confirm there's an uncommented RUN ./download-machine.sh line for the consensus tag your chain needs. This is the most common cause.
2. Run download-machine.sh with the correct version. Fetch the machine for the exact consensus tag and module root pair you recorded earlier.
3. Rebuild the replay binary. Required if you maintain a custom STF, where no published machine will ever match your root.

Then confirm the machines directory is actually discoverable, because two layout rules are enforced silently:

• A directory is ignored unless its name is either latest or exactly its own module root. The comparison is against the lowercase, 0x-prefixed form. A directory named with a truncated, uppercase, or otherwise reformatted root is skipped without an error, even though module-root.txt inside it is correct.
• Discovery stops at the first search path that yields any machine. One stale directory in an earlier path masks a correct one later in the search order. If --validation.wasm.root-path is set, it is the only path searched.

When --validation.wasm.root-path is unset, the search order is:

1. <project>/target/machines (source builds)
2. ./machines, relative to the working directory
3. ./target/machines
4. machines/ in the grandparent directory of the binary—for example, /usr/local/machines for a binary at /usr/local/bin/nitro

Setting --validation.wasm.root-path explicitly is the reliable fix when discovery picks the wrong directory.

Understand which roots your validator is required to have

A validator must have machines for both roots the block validator resolves, not just the current one:

• --node.block-validator.current-module-root (default current, read from the chain)
• --node.block-validator.pending-upgrade-module-root (default latest)

When these differ, both are required, and a missing machine for either produces cannot validate WasmModuleRoot. This is why a validator that ran fine yesterday can fail to start today: the chain's root changed, and the current root—read from the chain—now names a root with no machine in the image. Pending, by contrast, is resolved from what's on disk (via the machines/latest symlink), so it can't point at a machine you don't have; a pinned pending root that's missing fails startup with the same cannot validate WasmModuleRoot error.

By default, pending resolves through the machines/latest symlink, which download-machine.sh repoints to whatever it fetched last. If you'd rather not depend on that, pin it: --node.block-validator.pending-upgrade-module-root=<target root>. Either way the machine must be on disk, or startup fails with cannot validate WasmModuleRoot—pinning changes which root is required, not whether one is.

Clearing `pending-upgrade-module-root` is not a fix for a Rollup mismatch

Setting --node.block-validator.pending-upgrade-module-root= to an empty value only shrinks the set of roots required at startup. It has no effect on the runtime wasmroot doesn't match rollup error, which compares against the Rollup contract rather than against this flag.

So if your node starts but can't advance its bond, clearing this flag won't help—you need the correct machine on disk. Conversely, if you cleared it to work around a startup failure, you've disabled validation against the pending upgrade root, which means you won't discover a missing machine until the upgrade activates. Clearing it also means the validator has no pending root to match against, so it can't adopt a new onchain root at runtime either. Restore it once the correct machine is staged.

The validator starts but can't advance its bond

This error only occurs on chains still running the legacy staker—the BoLD staker doesn't perform this check. It means the node is running, but the root in the Rollup contract isn't among the roots the block validator actually validated with:

error advancing stake from node 18 (hash 0x4d42...): error generating node action:
wasmroot doesn't match rollup : 0x260f5fa5..., valid: [0x8b104a2e...]

Read it as: the Rollup contract expects the first root; your validator only has the second. Both roots map to consensus versions you can look up in the Nitro repository's Dockerfile—which usually reveals that the expected root corresponds to a consensus version your image never downloaded.

The fix is to stage the machine for the root the Rollup contract expects, then restart the validator. Confirm success by checking that a validator chosen line appears for that root.

A related message, block validation is still pending, is not a mismatch—it means the block validator hasn't validated anything yet and has no roots to compare. Wait for validation to progress.

Don't suppress the mismatch check

--node.staker.dangerous.ignore-rollup-wasm-module-root downgrades this error to a warning and lets the validator make assertions anyway. Its own help text describes it as dangerous, and for good reason: asserting against a module root the Rollup contract doesn't recognize risks producing assertions that can be challenged. Fix the machine, don't silence the check.

Likewise, --validation.wasm.enable-wasmroots-check=false disables only the non-fatal startup pre-check. It hides an early warning without addressing the cause.

Rollback options

How much you can undo depends entirely on whether the ArbOS upgrade has activated. The ArbOS upgrade applies in consensus once the chain's block timestamp reaches the scheduled timestamp, and the upgrade logic only ever moves the version upward.

What you changed	Reversible?
Scheduled ArbOS upgrade, before the timestamp	Yes. Call scheduleArbOSUpgrade again to push the timestamp out or to set a version at or below the current one
WASM module root	Yes. Call setWasmModuleRoot with the previous root. Roots are backward compatible
Nitro node version	Partially. You can downgrade, but not below a version that supports the chain's active ArbOS version
nitro-contracts	Difficult. Technically possible by re-pointing implementations through the upgrade executor, but treat it as a last resort
Activated ArbOS version	No. There is no downgrade path

Before the activation timestamp

A scheduled upgrade is just two stored values—a target version and a timestamp—and the upgrade only fires when the chain's timestamp reaches the target timestamp and the target version is higher than the current one. Until then, you can freely cancel or reschedule by calling scheduleArbOSUpgrade again through the upgrade executor. This is the safe window; if you have any doubt about validator readiness, postpone from here rather than pushing forward.

After the activation timestamp

The ArbOS version cannot be lowered. Recovery means rolling forward:

1. Restore node availability first. If nodes are failing to start, resolve that with the WASM root troubleshooting above. Getting nodes running on the new ArbOS version is almost always faster than any attempt to unwind.
2. Don't downgrade Nitro below the active ArbOS version. A node too old for the active ArbOS version stops with a fatal out-of-date error, so downgrading to "get back to a working state" makes the outage worse.
3. Escalate rather than improvise. If the chain is producing blocks but validation is broken, the chain is still live and withdrawals are the pressure point. Contact Offchain Labs support before attempting contract-level changes.

Reverting contracts after assertions exist

Rolling nitro-contracts back is only remotely viable if no assertions have been posted that depend on the new contracts' behavior. Once they have, a revert can invalidate chain history or strand in-flight withdrawals. Treat a contracts rollback as an incident requiring support involvement, not a routine operation.

Related documents

• ArbOS upgrade—step-by-step mechanics for each upgrade action
• ArbOS releases—per-release requirements, WASM module roots, and Nitro versions
• Ownership and access—which account holds which administrative role
• Monitoring tools and considerations—what to watch during and after an upgrade
• Batch poster troubleshooting—diagnosing batch poster errors
• Customize your chain's behavior—building a custom STF and its replay binary
• Nitro CLI flags reference—reference for the Nitro configuration flags named here (dangerous flags are not listed)