Upgrade Preparation

Purpose of this document

Technical instructions on preparing a protocol upgrade.

Instructions to create a new upgrade and apply it to both Alpha & Beta TesNet.

You MUST follow this PRIOR TO the MainNet Release Procedure: https://dev.poktroll.com/develop/upgrades/mainnet_release_procedure

Required Prerequisites Setup, Reading & Knowledge

• Ensure you've read and understood When is a Protocol Upgrade Needed?: https://dev.poktroll.com/develop/upgrades/upgrade_overview

• Ensure you have push access to pokt-network/poktroll: https://github.com/pokt-network/poktroll

• Familiarize yourself with the list of previous upgrade handlers: https://github.com/pokt-network/poktroll/tree/main/app/upgrades

• Familiarize yourself with how to test changes locally: https://dev.poktroll.com/develop/upgrades/localnet_upgrade_testing

• Ensure you have the required CLI tools: git, make, jq, sed, curl, go, brew, pocketd, etc.

1 Avoid On-Chain Non-Determinism

⚠️ The most common cause of chain halts is non-deterministic onchain behavior ⚠️

No amount of code reviews or testing can fully catch this. Here is a suggested and opinionated potential solution:

1

Use an LLM / CLI agent to review diffs

1. Start a session of Claude code or the CLI agent of your choice: https://www.anthropic.com/claude-code

2. Identify the tag of the previous MainNet release (e.g. v0.1.20)

3. Ask the agent the following prompt:

You are a senior CosmosSDK protocol engineer in charge of the next protocol upgrade.

Do a git diff v0.1.20.

Identify any potential bugs, edge cases or issues.

In particular, focus on any onchain behaviour that can result in non-deterministic outcomes. For example, iterating a map without sorting the keys first.

This is critical to avoid chain halts. Take your time and provide a comprehensive analysis.

2 Prepare a New Upgrade Handler

1

Identify the latest release version

Identify the version of the latest release from the full list of releases (e.g. v0.1.20): https://github.com/pokt-network/poktroll/releases

2

Copy vNEXT to the new upgrade

Prepare a new upgrade handler by copying vNEXT.go to the next release (e.g. v0.1.21):

cp app/upgrades/vNEXT.go app/upgrades/v0.1.21.go

3

Edit the new upgrade file

Open app/upgrades/v0.1.21.go and:

• Replace all instances of vNEXT with v0.1.21.

• Replace all instances of _NEXT_ with _0_1_21_.

• Remove all the general purpose template comments from v0.1.21.go.

4

Update app/upgrades.go

Open app/upgrades.go and:

• Comment out the old upgrade

• Add the new upgrade to allUpgrades

5

Recreate vNEXT template

Prepare a new vNEXT.go by copying vNEXT_Template.go to vNEXT.go:

cp app/upgrades/vNEXT_Template.go app/upgrades/vNEXT.go

Then open app/upgrades/vNEXT.go and remove all instances of Template.

6

Create a PR and merge

Create a PR with these changes and merge it. Example PR: https://github.com/pokt-network/poktroll/pull/1520

7

Business logic note

Note that the upgrade handler MAY need business logic related to onchain state changes. See other upgrades for reference: https://github.com/pokt-network/poktroll/tree/main/app/upgrades

3 Test Locally

Follow the instructions in Testing Protocol Upgrades Locally: https://dev.poktroll.com/develop/upgrades/localnet_upgrade_testing

If your upgrade handler had complex business logic, you MUST test it locally to avoid a chain halt.

Follow Testing Protocol Upgrades BEFORE submitting any transactions.

If you find an issue, you'll need to:

1

Delete the previous release

2

Delete the previous tag

3

Implement and merge in the fix

4

Prepare a new release

5

Regenerate the artifacts

6

Repeat the process above (you may need to jump back to earlier steps on this page)

4 Create a GitHub Release

1

Tag the release

Run:

make release_tag_minor

2

Publish the release

• Follow the onscreen instructions after running the command above

• Or draft a new release: https://github.com/pokt-network/poktroll/releases/new

• Use the tag above to auto-generate the release notes

• Set as a pre-release (change to latest release after upgrade completes)

3

Trigger release artifacts workflow (if needed)

Check if the Release Artifacts Workflow is running: https://github.com/pokt-network/poktroll/actions/workflows/release-artifacts.yml

If not, trigger it:

gh workflow run "Release artifacts"

4

Wait for CI

Wait for the Release Artifacts CI job to build artifacts for your release. It will take ~20 minutes and will be auto-attached to the release under the Assets section once complete.

5 Prepare the Upgrade Transactions

1

Generate upgrade tx JSON files

For example, for v0.1.21, run:

./tools/scripts/upgrades/prepare_upgrade_tx.sh v0.1.21

This will create:

• tools/scripts/upgrades/upgrade_tx_v0.1.21_alpha.json

• tools/scripts/upgrades/upgrade_tx_v0.1.21_beta.json

• tools/scripts/upgrades/upgrade_tx_v0.1.21_local.json

• tools/scripts/upgrades/upgrade_tx_v0.1.21_main.json

Make sure to commit these to GitHub once you're done.

Note: the height is not populated in the *.json files. This will be updated in subsequent steps.

2

Example JSON snippet

{
  "body": {
    "messages": [
      {
        "@type": "/cosmos.upgrade.v1beta1.MsgSoftwareUpgrade",
        "authority": "pokt10d07y265gmmuvt4z0w9aw880jnsr700j8yv32t",
        "plan": {
          "name": "v0.0.4",
          "height": "30",
          "info": "{\"binaries\":{...}}"
        }
      }
    ]
  }
}

3

Optional: Validate the Upgrade Binary URLs

Install go-getter if you don't have it:

go install github.com/hashicorp/go-getter/cmd/go-getter@latest

Check all binary URLs:

RELEASE_VERSION=<VERSION> # E.g. "v0.1.11"
for file in ./tools/scripts/upgrades/upgrade_tx_${RELEASE_VERSION}*; do
  echo "Processing $file"
  jq -r '.body.messages[0].plan.info | fromjson | .binaries[]' "$file" | while IFS= read -r url; do
    go-getter "$url" .
  done
done

Expected output:

2025/04/16 12:11:36 success!
2025/04/16 12:11:40 success!
2025/04/16 12:11:44 success!
2025/04/16 12:11:48 success!

6 Prepare Snapshots

Generate new snapshots for each network and ensure they are available here: https://snapshots.us-nj.poktroll.com/

This is currently a manual process maintained by Grove: https://grove.city/

Instructions are maintained in an internal Notion document: https://www.notion.so/buildwithgrove/Shannon-Snapshot-Playbook-1aea36edfff680bbb5a7e71c9846f63c?source=copy_link

7 Submit the Upgrade on Alpha & Beta TestNet

If you are submitting the upgrade for v0.1.21, run:

./tools/scripts/upgrades/submit_upgrade.sh alpha v0.1.21
./tools/scripts/upgrades/submit_upgrade.sh beta v0.1.21

Make sure to ONLY move to the next network after the prior one finished successfully.

8 Troubleshooting & Canceling an Upgrade

• Infrastructure Documentation (Grove): https://github.com/buildwithgrove/infrastructure/tree/main/docs

• Chain Halt Troubleshooting: https://dev.poktroll.com/develop/upgrades/chain_halt_troubleshooting

• Failed upgrade contingency plan: https://dev.poktroll.com/develop/upgrades/chain_halt_upgrade_contigency_plans

• Chain Halt Recovery: https://dev.poktroll.com/develop/upgrades/chain_halt_recovery