Disk Utilization FAQ

index_events = [] # Uncommented = index all events (default)

indexer = "null" # Disable indexing entirely

To limit indexing to specific events:

[tx_index]
indexer = "kv"
index_events = ["message.sender", "transfer.amount"]

Growth Factors:

  • Number of indexed events and attributes

  • Frequency of those events

Typical Behavior:

  • Can rival application.db in size

  • Redundant attributes (e.g., full addresses, denoms, etc.) multiply size

Mitigation:

  • Reduce index_events

  • Avoid emitting attributes that aren't needed off-chain

  • Set indexer = "null" for non-querying archival nodes

📂 state.db — Purpose, Contents, Growth Factors, Mitigation

Purpose: Tracks block height → AppHash + validator sets

Contents:

  • App hashes for every block height

  • Validator sets and historical changes

  • Consensus parameters and their history

  • IAVL tree root hashes and metadata

Growth Factors:

  • Number of blocks (stores app hash per height)

  • Validator set churn (frequent changes create snapshots)

  • Consensus parameter changes (governance proposals)

  • IAVL tree node storage (each node = separate DB record)

Typical Behavior:

  • Full Nodes/Validators: Grows slowly, usually < 100MB with proper pruning

  • Archival Nodes: Multi-GB growth is normal and expected

  • ⚠️ Warning: > 1GB for non-archival nodes indicates pruning issues

Common Issues Causing Large state.db:

  • For Non-Archival Nodes: Missing pruning configuration allowing indefinite retention

  • For All Node Types: Abnormally large ABCI responses (>10MB per block)

  • Excessive validator set changes storing complete metadata snapshots

  • IAVL tree inefficiencies with historical app hash storage

Mitigation:

  • Non-Archival Nodes: Enable pruning in app.toml and set min-retain-blocks

  • Archival Nodes: Focus on optimizing event emissions and ABCI response sizes

  • Use leveldb-inspector tool to identify large ABCI responses

  • Consider state sync recovery (non-archival nodes only)

  • Monitor for blocks with abnormally large ABCI responses

📂 application.db — Purpose, Contents, Growth Factors, Mitigation

Purpose: Stores the actual state of all SDK modules

Contents:

  • x/bank balances

  • x/staking delegations

  • Custom module state

Growth Factors:

  • Number of accounts, validators, contracts, etc.

  • Write volume to store

Typical Behavior:

  • Grows with app-level complexity

  • Compacted over time with IAVL pruning

Mitigation:

  • Enable pruning in app.toml

  • Periodic state sync or snapshots

📂 tx_index.db — Purpose & Notes

(Referenced earlier in the document — indexing behavior controlled via [tx_index] in config; disabling via indexer = "null" will reduce or eliminate this DB.)

🚨 Troubleshooting

When Database Sizes Indicate Problems — Normal ranges & warning signs

Normal database size ranges vary significantly by node type:

Full Nodes and Validators

Database
Typical Size
Warning Signs

application.db

Varies by chain activity

Growth matches network usage

blockstore.db

Linear with chain age (with pruning)

Sudden spikes indicate verbose events

tx_index.db

0 (if disabled) to large

Should be 0 if indexer = "null"

state.db

< 100MB even on large chains

> 1GB indicates pruning issues

Archival Nodes (Expected Larger Sizes)

Database
Expected Behavior
Investigation Threshold

application.db

Grows with chain activity

Same as other node types

blockstore.db

Linear growth, no pruning

Sudden spikes in block size

tx_index.db

Large (if indexing enabled)

Only if indexer should be disabled

state.db

Multi-GB growth over time

Individual blocks > 50MB

Investigating Large state.db — Causes & diagnostics

For Full Nodes/Validators (> 1GB) or Archival Nodes (individual blocks > 50MB):

Abnormally Large ABCI Responses (most common issue):

  • Individual blocks storing 10MB+ ABCI response data

  • Caused by excessive event emissions or verbose transaction logs

  • Check specific block heights with large responses

IAVL Tree Issues:

  • IAVL stores each tree node as separate database record

  • Historical app hashes retained indefinitely (normal for archival)

  • Potential state inconsistencies or corruption

Validator Set Churn:

  • Frequent validator changes create new validator set snapshots

  • Each change stores complete validator metadata (normal for archival)

Consensus Parameter Changes:

  • Governance proposals modifying consensus parameters

  • Historical parameter versions accumulate (normal for archival)

Incorrect Pruning Configuration (Full Nodes/Validators only):

  • No min-retain-blocks configured on non-archival nodes

  • Pruning disabled when it should be enabled

Diagnostic Tools — LevelDB Inspector, IAVL analysis, Configuration review

The Pocket Network codebase includes specialized tools for database analysis.

LevelDB Inspector — Analyzes database contents and identifies space usage:

# Navigate to your node directory
cd /path/to/your/pocketd/data

# Get database statistics and identify large entries
/path/to/poktroll/tools/leveldb-inspector/leveldb-inspector stats -d state.db
/path/to/poktroll/tools/leveldb-inspector/leveldb-inspector size -d state.db

# Check for abnormally large ABCI responses
/path/to/poktroll/tools/leveldb-inspector/leveldb-inspector keys -d state.db | grep "abciResponsesKey" | tail -10

IAVL Tree Analysis — For investigating IAVL tree issues:

# Check IAVL tree consistency (if available)
/path/to/poktroll/tools/iavl-tree-diff/main.go

# Query current consensus parameters
pocketd query consensus params

# Review validator set history
pocketd query staking validators --height <specific_height>

Configuration Review — Check your current settings:

# Review pruning configuration
grep -A 10 "pruning" ~/.pocketd/config/app.toml

# Check min-retain-blocks setting
grep "min-retain-blocks" ~/.pocketd/config/app.toml

# Verify indexing configuration
grep -A 5 "tx_index" ~/.pocketd/config/config.toml
Recovery Options for Oversized Databases

If databases have grown too large, consider these recovery methods.

Expandable: Snapshot-Based Recovery (Recommended)

# Stop your node
sudo systemctl stop pocketd

# Backup your validator key and node key (if validator)
cp ~/.pocketd/config/priv_validator_key.json ~/backup/
cp ~/.pocketd/config/node_key.json ~/backup/

# Remove old data (keep config)
rm -rf ~/.pocketd/data

# Download recent snapshot (replace with actual snapshot source)
wget https://snapshots.example.com/poktroll-latest.tar.gz

# Extract to data directory
tar -xzf poktroll-latest.tar.gz -C ~/.pocketd/data/

# Restart with proper pruning configuration
sudo systemctl start pocketd

Selectable: Selective Database Reset

# Stop node first
sudo systemctl stop pocketd

# Remove only problematic database (e.g., oversized tx_index.db)
rm -rf ~/.pocketd/data/tx_index.db

# Ensure indexer is disabled if not needed

# Edit config.toml: indexer = "null"

# Restart node
sudo systemctl start pocketd

Note: Avoid modifying links or query parameters — all links in this document remain unchanged.

PreviousMigration FAQNextPlaybooks

Was this helpful?