Performance troubleshooting

What is pprof

pprof is a tool for profiling and visualizing profiling data. In modern Go versions, it is included with the compiler (go tool pprof), but it can also be installed as a standalone binary from https://github.com/google/pprof

install (example)

go install

More information can be found in the pprof README: https://github.com/google/pprof/blob/main/doc/README.md

pprof and Dependencies - Installation

1

pprof (Required)

• pprof that comes with Golang is available via go tool pprof.

• A standalone binary can be installed with:

2

graphviz (Optional)

Recommended for visualization. It can be skipped if you're not planning to use visualizations.

• Installation guide: https://graphviz.readthedocs.io/en/stable/#installation

• On macOS:

How to Use pprof

pprof operates by connecting to an exposed endpoint in the software you want to profile. It can create snapshots for later examination, or can show information in a browser for an already running process.

Examples below use go tool pprof. If you installed a standalone binary, replace go tool pprof with pprof.

Available pprof Endpoints

Choose which profiling endpoint you need. Common endpoints:

• /debug/pprof/heap: Snapshot of the memory allocation of the heap.

• /debug/pprof/allocs: Similar to /debug/pprof/heap, but includes all past memory allocations, not just the ones currently in the heap.

• /debug/pprof/goroutine: All current go-routines.

• /debug/pprof/threadcreate: Records stack traces that led to the creation of new OS threads.

• /debug/pprof/block: Displays stack traces that led to blocking on synchronization primitives.

• /debug/pprof/profile: Collects 30 seconds of CPU profiling data — configurable via the seconds parameter.

• /debug/pprof/symbol: Looks up the program counters provided in the request, returning function names.

• /debug/pprof/trace: Provides a trace of the program execution.

Configure Software to Expose pprof Endpoints

It is recommended to never expose pprof to the internet, as this feature allows operational control of the software. A malicious actor could potentially disrupt or DoS your services if these endpoints are exposed to the internet.

Never expose pprof endpoints to the public internet.

Full Node & Validator pprof

In config.toml, you can configure pprof_laddr to expose a pprof endpoint on a particular network interface and port. By default, pprof listens on localhost:6060.

If the value has been modified, you must restart the process.

RelayMiner pprof

The RelayMiner can be configured to expose a pprof endpoint using a configuration file like this:

If any of these values have been modified, you must restart the process.

Save the Profiling Data

You can save profiling data to a file by running:

Example — save a heap profile:

That file can be shared with other people.

Explore the Profiling Data

Start an HTTP server and open a browser to inspect a saved profile file:

Example:

Explore without saving data

Visualize pprof data directly from a running endpoint:

Report Issues

If you believe you've found a performance problem, please open a GitHub Issue: https://github.com/pokt-network/poktroll/issues. Make sure to attach the profiling data.