# Profiling A profiler measures where a program spends its time, memory, allocations, or other resources, and attributes the cost to the code that incurred it. Sampling profilers ask the OS for stack traces at fixed intervals; instrumentation profilers insert counters at every function boundary. The output is usually visualized as a flame graph, a form introduced by Gregg (Gregg 2016)[^gregg2016]. Continuous profiling extends the technique by running profilers in production all the time, indexing the data, and letting engineers query by deployment, host, or release. ## What it catches - **Hot functions taking unexpected time.** A 1% function that is actually 40% of CPU. - **Pathological allocation patterns.** A loop that allocates per iteration instead of reusing a buffer; the heap profile names the call site. - **Tail-latency contributors.** Sampling profilers in continuous mode separate "what runs on the median request" from "what runs on the p99 request." - **Lock contention** (with the right profiler). Mutex profilers show which lock was held by which thread and for how long. - **Cold paths that should be hot.** A cache that never warms; a fast-path that never fires. - **Energy and battery costs** (mobile / embedded). Profilers that surface CPU frequency and core residency catch background-drain bugs that load tests miss. What profiling does **not** catch by itself: correctness bugs. A profile is silent on whether the output is right; it reports only the cost of producing it. ## Tools ### Sampling profilers (the default) - **Linux [perf](https://perfwiki.github.io/main/)** — low-overhead, kernel-supported sampling profiler for native code on Linux. - **[DTrace](https://dtrace.org/)** (macOS, FreeBSD, Solaris), **[bpftrace](https://github.com/bpftrace/bpftrace)** (Linux) — systems-tracing engines; broader than profiling alone. - **[macOS Instruments](https://developer.apple.com/tutorials/instruments)** — Time Profiler; the sampling profiler Apple ships. ### Per-ecosystem CPU profilers - **Go:** **[pprof](https://github.com/google/pprof)** (standard library) — sampling + heap + block + mutex profiles; trivial to enable in production. - **Python:** **[py-spy](https://github.com/benfred/py-spy)** (sampling, attach to running process), **[scalene](https://github.com/plasma-umass/scalene)** (CPU + memory + GPU + line-level), **[cProfile](https://docs.python.org/3/library/profile.html)** / **profile** (stdlib instrumentation). - **JVM (Java/Kotlin/Scala):** **[async-profiler](https://github.com/async-profiler/async-profiler)** — sampling profiler that handles JIT correctly. **[JFR](https://docs.oracle.com/en/java/javase/17/jfapi/)** (Java Flight Recorder, OpenJDK). - **Rust:** **[samply](https://github.com/mstange/samply)** (sampling; Firefox profiler UI), **perf** + `cargo flamegraph`, **[dhat](https://valgrind.org/docs/manual/dh-manual.html)** for heap. - **Node:** built-in `node --prof`; **[clinic.js](https://clinicjs.org/)**; **[0x](https://github.com/davidmarkclements/0x)** for flame graphs. - **Ruby:** **[stackprof](https://github.com/tmm1/stackprof)**, **[vernier](https://github.com/jhawthorn/vernier)**. - **C / C++:** **perf**, **[Intel VTune](https://www.intel.com/content/www/us/en/developer/tools/oneapi/vtune-profiler.html)**, **[AMD uProf](https://www.amd.com/en/developer/uprof.html)**, **macOS Instruments**. ### Memory and allocation profilers - **[Valgrind](https://valgrind.org/) massif** (C/C++), **[heaptrack](https://github.com/KDE/heaptrack)** (C/C++), **dhat** (Rust), **scalene** (Python), JFR / async-profiler allocation mode (JVM), **pprof** heap (Go). ### Continuous (always-on) profiling - **[Pyroscope](https://grafana.com/oss/pyroscope/)** — multi-language, continuous. - **[Parca](https://www.parca.dev/)** — Prometheus-shaped continuous profiler. - **Datadog Continuous Profiler** (commercial). - **Google Cloud Profiler**, **AWS CodeGuru Profiler**. ### Flame-graph rendering - **[flamegraph.pl](https://github.com/brendangregg/FlameGraph)** (Brendan Gregg's original Perl script). - **[inferno](https://github.com/jonhoo/inferno)** (Rust port). - **[speedscope](https://www.speedscope.app/)** — interactive web viewer that reads many formats. ## When to use, when not **Use:** - Before optimizing anything. The bottleneck is almost never where the author guessed. - After a measured regression. Comparing the new profile with the previous release's profile names the code whose cost changed. - For continuous profiling in any production system where performance is a quality dimension. Sampling overhead is low enough to leave a profiler running permanently. - For memory leaks and allocation hotspots. Heap profilers turn a vague "process keeps growing" into a stack trace. **Don't:** - For correctness bugs. A profile of a wrong program is still a valid profile, so the fault needs a correctness method ([property-based](https://quality.stereobooster.com/property-based-testing.md), [example](https://quality.stereobooster.com/example-tests.md)) over the workload [load and stress testing](https://quality.stereobooster.com/load-and-stress-testing.md) produces. - Without representative load. Profiling an idle service reports the idle service's costs, not the ones under load. - Instrumentation profilers in production for non-trivial code. The overhead distorts the measurement it is meant to produce; a sampling profiler distorts less. ## Evidence The support here is definitional rather than statistical: an optimization has no target until the cost is measured. Google-Wide Profiling ran continuous profiling across machines in multiple Google data centers at an aggregate profiling overhead below 0.01% (Ren et al. 2010)[^ren2010]. ## Related **Performance** These aren't competing choices — they answer different performance questions, at different scopes. [Microbenchmarking](https://quality.stereobooster.com/microbenchmarking.md) measures the cost of one small unit in isolation. Profiling explains *where* time or memory goes in a real workload. [Load and stress testing](https://quality.stereobooster.com/load-and-stress-testing.md) drives the whole system under traffic to find where it degrades. You reach for whichever fits the question. ## Classification - **Quality dimensions:** Performance. - **Area:** Performance optimization; any runtime language; production hot-path investigation and capacity planning. - **Guarantee:** Heuristic — the profile reports *where time and resources went on the samples taken*, not *where they will go on every workload*. ## Referenced by - [Quality dimensions](https://quality.stereobooster.com/quality-dimensions.md) · Quality dimensions - [Effect scope](https://quality.stereobooster.com/effect.md) · The axes - [Algorithmic complexity testing](https://quality.stereobooster.com/algorithmic-complexity.md) · Methods - [Change-point detection](https://quality.stereobooster.com/change-point-detection.md) · Methods - [Load and stress testing](https://quality.stereobooster.com/load-and-stress-testing.md) · Methods - [Microbenchmarking](https://quality.stereobooster.com/microbenchmarking.md) · Methods - [Monitoring and observability](https://quality.stereobooster.com/monitoring-and-observability.md) · Methods - [Verifying memory safety](https://quality.stereobooster.com/memory.md) · Methods - [Worst-case execution-time analysis](https://quality.stereobooster.com/wcet-analysis.md) · Methods - [How AI fits into software quality](https://quality.stereobooster.com/ai.md) · AI - [Choosing methods](https://quality.stereobooster.com/choosing.md) · Overview ## References [^gregg2016]: Gregg, Brendan. 2016. "[The Flame Graph](https://dl.acm.org/doi/10.1145/2909476)." *Communications of the ACM* 59 (6): 48–57. . [^ren2010]: Ren, Gang, Eric Tune, Tipp Moseley, Yixin Shi, Silvius Rus, and Robert Hundt. 2010. "[Google-Wide Profiling: A Continuous Profiling Infrastructure for Data Centers](https://storage.googleapis.com/gweb-research2023-media/pubtools/pdf/36575.pdf)." *IEEE Micro* 30 (4): 65–79. .