This is general knowledge or an external-engine reference, not an Apollo architecture or implementation claim. See Apollo's architecture status for target terminology and verified implementation boundaries.
Q56: How to Do Performance Profiling? What Tools Are There?
Implementation status: Apollo has self-built logging, MetricRegistry, and span/trace primitives. A dedicated Profiler, OpenTelemetry SDK, and OTLP exporter are not integrated. Tool comparisons and observability designs on this page are not claims of deployed functionality.
Core Conclusion
The first principle of performance profiling is not "optimize first", but "locate first".
Effective performance profiling usually has to answer three things:
- Where it is slow
- Why it is slow
- Whether this bottleneck is worth optimizing
Without clear metrics and evidence, the vast majority of optimizations become guessing.
1. First Distinguish "Feels Slow" from "Slow by Metric"
Many performance problems start from a felt sensation:
- Latency suddenly goes high
- Frame time jitter
- Memory keeps rising
- A certain scene gets laggy when crowded
But once you actually start analyzing, you must turn it into quantifiable metrics, for example:
- TPS
- Response time percentiles
- CPU hotspots
- Memory growth curves
- Network throughput
Otherwise it is hard to know whether the problem has been reproduced, and hard to verify whether an optimization really works.
2. Performance Profiling Is Usually Split by Layer
2.1 System Layer
Focus on:
- CPU
- Memory
- Disk I/O
- Network
- Context switching
2.2 Process Layer
Focus on:
- Which thread is busy
- Which module eats CPU
- Which allocations are growing
2.3 Business Layer
Focus on:
- Which message is heaviest
- Which scene is hottest
- Which kind of entity update is most expensive
- Which call chain times out most
Real production performance problems usually require looking at these three layers connected together.
3. Define the Analysis Path First, Then Choose Tools
There are many tools, but more is not always better.
A more practical approach is usually:
- First use lightweight metrics to discover anomalies
- Then use profiling to narrow the scope
- Finally do fine-grained analysis on the hotspots
In other words, monitoring is the entrance and the profiler is the magnifier; do not turn on heavy tools at full volume from the start.
4. Common Analysis Targets and Tool Directions
4.1 CPU Hotspots
Common things to look at:
- Which functions take the largest share
- Whether high-frequency small functions accumulate
- Whether there is lock waiting or spinning
4.2 Memory Problems
Common things to look at:
- Whether resident memory keeps rising
- Whether there are leaks
- Whether allocations are too frequent
- Where the large objects and hot containers are
4.3 Threads and Locks
Common things to look at:
- Which threads are blocked
- Whether there is lock contention
- Whether there is queue backlog
4.4 Network and I/O
Common things to look at:
- Which connections or message types are heaviest
- Whether there is send queue backlog
- Whether I/O waiting is abnormal
5. Tools Are Only Means; the Key Is the Chain of Evidence
A complete performance localization should ideally form a chain like this:
- Monitoring discovers an anomalous metric
- Logs or tracing narrow it to a specific scene
- The profiler finds the hotspot function or hotspot path
- After the fix, re-run stress tests or replay to verify
If the verification step is missing, it is easy to mistake "correlation" for "causation".
6. Production and Local Should Be Treated Separately
Local profiling is suitable for:
- Reproducing stable problems
- Deep analysis of hotspots
Production analysis is better suited for:
- Lightweight sampling
- Metric comparison
- Localized capture
Do not casually turn on high-overhead tools in production, otherwise the analysis itself may become a new source of problems.
7. Common Tool Categories
They differ by platform and language, but common ones can be categorized by use:
- System metrics tools
- CPU profiler
- Memory profiler
- Thread and lock analysis tools
- Network packet capture and throughput analysis tools
- tracing and flame graph
What truly matters is not memorizing tool names, but knowing which kind of tool to use for which kind of problem.
8. Common Misconceptions
8.1 First Optimize from Experience, Then See the Result
Often wasted effort, or even made worse.
8.2 High CPU Means the Algorithm Is Bad
It could also be caused by lock contention, ineffective polling, frequent allocations, or I/O waiting.
8.3 The Function with the Highest Share in Tool Output Must Be Optimized First
Not necessarily. You also need to check whether it is on the core path and whether it really affects user experience.
References
- Linux perf / flame graph / tracing related materials
- Various server-side profiling and production observability practice materials
