This is general knowledge or an external-engine reference, not an Apollo architecture or implementation claim. See Apollo's architecture status for target terminology and verified implementation boundaries.
Q68: How to Design Memory Monitoring?
Core Conclusion
The goal of memory monitoring is not "printing a memory value once per second," but being able to answer:
- Why is memory growing
- Which category of objects is growing
- Is it a leak, a peak, or normal cache fluctuation
- When should we alert, and when is it just observation
If monitoring can only see the total but not the structure and trend, its value for online troubleshooting is very low.
I. First Distinguish Several Types of Memory Problems
Before designing monitoring, first clarify what you want to identify:
- Continuously growing leaks
- Periodic peaks
- Cache bloat
- Fragmentation or allocation jitter
All of these look like "memory is high" from total RSS, but their root causes are completely different.
II. Looking Only at Total Memory Is Far from Enough
At minimum, observation should be layered:
- Process-level totals
- Module-level usage
- Allocation frequency
- Key object counts
- Sources of large memory blocks
Otherwise you can only know "the machine is almost full," without knowing what to change.
III. Memory Monitoring Must Track Both Volume and Trends
Valuable monitoring is not just the current value, but also:
- Growth rate
- Peaks and drops
- Comparison between versions
- Differences before and after peak scenarios
Many leaks are not obvious in the short term, but the trend is clear.
IV. Key Object Statistics Are Often More Useful Than Allocator Details
In online games, what is usually worth counting includes:
- Entity count
- Timer count
- Message queue backlog
- Cache entry count
- Usage of specific large object pools
Because many memory problems are not the underlying allocator being broken, but abnormal counts of business objects.
V. Alert Strategy Must Be Restrained
If memory alerts are triggered by a single threshold, false positives are very likely.
A more reasonable approach usually combines:
- Current value
- Growth trend
- Duration
- How close to the system limit
Only this can distinguish "normal peak" from "abnormal runaway."
VI. Online and Offline Analysis Must Work Together
Online monitoring is better suited for:
- Lightweight statistics
- Trend observation
- Fast alerting
Offline analysis is better suited for:
- Heap snapshots
- Leak location
- Object distribution analysis
The two are not replacements for each other, but complementary.
VII. A More Robust Engineering Design
Common practice is:
- Process-level metrics collected residently
- Module and object-level statistics reported periodically
- On-demand sampling supported for high-risk scenarios
- Leak problems further located through snapshots and profiling
This keeps the monitoring itself from being too heavy while retaining troubleshooting capability.
VIII. Common Misconceptions
1. Stable RSS means no problem
Not necessarily. Object pool bloat, fragmentation, and hotspot peaks can also hide within a stable total.
2. A lower alert threshold is safer
Too low creates noise and drowns out real anomalies.
3. Only leaks are worth monitoring
Peaks, cache bloat, and queue backlog can also bring the system down.
References
- Memory monitoring, heap analysis, and online alerting practice materials
