Skip to content

This is general knowledge or an external-engine reference, not an Apollo architecture or implementation claim. See Apollo's architecture status for target terminology and verified implementation boundaries.

Q68: How to Design Memory Monitoring? ​

Core Conclusion ​

The goal of memory monitoring is not "printing a memory value once per second," but being able to answer:

  • Why is memory growing
  • Which category of objects is growing
  • Is it a leak, a peak, or normal cache fluctuation
  • When should we alert, and when is it just observation

If monitoring can only see the total but not the structure and trend, its value for online troubleshooting is very low.

I. First Distinguish Several Types of Memory Problems ​

Before designing monitoring, first clarify what you want to identify:

  • Continuously growing leaks
  • Periodic peaks
  • Cache bloat
  • Fragmentation or allocation jitter

All of these look like "memory is high" from total RSS, but their root causes are completely different.

II. Looking Only at Total Memory Is Far from Enough ​

At minimum, observation should be layered:

  • Process-level totals
  • Module-level usage
  • Allocation frequency
  • Key object counts
  • Sources of large memory blocks

Otherwise you can only know "the machine is almost full," without knowing what to change.

Valuable monitoring is not just the current value, but also:

  • Growth rate
  • Peaks and drops
  • Comparison between versions
  • Differences before and after peak scenarios

Many leaks are not obvious in the short term, but the trend is clear.

IV. Key Object Statistics Are Often More Useful Than Allocator Details ​

In online games, what is usually worth counting includes:

  • Entity count
  • Timer count
  • Message queue backlog
  • Cache entry count
  • Usage of specific large object pools

Because many memory problems are not the underlying allocator being broken, but abnormal counts of business objects.

V. Alert Strategy Must Be Restrained ​

If memory alerts are triggered by a single threshold, false positives are very likely.

A more reasonable approach usually combines:

  • Current value
  • Growth trend
  • Duration
  • How close to the system limit

Only this can distinguish "normal peak" from "abnormal runaway."

VI. Online and Offline Analysis Must Work Together ​

Online monitoring is better suited for:

  • Lightweight statistics
  • Trend observation
  • Fast alerting

Offline analysis is better suited for:

  • Heap snapshots
  • Leak location
  • Object distribution analysis

The two are not replacements for each other, but complementary.

VII. A More Robust Engineering Design ​

Common practice is:

  • Process-level metrics collected residently
  • Module and object-level statistics reported periodically
  • On-demand sampling supported for high-risk scenarios
  • Leak problems further located through snapshots and profiling

This keeps the monitoring itself from being too heavy while retaining troubleshooting capability.

VIII. Common Misconceptions ​

1. Stable RSS means no problem ​

Not necessarily. Object pool bloat, fragmentation, and hotspot peaks can also hide within a stable total.

2. A lower alert threshold is safer ​

Too low creates noise and drowns out real anomalies.

3. Only leaks are worth monitoring ​

Peaks, cache bloat, and queue backlog can also bring the system down.

References ​

  • Memory monitoring, heap analysis, and online alerting practice materials

Released under the MIT License