This is general knowledge or an external-engine reference, not an Apollo architecture or implementation claim. See Apollo's architecture status for target terminology and verified implementation boundaries.
Q23: How Do You Implement an RPC Call?
Core Conclusion
The essence of RPC is not "being as elegant as calling a local function", but "wrapping a remote request into a governable calling protocol".
A real RPC design must at least solve:
- How to locate the target
- How to serialize parameters
- How to correlate requests with responses
- How to handle timeout, failure, retry, and idempotency
- How to observe call quality
If you stay at the surface of "remote function calling", problems will surface in engineering very quickly.
I. What an RPC Actually Does
A single RPC call usually contains these steps:
- The caller constructs the request
- The method name, target object, and parameters are serialized
- It is sent over the network to the target node
- The target node deserializes and dispatches to the corresponding handler
- After execution, a result or error is returned
- The caller looks up the waiting context by request ID
So RPC is not just a function-call interface, but a whole set of communication conventions.
II. The Core Parts of an RPC
1. Service Discovery or Target Addressing
The caller must know where to send the request.
Common approaches include:
- Fixed node address
- Routing table
- Gateway forwarding
- Registry center
- Routing based on entity ID or shard key
In game services, many RPCs are not "find a service name" but "find the node responsible for this player, scene, instance, or Cell".
2. Protocol and Serialization
An RPC needs to define the request packet structure, usually containing at least:
request_idmethodtargettimeoutpayload
Serialization can use:
- Protobuf
- FlatBuffers
- MessagePack
- A custom binary protocol
The point here is not "which one is most advanced", but whether the team can evolve and debug it stably.
3. Request Context Management
After the caller sends the request, it must correlate the response back with the original request.
So you need to maintain:
- A table of outstanding requests
- Timeout timers
- Cancellation or invalidation logic
If this layer is not done well, things get messy during timeouts, retries, and node switching.
4. Error Model
RPC failure is not just one kind of failure.
At minimum you should distinguish:
- Network unreachable
- Target does not exist
- Business rejection
- Call timeout
- Node busy
- Result unknown
"Result unknown" is especially important, because a timeout does not necessarily mean the operation did not execute; it may only mean the result did not come back.
III. How to Choose Between Synchronous RPC and Asynchronous RPC
1. Synchronous RPC
The benefit is that the calling code is straightforward, suitable for:
- Initialization flows
- Admin backends
- Low-frequency tooling
The downsides are equally clear:
- Easy to block threads
- Easy to lengthen call chains
- A slightly slower upstream causes cascading buildup layer by layer
2. Asynchronous RPC
Asynchronous is better suited to the main chain of a game service, because it makes it easier to:
- Control thread occupancy
- Do timeouts and cancellation
- Fit an event-driven model
The cost is:
- More complex code structure
- Error handling is easier to miss
In online games, most core chains eventually lean toward asynchronous or message-driven, rather than widespread synchronous waiting.
IV. How RPC in Game Services Differs from Generic Microservice RPC
RPC in game services often emphasizes more:
- Entity- or partition-based routing
- High-frequency short messages
- Tight coupling with state machines, AOI, instances, and sessions
- Very high sensitivity to tail latency
It is not exactly the same as backend business services. Many times "method calling" is only the surface form; underneath it looks more like a targeted messaging mechanism.
For example, when a player uses a skill, the real chain may be:
- The gateway receives the packet
- It is routed to the logical node the player is on
- The logical node then calls the scene node
- The scene node triggers nearby broadcast or settlement
What matters most in this chain is not whether the function interface looks local, but whether each hop clearly knows its responsibilities and timeout semantics.
V. What Is Most Often Overlooked When Implementing RPC
1. A Timeout Does Not Equal a Failed Rollback
If a request times out, the caller can only confirm "the result did not return within the time limit", and cannot directly infer "the target definitely did not execute".
This directly affects:
- Whether retry is allowed
- Whether idempotency is needed
- Whether the final state needs to be queried
2. Retries Must Look at Operation Semantics
Not all RPCs can be retried automatically.
Usually suitable for retry:
- Read-only queries
- Idempotent writes
- Operations that explicitly support de-duplication
Not suitable for blind retry:
- Charging
- Granting rewards
- Migration and leader switching
- Multi-stage state advancement
3. Observability Must Be Built In
Once an RPC crosses process boundaries, troubleshooting depends heavily on:
- Request ID
- Call chain logs
- Timeout distribution
- Error codes
- Per-node statistics
Without these, online problems are almost untraceable.
VI. A Practical RPC Packet Structure
Most custom RPCs eventually converge to a similar structure:
- Packet header: magic number, version, length, flags
- Routing fields: service name, node ID, entity ID, partition key
- Request fields:
request_id,method_id - Metadata: timeout, trace_id, retry flag
- Payload: serialized parameters
The response packet then contains:
request_idstatuserror_codepayload
The structure is not complex, but you must always reserve room for version evolution.
VII. A Safer Practice in Game Services
Many times, a better practice than "writing RPC everywhere" is to layer:
- Query-type requests can use RPC
- High-value state advancement uses message-driven or single-threaded entity mailboxes
- Broadcast or event propagation uses an event bus
- Strong consistency chains should minimize cross-node synchronous calls
The more cross-node synchronous waits there are, the easier it is to form long call chains and cascading timeouts.
VIII. Common Misconceptions
1. RPC Is Just Syntactic Sugar for Remote Function Calls
Wrong. The truly hard parts are timeout, routing, error models, retry semantics, and observability.
2. All Service Interaction Is Suitable for RPC
Wrong. Large fan-out, asynchronous broadcast, and event propagation scenarios often suit message queues or event mechanisms better.
3. Automatic Retry After Timeout Is Safest
Wrong. Many write operations end up in a "result unknown" state after timeout, and retry may create double writes.
References
- Birrell and Nelson, Implementing Remote Procedure Calls
- gRPC design documents
- Various materials on entity routing and asynchronous calling practices in online game services
