Can Apollo Be More Complete Than BigWorld? An Auditable Assessment
Scope: compare Apollo's fit and verifiability for instance-based multiplayer game servers, not total product features or historical production scale. This is an engineering note. “Can exceed” means a measurable direction, not a claim that Apollo already exceeds BigWorld. Repository status was checked against current
main; external sources retrieved 2026-10-19.
Conclusion
Apollo can become a better fit and provide stronger evidence in selected dimensions; there is no basis today to claim that Apollo is more complete overall. Public BigWorld material shows distributed Cells, Ghosts, CellAppMgr, and load adjustment. Apollo focuses on instance-based gameplay, deterministic verification, and engineering observability, while reserving boundaries for future multi-Cell operation. These are different goals and require separate acceptance evidence.
A. Dimensions Apollo could exceed
These are potential engineering advantages. Current evidence proves only that Apollo has an implementation or design foundation. “Exceeds” is justified only after meeting the criterion in the last column.
| # | Dimension and Apollo evidence | BigWorld counterpart | Acceptance criterion for “exceeds” |
|---|---|---|---|
| A1 | Instance gameplay lifecycle and settlement boundary. Apollo implements eight Instance states and adjacent-transition validation: instance.hpp, instance.cpp, test_instance.cpp; goal is described in design §Instance state machine. | The reviewed BigWorld spatial model uses Space/Cell; the source mapping found no built-in eight-state gameplay lifecycle equivalent. CellAppMgr manages Space/Cell and is not itself a gameplay lifecycle state machine. 14.4.1 Space | Test every state and illegal transition, disconnect/cancel/repeated settlement and cleanup combinations; make settlement IDs idempotent and prove restart reconciliation grants no duplicate rewards. Any claim must stay limited to the compared gameplay lifecycle. |
| A2 | Deterministic battle recomputation evidence. Tests cover stable input ordering, seed sensitivity, and hash-chain recomputation: test_battle_runtime.cpp. The four constraints and replay format are specified in Battle Determinism §§1–5; a complete offline replay tool remains planned (§7). | The reviewed BigWorld 14.4.1 Cell/Witness/CellAppMgr sources cover spatial simulation and management. This search of public source/manuals found no equivalent battle-input recording and deterministic recomputation contract. This is a finding about reviewed material, not proof that every BigWorld deployment lacks one. | With a fixed build, seed, snapshot, and input, repeated and post-restart runs produce identical per-tick hashes; a changed input identifies the first divergent tick. Archive tool version, recording, and report. Claim superiority only over the reviewed reference workflow after the corpus passes. |
| A3 | Reproducible capacity and regression benchmarks. Capacity and Benchmark §§2.1–2.4 records hardware, workload, measurement methods, and multi-model results, while documenting gaps in network sends, full capacity, and mixed workloads. | BigWorld's operations guide describes CellApp CPU-load-based adjustment/balancing, but the reviewed public material has no directly rerunnable report at Apollo's measurement granularity. Server Operations Guide §CellAppMgr | Freeze environment, workload seed, sampling, and the full path; report p50/p95/p99, throughput, dropped ticks, queues, network, and resource costs, with raw results and rerun commands. Compare on equivalent hardware and semantics, repeat within a declared variance, and show better target metrics. No same-basis data means no performance claim. |
| A4 | Fault diagnosis and retained engineering evidence. Apollo has local Crashpad capture, symbolication workflow, and logging/metrics conventions: Crash Capture §§1–4, Capacity and Benchmark §5. Upload, quotas, and platform validation remain open boundaries. | BigWorld has BWMachined/Reviver supervision and CellApp fault-tolerance mechanisms; see Server Overview §§5.1, 5.8–5.9 and 14.4.1 process sources. | Run the same fault-injection set and quantify capture rate, symbolication success, time to diagnosis, and recovery decisions. Correlate every crash with process generation, logs, metrics, and instance/settlement records. Claim superiority only if comparative results show better diagnostic coverage and no recovery data errors. |
| A5 | Modern client protocol and observability governance. Apollo's network design covers protocol versions, boundaries, metrics, and load testing: Network Abstraction §§4–5, Gateway Responsibilities. Compatibility windows and business protocols still need scenario acceptance. | BigWorld 14.4.1 public source uses Mercury message interfaces (for example, CellAppMgr interface). It has mature messaging/process mechanisms, but this review found no equivalent modern client-evolution and telemetry acceptance data. | Automate old/new client handshakes, unknown fields/messages, rolling upgrades, rejection reasons, and end-to-end traces; expose metrics/alerts for critical states. Prove clearer compatibility windows, complete diagnostics, and no mis-decoding under equivalent upgrade/fault scenarios. |
B. Dimensions that cannot yet be claimed
These require long-running production and scale evidence. Designs, unit tests, simulations, and single-host benchmarks do not substitute for real-scale proof. For now, say only that the design has the relevant boundaries or plan.
| # | Dimension | Apollo evidence and limit | BigWorld capability/source | Evidence needed |
|---|---|---|---|---|
| B1 | Ghost/Witness and seamless visibility at 10k-player scale | Scene AOI is local; inter-Cell Ghost lifecycle, sequence repair, and continuous Witness visibility remain target design, not implementation. Witness/Ghost; Seamless Map §§3–4, 8. | BigWorld documentation describes multiple Cells/CellApps, Real/Ghost replication, and neighboring visibility: Server Overview §5.1. | Long-running multi-host tests with declared concurrency, entity density, AOI/message load, jitter, and faults; record boundary traversal without duplicates/flicker or dual Real, and compare with BigWorld under equivalent load. |
| B2 | Cross-Cell migration in practice and fault takeover | Freeze, snapshot, commit, generation, and fencing are designed, but the real inter-process transfer path and production drill are incomplete. Seamless Map §§5–8; Mapping §§1, 6. | BigWorld CellAppMgr source includes CellApp failure/recovery, Cell update, and placement management interfaces; the Cell model supports partition-migration mechanisms: CellAppMgr, Cell. | Cover pre/post-commit failures, network partitions, old-process resurrection, and manager restart across processes/hosts; prove single-writer authority, no missing/duplicate settlement, and continuous active state over repeated runs. |
| B3 | Long-running CellAppMgr balancing and mature operations scale | WorldApp still advances Scene directly. World/cell separation, capacity scheduling, dynamic migration, and production data are absent. Mapping §§1, 6; World/cell management. | BigWorld CellAppMgr receives CellApp load and adjusts Cell boundaries; the public manual describes CPU-load-based balancing. Server Operations Guide §2.8 | Long-running multi-host operation with hot-spot movement, node changes, failures, and upgrades; report skew, migration cost/failure rate, stabilization time, and operator intervention against an equivalent environment. |
C. Current gaps against BigWorld
Grouped from the implementation-boundary column in BigWorld–Apollo Mapping §§1–6. These are current gaps, not permanent limitations.
| # | BigWorld capability currently absent or incomplete in Apollo | Apollo boundary |
|---|---|---|
| C1 | Independent CellApp execution process and separation of World management from Cell simulation. | Mapping §§1, 6: WorldApp still advances Scene; independent cell entry, registration, and remote dispatch are incomplete. |
| C2 | Production partitioning of multi-Cell Spaces, topology/boundary management, and automatic load adjustment. | Mapping §§2, 5: current path is one process/Cell; Seamless Map §§2, 8: static design stage. |
| C3 | Full Ghost replication, neighboring-Cell visibility merging, and continuous cross-Cell Witness. | Mapping §§3, 5; Witness/Ghost: only local AOI partial responsibilities exist. |
| C4 | Closed-loop cross-Cell authority transfer, fencing, fault takeover, and state recovery. | Mapping §§5–6; Seamless Map §§6, 9: design constraints are not runtime capability. |
| C5 | Full BaseAppMgr-equivalent Base registration, player ownership, recovery, and placement service. | Mapping §1: loginServer has partial primitives; Base ownership/recovery remain pending. |
| C6 | General Base/Cell entity system, distributed Mailbox/EntityCall routing, and automatic property replication. | Mapping §3: RemoteEntityRef and related primitives lack complete authority versioning, receipts, and inter-process wiring. |
| C7 | BigWorld Python compatibility, official-client network compatibility, and full EntityDefs/IDL behavior. | BigWorld Compatibility §Not Implemented: current layer is a single-threaded C++ API facade and explicitly excludes these. |
| C8 | CellApp fault tolerance and long-term balancing validated at operational scale. | Mapping §§1, 5–6: reference boundaries/plans exist, not field evidence; related to B2/B3 maturity requirements. |
Acceptance matrix: minimum falsifiable evidence
Retain the input, environment, rule, and result for each experiment. A failure disproves the corresponding “exceeds” claim; documents and diagrams alone are insufficient.
| Dimension | Minimum acceptance form | Falsification condition |
|---|---|---|
| A1 Instance lifecycle | Full transition table, repeated settle/destroy, and post-failure ledger reconciliation; assert unique rewards and cleared terminal resources. | An illegal transition succeeds, a reward is duplicated/lost, or an active instance cannot converge to terminal state. |
| A2 Deterministic replay | Repeat with fixed binary/seed/snapshot/input; change one input and identify the first per-tick hash divergence. | Same input produces different hashes or an intentional change cannot be localized. |
| A3 Capacity/regression | Rerunnable workload, hardware/compiler parameters, raw samples, and p95/p99/throughput/dropped-tick/queue report; same-basis BigWorld comparison. | Cannot reproduce, critical data is missing, target metric is not better, or variance exceeds declared bounds. |
| A4 Fault diagnosis | Same injection list with dump, symbolized stack, generation, correlated logs/metrics, and recovery decision; compare item by item. | Missing crash evidence, unusable stack, uncorrelated generation, or state/settlement error during recovery. |
| A5 Protocol governance | Automated old/new protocol matrix, rolling upgrade, unknown fields, reconnect behavior, plus trace/alert records; compare equivalent BigWorld scenarios. | Valid old clients are silently misread, upgrades misroute messages, or faults cannot be diagnosed. |
Recommended order (pre-decided)
First complete A1/A2 automated tests and durable settlement/recording evidence → establish A3/A4 baselines (measurement and fault records) → close A5 protocol compatibility and observability → separate World/cell and implement static multi-Cell → Ghost/Witness and boundary AOI → fenced migration/takeover → automatic balancing and large-scale soak comparison.
Migration and automatic balancing depend on topology identity, single-writer authority, replayable state, capacity telemetry, and fault reconciliation. Expanding distributed runtime before these are evidenced makes correctness hard to prove. The first four items improve verifiability for Apollo's instance-based target; multi-Cell is later work to catch up with BigWorld's spatial runtime, not a present advantage.
External references (retrieved 2026-10-19)
- Public BigWorld 14.4.1 source: CellAppMgr, Space, Cell, Cell Entity, Witness, CellAppMgr interface. These establish mechanisms, not commercial deployment scale.
- BigWorld Server Overview, public manual mirror, covering CellApp, BaseAppMgr, DBMgr, BWMachined, and Ghosts: Chapter 5. This is an older manual used for mechanism background; use the pinned 14.4.1 source for version-specific claims.
- BigWorld Server Operations Guide, public manual mirror, CellAppMgr configuration and load balancing: Chapter 2. Historical documentation; it does not replace runtime measurements for a specified version.
- Apollo benchmark, determinism, crash-capture, and implementation-boundary references are linked in each A/B/C row; designed and implemented status are distinguished.
