Skip to content

world and cell Management Design ​

Status: review draft. This refines the agreed management/execution boundary. It does not claim that standalone cell execution, remote placement, or continuous migration exists. Review the design before revising the implementation backlog and code.

1. Recommendation and Scope ​

Use world for placement, cell for simulation, and Base for persistent player state. The default deployment has one World manager on a host, automatically maintaining one or more cell processes. Management uses world; execution uses the standalone cellServer by default (world --role=cell is no longer accepted), with explicit registration, commands, and authority checks.

CellApp refers to the BigWorld comparison; Apollo's execution role remains cell. The executor defaults to the standalone cellServer entry (world --role=cell is no longer accepted), adding no cellapp role alias. Uppercase Cell is a spatial partition; Space maps to the spatial meaning of the existing Scene; Instance retains its eight-state gameplay lifecycle.

Initially, each Space has exactly one complete Cell. Scaling starts additional cell processes and places new Spaces on ready executors. Shrinking drains existing activities naturally. Continuous whole-Space/Instance migration comes later, followed by multiple Cells per Space, Ghost replication, and seamless boundaries when needed.

Apollo process architecture: world is the manager (register/topology/capacity/migration) and cellServer the standalone simulation executor (Scene/Avatar/AOI/battle/Lua), coordinated over 960 register/961 verdict/962 assign/963 retire/967 self-report; Base owns player profiles and base gameplay persisted via database; gateway routes directly to Base and cell; interfaces receives platform callbacks; machined spawns and supervises; logger/admin collect logs and metrics.Apollo process architecture: world is the manager (register/topology/capacity/migration) and cellServer the standalone simulation executor (Scene/Avatar/AOI/battle/Lua), coordinated over 960 register/961 verdict/962 assign/963 retire/967 self-report; Base owns player profiles and base gameplay persisted via database; gateway routes directly to Base and cell; interfaces receives platform callbacks; machined spawns and supervises; logger/admin collect logs and metrics.
Target design. Implementation status is documented separately.Open full-size diagram

The BigWorld CellAppMgr source places executor, Space/Cell, and load management in the manager; the CellApp source provides the executor reference. This reuses boundaries and terminology, not all distributed capabilities. See the mapping.

1.1. Application Directories and Binaries ​

This draft adopts short names for the application directory, executable CMake target, and published program, without Server, -server, or -app suffixes; the naming migration is complete. world defaults to the management role; the executor entry is cellServer—that name keeps the Server suffix as an explicit exception mirroring the BigWorld executor entry name, distinct from the short-name rule above—and the world --role=cell delegation is removed, with cellServer as the sole executor entry.

Directory / programResponsibility
apps/gateway / gatewayClient connections, sessions, routing
apps/login / loginAuthentication, directory, location projections
apps/base / basePersistent player objects and business
apps/world / worldDefault management role; executor runs via the standalone cellServer
apps/database / databaseStorage and durable results
apps/machined / machinedLocal supervision

Keep cell assembly within the World application or shared libraries with module boundaries; the executor entry is cellServer (world --role=cell delegation removed) and the manager shell no longer inlines executor assembly. The former host demo entry (apps/game-server) was removed; its hosting duties are carried by world and the shared libraries, not by another deployment role. Operations tools such as logger, admin, and fleetctl retain their own responsibilities and names.

C++ type names do not follow binary names mechanically: the Base part is base::Avatar, the Cell part cell::Avatar.

2. Ownership and Calls ​

RoleOwned stateNormal path
worldExecutor registration, Space/Cell placement, capacity reservations, topology, creation/migration decisions, drainingBase requests placement; World commands cell and publishes committed locations
cellLocal Entity/Avatar, AOI, combat, Lua, simulation clock, active InstanceGateway inputs directly to cell; simulation output to gateway; business requests to original Base
basebase::Avatar, persistent player state, business rules, settlement deduplicationBasic business and settlement through Base to database
loginAuthentication, service directory, player-location projectionsAddress discovery and projections, without Cell-placement authority
gatewayClient connection, session, Base/cell route cachesSimulation traffic bypasses the World manager
Local supervision / machinedProcess startup, liveness, exit, restart backoffWorld embeds reusable supervision by default; standalone machined can execute local lifecycle requests

World neither holds another Avatar nor ticks Scene. cell holds no complete player profile or home-database credentials and uses no remote equivalent of a local Anchor pointer. Instance shared goals, clocks, and settlement have one executor owner: initially the executor hosting the entire Space. World stores location and lifecycle projections.

One World per host is the default deployment, not a requirement that World identity equals a machine address or that all future executors must be local. Each cell belongs to one World. Multiple Worlds manage non-overlapping configured scopes; each logical World has one active manager. A shared cross-server pool has an explicit owning World, avoiding concurrent capacity allocation by two managers.

3. Identity and Minimum Records ​

These are semantic fields, not completed wire additions. Allocate message IDs and evolve fields through the existing contract generator.

RecordRequired informationConstraint
ExecutorLogical cell ID, process incarnation, owning World, control/simulation endpoints, versions/capabilities, readiness/drain statePID/IP is not stable identity; stale registrations cannot overwrite a new incarnation
SpaceStable Space ID, map asset reference, owning World, optional Instance, lifecycleIndependent of executor, map ID, and Instance ID
Cell placementSpace/Cell IDs, bounds, executor/incarnation, topology version, allocation stateInitially covers the whole Space; reserves adjacency and multiple partitions
Entity locationStable entity ID, Space/Cell, executor, simulation authority epoch, session/route versionsIndependent of Base ownership/version
Creation/migrationStable request ID, source/target, expected versions, phase, snapshot digest, decision, cleanup progressRetries query the same operation rather than create another activity

Space/entity/operation IDs must be unique across the communicating deployment. If player IDs are unique only within a home server, cross-server keys must include that server or adopt a validated global ID rule. Existing IDs are not assumed sufficient.

World owns placement, cell executes under valid authorization, and directory/gateway cache committed results. Process incarnation, World-management epoch, simulation authority, topology, Base write ownership, and session versions are separate concepts; one route_version cannot replace them all.

4. Startup, Registration, and Disconnection ​

  1. World obtains exclusive ownership, restores management records, and completes recovery before allocating new Spaces.
  2. cell resolves its World using configuration or directory, authenticates its service identity, and registers incarnation, protocol/snapshot capabilities, and capacity. Same-host machined is unnecessary.
  3. World validates ownership, identity, and versions and returns its management epoch. cell becomes ready only after dependencies, resource/script loading capabilities, and simulation ingress are checked.
  4. Only ready, non-draining executors receive placements. Heartbeats report allocations, tick delay, queues, memory, and reservations. Reject stale incarnations and replies.
  5. On disconnection, stop new allocations and mark the executor uncertain. Reconcile before creating replacement authority elsewhere.

Initial policy preserves a single writer without automatic failover. Heartbeat timeout establishes unreachability, not death. During a bounded configured grace period, cell continues existing activities. At expiry, freeze at a tick boundary, stop new external business effects, and close input acceptance at the IO boundary. Reconnection requires allocation/epoch reconciliation before unfreezing.

Use a local monotonic clock and calibrate grace periods against heartbeat timing, jitter, and acceptable pauses. Expiry or a higher epoch alone does not prove fencing: an old process may resume from suspension while receivers retain stale caches. Activate replacement authority only after acknowledged source freezing or verified termination/access isolation. Otherwise remain blocked. Automatic failover needs separate arbitration, authorization timing, and fencing at every side-effect receiver.

4.1. Separate Entries and Startup Roles ​

Startup forms are shown below; the world manager and cellServer executor entries are implemented, while automatic World-spawned children remain future work:

text
world
world --world-id 1 --m1-port 9101 --assign-spaces 401,402 \
  --service-register-host 127.0.0.1 --service-register-port 9001
cellServer --world-id 1 --m1-port 9301 --cell-id 21 \
  --register-host 127.0.0.1 --register-port 9101 --add-space 501:arena

--service-register-host/--service-register-port point world at the login service-register port; world registers directly via 964/965 (no machined). --settlement-credential-dir <dir> supplies the settlement credential directory (default off).

No arguments, or no --role, makes world default to the manager role (--role accepts only manager; there is no world form or cell form on world; the standalone cellServer accepts only cell for --role). A missing role value, unknown role, or invalid configuration fails explicitly. Role is fixed at startup, never inferred from a port.

World generates each child's role, logical identity, manager endpoint, separate runtime directory, and bootstrap credentials; developers do not maintain a handwritten configuration per executor. Do not log bootstrap secrets. Executors can bind ports and report them through authenticated registration rather than race between probing and binding a free port.

World starts management and local supervision without a simulation host. cell starts simulation and registration without spawning more executors. The shared executor body performs validation and executor assembly; management, simulation, and supervision retain module boundaries. Logs, metrics, and crash reports identify role, logical ID, and incarnation. cell opens neither World-management logs nor Base profile storage.

World chooses desired executor count and invokes local supervision by default. Extract reusable functionality from apps/machined and add dynamic process membership. The existing supervisor is principally POSIX with a fixed roster; cross-platform autoscaling is not implemented. Use structured argv, a spawn/exec wrapper appropriate for a multithreaded POSIX host, and separately implemented Windows process handles and cleanup.

With standalone machined, World requests desired lifecycle changes and machined executes them. Each process has exactly one restart supervisor. The World manager itself is supervised by a system service, container, or machined. Remote startup requires a supervisor on the target machine or external orchestration; local spawning does not provide remote startup.

4.2. Automatic Growth and Natural Shrinking ​

DecisionRule
LimitsDefault minimum one executor; maximum bounded by CPU/memory and host reserves
GrowthSustained capacity pressure/high watermark starts executors within cooldown and maximum; starting processes count toward desired capacity
ReadinessSpawn success is only starting; allocation waits for authenticated registration and readiness; startup timeout cleans up with bounded backoff
ShrinkingSustained low watermark selects an executor for draining; exit only after active/prepared Spaces, migrations, and unconfirmed settlements are cleared
Drain cancellationCancel before exit is issued if capacity pressure returns; never allocate to an exiting process
Blocked drainingPersistent Spaces or unfinished activities keep the process alive; timeout reports blocked shrink rather than killing gameplay
RestartDistinguish intended shrink from crash; retries are bounded, incarnation changes, and old activity restoration is not assumed

Use sustained windows, separate high/low thresholds, and cooldown. More processes distribute independent Spaces; they do not accelerate one hot Space with a single Cell. At host resource limits, reject new activities or add hosts through deployment management.

Persist executor identity/incarnation, startup/drain/exit intent, and supervision ownership. After World restart, reconcile surviving executors before filling deficits. Do not spawn a duplicate fleet merely because surviving processes are not children of the new manager. Validate startup identity and authenticated registration rather than PID alone.

Publish immutable version directories or equivalent release artifacts. World starts the selected release and records build/protocol capabilities without overwriting running files. Old/new cell processes may use different versions of the same binary. An old World can launch a new cell only when actual control-protocol compatibility exists.

5. Space Creation, Capacity, and Destruction ​

PhaseAction and acknowledgement
RequestBase validates party, eligibility, version, and gameplay rules and supplies a stable request ID
ReserveWorld filters ready/compatible/admitted executors, reserves capacity, and records intent; no capacity produces explicit rejection
Preparecell idempotently loads resources and creates the complete Cell/local Scene and optional Instance, without inputs or activity ticks
CommitWorld durably records placement and requests activation; publish usable route/admission only after activation acknowledgement
EnterBind original Base, read-only gameplay snapshot, session, and expiring admission; create one authoritative Avatar
DestroyStop admission, complete activity and durable settlement handling, release simulation resources, record terminal state, reclaim capacity

On timeout, query the same operation. Cancel prepared resources only after confirming no commit. Lost replies must not duplicate creation; committed activation failure requires recovery or explicit failed cleanup. Destruction checks placement epoch so stale requests cannot destroy a new allocation.

Budget tick time, entities, memory, input backlog, and reserved resources rather than just players. World reserves capacity atomically against concurrent requests; cell enforces a local hard limit. Overload rejects new activities while preserving active budgets. Initially, multiple Spaces use one sequential simulation thread, with its shared failure/stall scope; parallelism is separate work.

Retain recoverable settlement records until durable confirmation. Large map resources may unload, but the only settlement evidence cannot be deleted. World cannot infer crash compensation from an in-memory projection; see cross-server design.

6. Management Persistence and World Restart ​

Persist allocation/version records, operation decisions, capacity reservations, Instance ownership and settlement-recovery record locations, and draining. Heartbeat loads and address caches are rebuildable; commit decisions are not disposable cache or ordinary log lines.

Use a World-owned persistent management journal/checkpoint initially, separate from player database and without another business process. Successful commit replies wait for the declared durability level. Require idempotent replay, integrity checks, incomplete-tail handling, and single-writer locking. Existing player PersistJournal is a reference, not proof of management transactions or power-loss safety.

Initially support one active manager restarting with the same persistent volume, without automatic cross-host standby. Verify the old manager has stopped, restore records, increase management epoch, and reconcile executor manifests. Executor self-reports cannot replace commit history. Conflicting/missing records or uncertain source isolation keep activities frozen; unavailable records block new allocation instead of silently starting empty.

Local supervision also needs surviving-process reassociation and supervision continuity. Parent-child spawning alone cannot deliver continuous World upgrades. Without that capability, explicitly use a stopped upgrade rather than blindly killing or duplicating existing executors.

For planned updates, stop new creation/migration, complete or record in-flight operations, hand off the journal, stop the old manager, and reassociate executors. Existing activities continue only within validated grace/authorization bounds; otherwise freeze. Simulation pause and client TCP loss are separate outcomes.

The reconciliation record formats (journal v2), protocol extension (967 cell_space_report), and per-item adjudication are detailed in the World Restart Reconciliation design (Batches A/B implemented, Batch C pending).

7. Whole-Space/Instance Migration ​

New placements and natural draining can ship first, but do not establish continuous hot updates.

  1. Reserve target capacity and validate protocol, assets, and snapshot conversion; record migration ID and expected source placement.
  2. Freeze the migration unit at a source tick boundary and record the last executed input sequence. Complete or record Base requests and pending settlements, producing no new side effects.
  3. Snapshot entities, monsters, Instance phase/goals, clocks, randomness, timers, serializable script state, input watermarks, and pending work. Reject unsupported activities rather than copy Lua memory pointers.
  4. Target validates version/digest and restores inactive state, without ticks, rewards, or authoritative output.
  5. After confirming source freeze, World durably commits target authority and new epochs. Target activates from this decision; directory/gateway update versioned routes. Source stays frozen. Stale routes return migration/stale status rather than unbounded forwarding.
  6. Target resumes with input deduplication and acknowledges. Source cleans up, capacity is released, and World records completion. Precommit cancellation first verifies no commit; postcommit timeout cannot unilaterally unfreeze the source.

Initially migrate the complete Space and associated Instance. Shared activity dependencies across Spaces must move together or make that activity unsupported. Ordinary cross-server entry moves a player to another Space without moving the entire original activity.

Input buffering is bounded by size and time; reject overflow. Replay depends on request sequences and execution watermarks. Legitimate settlements executed before freezing may arrive late: Base validates authorization history, stable settlement IDs, and allowed operation scope. Neither blindly rejecting every old epoch nor permitting new effects from an obsolete source is correct.

8. Cross-server Play and Both Update Modes ​

The destination World allocates its own cell while players retain their original Base. Resolve Base service identities, not home-database endpoints or credentials. Base replacement changes incarnation, not home affiliation or storage ownership.

For cross-World player migration, the source World owns the durable decision; destination World reserves/prepares and activates according to it. If the coordinator is unavailable, query the same migration ID and do not unilaterally cancel or resume both ends. Retain decision evidence and completion records. Live transfer of an entire Space's management ownership between Worlds is deferred pending management-record and in-flight-operation handoff design.

Update modeRequired capabilitiesLimits
Protocol-changing process replacementCoexisting releases, actual old/new decoders/adapters, snapshot conversion, freeze/commit/route handoffValidate internal and client protocols separately; unchanged clients still need actual compatibility, not just a schema allowlist
Code-only, unchanged wireLua updates at safe points with private-state/coroutine handling; C++ replacement uses the same migration flowIn-place Lua updates do not prove arbitrary state migration; C++ replacement still requires snapshots and authority transfer

Base transfers persistent state/write ownership, World transfers management records, and cell transfers simulation. An unchanged gateway may preserve TCP while simulation pauses. Replacing gateway still needs independent connection handoff or reconnection.

9. Seamless-map Compatibility ​

Reserve stable Space/entity IDs, common coordinates, explicit Cell bounds/adjacency, topology, and entity authority epochs. Never identify Space by executor ID. Keep Scene as the local simulation container and later separate local bounds from logical Space identity, without another synonymous Scene→Space container chain.

Multiple Cells per Space will additionally require Real/Ghost replication, continuous AOI/Witness, entity-level handoff, cross-Cell combat adjudication, and one owner of shared Instance state. Initially migration may advance authority for the whole Space, but contracts must allow entity authority to diverge later.

Current SceneTransfer recreates Avatar on explicit space change. It cannot implement same-Space handoff preserving entity identity, health, cooldowns, timers, and observation continuity. Reserved interfaces are not evidence that seamless maps work.

10. Risks and Review Findings ​

RiskIncorrect shortcutRecommendation
Excess complexity for lightweight gamesBegin with partition balancing, Ghosts, and automatic failoverStandalone cellServer executor, local supervision, single-Cell Spaces, natural draining first
Manager bottleneckProxy movement/combat/visibility through WorldDirect executor traffic, bounded control queues, summarized heartbeats, throttled migration
Split simulationActivate a copy on heartbeat timeoutConfirm source freeze/isolation; uncertainty blocks activation
Manager failureTrust executor self-reports after losing memory topologyExclusive management, durable decisions, reconciliation; no initial automatic standby
Incomplete migrationMove only Avatar and recreate other runtime stateComplete dependency snapshots; reject unsupported activities
Cross-server writes/duplicate rewardsDirect database writes or validate only the current epochOriginal Base, stable settlement IDs, authorization history, durable deduplication
Incorrect durability claimTreat file flush as power-loss safetyDeclare durability level, test crash points, refuse unsafe recovery

Implementation parameters decided (2026-10-30; docs/review/2026-10-30-decisions.md): management-log durability level = A lines fsync, R/B lines flush + failure downgrade (S05); heartbeat/grace intervals = 5s/15s/30s as a dev-validation baseline (D24, not a production promise). Implementation parameters still pending measurement: capacity budgets, snapshot size/pause budgets, and each gameplay script's migratable-state inventory — these are not written as performance promises without measurement. Automatic failover, live management-scope transfer, and multi-Cell Spaces are separately scoped work.

10.1. Is interfaces Needed? ​

The KBEngine configuration reference describes interfaces as an external-account, login, and billing integration service. Its source declarations include account creation/login and charge requests/callbacks. It is separate from client gateway and World simulation management.

Reserve asynchronous integration interfaces without requiring another default process. Simple platform-login checks can initially run through a login adapter with bounded timeout, concurrency, and queues. External latency must not block authentication processing or simulation ticks. Multiple channels, payment callbacks, separate credentials, or deployment isolation may later justify optional apps/interfaces / interfaces without inventing another name.

A separate service verifies signatures, maps external identities, transforms formats, and forwards bounded requests. login decides game authentication/session outcomes, base decides rewards and consumption, and database persists them. Define order/event IDs, retries, and durable deduplication end to end. Acknowledging external acceptance requires recoverable delivery records rather than an in-memory queue. interfaces neither mutates player profiles directly nor simulates gameplay or acquires home-database credentials. Enabling this optional service and designing payments require separate review.

11. Current Code and Implementation Order ​

Current codeTarget boundary/gap
WorldApp gameLoop, acceptanceQueue_, WorldHost, World/SceneMove simulation and bounded intent consumption into cell; preserve single-writer and shutdown draining
WorldApp PlayerStore/Anchor and direct settlement callsReturn long-term ownership to Base; replace with read-only snapshots and remote business requests
SquadInstanceService and InstanceRuntime belongs to cell; World reserves/commits placement; local pointers cannot span processes
WorldAssignmentMissing complete executor/Cell and management/simulation epochs
SceneTransfer and old profile migrationsExplicit space change is separate; moving full profiles does not implement fixed-Base simulation migration
M1, directory, machinedReuse transport/supervision primitives; add authenticated registration, wire, idempotency, authorization

Complete Base/simulation data separation, then standalone cellServer executor and world management roles, local supervision, registration, whole-Space creation/destruction, and failure freezing. Add multiple executors and bounded autoscaling, then management recovery, continuous activity migration, and both update drills. Until migration works, upgrading by natural draining is not continuous hot-update completion.

No executables, C++ types, or protocol fields change in this draft, and no new decision is automatically accepted. Root todo.md remains the only backlog; update P2/P3/P4/P5 acceptance conditions after review.

12. Acceptance ​

  • Standalone cellServer executor and world management roles run as real processes; no recursive cell spawning, bounded startup retries, no restart after intentional shrinking.
  • No executor means explicit creation failure; World neither ticks simulation nor loads complete profiles.
  • Validate thresholds, cooldown, starting counts, limits, natural draining, persistent Spaces blocking shrink, and no duplicate spawning after manager restart.
  • Test concurrent placement, Space isolation, reservations/rejection, duplicate requests, lost replies, and delayed destruction.
  • Reject forged/stale registration, stale input/output, and paused processes resuming after disconnection; never produce two authorities.
  • Crash before/after each durable creation/migration point; recover one decision, and never silently activate replacement state after losing management records.
  • Continuous migration preserves members, monsters, goals, health, skills, clocks, randomness, and required scripts; cover unknown decisions, target failures, and source-cleanup failures.
  • Two home Bases retain ownership; lost settlement ACKs, duplicates, late results, unavailable Base, and restart cannot duplicate rewards or erase unconfirmed evidence.
  • Drill real protocol/snapshot evolution and unchanged-wire code updates separately; record activity pause, login continuity, and TCP continuity independently.

Released under the MIT License