process. Each actor has its own message queue and processes messages sequentially, scheduled across a thread pool.

The key performance principle: **distribute work across many actors so all 72 cores stay busy.**

But in live service, we hit an unexpected bottleneck: **field bosses.**

The game originally designed all zones as PvP-enabled, so field bosses were meant to be PvPvE encounters — players would fight each other while fighting the boss, naturally distributing load. But a late design change introduced "peace mode" (no PvP). Result: 2000+ players stood still in one spot, spamming skills at a single NPC. That NPC's actor became a Monolithic Actor — hundreds of message producers, one sequential consumer. Its message queue grew faster than one core could drain it, while the other 71 cores sat idle waiting.

Our general strategies for preventing singleton actor bottlenecks:

1. **Split by purpose** (ClientRepo, GuildRepo, SessionRepo — never one god-repository)
2. **Shard by hash** (N actors with modulo routing for request-heavy workloads)
3. **Per-thread read copies** (for read-heavy data like spatial indexes — reads are lock-free, only writes go through the actor)

For the field boss specifically, we added **MulticastProxyActors**. Profiling showed the dominant cost inside the boss actor was broadcasting packets to hundreds of nearby players (N² fan-out). The boss now delegates packet broadcasting to a pool of proxy actors, keeping its own queue focused on game logic.

# 4. TCP Bandwidth-Delay Product bit us in production

During a beta test hosted on AWS in a distant region, players with high-latency connections (\~200ms RTT) kept disconnecting during large-scale battles. The symptom: throughput capped at \~50 KB/sec.

After ruling out client issues with a headless client test, we traced the problem to our network layer using Windows Registered I/O (RIO). The send buffer was sized at only 8KB. Since sends complete only after the data is ACKed, with 200ms RTT the pipeline stalls: **8KB / 200ms = 40 KB/sec maximum throughput.**

The fix was simply increasing the RIO send buffer size. That's it — a one-line config change. But it took days of investigation across network, client, and server teams to find it. The deeper lesson was: **understand TCP fundamentals (BDP = Bandwidth × RTT) when sizing your I/O buffers**, especially when deploying to regions with higher latency than your test environment.

# 5. C++20 Coroutines: powerful but deceptive

We adopted C++20 coroutines (`co_await`) alongside our existing Promise-based async. Coroutines are great for readability, but they create a **dangerous illusion of synchronous code.**

Task<void> OnRecvSellWeapon() {
const Item* weapon = pc->GetWeapon(id);
if (!pc->CanSell(weapon)) co_return;

co_await *shop_npc; // ← context switch!
if (!shop_npc->CanBuy(weapon)) co_return; // ← weapon may be deleted!

co_await *pc; // ← context switch!
pc->AddGold(weapon->price); // ← weapon may be deleted!
}

The code reads like a synchronous function, but each `co_await` switches to a different actor's context. Between suspension points, **any pointer or reference may have been invalidated** — other actors keep running. Developers naturally forget this because the code *looks* sequential.

**The coroutine paradox:**

* Pro: "Reads like synchronous code"
* Con: "Developers forget it's asynchronous"

Solutions: acquire ownership (`unique_ptr`), re-validate after resume, or copy values before suspension points.

We follow C++ Core Guidelines strictly: [CP.51](https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#cpcoro-coroutines) (no capturing lambda coroutines), CP.52 (no locks across suspension), CP.53 (no reference params to coroutines).

# 6. Protobuf serialization: serialize once, copy many

For multicast packets (same message to hundreds of players), we learned that:

* **Serialization cost >> memory copy cost**
* Initial approach: serialize directly