Lessons learned from 2 years of operating a C++ MMO game server in production 1/N

I've been working as a server team lead and technical director on a large-scale MMO for the past 2+ years, from launch through live service.

our server uses an in-house actor model.
The core parts of our code are over 15 years old.
It started with C++98. Now we use C++20.
We use modern C++, but old style code is still in many places.

One example is that we allow certain race conditions.
This codebase is older than the C++ memory model.
It worked then, and it still works now.

// attacker context
int32_t hp = target->hp(); // violates actor context, but allowed

Another example is our enum convention.
For enums that can have an empty state, we use INVALID as the first value.

enum class Weapon {
INVALID = 0,
Dagger,
Sword,
Bow,
};

We replaced INVALID with std::optional to separate the logic of "no value" from the actual data set.

// before
Weapon weapon = target->weapon();
if (weapon != Weapon::INVALID) {
Process(weapon);
}

// after
if (target->weapon().has_value()){
Process(target->weapon().value());
}

One month after we deployed it, we got a crash.
We identified the root cause through crash dump analysis. It was a TOCTOU problem.

// reading from another actor context
if (target->weapon().has_value()){
// another thread had changed the weapon.
Process(target->weapon().value()); // bad_optional_access
}

With INVALID, this would not have crashed.
We had to review all the code we changed.
We reverted the code that reads optional values from other actors back to the INVALID convention.

Now INVALID and std::optional coexist in our codebase.
We know that switching to std::atomic is the right fix.
But a simple swap could introduce other problems, so we are thinking it through carefully.

we did not review the existing code well enough before applying modern C++.
Legacy code has implicit rules.
Making code more explicit is what modern C++ is really for.

https://redd.it/1rotys6
@r_cpp