Sub-microsecond timing on EC2 is way messier than I expected

Been doing sub-microsecond profiling on EC2 and kept getting wildly inconsistent cycle counts.

One mistake was using cpuid as the serialization barrier before rdtsc. On a VM that can be a mess, since cpuid often traps so the hypervisor can fake feature flags. So now the "measurement overhead" includes a VM exit, which is thousands of cycles on some runs.

Switching to lfence + rdtsc made the numbers a lot more stable.

Then I hit the calibration problem. Measuring TSC frequency with a short sleep() looked simple, but the results were all over the place. Scheduler delay, timer granularity, and probably vCPU steal time were enough to make the calibration useless at this scale. A busy-wait loop with pause gave me a much saner number.

Also forgot to pin the thread at first. rdtscp at least tells you when you migrated, but those samples are basically trash. Same with the first few iterations before icache/branch predictor warm up.

Curious what people here actually use for sub-microsecond timing. Do you just trust nanobench / Google Benchmark, or do you still end up writing your own rdtsc wrappers once VMs get involved?

https://redd.it/1t0o16n
@r_cpp