Reducing FFmpeg build times in practice

We compile FFmpeg from source regularly for custom codec work and video pipelines, so build time directly affects iteration speed. Baseline was a 24 minute clean build on a 16 core Xeon. During active development we were running multiple builds per day, and CI was consistently blocked on compilation.

ccache helped incremental builds but not clean CI runs. Disabling unused codecs with --disable-everything and enabling only what we needed saved about three minutes. NVMe storage produced marginal gains. Scaling cores and RAM helped up to 16 cores, then flattened out.

Profiling with ninja -d stats showed compilation at \~80 percent of wall time, linking at \~15 percent and mostly serial, configure at \~5 percent and serial.

We then tested distributed builds. distcc delivered roughly a 60 percent improvement but required nontrivial setup. icecc performed slightly better in our environment at around 65 percent. Incredibuild produced the largest gain at about 88 percent over baseline.

Final numbers:

Clean build: 24 minutes to 2 minutes 50 seconds

Incremental: 8 minutes to 45 seconds

Full CI pipeline: 35 minutes to 6 minutes

How far are you pushing distcc or icecc in workloads? Anyone managed to squeeze more out of them?

How are you handling LTO in distributed setups? Is there an approach that preserves most of the distributed gains without turning the final link into a long serial step?

For anyone working on other large C or C++ codebases, did distributed compilation create a similar step change, or did you hit a different ceiling first?

https://redd.it/1rp23xe
@r_cpp