interesting writeup on inference economics
tldr it doesn't scale as good as "regular web services" where it's possible but hard to get to 80-90% utilization, and is more like 40-50% even with an idealized load
https://www.anjalishriva.com/fat-tails
Post #972
183