Here's another good question, that Daniil you may ask shortly: Why logit is slower than relu

https://www.coursera.org/learn/neural-networks-deep-learning/discussions/weeks/1/threads/OC2Bh8ahEeetfwrt7EZTMA

We've been discussing it with Agerke last week.