"The User-Agent request-header field contains information about the user agent originating the request. This is for statistical purposes, the tracing of protocol violations, and automated recognition of user agents for the sake of tailoring responses to avoid particular user agent limitations. User agents SHOULD include this field with requests. "

Создатели протокола HTTP знали, какую дрянь они делают, и всё равно её сделали. Подробности.