Logging is the oldest tool to troubleshoot issues with the software. But relevant information is spread across many individual log lines, making it difficult or even impossible to quickly search right details, perform some aggregation or analysis. That's where canonical logs concept can help.
Canonical log is the structured one long log line at the end of the request (or any other type of work) that includes fields with request’s key characteristics. Having that data collocated in single information-dense lines makes queries and aggregations over it faster to write, and faster to run.
Canonical log can include the following information:
- HTTP verb, path, response code and status
- Authentication related information
- Request ID, Trace ID
- Error ID and error message
- Service info: name, version, revision
- Timing information: operation duration, percentiles, time spent in database queries and others
- Remaining and total rate limits
- Any other useful information for your service
I want to highlight that log must be structured (key-value, json) to make it machine readable. Structured logs can be easily indexed by many of existing tools, providing an ability to search and aggregate collected data.
Simple canonical log sample:
[2019-03-18 22:48:32.999] canonical-log-line alloc_count=9123 auth_type=api_key database_queries=34 duration=0.009 http_method=POST http_path=/v1/charges http_status=200 key_id=mk_123 permissions_used=account_write rate_allowed=true rate_quota=100 rate_remaining=99 request_id=req_123 team=acquiring user_id=usr_123
Good practice is to formalize log contract across services and applications. As an example protobuf structure can be used for that purposes.
Canonical logs seems to be a lightweight, flexible, and technology-agnostic technique to improve overall system observability. It's easy to implement and extend any existing logging capabilities.
References:
- Using Canonical Log Lines for Online Visibility
- Fast and flexible observability with canonical log lines
- Logs Unchained: Exploring the benefits of Canonical Logs
#engineering #observability