The primary source of data for gaming is #telemetry. What is telemetry and how to collect and analyze it?



The word Telemetry is derived from the Greek roots tele, "remote", and metron, "measure".



Games are state machines - a person creates a continual loop of actions and responses which keep the game state changing. Often loops keeping the user engages over a period of time.



Telemetry helps to discovering #who is performing #what action #when and where in the game. It cannot provide #why.



Back in the day, I built Delta Lake using Azure Databricks for one of the largest Xbox game studios. It was a new concept for me, especially coming from Amazon, where we faced numerous privacy-related issues with the traditional data lake on S3 and EMR. Delta Lake emerged as a promising technology to address these problems, and it succeeded.



The Delta Lake approach worked brilliantly for game analytics. It enabled us to stream telemetry data into the Bronze and Silver layers in near real-time using Delta Streaming, and to compute metrics in the Gold layer every hour.

Databricks allowed us to unify data engineering, analytics, and machine learning cases in the same workspace.



Using Azure Bicep and Azure DevOps, we maintained our infrastructure as code.



The bottom line is that Apache Spark and the Lakehouse architecture remain significant concepts today, enabling the effective tackling of real business processes.



A close alternative to Databricks would be Snowflake, if you're willing to incur the associated costs. If not, there are numerous open-source technologies available that can serve your needs, assuming you have a competent engineering team.



Link: https://www.linkedin.com/posts/dmitryanoshin_telemetry-who-what-activity-7140747632137674752-ssRL