🤦🏼‍♀️Too many datasets? Label their metadata and divide them onto 3 categories: Technical (logical and physical), Operational (Lineage and Data Profiling stats) and Team metadata from data scientists and analysts.
To understand the each dataset better ask next questions:
1. What does the data represent logically? What is the meaning of the attributes? Is it the source of truth, or derived from another dataset?
2. What is the schema of data? Who manages it? How was it transformed?
3. When was it last updated? Is the data tiered? Where are the previous versions? Can I trust this data? How reliable is the data quality?
4. Who and/or which team is the owner? Who are the common users?
5. What query engines are used to access the data? Are the datasets versioned?
6. Where is the data located? Where is it replicated, and what is the format?
7. How is the data physically represented, and can it be accessed?
8. Are there similar datasets with common similar or identical content, both overall as well as for individual columns?
When you have answers about all datasets you can build a metadata catalog service is a critical building block of Data Lake/Data Mesh/Data Lakehouse platforms. This service typically post-hoc collects metadata after the datasets have been created or updated by various pipelines without interfering with dataset owners or users.
https://medium.com/wrong-ml/why-is-understanding-datasets-hard-in-the-real-world-6eec47cafaa1