📝Dataframe validation with Pandera
In large DS projects, the Great Expectations framework can be used to validate the dataset and check the quality of the data. However, smaller tasks require simpler tools. For example, the lightweight Python library Pandera, which explicitly checks information in dataframes at runtime. Pandera allows you to define a data schema once using a class-based API with pydantic syntax and use it to validate various types of dataframes, including pandas, dask, modin, and pyspark.pandas. You can check the types and properties of columns in pd.DataFrame or values in pd.Series, perform more complex statistical testing such as hypothesis testing. You can synthesize data from schema objects for property-based testing using pandas data structures.
Function decorators allow you to integrate with existing data analysis/processing pipelines using function decorators. With lazy validation, you can validate dataframes before errors occur. Finally, compatibility with other Python tools such as pydantic, fastapi, and mypy makes Pandera a useful tool for the ML developer and data analyst.
Documentation: https://pandera.readthedocs.io/en/stable/
Example: https://towardsdatascience.com/validate-your-pandas-dataframe-with-pandera-2995910e564