31. What is ETL process? ๐
ETL stands for Extract, Transform, Load.
- Extract: Pulling data from sources (databases, APIs, files) ๐ค
- Transform: Cleaning, formatting, and applying business logic ๐ ๏ธ
- Load: Saving the transformed data into a data warehouse or system ๐ฅ
It helps consolidate data for reporting and analysis.
32. What are some challenges in data cleaning? ๐ซ
- Missing values ๐คท
- Duplicates ๐ฏ
- Inconsistent formats (e.g., date formats, units) ๐งฉ
- Outliers ๐
- Incorrect or incomplete data โ
- Merging data from multiple sources ๐ค
Cleaning is time-consuming but critical for accurate analysis.
33. What is data wrangling? ๐งน
Also known as data munging, itโs the process of transforming raw data into a usable format.
Includes:
- Cleaning โจ
- Reshaping ๐
- Combining datasets ๐
- Dealing with missing values or outliers ๐๏ธ
34. How do you handle missing data? โ
- Remove rows/columns (if missingness is high) โ๏ธ
- Imputation (mean, median, mode) ๐ข
- Forward/backward fill โก๏ธโฌ ๏ธ
- Using models (KNN, regression) ๐ค
- Always analyze why data is missing before deciding.
35. What is data normalization in Python? โ๏ธ
Normalization scales numerical data to a common range (e.g., 0 to 1).
Common methods:
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
normalized_data = scaler.fit_transform(data)
Useful for ML models to prevent bias due to varying value scales.
36. Difference between .loc and .iloc in Pandas ๐๐ข
- .loc[]: Label-based indexing
df.loc[2] # Row with label 2
df.loc[:, 'age'] # All rows, 'age' column
- .iloc[]: Integer position-based indexing
df.iloc[2] # Third row
df.iloc[:, 1] # All rows, second column
37. How do you merge dataframes in Pandas? ๐ค
Using
merge() or concat()pd.merge(df1, df2, on='id', how='inner') # SQL-style joins
pd.concat([df1, df2], axis=0) # Stack rows
Choose keys and join types (inner, left, outer) based on data structure.
38. Explain groupby() in Pandas ๐
Used to group data and apply aggregation.
df.groupby('category')['sales'].sum()Steps:
1. Split data into groups ๐งฉ
2. Apply function (sum, mean, count) ๐งฎ
3. Combine result ๐
39. What are NumPy arrays? โ
N-dimensional arrays used for fast numeric computation.
Faster than Python lists and support vectorized operations.
import numpy as np
a = np.array([1, 2, 3])
40. How to handle large datasets efficiently? ๐
- Use chunking (
read_csv(..., chunksize=10000))- Use NumPy or Dask for faster ops
- Filter unnecessary columns early
- Use vectorized operations instead of loops
- Work with cloud data tools (BigQuery, Spark)
๐ฌ Tap โค๏ธ if this was helpful!