Data Engineers on Telegram: “Two Commonly Asked Pyspark Inrerview Questions!!…”

Two Commonly Asked Pyspark Inrerview Questions!!:

Scenario 1: Handling Missing Values

Interviewer: "How would you handle missing values in a PySpark DataFrame?"

Candidate:

from pyspark.sql.functions import when, isnan

# Load the DataFrame
df = spark.read.csv("path/to/data.csv", header=True, inferSchema=True)

# Check for missing values
missing_count = df.select([count(when(isnan(c), c)).alias(c) for c in df.columns])

# Replace missing values with mean
from pyspark.sql.functions import mean
mean_values = df.agg(*[mean(c).alias(c) for c in df.columns])
df_filled = df.fillna(mean_values)

# Save the cleaned DataFrame
df_filled.write.csv("path/to/cleaned/data.csv", header=True)

Interviewer: "That's correct! Can you explain why you used the fillna() method?"

Candidate: "Yes, fillna() replaces missing values with the specified value, in this case, the mean of each column."

*Scenario 2: Data Aggregation*

Interviewer: "How would you aggregate data by category and calculate the average sales amount?"

Candidate:

# Load the DataFrame
df = spark.read.csv("path/to/data.csv", header=True, inferSchema=True)

# Aggregate data by category
from pyspark.sql.functions import avg
df_aggregated = df.groupBy("category").agg(avg("sales").alias("avg_sales"))

# Sort the results
df_aggregated_sorted = df_aggregated.orderBy("avg_sales", ascending=False)

# Save the aggregated DataFrame
df_aggregated_sorted.write.csv("path/to/aggregated/data.csv", header=True)

Interviewer: "Great answer! Can you explain why you used the groupBy() method?"

Candidate: "Yes, groupBy() groups the data by the specified column, in this case, 'category', allowing us to perform aggregation operations."