🧠 1️⃣ Tell me about your data engineering experience and key projects
✅ Sample Answer:
"I have 4+ years as a data engineer building scalable ETL pipelines, data lakes, and real-time streaming systems. Expert in PySpark, Airflow, Snowflake, Kafka, and dbt. Recently built a 10TB customer 360 pipeline processing 1B+ events daily with 99.99% uptime. Reduced data latency from 6 hours to 15 minutes using streaming and optimized warehouse costs by 68% through partitioning and Z-ordering."
📊 2️⃣ What is the difference between batch processing and stream processing? When to use each?
✅ Answer:
Batch: Process large volumes at scheduled intervals (hourly/daily). Use for reports, ML training, data warehousing. Tools: Airflow, Spark batch jobs.
Stream: Process data in real-time as it arrives. Use for fraud detection, live dashboards, recommendations. Tools: Kafka Streams, Flink, Spark Streaming.
Hybrid: Lambda architecture (batch + stream layers).
🔗 3️⃣ Explain ETL vs ELT. What factors determine your choice?
✅ Answer:
ETL (Extract→Transform→Load): Transform in staging layer, load clean data to warehouse. Good for simple transformations, low-volume, strict data quality.
ELT (Extract→Load→Transform): Load raw data, transform in warehouse. Better for cloud warehouses (Snowflake, BigQuery), complex transformations, data lake use cases.
Choose ELT for modern stacks (80% current jobs), ETL for legacy/strict compliance.
🧠 4️⃣ What is a data lake vs data warehouse? When would you use each?
✅ Answer:
Data Lake: Raw, semi-structured data at scale (S3, ADLS). Schema-on-read, good for ML, data science, unknown future use cases.
Data Warehouse: Clean, structured data optimized for analytics (Snowflake, Redshift). Schema-on-write, SQL analytics, BI dashboards.
Use lake for raw storage + warehouse for consumption. Lakehouse (Databricks) combines both.
📈 5️⃣ How do you design idempotent data pipelines?
✅ Answer:
Idempotent: Run multiple times → same result.
Techniques:
- Unique keys/checksums for deduplication
- Upsert (MERGE) instead of INSERT
- Watermarking (process only new data)
- Transactional outbox pattern
- Exactly-once Kafka semantics
Example:
MERGE target t USING staging s ON t.id = s.id WHEN MATCHED THEN UPDATE WHEN NOT MATCHED THEN INSERT📊 6️⃣ What is Apache Airflow? Key components and DAG best practices
✅ Answer:
Airflow: Workflow orchestration platform. DAGs (Directed Acyclic Graphs) define pipeline dependencies.
Components: Scheduler, Webserver, Metadata DB, Workers (Celery/Kubernetes).
Best practices:
- Small, focused tasks (<15min)
- Idempotent tasks
- Retry logic + SLAs
- XComs for lightweight data passing
- Dynamic DAGs via Jinja templating
📉 7️⃣ Explain partitioning vs bucketing vs clustering in big data systems
✅ Answer:
Partitioning: Split data by column values (date, region) → directory structure. Prunes I/O for queries.
Bucketing: Hash-based file grouping within partitions. Optimizes JOINs (same bucket).
Clustering: Multi-dimensional sorting (Snowflake Z-order). Dynamic, query-optimized.
Example:
PARTITIONED BY (year, month) CLUSTERED BY (customer_id) balances prune + sort.📊 8️⃣ How do you handle schema evolution in data pipelines?
✅ Answer:
Schema evolution: Handle changing upstream data structures.
Strategies:
- Avro/Protobuf (schema in file metadata)
- dbt schema.yml + tests
- Delta Lake/Apache Iceberg (ACID + schema evolution)
- Flexible staging layer (JSON → structured)
- Versioned tables (table_v1, table_v2)
🧠 9️⃣ What is Spark? Compare DataFrames vs RDDs vs Datasets
✅ Answer:
Spark: Distributed data processing engine.
RDD: Low-level, resilient distributed datasets (Python objects).
DataFrame: Structured, optimized (Tungsten + Catalyst).
Dataset: Type-safe DataFrame (Scala/Java only\