Azure Data Engineer interview questions:

Project Questions:

1) Tell me about your project. Explain it end to end.
2) On what measure you have mentioned about the storage cost decreased and cost optimization has been done?

ADF and ADLS

1) There are 10 million CSV files in ADLS Gen2. How will you read them and process using Azure Data Factory? Explain in detail what and all would be used.
2) What is Integration Runtime?
3) What is variable and parameter in ADF?
4) Explain different activities and brief about what each activity does.
5) The pipeline is scheduled and it got failed. I need to automate it by sending the mail when it gets failed. How would you implement it?
6) How do you handle the exceptions in ADF?
7) If the ADF pipeline is running very slow, how would you approach and fix it?
8) What is the difference between Blob storage and ADLS Gen2? Why ADLS Gen2 is required?

Azure Databricks

9) How do you connect ADLS Gen 2 with databricks? In where we mention the role assignments?
10) If you are using Service Principal to connect with ADLS from Azure Databricks explain the steps and how would you code it?
11) Why using service principal? How would you create it?
12) What is Databricks runtime? Why we need it?
13) What are Workflows?
14) Explain about the Medallion Architecture in brief.
15) Explain about Delta file format briefly.
16) Consider you are working in Facebook. User is writing data for each record. Since each record is been written, a new json transaction log will get created for each write. But we can use Datalake only right. Why do we require delta file format? It can decrease the performance every now and then right?
17) Consider a job is running very slow in Azure Databricks. How would you approach the issue and make it faster?
18) What are the optimization techniques you have worked on? Explain them in brief.
19) How would you optimize the job with respect to memory management in azure databricks?