⚡ Parallelism In Databricks ⚡

1️⃣ DEFINITION

Parallelism = running many tasks 🏃‍♂️🏃‍♀️ at the same time
(instead of one by one 🐢).
In Databricks (via Apache Spark), data is split into
📦 partitions, and each partition is processed
simultaneously across worker nodes 💻💻💻.

2️⃣ KEY CONCEPTS

🔹 Partition = one chunk of data 📦
🔹 Task = work done on a partition 🛠️
🔹 Stage = group of tasks that run in parallel ⚙️
🔹 Job = complete action (made of stages + tasks) 📊

3️⃣ HOW IT WORKS

✅ Step 1: Dataset ➡️ divided into partitions 📦📦📦
✅ Step 2: Each partition ➡️ assigned to a worker 💻
✅ Step 3: Workers run tasks in parallel ⏩
✅ Step 4: Results ➡️ combined into final output 🎯

4️⃣ EXAMPLES

# Increase parallelism by repartitioning
df = spark.read.csv("/data/huge_file.csv")
df = df.repartition(200) # ⚡ 200 parallel tasks

# Spark DataFrame ops run in parallel by default 🚀
result = df.groupBy("category").count()

# Parallelize small Python objects 📂
rdd = spark.sparkContext.parallelize(range(1000), numSlices=50)
rdd.map(lambda x: x * 2).collect()

# Parallel workflows in Jobs UI ⚡
# Independent tasks = run at the same time.

5️⃣ BEST PRACTICES

⚖️ Balance partitions → not too few, not too many
📉 Avoid data skew → partitions should be even
🗃️ Cache data if reused often
💪 Scale cluster → more workers = more parallelism

====================================================
📌 SUMMARY
Parallelism in Databricks = split data 📦 →
assign tasks 🛠️ → run them at the same time ⏩ →
faster results 🚀