💥Apache Spark on Google Colab? Installing PySpark on DS Cloud
Apache Spark is one of the most demanded computational frameworks in the Big Data field. With its clustered architecture and in-memory MapReduce jobs, it quickly processes huge amounts of data. Spark has an API for the popular DS Python language, PySpark, and automatically parallelizes code generated on the local machine to all nodes in the cluster. But what if you don't have a cluster and need to process a lot of data?
Cloud solutions will help - Google Colab, where you can install Spark. First you need to download Java, because the framework is written in Scala and runs on JVM:
!apt-get install openjdk-8-jdk-headless -qq > /dev/null
Then you should download the Spark framework itself from the official website of the Apache Software Foundation, along with Hadoop
!wget -q https://www-us.apache.org/dist/spark/spark-3.1.2/spark-3.1.2-bin-hadoop2.7.tgz
Unzip the downloaded tgz-file
!tar xf spark-3.1.2-bin-hadoop2.7.tgz
The next step is to install the findspark library to find Spark on the system and import it.
!pip install -q findspark
Then you need to set the path to the Colab environment to run PySpark there:
import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = "/content/spark-3.1.2-bin-hadoop2.7"
To find Spark on the system, import findspark and use the findspark.init() method. The findspark.find() method will help you find out where Spark is installed:
import findspark
findspark.init()
findspark.find()
Finally, you can import the Spark - SparkSession from PySpark.sql and create an entry point to the framework, optionally specifying the application name in the appName (“”) configuration parameter
from pyspark.sql import SparkSessionspark= SparkSession \
.builder \
.appName("Spark Application Name") \
.getOrCreate()
https://medium.com/geekculture/how-to-get-your-spark-installation-right-every-time-on-colab-218d57b6091d