Let's Unblock: Spark Setup (Intellij)

Let's Unblock: Spark Setup (Intellij)
Added a month ago
Summary: Starting a series 'Let's Unblock' for the common blockers that programmers/developers face while developing an application specifically Spark with Scala.
Sentiment: positive
1 minute read
2 people have read this
Stocks: $DNOW
Cryptos: $XLA

Before getting our hands dirty into code let's set up our environment and prepare our IDE to understand the Scala language and SBT plugins.


  1. Java (preferably JDK 8+).
  2. Intellij IDE (community or ultimate).

IDE Setup:

So let's start configuring the plugin required for Scala and SBT environment by the following steps:

  1. Click on Configure in the bottom right of the IDE.
  2. Then click plugins.
  3. Click on Browse Repositories and search Scala.
  4. Now, enter SBT in the search section and install the SBT plugin there.

Woohoo, the IDE is now ready to understand the language you try to communicate it with.

The Project Setup:

We will learn here to set up a scala spark project using 2 build tools one by using our favorite maven and the other one using the Scala family's SBT.

Let's Start With The SBT

  1. Click on create new Project.
  2. Select Scala and then choose SBT from the options shown in the right pane:Select Scala and Then Choose SBT
  3. Add a name to your project:Add a Name to Your Project
  4. Click on the finish button. 
  5. Congrats, your project structure is created and you will be shown a new window with its structure.
  6. Let's edit the build.sbt to make our scala project a spark one.
  7. Add this spark dependency in the project to make it one: libraryDependencies ++= Seq( "org.apache.spark" % "spark-core_2.11" % "2.1.0" )
  8. Adding Spark DependencyClick on the refresh icon appearing on the top right in the editor to download the relevant dependency.
  9. Create a Scala class (e.g., Master) by right-clicking on the Scala folder in the project structure window.Adding Scala Class
  10. To see your code running paste this code in your master class:

Click on the play button and you will be able to see your output in the Run window below.

Output in Run Window


Maven (Java Friends favorite) way of creating a project:

  1. Click on create new Project from the file menu.
  2. Click on Maven, proceed with create from an archetype, and hit the checkbox
  3. Select archetype scala-archetype-simple:1.2Selecting Archetype
  4. Woohoo, no need to make the main class here, it will already be freshly prepared for you.
  5. Let's make our Scala project a Spark one by adding the following dependencies to pom.xml:

Follow the remaining steps from above by copying the same code snippet.

spark-demo pom.xml Screenshot

App.scala Screenshot

Github Links:

SBT project: https://github.com/dheerajgupta217/spark-demo-sbt.

Maven Project: https://github.com/dheerajgupta217/spark-demo-maven.

Opinions expressed by DZone contributors are their own.