Getting started
What you will need
If you want to use Spark-ZIO, you will need these:
val runtime: DefaultRuntime = new DefaultRuntime {}
val ss: SparkSession = SparkSession.builder.master("local[*]").getOrCreate()
val sparkEnv: SparkZIO = new SparkZIO(ss)
You can obviously change the configuration for your runtime and your SparkSession.
Now that you wield the power of Spark and ZIO together, let’s make a simple local Spark job:
import io.univalence.sparkzio.SparkEnv.TaskS
val program: TaskS[Unit] = for {
df <- sparkEnv.read.parquet(pathToParquet)
_ <- Task(df.show())
} yield ()
runtime.unsafeRun(program)