Spark-Examples

Interactive Spark Cluster

Script start_spark_cluster.sh.

Spawns a Spark Master and Spark Workers and waits until Job Timeout. Weird parametrization (e.g. 7 workers threads) to see that it actually works. Unclear:

• In the Scala Example, is uses 7 worker instances as expected. The Python Example uses only 2. Why?
• How to run workers on more than one node? Probably with srun
• Executing srun -n 1 start-worker.sh $MASTER_URL leads to a quiet failure (?) of the workers. Why?

References

• Pi Estimate (Python + Scala):
• Simple Slurm Example (not completely compatible):