Skip to content
Snippets Groups Projects
Commit 481941c0 authored by Stefan Kesselheim's avatar Stefan Kesselheim
Browse files

added history server

parent 61b6f9b9
Branches
Tags
No related merge requests found
...@@ -25,6 +25,11 @@ python pyspark_pi.py ...@@ -25,6 +25,11 @@ python pyspark_pi.py
``` ```
Note the `i` that that has been added to the master hostname. Note the `i` that that has been added to the master hostname.
To connect to the master and workers with a browser, you need a command of the following form:
```bash
ssh -L 18080:localhost:18080 -L 8080:localhost:8080 kesselheim1@jwb0085i.juwels -J kesselheim1@juwels-booster.fz-juelich.de
```
Then you can navigate to (http://localhost:8080) to the the output.
Open Questions Open Questions
- In the Scala Example, is uses all worker instances as expected. The Python Example uses only 2. Why? - In the Scala Example, is uses all worker instances as expected. The Python Example uses only 2. Why?
...@@ -32,6 +37,11 @@ Open Questions ...@@ -32,6 +37,11 @@ Open Questions
ToDos: ToDos:
- Include a Python Virtual Environment - Include a Python Virtual Environment
- Create a Notebook that illustrates how to run the Pi example in Juypter - Create a Notebook that illustrates how to run the Pi example in Juypter
- The history server does not work yet. It crashed with this error message:
```
Exception in thread "main" java.io.FileNotFoundException: Log directory specified does not exist: file:/tmp/spark-events Did you configure the correct one through spark.history.fs.logDirectory?
```
The logdir config is not configured in the right way.
## References ## References
- Pi Estimate (Python + Scala): [](https://spark.apache.org/examples.html) - Pi Estimate (Python + Scala): [](https://spark.apache.org/examples.html)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment