Download exercise update, add exercise for decoding BUFR using TC tra…

…cks example (#117)

* init new exercise version

* add initial notebook

* update

* commit WIP and switch branch

* download-exercise-update add TC decode example

* update topics, probabilistic instead of deterministic

* update downloader settings

* need to restart stack to apply changes ...

* explicitly mention training instructor

* check logs rather than directory ...

documentation/docs/practical-sessions/downloading-data-from-wis2.en.md

@@ -8,7 +8,9 @@ title: Downloading data from WIS2 notifications
 
     By the end of this practical session, you will be able to:
 
-    - use the "wis2downloader" to subscribe to WIS2 data notifications and download data to your local system.
+    - use the "wis2downloader" to subscribe to WIS2 data notifications and download data to your local system
+    - view the status of the downloads in the Grafana dashboard
+    - decode some downloaded data using the "decode-bufr-jupyter" container
 
 ## Introduction
 
@@ -20,6 +22,13 @@ In this session you will learn how to setup a subscription to a WIS2 Broker and
 
      If you like to develop your own service for subscribing to WIS2 notifications and downloading data, you can use the [wis2downloader source code](https://github.com/wmo-im/wis2downloader) as a reference.
 
+!!! Other tools for accessing WIS2 data
+
+    The following tools can also be used to discover and access data from WIS2:
+
+    - [pywiscat](https://github.com/wmo-im/pywiscat) provides search capability atop the WIS2 Global Discovery Catalogue in support of reporting and analysis of the WIS2 Catalogue and its associated discovery metadata
+    - [pywis-pubsub](https://github.com/wmo-im/pywis-pubsub) provides subscription and download capability of WMO data from WIS2 infrastructure services
+
 ## Preparation
 
 Before starting please login to your student VM and ensure your wis2box instance is up and running.
@@ -135,12 +144,159 @@ wis2downloader remove-subscription --topic cache/a/wis2/de-dwd-gts-to-wis2/#
 
 Check the wis2downloader dashboard in Grafana to see the subscription removed. You should see the downloads stopping.
 
-!!! More command line tools
+## Exercise 6: subscribe to the wis2training-broker and setup a new subscription
 
-    The following tools can also be used to discover and access data from WIS2:
+For the next exercise we will subscribe to the wis2training-broker.
+
+This demonstrates how to subscribe to a broker that is not the default broker and will allow you to download some data published from the WIS2 Training Broker.
+
+Edit the wis2box.env file and change the DOWNLOAD_BROKER_HOST to `wis2training-broker.wis2dev.io`, change DOWNLOAD_BROKER_PORT to `1883` and change DOWNLOAD_BROKER_TRANSPORT to `tcp`:
+
+```copy
+# downloader settings
+DOWNLOAD_BROKER_HOST=wis2training-broker.wis2dev.io
+DOWNLOAD_BROKER_PORT=1883
+DOWNLOAD_BROKER_USERNAME=everyone
+DOWNLOAD_BROKER_PASSWORD=everyone
+# download transport mechanism (tcp or websockets)
+DOWNLOAD_BROKER_TRANSPORT=tcp
+```
+
+Then restart the wis2box-stack to apply the changes:
+
+```bash
+python3 wis2box-ctl.py start
+```
+
+Check the logs of the wis2downloader to see if the connection to the new broker was successful:
+
+```bash
+docker logs wis2downloader
+```
+
+You should see the following log message:
+
+```copy
+...
+INFO - Connecting...
+INFO - Host: wis2training-broker.wis2dev.io, port: 1883
+INFO - Connected successfully
+```
+
+Now we will setup a new subscription to the topic to downloaded cyclone-track data from the WIS2 Training Broker.
+
+Login to the **wis2downloader** container:
+
+```bash
+python3 wis2box-ctl.py login wis2downloader
+```
+
+Then add the subscription to the topic `origin/a/wis2/int-wis2-training/data/core/weather/prediction/forecast/medium-range/probabilistic/trajectory`:
+
+```bash
+wis2downloader add-subscription --topic origin/a/wis2/int-wis2-training/data/core/weather/prediction/forecast/medium-range/probabilistic/trajectory
+```
+
+Exit the **wis2downloader** container by typing `exit`.
+
+Wait until you see the downloads starting in the wis2downloader dashboard in Grafana. The training-instructor will push some data to the WIS2 Training Broker for you to download.
+
+!!! note "Downloading data from the WIS2 Training Broker"
+
+    The WIS2 Training Broker is a test broker that is used for training purposes and may not publish data all the time.
+
+    During the in-person training sessions, the local training will publish data to the WIS2 Training Broker for you to download.
+
+    If you are doing this exercise outside of a training session, you may not see any data being downloaded.
+
+Check that the data was downloaded by checking the wis2downloader logs again with:
+
+```bash
+docker logs wis2downloader
+```
+
+And you should see a log message similar to the following:
+
+```copy
+[...] INFO - Message received under topic origin/a/wis2/int-wis2-training/data/core/weather/prediction/forecast/medium-range/probabilistic/trajectory
+[...] INFO - Downloaded A_JSXX05ECEP020000_C_ECMP_...
+```
+
+## Exercise 7: decoding the downloaded data
+
+In order to demonstrate how you can decode the downloaded data, we will start a new container using 'decode-bufr-jupyter' image.
+
+This container will be start Jupiter notebook server on your instance that include the eccodes library that you can use to decode the downloaded data.
+
+We will the example notebooks included in `~/exercise-materials/notebook-examples` to decode the downloaded data for the cyclone tracks.
+
+To start the container, use the following command:
+
+```bash
+docker run -d --name decode-bufr-jupyter \
+    -v ~/wis2box-data/downloads:/root/downloads \
+    -p 8888:8888 \
+    -e JUPYTER_TOKEN=dataismagic! \
+    mlimper/decode-bufr-jupyter
+```
+
+!!! note "About the decode-bufr-jupyter container"
+
+    The `decode-bufr-jupyter` container is a custom container that includes the eccodes library and Jupyter notebook server. The container is based on a image that includes the `eccodes` library for decoding BUFR data, along with libraries for plotting and data analysis.
+
+    The command above starts the container in detached mode, with the name `decode-bufr-jupyter`, the port 8888 is mapped to the host system and the environment variable `JUPYTER_TOKEN` is set to `dataismagic!`.
+
+    The command above also mounts the `~/wis2box-data/downloads` directory to `/root/downloads` in the container. This ensures that the downloaded data is available to the Jupyter notebook server.
+
+Once the container is started, you can access the Jupyter notebook server by navigating to `http://<your-host>:8888` in your web browser.
+
+You will see a screen requesting you to enter a "Password or token".
+
+Provide the token `dataismagic!` to login to the Jupyter notebook server.
+
+After you login, you should see the following screen listing the directories in the container:
+
+![Jupyter notebook home](../assets/img/jupyter-files-screen1.png)
+
+Double click on the `example-notebooks` directory to open it.
+
+You should see the following screen listing the example notebooks, double click on the `tropical_cyclone_track.ipynb` notebook to open it:
+
+![Jupyter notebook example notebooks](../assets/img/jupyter-files-screen2.png)
+
+You should now be in the Jupyter notebook for decoding the tropical cyclone track data:
+
+![Jupyter notebook tropical cyclone track](../assets/img/jupyter-tropical-cyclone-track.png)
+
+Read the instructions in the notebook and run the cells to decode the downloaded data for the tropical cyclone tracks. Run each cell by clicking on the cell and then clicking the run button in the toolbar or by pressing `Shift+Enter`.
+
+At the end you should see a plot of the strike probability for the tropical cyclone tracks:
+
+![Tropical cyclone tracks](../assets/img/tropical-cyclone-track-map.png)
+
+!!! question 
+
+    The result displays the predicted probability of tropical storm track within 200 km. How would you update the notebook to display the predicted probability of tropical storm track within 300 km ?
+
+??? success "Click to reveal answer"
+
+    To update the notebook to display the predicted probability of tropical storm track within a different distance, you can update the `distance_threshold` variable in the code-block that calculates the strike probability.
+
+    To display the predicted probability of tropical storm track within 300 km, 
+
+    ```python
+    # set distance threshold (meters)
+    distance_threshold = 300000  # 300 km in meters
+    ```
+
+    Then re-run the cells in the notebook to see the updated plot.
+
+!!! note "Decoding BUFR data"
+
+    The exercise you just did provided one specific example of how you can decode BUFR data using the eccodes library. Different data types may require different decoding steps and you may need to refer to the documentation for the data type you are working with.
+
+    For more information please consult the [ECCODES documentation](https://confluence.ecmwf.int/display/ECC).
 
-    - [pywiscat](https://github.com/wmo-im/pywiscat) provides search capability atop the WIS2 Global Discovery Catalogue in support of reporting and analysis of the WIS2 Catalogue and its associated discovery metadata
-    - [pywis-pubsub](https://github.com/wmo-im/pywis-pubsub) provides subscription and download capability of WMO data from WIS2 infrastructure services
 
 
 ## Conclusion
@@ -150,3 +306,5 @@ Check the wis2downloader dashboard in Grafana to see the subscription removed. Y
     In this practical session, you learned how to:
 
     - use the 'wis2downloader' to subscribe to a WIS2 Broker and download data to your local system
+    - view the status of the downloads in the Grafana dashboard
+    - decode some downloaded data using the 'decode-bufr-jupyter' container