Skip to content

Commit

Permalink
Refactor - non dev mode, disk space limits and file naming (#18)
Browse files Browse the repository at this point in the history 
* Updates to address
- #15
- #14
- #13

* - Refactor of app.py allow running in none dev mode.
- Change to use get, post and delete to manage subscriptions
- addition of openapi doc

* Addition of swagger template to render openapi doc.
Update of README.md

* conditional tls-set

* Update of readme. Update ot GET, POST and DELETE methods, e.g. POST content now via request body.

Improved error handling and response codes.

* Tweaks to readme.

* Tweaks to readme.

* Update validate_topic.py

print input-topic in case of validation error

* Option to disable topic validation added.

* Unlimited disk usage possible by setting limit to 0.

* Reversion to using filename from global cache.
Switch to using minimum free space rather than maximum download dir size.

* Update to readme to include validat_topics option.

* Update to config.json and swagger page to allow customisable base url.

* Log levels configurable.

---------

Co-authored-by: Maaike <mlimper@wmo.int>
  • Loading branch information
@david-i-berry @maaikelimper
david-i-berry and maaikelimper authored Jun 12, 2024
1 parent 6d1611b commit 2dfa7cc
Show file tree
Hide file tree
Showing 15 changed files with 768 additions and 339 deletions.
226 changes: 179 additions & 47 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,111 +5,243 @@

The WIS2 Downloader is a Flask-based Python application that allows you to connect to a WIS2 Global Broker, manage subscriptions to topic hierarchies, and configure their associated download directories.

**Note**: This repository does *not* contain the desktop application GUI that can be used to aid maintenance of subscriptions and explore Global Discovery Catalogues. <a href="https://github.com/wmo-im/wis2-downloader-gui">The WIS2 Downloader GUI can be found here.</a>

## Features

- **Dynamic Subscription Management**: Quickly add or remove subscriptions ad hoc without needing to restart the service or change configuration files.
- **Monitor Download Statistics**: Access the Prometheus metrics through the `/metrics` endpoint, ideal for <a href="https://prometheus.io/docs/visualization/grafana/">Grafana visualization</a>.
- **Multi-Threading Support**: Configure the number of download workers for more efficient data downloading.

## Demo
![backend-demo](https://github.com/wmo-im/wis2-downloader/assets/47696929/f9eb9eb3-07bd-49df-9714-61d952000f2e)

*The GET requests are demonstrated here using <a href="https://www.postman.com/">Postman</a>, but the terminal or your browser will suffice too.*

## Getting Started

### 1. Installation
You can install using Pip:

__NOTE__: Thw downloader has not yet been uploaded to PyPI and needs to be installed directly from github:

```bash
pip install wis2downloader
pip install https://github.com/wmo-im/wis2downloader/archive/main.zip
```

This will install the version from the main development branch.

### 2. Configuration

Create a file `config.json` in your local directory, with the following contents:
Create a file `config.json` in your local directory that conforms with the following schema:

```yaml
schema:
type: object
properties:
base_url:
type: string
description:
Base URL for the wis2downloader service.
example: http://localhost:5050
broker_hostname:
type: string
description: The hostname of the global broker to subscribe to.
example: globalbroker.meteo.fr
broker_password:
type: string
description: The password to use when connecting to the specified global broker.
example: everyone
broker_port:
type: number
description: The port the global broker is using for the specified protocol.
example: 443
broker_protocol:
type: string
description: The protocol (websockets or tcp) to use when connecting to the global broker.
example: websockets
broker_username:
type: string
description: The username to use when connecting to the global broker.
example: everyone
download_workers:
type: number
description: The number of download worker threads to spawn.
example: 1
download_dir:
type: string
description: The path to download data to on the server/computer running the wis2downloader.
example: ./downloads
flask_host:
type: string
description: Network interface on which flask should listen when run in dev mode.
example: 0.0.0.0
flask_port:
type: number
description: The port on which flask should listen when run in dev mode.
example: 5050
log_level:
type: string
description: Log level to use
example: DEBUG
log_path:
type: string
description: Path to write log files to.
example: ./logs
min_free_space:
type: number
description:
Minimum free space (GB) to leave on download volume / disk after download.
Files exceeding limit will not be saved.
example: 10
save_logs:
type: boolean
description: Write log files to disk (true) or stdout (false)
example: false
mqtt_session_info:
type: string
description:
File to save session information (active subscriptions and MQTT client id) to.
Used to persist subscriptions on restart.
example: mqtt_session.json
validate_topics:
type: boolean
description: Whether to validate the specified topic against the published WIS2 topic hierarchy.
example: true
```
An example is given below:
```json
{
"broker_url": "replace with url of the global broker, e.g. globalbroker.meteo.fr",
"broker_port": "replace with the port to use on the global broker, e.g. 443",
"username": "username to use on the global broker, default everyone",
"password": "password to use on the global broker, default everyone",
"protocol": "transport protocol to use, either tcp or websockets",
"topics": {"initial topic 1": "associated download folder", ...},
"download_dir": "default base download directory",
"flask_host": "127.0.0.1",
"flask_port": 8080,
"base_url": "http://localhost:5050",
"broker_hostname": "globalbroker.meteo.fr",
"broker_password": "everyone",
"broker_port": 443,
"broker_protocol": "websockets",
"broker_username": "everyone",
"download_workers": 1,
"save_logs": false,
"log_dir": "default base directory for logs to be saved"
"download_dir": "downloads",
"flask_host": "0.0.0.0",
"flask_port": 5050,
"log_level": "DEBUG",
"log_path": "logs",
"min_free_space": 10,
"mqtt_session_info" : "mqtt_session.json",
"save_logs": false,
"validate_topics": true
}
```

This will be used when starting the WIS2 Downloader service.

### 3. Running

In your terminal, run:
1. Set an environment variable specifying the path to the config.json file.

*Linux (bash)*
```bash
export WIS2DOWNLOADER_CONFIG=<path_to_your_config_file>
```
wis2downloader --config <path to configuration file>

*Windows (Command Prompt)*
```
set WIS2DOWNLOADER_CONFIG=<path_to_your_config_file>
```

The Flask application should now be running. If you need to stop the application, you can do so in the terminal with `Ctrl+C`.

## Maintaining and Monitoring Subscriptions
2. Start the downloader

### Adding subscriptions
Subscriptions can be added via a GET request to the `./add` endpoint on the Flask app, with the following form:
*Dev mode (Windows and Linux)*

```bash
curl http://<flask-host>:<flask-port>/add?topic=<topic-name>&target=<download-directory>
wis2downloader
```

- `topic` specifies the topic to subscribe to. *Special characters (+, #) must be URL encoded, i.e. `+` = `%2B`, `#` = `%23`.*
- `target` specifies the directory to save the downloads to, relative to `download_dir` from `config.json`. *If this is not provided, the directory will default to that of the topic hierarchy.*
*Using gunicorn (Linux only)*
```
gunicorn --bind 0.0.0.0:5050 -w 1 wis2downloader.app:app
```

For example:
```bash
curl http://localhost:8080/add?topic=cache/a/wis2/%2B/data/core/weather/%23&target=example_data
**Note**: Only one worker is supported due to the downloader spawning additional threads and persistence of MQTT
connections.

The Flask application should now be running. If you need to stop the application, you can do so in the terminal
with `Ctrl+C`.

## Maintaining and Monitoring Subscriptions

The API defintion of the downloader can be found at the `/swagger` endpoint, when run locally see
http://localhost:5050/swagger. this includes the ability to try out the different end points.

### Adding subscriptions
Subscriptions can be added via a POST request to the `/subscriptions` endpoint.
The request body should be JSON-encoded and adhere to the following schema:

```yaml
schema:
type: object
properties:
topic:
type: string
description: The WIS2 topic to subscribe to
example: cache/a/wis2/+/data/core/weather/surface-based-observations/#
target:
type: string
description: Sub directory to save data to
example: surface-obs
required:
- topic
```
The list of active subscriptions should be returned as a JSON object.
In this example all notifications published to the `surface-based-observations` topic from any WIS2 centre will be
subscribed to, with the downloaded data written to the `surface-obs` subdirectory of the `download_dir`.

Notes:
1. If the `target` is not specified it will default to the topic the data are published on.
1. The `+` wild card is used to specify any match at a single level, matching as WIS2 centre in the above example.
1. The `#` wild card matches any topic at or below the level it occurs. In the above example any topic published below
cache/a/wis2/+/data/core/weather/surface-based-observations will be matched.

#### Example CURL command:

### Deleting subscriptions
Subscriptions are deleted similarly via a GET request to the `./delete` endpoint, with the following form:
```bash
curl http://<flask-host>:<flask-port>/delete?topic=<topic-name>
curl -X 'POST' \
'http://127.0.0.1:5050/subscriptions' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"topic": "cache/a/wis2/+/data/core/weather/surface-based-observations/#",
"target": "surface-obs"
}'
```

For example:
### Deleting subscriptions
Subscriptions are deleted via a DELETE request to the `/subscriptions/{topic}` endpoint where `{topic}` is the topic
to unsubscribe from.

#### Example CURL command

```bash
curl http://localhost:8080/delete?topic=cache/a/wis2/%2B/data/core/weather/%23
curl -X DELETE http://localhost:5050/subscriptions/cache/a/wis2/%2B/data/core/weather/%23
```

The list of active subscriptions should be returned as a JSON object.
This cancels the `cache/a/wis2/+/data/core/weather/#` subscription. Note the need to url encode the `+` (`%2B`)
and `#` (`%23`) symbols.

### Listing subscriptions
Subscriptions are listed via a GET request to `./list`:
Current subscriptions can listed via a GET request to `/subscriptions` end point.

#### Example CURL command

```bash
curl http://<flask-host>:<flask-port>/list
curl http://localhost:5050/subscriptions
```

The list of active subscriptions should be returned as a JSON object.

### Viewing download metrics
Prometheus metrics for the downloader are found via a GET request to `./metrics`, e.g.:
Prometheus metrics for the downloader are found via a GET request to the `/metrics` end point.

#### Example CURL command

```bash
curl http://<flask-host>:<flask-port>/metrics
curl http://localhost:5050/metrics
```

## Bugs and Issues

All bugs, enhancements and issues are managed on [GitHub](https://github.com/wmo-im/wis2-downloader/issues).
All bugs, enhancements and issues are managed on [GitHub](https://github.com/wmo-im/wis2downloader/issues).

## Contact

Expand Down
8 changes: 5 additions & 3 deletions pyproject.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,19 +17,21 @@ requires-python = ">=3.8"
keywords = ["WIS2.0", "MQTT", "subscribe", "download"]
license = {file = "LICENSE"}
dependencies = [
"click",
"certifi>=2024.2.2",
"flask>=3.0.3",
"flask-cors>=4.0.0",
"paho-mqtt>=2.0.0 ",
"urllib3>=2.2.1",
"prometheus_client>=0.20.0",
"pywis-topics>=0.3.2"
"urllib3>=2.2.1",
"pywis-topics>=0.3.2",
"pyyaml"
]
dynamic = ["version"]

[project.scripts]

wis2downloader = "wis2downloader:app.main"
wis2downloader = "wis2downloader:app.run"

[tool.setuptools.dynamic]
version = {attr = "wis2downloader.__version__"}
2 changes: 1 addition & 1 deletion wis2downloader/__init__.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,4 @@

__version__ = '0.1.dev2'

shutdown = Event()
stop_event = Event()
Loading

0 comments on commit 2dfa7cc

Please sign in to comment.