Skip to content

Commit

Permalink
Updated batchmap docs (#3649)
Browse files Browse the repository at this point in the history
I've just set this up on a new machine and confirmed it works. I've included the steps I took here because this is likely to work for other people, and it's all a bit mysterious without a script to follow.

Future work will involve updating this demo to actually explain what it is doing, because it's only just accessible to me, and I wrote it.
  • Loading branch information
mhutchinson authored Oct 16, 2024
1 parent cfc7b13 commit 279a669
Showing 1 changed file with 28 additions and 6 deletions.
34 changes: 28 additions & 6 deletions experimental/batchmap/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,18 +51,40 @@ are identified.

## Running the demo

### Building the map
### Setup

The map uses Apache Beam to assemble and run a pipeline of tasks to construct the map.
The following instructions show how to run this code locally using the Python portable runner.
Comprehensive documentation for setting up a Python environment is beyond the scope of this demo, but instructions can be found at [Beam Python Tips](https://cwiki.apache.org/confluence/display/BEAM/Python+Tips).
Steps that worked for me using python 3.10.12 on a Debian machine are:

```shell
python3 --version # Confirm 3.10.12
sudo apt install python3-venv

# Create a virtual python env for configuring the runner and deps
python3 -m venv ~/Virtualenvs/beampy31012
source ~/Virtualenvs/beampy31012/bin/activate
python3 -m pip install --upgrade pip
pip install wheel
pip install apache-beam

# Now run the portable runner, and leave it listening for work on 8099
python -m apache_beam.runners.portability.local_job_service_main --port=8099
```

The following instructions show how to run this code locally using the Python
portable runner. Setting up a Python environment is beyond the scope of this
demo, but instructions can be found at [Beam Python Tips](https://cwiki.apache.org/confluence/display/BEAM/Python+Tips).
These instructions are for Linux/MacOS but can likely be adapted for Windows.
Alternate steps that may work from checking out the source repo:

Ensure that the Python runner is running:
1. Check out `github.com/apache/beam` (tested at branch `release-2.24.0`)
2. `cd sdks/python` within that repository
3. `python -m apache_beam.runners.portability.local_job_service_main --port=8099`


### Building the map

These instructions are for Linux/MacOS but can likely be adapted for Windows.
Ensure that you have a python portable runner listening on port 8099, as documented above.

In another terminal:
1. Check out this repository and `cd` to the folder containing this README
2. `go run ./cmd/build/mapdemo.go --output=/tmp/mapv1 --runner=universal --endpoint=localhost:8099 --environment_type=LOOPBACK`
Expand Down

0 comments on commit 279a669

Please sign in to comment.