This is not an officially supported Google product, though support will be provided on a best-effort basis.
Copyright 2020 Google LLC
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
cogbeam is a simple python based Apache Beam pipeline, optimized for Google Cloud Dataflow, which aims to expedite the conversion of traditional GeoTIFFs into Cloud Optimized GeoTIFFS by leveraging on-demand parallel processing.
The impetus for cogbeam came from Google Earth Engine's (GEE) support of COGs as Cloud GeoTIFF Backed Earth Engine Assets as a means for users of GEE to analyze raster data in situ, directly from Google Cloud Storage (GCS).
There are decades worth of raster geospatial data that could easily be made accessible and made ready for analysis if it were converted to the open COG standard, and it would be a drag to have to do all of that using gdal_translate in serial.
Therefore, cogbeam is designed to simply point to bucket of regular geotiffs in a GCS bucket, specify a new bucket where you'd like your COGs to appear, spin up an appropriate number of workers to batch convert your geotiffs into COGs and then immediately shut down those workers. At that point you can delete the source images for cost savings.
Clone this repository and configure Cloud Shell or a Compute Engine Virtual Machine to launch Dataflow Pipelines
Note: Cloud Shell is limited to sessions that are less than 20 minutes. If you're running a large conversion pipeline that is likely to take more than 20 minutes, use tmux and a Virtual Machine on Compute Engine.
Choose a Google Cloud Platform project where you will launch this Dataflow pipeline.
You don't need much of a machine to launch the pipeline as none of the processing of the pipeline is done locally.
- Start with an Ubuntu 20.04 Image
- start a tmux session (this will ensure that your pipeline finishes correctly even if you lose connection to your VM).
tmux - Update aptitude.
sudo apt-get update - Install pip3
sudo apt-get install python3-pip - Clone this repository
git clone XXXXXX - Change directories into the new "cogbeam" folder:
cd cogbeam - Install local python3 dependencies
pip install -r requirements.text - You'll need to authenticate so that the python code that runs locally can use your credentials.
gcloud auth application-default login
- Launch a Cloud Shell from that project.
- Clone this repository:
git clone XXXXXXXX - Change directories into the new "cogbeam" folder:
cd cogbeam - You will need to install the python dependencies locally:
pip3 install -r requirements.text
By default, GCP projects will automatically create a VPC network called 'default' upon creation that have a subnet in each region. However, some organizations have policies that prohibit the creation of a default network. You can check to see if you already have a 'default' or other VPC network in your project here. If you already have a network in the project, you can skip this step. Otherwise, you'll want to create a VPC network.
Click "CREATE VPC NETWORK"
Choose a name for the network. ('default' is fine, you'll just need to remember what it is later when you launch the pipeline.)
Choose 'Automatic' for the 'Subnet creation mode'.
Select some of the default Firewall rules for the network, but at least choose "default-allow-internal" so the workers can communicate with each other. Note: this example only really worries about running Dataflow; you may have other requirements.
Choose 'Regional' for the Dynamic routing mode. Click 'Create' to create the network.
You need to enable the Dataflow API for your project here.
Click "Enable"
If you're running cogbeam in a project your account owns, there shouldn't be anything special that you need to do with regard to permissions on either the source or destination GCS Buckets.
However, you may wish to read from GCS buckets in another GCP project, and write to buckets in another project. If so, you'll need to consider the following permission requirements, and make sure the following accounts have the following minimum permissions on the buckets.
- Source Bucket
- Storage Object Viewer
- Your project's Compute Engine default service account
- Because the Dataflow workers need to be able to read the source bucket and find the source images.
- Your project's Compute Engine default service account
- Storage Object Viewer
- Destination Bucket
- Storage Object Admin
- Your Google account
- Because you have to upload the staging files to launch the Dataflow pipeline
- Your project's Compute Engine default service account
- Because the Dataflow workers need to be able to write and delete objects to/from the destination bucket.
- Your Google account
- Storage Object Admin
Also, before running cogbeam, if you want the resulting COGs to be publicly available you should add the account 'allUsers' as a Storage Object Viewer on the bucket as discussed here. That way, every COG generated will automatically be publicly viewable.
By default, workers use your project's Compute Engine default service account as the controller service account. This service account (-compute@developer.gserviceaccount.com) is automatically created when you enable the Dataflow API for a project. You can verify this service account exists and find it's full address under the IAM & Admin section of your project.
python3 cogbeam.py -h
| Flag | Required | Default | Description |
|---|---|---|---|
| --project | Yes | Specify GCP project where pipeline will run and usage will be billed | |
| --cog_bucket | Yes | GCS Output Bucket ONLY (no folders) e.g. "bucket_name" for Converted COGs | |
| --cog_folder | Yes | GCS Output Bucket folder(s) e.g. "folder_1/folder_2" for Converted COGs - can be blank for root of bucket | |
| --source_bucket | Yes | GCS Source Bucket ONLY (no folders) e.g. "bucket_name" | |
| --source_folder | Yes | GCS Bucket folder(s) e.g. "folder_1/folder_2" | |
| --file_pattern | Yes | File pattern to search e.g. "*.tif" | |
| --network | Yes | GCE Network to use | |
| --profile | No | lzw | COG Profile to use (jpeg, webp, lzw, deflate, etc.) JPEG is default but only supports 3 band rasters. WebP not well supported yet. |
| --max_num_workers | No | 20 | The maximum number of workers to scale to in the Dataflow pipeline. |
| --machine_type | No | n1-standard-2 | The Compute Engine machine type to use for each worker |
| --region | No | us-central1 | The GCP region in which the pipeline will run |
This example shows using cogbeam to convert the 812 regular GeoTIFFs from the 2016 NAIP for the US state of Vermont,located in the publicly accessible gs://vermont-naip-2016 bucket, into 812 COGs located in the publicly accessible gs://vermont-naip-2016-cog bucket.
python3 cogbeam.py --source_bucket "vermont-naip-2016" --source_folder "" --file_pattern "*.tif" --cog_bucket "vermont-naip-2016-cog" --cog_folder "" --network default --project cogbeam --profile lzw
The pipeline should start off by uploading the staging files to the '\tmp' directory of your 'cog_bucket' folder and then you will get a link to track the pipeline in the GCP console. The 'Job status' will be 'Running.'
You'll also see a constant stream of communication from Dataflow in your terminal window, as the pipeline scales workers up and down to handle the conversion of the COGs.
When the pipeline completes, all the workers will be turned down and you should see the 'Job status' as 'Succeeded.'
At the end of the script, a sample Google Earth Engine script is generated and uploaded to the cog_bucket. In this example, it is uploaded to: gs://vermont-naip-2016-cog/gee-script.js
You can simply cut and paste the contents of that script to examine the COGs in Google Earth Engine.
But don't forget, these are 4 band COGs, and Google Earth Engine isn't just for visualization. You can treat a COG in GEE just like a native asset so it's easy to create an NDVI analysis against the COGs without moving them from Google Cloud Storage.
*These figures are estimates using GCP pricing as of 10/2020
Generating COGs slightly increases the size of each file because it adds tiled overviews to the GeoTIFFs. Below we'll review some of the costs for the 2016 Vermont NAIP example.
| Image Type | Storage Size | Monthly Storage Cost (us multiregion) |
|---|---|---|
| Standard GeoTIFF | 295.32 GiB | $7.68 |
| COG | 440.21 GiB | $11.45 |
Running the Dataflow pipeline, with 20 n1-standard-2 machines resulted in the following billable metrics for the 27 minute 11 second run.
| Metric | Count |
|---|---|
| Total vCPU time | 12.594 vCPU hr |
| Total memory time | 47.227 GB hr |
| Total HDD PD time | 1,574.233 GB hr |
This equates to roughly $1.44 in processing costs for the conversion.
A final consideration is egress. When you're sharing datasets on cloud storage, presumably it is so analysts can access it. It can be extremely difficult though to figure out how many analysts will download the data, and how much of it, over the course of a year. Once a dataset leaves any cloud provider's network you are charged an egress fee. These costs can mount quickly and unpredictably. Here's a table showing the estimated egress charges for the example dataset in terms of number of times the entire dataset is downloaded.
| Number of Downloads | Price |
|---|---|
| 1 | $31.01 |
| 2 | $62.02 |
| 5 | $155.05 |
| 10 | $310.10 |
We can look at similar estimates for the slightly larger COGs, which are backwards compatible and somewhat likely to still be downloaded for some applications. But, an important difference with COGs is that they're already "analysis ready" as we've shown with Google Earth Engine. That means that end users of the data don't actually have to download the data off of the cloud - they can analyze it in place and on GCP, that doesn't represent a billable egress event.
| Number of Downloads | Price |
|---|---|
| 1 | $46.22 |
| 2 | $92.44 |
| COGs used in GEE | $0 |