Skip to content

Using GTG

Jump to bottom
Kim Whitehall edited this page Jan 20, 2016 · 21 revisions

Using GTG

##1. Install all the Python Modules listed in the what is needed section of the Project Home page.

##2. Download the latest release from the Release Page Grab the release you want in either tar.gz or zip format and uncompress them on your local machine.

Note the following files:

  • config.json: contains the variables for the GTG. The config variables are broken up into two types, simple variables you'd change to adjust the temporal and spatial domains and the DIRS to input files and the advanced variables that will change how the algorithm detects MCCs.

Simple Variables (you want to change these before running the code):

LATMAX
LATMIN
LONMIN
LONMAX
DIRS.CEoriDirName
DIRS.mainDirStr
DIRS.TRMMdirName
startDateTime
endDateTime

  • baselineDataDir.zip: contains sample data.

  • In code/

  • UserInterface.ipynb: notebook to use the GTG

  • iomethods.py: contains the I/O functions needed for the GTG

  • metrics.py: contains the core (atmospheric) metrics performed by the GTG

  • mccSearch.py: contains all the functions needed

  • mccSearchUI.py: contains a command line wizard for running the program

  • mainProg.py: contains a sample of the general workflow of the order the modules should be called.

  • mainProgTemplate.py: the general workflow of the order the modules should be called in order to complete an evaluation.

  • plotting.py: contains the core plotting functions for the GTG

  • utils.py: contains misc. functions needed by the GTG

  • variables.py: contains the classes for to set up the variables needed by the GTG

Scripts for developers

  • gtg_lint_file: the config file for the GTG linting standards.
  • code/baselineTiming.py: script for timing individual components of the GTG
  • code/checkMERGBaseline.py: script for checking the differences between outputs from various GTG runs

##3. Run the GTG

New Users (Jupyter Notebook)

See our Quick Start GTG Jupyter Notebook

Advanced Users (Python Script)

  1. Edit the variables in the config.json file as necessary.
  2. Copy mainProgTemplate.py to another file e.g. cp mainProgTemplate.py firstGTGScript.py
  3. Choose the functions to be called in the script including any metrics or plots.
  4. Run the script. e.g. python firstGRGSCript.py

##4. GTG workflow in weather application mccSearchUI.py The general workflow of the program. The dashed lines indicate optional paths.

The findCloudElements (find_cloud_elements) can be completed in a parallel fashion leveraging Python's multiprocessing libraries on the vectorization branch of the code. This design was contributed by Gabriel Mel de Fontenay, who was a final year CS student at USC during his capstone project. The design is illustrated below.The parallelized implementation of find_cloud_elements that leverages the Python multiprocessing library.

##5. Successful run Once everything went well, the directory you indicated where outputs should be stored will be generated, and four folders should appear in it.

Example of GTG outputs file structure

  • image/: stores all images generated from plots during the GTG.
  • textFiles/: stores all the text files generated during the run.The core ones are:

      (1) cloudClusters.txt: a text file indicating the full path of each MCS identified i.e. all the cloud elements in a MCS

      (2) cloudElements.txt: a text file (for debugging purposes) that contains information about interesting areas that were identified, but not kept as cloud elements in the GTG.

      (3) cloudElementsUserFile.txt: a text file that contains information about each cloud element identified

      (4) graphJSON.txt: a JSON file with each cloudElement's time identified, location, temperature range, convective fraction, shape, size, and elements it is directly connected to. This file can be used to generate D3 network graphics.

      (5) unittestResults.txt: a report of the runtimes for each core area of the GTG, a report of the number of nodes found at each stage and the number of MCCs and MCSs found, and a list of the nodes found.

  • MERGnetcdfCEs/: contains the infrared data in masked netCDF files that have been generated for each cloud element identified
  • TRMMnetcdfCEs/: contains the precipitation data in clipped netCDF files that have been generated for each cloud element identified.
Clone this wiki locally