updated v0.5.4

- Major version improvements to performance and codebase - Added rename tool to allow file renaming to EE rules - Added natural sorting to sort filenames to be ingested - Added capability for image and table upload to check for both existing assets and assets in task queue before retrying - Added task check capability to avoid 3000 tasks in queue - Updated and optimized failure checks and logging - Added path and asset schema check for EE rulesets - Updated docs and readme
samapriya · May 23, 2022 · b453ac2 · b453ac2
1 parent aafcdad
commit b453ac2
Show file tree

Hide file tree

Showing 11 changed files with 222 additions and 117 deletions.
diff --git a/README.md b/README.md
@@ -1,8 +1,7 @@
 # geeup: Simple CLI for Earth Engine Uploads
 
 [![Twitter URL](https://img.shields.io/twitter/follow/samapriyaroy?style=social)](https://twitter.com/intent/follow?screen_name=samapriyaroy)
-![](https://tokei.rs/b1/github/samapriya/geeup?category=code)
-![](https://tokei.rs/b1/github/samapriya/geeup?category=files)
+[![Hits-of-Code](https://hitsofcode.com/github/samapriya/geeup?branch=master)](https://hitsofcode.com/github/samapriya/geeup?branch=master)
 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.5814026.svg)](https://doi.org/10.5281/zenodo.5814026)
 [![PyPI version](https://badge.fury.io/py/geeup.svg)](https://badge.fury.io/py/geeup)
 ![PyPI - Downloads](https://img.shields.io/pypi/dm/geeup)
@@ -27,6 +26,7 @@ Zenodo. https://doi.org/10.5281/zenodo.5814026
 * [Getting started](#getting-started)
 * [geeup Simple CLI for Earth Engine Uploads](#geeup-simple-cli-for-earth-engine-uploads)
     * [geeup Quota](#geeup-quota)
+    * [geeup Rename](#geeup-rename)
     * [geeup Zipshape](#geeup-zipshape)
     * [geeup getmeta](#geeup-getmeta)
     * [Cookie Setup](#cookie-setup)
@@ -136,6 +136,26 @@ Optional named arguments:
 
 ```
 
+### geeup Rename
+
+This tool is simply designed to rename filenames to confirm to GEE rules about path renaming including allowing for only hypens or underscores and letters and numbers with no spaces. The tool does do in replace replacement which means it will not create a copy but rename to the same location they are in so use with caution
+
+![geeup_rename](https://user-images.githubusercontent.com/6677629/169738141-b032f14b-7b26-441a-96bd-a4eadeaeba7a.gif)
+
+```
+geeup rename -h
+
+usage: geeup rename [-h] --input INPUT
+
+optional arguments:
+  -h, --help     show this help message and exit
+
+Required named arguments.:
+  --input INPUT  Path to the input directory with all files to be
+                 uploaded
+
+```
+
 ### geeup Zipshape
 So here's how table upload in Google Earth Engine works, you can either upload the component files shp, shx, prj and dbf or you can zip these files together and upload it as a single file. The pros for this is that it reduces the overall size of the shapefile after zipping them along, this tool looks for the shp file and finds the subsidiary files and zips them ready for upload. It also helps when you have limited upload bandwidth. Cons you have to create a replicate structure of the file system, but it saves on bandwidth and auto-arranges your files so you don't have to look for each additional file.
 
@@ -264,13 +284,18 @@ geeup tabup --source "full path to folder with Zipped Shapefiles/CSV files" --de
 ```
 
 ### geeup tasks
-This script counts all currently running, ready, completed, failed and canceled tasks along with failed tasks. This tool is linked to your google earth engine account with which you initialized the earth engine client. This takes no argument.
+This tasks tool gives a direct read out of different Earth Engine tasks across different states currently running, cancelled, pending and failed tasks and requires no arguments. However you could pass the state and get stats like SUCCEEDED along with item description or path, number of attempts and time taken along with task ID as a JSON list. This could also simply be piped into a JSON file using ">"
+
+![geeup_tasks_enhanced](https://user-images.githubusercontent.com/6677629/169737348-abf13334-e360-487e-b4ce-d25aa677404c.gif)
 
 ```
-usage: geeup tasks [-h]
+usage: geeup tasks [-h] [--state STATE]
 
 optional arguments:
-  -h, --help  show this help message and exit
+  -h, --help     show this help message and exit
+
+Optional named arguments:
+  --state STATE  Query by state type SUCCEEDED|PENDING|RUNNING|FAILED
 ```
 
 ### geeup delete
@@ -288,6 +313,16 @@ optional arguments:
 ```
 # Changelog
 
+### 0.5.4
+- Major version improvements to performance and codebase
+- Added rename tool to allow file renaming to EE rules
+- Added natural sorting to sort filenames to be ingested
+- Added capability for image and table upload to check for both existing assets and assets in task queue before retrying
+- Added task check capability to avoid 3000 tasks in queue
+- Updated and optimized failure checks and logging
+- Added path and asset schema check for EE rulesets
+- Updated docs and readme
+
 ### 0.5.3
 - Major version removed selenium support as stable method
 - Overall improvements to performance and codebase

diff --git a/docs/changelog.md b/docs/changelog.md
@@ -1,5 +1,15 @@
 # Changelog
 
+### 0.5.4
+- Major version improvements to performance and codebase
+- Added rename tool to allow file renaming to EE rules
+- Added natural sorting to sort filenames to be ingested
+- Added capability for image and table upload to check for both existing assets and assets in task queue before retrying
+- Added task check capability to avoid 3000 tasks in queue
+- Updated and optimized failure checks and logging
+- Added path and asset schema check for EE rulesets
+- Updated docs and readme
+
 ### 0.5.3
 - Major version removed selenium support as stable method
 - Overall improvements to performance and codebase

diff --git a/docs/projects/rename.md b/docs/projects/rename.md
@@ -0,0 +1,19 @@
+# geeup Rename
+
+This tool is simply designed to rename filenames to confirm to GEE rules about path renaming including allowing for only hypens or underscores and letters and numbers with no spaces. The tool does do in replace replacement which means it will not create a copy but rename to the same location they are in so use with caution
+
+![geeup_rename](https://user-images.githubusercontent.com/6677629/169738141-b032f14b-7b26-441a-96bd-a4eadeaeba7a.gif)
+
+```
+geeup rename -h
+
+usage: geeup rename [-h] --input INPUT
+
+optional arguments:
+  -h, --help     show this help message and exit
+
+Required named arguments.:
+  --input INPUT  Path to the input directory with all files to be
+                 uploaded
+
+```
diff --git a/docs/projects/task_status.md b/docs/projects/task_status.md
@@ -1,13 +1,15 @@
 # Task status
 
-This script counts all currently running, cancelled, pending and failed tasks and requires no arguments.
+This tasks tool gives a direct read out of different Earth Engine tasks across different states currently running, cancelled, pending and failed tasks and requires no arguments. However you could pass the state and get stats like SUCCEEDED along with item description or path, number of attempts and time taken along with task ID as a JSON list. This could also simply be piped into a JSON file using ">"
 
-![geeup_tasks](https://user-images.githubusercontent.com/6677629/114294401-ff43b800-9a63-11eb-8b2b-50811fa371ef.gif)
+![geeup_tasks_enhanced](https://user-images.githubusercontent.com/6677629/169737348-abf13334-e360-487e-b4ce-d25aa677404c.gif)
 
 ```
-> geeup tasks -h
-usage: geeup tasks [-h]
+usage: geeup tasks [-h] [--state STATE]
 
 optional arguments:
-  -h, --help  show this help message and exit
+  -h, --help     show this help message and exit
+
+Optional named arguments:
+  --state STATE  Query by state type SUCCEEDED|PENDING|RUNNING|FAILED
 ```
diff --git a/geeup/__init__.py b/geeup/__init__.py
@@ -2,4 +2,4 @@
 
 __author__ = "Samapriya Roy"
 __email__ = "samapriya.roy@gmail.com"
-__version__ = "0.5.3"
+__version__ = "0.5.4"
diff --git a/geeup/batch_uploader.py b/geeup/batch_uploader.py
@@ -81,7 +81,7 @@ def __init__(self, schema):
 
     def _format_message(self, field, error):
         print("")
-        return "GEE file name & path cannot have spaces & can only have letters, numbers, hiphens and underscores"
+        return "GEE file name & path cannot have spaces & can only have letters, numbers, hyphens and underscores"
 
 
 def upload(
@@ -93,7 +93,8 @@ def upload(
     nodata_value=None,
     bucket_name=None,
 ):
-    schema = {"collection_path": {"type": "string", "regex": "^[a-zA-Z0-9/_-]+$"}}
+    schema = {"collection_path": {
+        "type": "string", "regex": "^[a-zA-Z0-9/_-]+$"}}
     collection_validate = {"collection_path": destination_path}
     v = Validator(schema, error_handler=CustomErrorHandler(schema))
     if v.validate(collection_validate, schema) is False:
@@ -158,7 +159,8 @@ def upload(
         properties = metadata[filename] if metadata else None
         try:
             if user is not None:
-                gsid = __upload_file_gee(session=google_session, file_path=image_path)
+                gsid = __upload_file_gee(
+                    session=google_session, file_path=image_path)
 
             df = pd.read_csv(metadata_path)
             dd = (df.applymap(type) == str).all(0)
@@ -233,7 +235,8 @@ def upload(
                             }
                         }
                         asset_validate = {"asset_path": asset_full_path}
-                        v = Validator(schema, error_handler=CustomErrorHandler(schema))
+                        v = Validator(
+                            schema, error_handler=CustomErrorHandler(schema))
                         if v.validate(asset_validate, schema) is False:
                             print(v.errors)
                             raise Exception
@@ -255,7 +258,8 @@ def upload(
 
 
 def __find_remaining_assets_for_upload(path_to_local_assets, path_remote):
-    local_assets = [__get_filename_from_path(path) for path in path_to_local_assets]
+    local_assets = [__get_filename_from_path(
+        path) for path in path_to_local_assets]
     if __collection_exist(path_remote):
         remote_assets = __get_asset_names_from_collection(path_remote)
         tasked_assets = []
@@ -339,9 +343,11 @@ def __get_google_auth_session(username):
             cookie_list = cookie_list
         elif cookie_check(cookie_list) is False:
             try:
-                cookie_list = raw_input("Cookies Expired | Enter your Cookie List:  ")
+                cookie_list = raw_input(
+                    "Cookies Expired | Enter your Cookie List:  ")
             except Exception:
-                cookie_list = input("Cookies Expired | Enter your Cookie List:  ")
+                cookie_list = input(
+                    "Cookies Expired | Enter your Cookie List:  ")
             finally:
                 with open("cookie_jar.json", "w") as outfile:
                     json.dump(json.loads(cookie_list), outfile)
@@ -360,7 +366,8 @@ def __get_google_auth_session(username):
     session = requests.Session()
     for cookies in cookie_list:
         session.cookies.set(cookies["name"], cookies["value"])
-    response = session.get("https://code.earthengine.google.com/assets/upload/geturl")
+    response = session.get(
+        "https://code.earthengine.google.com/assets/upload/geturl")
     if (
         response.status_code == 200
         and ast.literal_eval(response.text)["url"] is not None
@@ -448,7 +455,8 @@ def __create_image_collection(full_path_to_collection):
     if __collection_exist(full_path_to_collection):
         print("Collection " + str(full_path_to_collection) + " already exists")
     else:
-        print("Collection does not exist: Creating {}".format(full_path_to_collection))
+        print("Collection does not exist: Creating {}".format(
+            full_path_to_collection))
         try:
             ee.data.createAsset(
                 {"type": ee.data.ASSET_TYPE_IMAGE_COLL_CLOUD}, full_path_to_collection

diff --git a/geeup/config.py b/geeup/config.py