Python 3.9 Builds (#205)

* made tf 1.1.2 compatible

* tf formatted

* upgraded to sls v2

* container build

* added new GSIs for multi-version

* multi-version build

* multi-version python

* include python version

* added secondary db

* fixed additional # issue

* db migration scripts for multi python

* download config to dynamo

* new configuration multi python

* fixed sk for builds

* test script

* added type hints

* latest in region

* multi python api

* linted with black

* added formats

* added force builds

* linted and corrected

* added READMEs

* minor changes to README

* once a month

* fixed typo in name

* fixed typo in name

* removed errored packages

* test double-bracket

* fixed double bracket

* fixed double bracket

* updated for python3.9

* new README.MD

* added python3.9 README

* cropped diagram

* updated docs

* new diagram

* new diagram

* removed unneccessary pass

README.MD

@@ -2,63 +2,31 @@
 
 🐍 A collection of Python Packages as AWS Lambda(λ) Layers 🐍
 
-[![Python 3.8](https://img.shields.io/badge/python-3.8-green.svg)](https://www.python.org/downloads/release/python-380/) [![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/keithrozario/Klayers.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/keithrozario/Klayers/context:python) [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) 
+[![Python 3.8](https://img.shields.io/badge/python-3.8-green.svg)](https://www.python.org/downloads/release/python-380/) [![Python 3.8](https://img.shields.io/badge/python-3.9-green.svg)](https://www.python.org/downloads/release/python-390/) [![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/keithrozario/Klayers.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/keithrozario/Klayers/context:python) [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) 
 
-## Layer List
+## List of ARNs
 
-List of all layer version arns are available by region [here](deployments/python3.8/arns)
+List of the latest layer version arns are available by region:
 
-There might be multiple versions for a single package, as we publish a new layer version if the package dependencies (`requirements.txt`) has been updated. Refer to [Layer expiry](#Layer-expiry) for more data.
+* [Python 3.8](deployments/python3.8)
+* [Python 3.9](deployments/python3.9)
 
-*Note: We have deprecated layers for python3.6 and python 3.7, please use these newer versions that are built for python 3.8 going forward.*
+*Note: We have deprecated layers for python3.6 and python 3.7, please use these newer versions that are built for python 3.8 and python 3.9 going forward.*
 
 ## Python Packages
 
-Just some examples of the ~70 packages we build every week, for a full list click [here](pipeline/config/Klayers-prodp38/packages.csv):
-
-* aws-lambda-powertools
-* aws-xray-sdk
-* bcrypt
-* beautifulsoup4
-* boto3
-* dynamodb-encryption-sdk
-* idna
-* lambda-cache
-* nltk
-* numpy
-* opencv-python-headless
-* openpyxl
-* pandas
-* parliament
-* Pillow
-* pulp
-* pycryptodome
-* PyJWT
-* PyMUPDF
-* pyqldb
-* pytesseract
-* pytz
-* PyYAML
-* requests
-* slackclient
-* spacy
-* ...
+For the full list of Python packages, refer to the following, feel free to make a pull requests modifying the files below to requests for newer python versions.
 
-## Status of layers
-
-Click [here](https://pyup.io/repos/github/keithrozario/Klayers/) to see specific dependencies of all the latest builds. Layers are built with the latest package version at 2am UTC every Sunday.
+* [Python 3.8](pipeline/config/packages_p38.csv)
+* [Python 3.9](pipeline/config/packages_p39.csv)
 
 ## Using the Layers
 
-You can use the layers anyway you see fit, here are 3 proposed options:
+You can use the layers anyway you see fit, here are 4 options based on what method you use to deploy your lambda functions:
 
 ### Option 1: Set ARN as layer
 
-Click links below for your preferred version of python, and then select your region of choice, you will see a full list of layer version ARNs to use.
-
-* [Arns](deployments/python3.8/arns)
-
-Once selected, you can add the arn directly from the console, by selecting Layers->Add a Layer->Specify an Arn:
+Add the arn directly from the console, by selecting Layers->Add a Layer->Specify an Arn:
 
 ![Screenshot](documentation/add_arn.png)
 
@@ -76,84 +44,80 @@ You can include layers in your deployments, by utilizing the `layers` property a
 check:
   handler: 02_pipeline/check.main
   description: Checks for package on PyPi via the API
-  runtime: python3.8
+  runtime: python3.9
   timeout: 30
   memorySize: 256
   layers:
   - arn:aws:lambda:${self:provider.region}:113088814899:layer:Klayers-python37-packaging:1
   - arn:aws:lambda:${self:provider.region}:113088814899:layer:Klayers-python38-aws-lambda-powertools:23
 ```
 
-## Layer expiry
+### Option 4: Using AWS Serverless Application Model (SAM)
 
-Some layer versions will have a `expiry_date` field. This is the date for when the layers will be deleted.
+Using AWS SAM, you can include layers in your serverless applications.You must use layers from the same region as your function:
 
-In general, layers are scheduled for deletion 60 days after a new layer version has been published for that package. If you use that latest version of a layer, you're guaranteed at least 60 days before the layer is deleted.
+```yaml
+ServerlessFunction:
+  Type: AWS::Serverless::Function
+  Properties:
+    CodeUri: .
+    Handler: my_handler
+    Runtime: Python3.9
+    Layers:
+        - arn:aws:lambda:ap-southeast-1:113088814899:layer:Klayers-p39-packaging:1
+```
 
-All functions deployed with a layer will still work indefinitely, but you won't be able to deploy new functions referencing a deleted layer version.
+## Status of layers
 
-## Binaries
+Layers are built with the latest package version at 2am UTC on the first day of the Month.
 
-Special hand-crafted binaries for layers. These layers are not automatically built, and hence slower update cycles, and will only work with python3.7 functions as the underlying OS is Amazon Linux 1.
+If there's a critical issue with a package, which needs a new version of it published, raise an issue for the package, and I'll try my best to perform the deploy.
 
-| Package        | ARN                                                                             | Version    |
-| :------------- |:------------------------------------------------------------------------------- | ---------- |
-| pip            | arn:aws:lambda:\<*region*>:113088814899:layer:Klayers-python37-pip:2            | 19.2.1 <sup>1</sup>|     |
-| en_core_web_sm | arn:aws:lambda:\<*region*>:770693421928:layer:Klayers-python38-spacy_model_en_small:1 | 2.2.5 <sup>2</sup>|
-| es_core_news_sm | arn:aws:lambda:\<*region*>:770693421928:layer:Klayers-python38-spacy_model_es_small:1 | 2.3.1 <sup>2</sup>|
+## Layer expiry
 
-<sup>1</sup> Python3.8 layers already have new version of pip, use only if on Python3.7
+Some layer versions will have a `expiry_date` field. This is the date for when the layers will be deleted.
 
-<sup>2</sup> Spacy [en_core_web_sm](https://spacy.io/models/en) model, for use with the spacy layer refer [here](https://github.com/keithrozario/Klayers/issues/97) for more info.
+In general, layers are scheduled for deletion 365 days after a new layer version has been published for that package. If you use that latest version of a layer, you're guaranteed at **least** 365 days before the layer is deleted.
 
+All functions deployed with a layer will still work indefinitely, but you won't be able to deploy new functions referencing a deleted layer version.
 
 ## Architecture Diagram
 
 ![Screenshot](documentation/Klayers-Architecture.png)
 
-## API (beta)
+## API
+
+API v2 has been released. Please use v2 going forward, as we'll end v1 by June-2022.
 
-We've recently added an API under beta. All API calls are http-based, and work only with https (TLS1.2 and above). The API is heavily cached, so there could be minor delays in updates.
+All API calls are http-based, and work only with https (TLS1.2 and above). The API is heavily cached, so there could be minor delays in updates.
 
 ### Get latest ARN for specific package in region
 
-Returns data on the latest layer for a specific *{package}* in a specific *{region}*
+Returns data on the latest layer for a all packages in a specific *{region}* for your *{python_version}*
 
-*https://api.klayers.cloud/api/v1/layers/latest/{region}/{package}*
+*https://api.klayers.cloud/api/v2/{python_version}/layers/latest/{region}/*
 
 example:
 
-  * https://api.klayers.cloud/api/v1/layers/latest/us-east-1/requests
-  * https://api.klayers.cloud/api/v1/layers/latest/ap-southeast-1/boto3
-
+* [https://api.klayers.cloud/api/v2/p3.9/layers/latest/us-east-1/](https://api.klayers.cloud/api/v2/p3.9/layers/latest/us-east-1/)
+* [https://api.klayers.cloud/api/v2/p3.8/layers/latest/ap-southeast-1/](https://api.klayers.cloud/api/v2/p3.9/layers/latest/ap-southeast-1/)
 
 ### Get all ARNs for specific package in region
 
-Returns data on the all layers (latest and deprecated) for a specific *{package}* in a specific *{region}*
-
-*https://api.klayers.cloud/api/v1/layers/{region}/{package}*
+Returns data on the all layers (latest and deprecated) for a **specific** *{package}* in a specific *{region}* for your *{python_version}*
 
+*https://api.klayers.cloud/api/v2/{python_version}/layers/{region}/{package}*
 
 example:
 
-  * https://api.klayers.cloud/api/v1/layers/us-east-1/requests
-  * https://api.klayers.cloud/api/v1/layers/ap-southeast-1/boto3
-
-### Get list of all built packages
-
-Returns list of all packages currently being built, including dependencies and build date.
-
-*https://api.klayers.cloud/api/v1/builds/latest*
-
-example:
-
-  * https://api.klayers.cloud/api/v1/builds/latest
+* [https://api.klayers.cloud/api/v2/p3.8/layers/us-east-1/requests](https://api.klayers.cloud/api/v2/p3.8/layers/us-east-1/requests)
+* [https://api.klayers.cloud/api/v2/p3.9/layers/ap-southeast-1/boto3](https://api.klayers.cloud/api/v2/p3.9/layers/ap-southeast-1/boto3)
 
 ## Special Thanks
 
-* [Chahna107](https://github.com/chahna107) for adding tesseract config files into the tesseract layer.
+* [Chahna107](https://github.com/chahna107) 
 * [~ Dependencies scanned by PyUp.io ~](https://pyup.io/)
 
 ## Asking for additional layers
 
-If you would like a new package to be made a layer, raise a `pull request` modifying the `pipeline/config/Klayers-prodp38` file. 
+If you would like a new package to be made a layer, raise a `pull request` modifying the `pipeline/config/packages_p39.csv` or `pipeline/config/packages_p38.csv` file (depending on which version of python you use). By default, please request for the latest version of python unless you absolutely need an older version.

deployments/python3.8/README.MD

@@ -0,0 +1,27 @@
+# Layers for Python 3.8
+
+Use the following links to download the latest Arns for your specific region in the format you prefer (json, csv or html). 
+
+| Region (Code) | Region (Name)| ARNs|
+| :------------- |:--------|:--------|
+| af-south-1 |Africa (Cape Town)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/af-south-1/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/af-south-1/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/af-south-1/html)|
+| ap-east-1 | Asia Pacific (Hong Kong)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-east-1/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-east-1/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-east-1/html)|
+| ap-northeast-1 |Asia Pacific (Tokyo)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-northeast-1/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-northeast-1/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-northeast-1/html)|
+| ap-northeast-2 |Asia Pacific (Seoul)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-northeast-2/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-northeast-2/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-northeast-2/html)|
+| ap-south-1 |Asia Pacific (Mumbai)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-south-1/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-south-1/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-south-1/html)|
+| ap-southeast-1 |Asia Pacific (Singapore)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-southeast-1/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-southeast-1/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-southeast-1/html)|
+| ap-southeast-2 |Asia Pacific (Sydney)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-southeast-2/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-southeast-2/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-southeast-2/html)|
+| ap-southeast-3 |Asia Pacific (Jakarta)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-southeast-3/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-southeast-3/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ap-southeast-3/html)|
+| ca-central-1 |Canada (Central)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ca-central-1/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ca-central-1/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/ca-central-1/html)|
+| eu-central-1 |EU (Frankfurt)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/eu-central-1/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/eu-central-1/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/eu-central-1/html)|
+| eu-north-1 |EU (Stockholm)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/eu-north-1/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/eu-north-1/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/eu-north-1/html)|
+| eu-south-1 |EU (Milan)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/eu-south-1/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/eu-south-1/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/eu-south-1/html)|
+| eu-west-1 |EU (Ireland)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/eu-west-1/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/eu-west-1/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/eu-west-1/html)|
+| eu-west-2 |EU (London)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/eu-west-2/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/eu-west-2/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/eu-west-2/html)|
+| eu-west-3 |EU (Paris)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/eu-west-3/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/eu-west-3/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/eu-west-3/html)|
+| me-south-1 |Middle East (Bahrain)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/me-south-1/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/me-south-1/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/me-south-1/html)|
+| sa-east-1 |South America (São Paulo)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/sa-east-1/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/sa-east-1/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/sa-east-1/html)|
+| us-east-1 |US East (N. Virginia)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/us-east-1/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/us-east-1/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/us-east-1/html)|
+| us-east-2 |US East (Ohio)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/us-east-2/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/us-east-2/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/us-east-2/html)|
+| us-west-1 |US West (N. California)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/us-west-1/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/us-west-1/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/us-west-1/html)|
+| us-west-2 |US West (Oregon)| [json](https://api.klayers.cloud//api/v2/p3.8/layers/latest/us-west-2/json) \| [csv](https://api.klayers.cloud//api/v2/p3.8/layers/latest/us-west-2/csv) \| [html](https://api.klayers.cloud//api/v2/p3.8/layers/latest/us-west-2/html)|