Backend release branch to main

.github/CODEOWNERS

@@ -1,2 +1 @@
 *       @esfoobar-usds @vim-usds @emma-nechamkin @mattbowen-usds
-

.github/ISSUE_TEMPLATE/dataset_request.yaml

@@ -56,7 +56,7 @@ body:
     id: other-datasource-type
     attributes:
       label: If "other" above, please specify
-      description: If you have selected "other" above, please give as much detail as you can as to where this data can live 
+      description: If you have selected "other" above, please give as much detail as you can as to where this data can live
       placeholder: www.example.com and www.example2.com
     validations:
       required: false
@@ -110,14 +110,14 @@ body:
     attributes:
       label: Known data quality issues
       description: Please describe any data quality issues you know about with this dataset.
-      placeholder: Limited accuracy in rural areas, etc. 
+      placeholder: Limited accuracy in rural areas, etc.
     validations:
-      required: false  
+      required: false
   - type: input
     id: geographic-coverage
     attributes:
       label: What is the geographic coverage (in percent) of this dataset
-      description: If known, provide an estimate of the coverage of this dataset vis-a-vis the full United States 
+      description: If known, provide an estimate of the coverage of this dataset vis-a-vis the full United States
       placeholder: 90%
     validations:
       required: false
@@ -126,9 +126,9 @@ body:
     attributes:
       label: Description of geographic coverage estimate
       description: (If relevant) Please explain your reasoning behind the above estimate of geographic coverage
-      placeholder: e.g. Missing information for Puerto Rico, etc. 
+      placeholder: e.g. Missing information for Puerto Rico, etc.
     validations:
-      required: false  
+      required: false
   - type: input
     id: last-updated-date
     attributes:
@@ -151,10 +151,10 @@ body:
     id: link-to-documentation
     attributes:
       label: Link to more documentation
-      description: Please provide a link where one can find additional information 
+      description: Please provide a link where one can find additional information
       placeholder: www.example.com
     validations:
-      required: true   
+      required: true
   - type: dropdown
     id: can-go-in-cloud
     attributes:
@@ -167,8 +167,8 @@ body:
   - type: textarea
     id: additional-information
     attributes:
-      label: Additional Information 
+      label: Additional Information
       description: Please provide any additional information you have about this dataset
-      placeholder: e.g. Missing information for Puerto Rico, etc. 
+      placeholder: e.g. Missing information for Puerto Rico, etc.
     validations:
       required: false

.github/ISSUE_TEMPLATE/pull_request_template.md

@@ -19,4 +19,4 @@ Fixes # (issue number)
 - [ ] My changes generate no new warnings
 - [ ] I have added tests that prove my fix is effective or that my feature works
 - [ ] New and existing unit tests pass locally with my changes
-- [ ] Any dependent changes have been merged and published in downstream modules
+- [ ] Any dependent changes have been merged and published in downstream modules

.github/workflows/create-score-version.yml

@@ -5,11 +5,12 @@ on:
       score_version:
         description: "Which version of the score are you generating?"
         required: true
-        default: 'beta'
+        default: '1.0'
         type: choice
         options:
         - beta
         - 1.0
+        - test
 
 env:
   CENSUS_API_KEY: ${{ secrets.CENSUS_API_KEY }}
@@ -54,7 +55,6 @@ jobs:
           aws-region: us-east-1
       - name: Install GDAL/ogr2ogr
         run: |
-          sudo add-apt-repository ppa:ubuntugis/ppa
           sudo apt-get update
           sudo apt-get -y install gdal-bin
           ogrinfo --version
@@ -64,15 +64,40 @@ jobs:
       - name: Generate Score Post
         run: |
           poetry run python3 data_pipeline/application.py generate-score-post -s aws
+      - name: Confirm we generated the version of the score we think we did
+        if: ${{ env.J40_VERSION_LABEL_STRING == '1.0' || env.J40_VERSION_LABEL_STRING == 'test' }}
+        run: |
+          grep "Identified as disadvantaged due to tribal overlap" data_pipeline/data/score/downloadable/* > /dev/null
+      - name: Confirm we generated the version of the score we think we did
+        if: ${{ env.J40_VERSION_LABEL_STRING == 'beta' }}
+        run: |
+          grep -v "Identified as disadvantaged due to tribal overlap" data_pipeline/data/score/downloadable/* > /dev/null
       - name: Generate Score Geo
         run: |
           poetry run python3 data_pipeline/application.py geo-score
+      - name: Run smoketest for 1.0
+        if: ${{ env.J40_VERSION_LABEL_STRING == '1.0' || env.J40_VERSION_LABEL_STRING == 'test' }}
+        run: |
+          poetry run pytest data_pipeline/ -m smoketest
       - name: Deploy Score to Geoplatform AWS
         run: |
           poetry run s4cmd put ./data_pipeline/data/score/csv/ s3://justice40-data/data-versions/${{env.J40_VERSION_LABEL_STRING}}/data/score/csv --recursive --force --API-ACL=public-read
           poetry run s4cmd put ./data_pipeline/files/ s3://justice40-data/data-versions/${{env.J40_VERSION_LABEL_STRING}}/data/score/downloadable --recursive --force --API-ACL=public-read
           poetry run s4cmd put ./data_pipeline/data/score/downloadable/ s3://justice40-data/data-versions/${{env.J40_VERSION_LABEL_STRING}}/data/score/downloadable --recursive --force --API-ACL=public-read
-
+      - name: Confirm we generated the version of the score we think we did
+        if: ${{ env.J40_VERSION_LABEL_STRING == '1.0' || env.J40_VERSION_LABEL_STRING == 'test' }}
+        run: |
+          curl "https://static-data-screeningtool.geoplatform.gov/data-versions/1.0/data/score/downloadable/1.0-shapefile-codebook.zip" -s -f -I -o /dev/null && \
+          curl "https://static-data-screeningtool.geoplatform.gov/data-versions/1.0/data/score/downloadable/1.0-communities.xlsx" -s -f -I -o /dev/null && \
+          curl "https://static-data-screeningtool.geoplatform.gov/data-versions/1.0/data/score/downloadable/1.0-communities.csv" -s -f -I -o /dev/null && \
+          curl "https://static-data-screeningtool.geoplatform.gov/data-versions/1.0/data/score/downloadable/1.0-shapefile-codebook.zip" -s -f -I -o /dev/null && \
+          curl "https://static-data-screeningtool.geoplatform.gov/data-versions/1.0/data/score/downloadable/cejst-technical-support-document.pdf" -s -f -I -o /dev/null && \
+          curl "https://static-data-screeningtool.geoplatform.gov/data-versions/1.0/data/score/downloadable/draft-communities-list.pdf" -s -f -I -o /dev/null
+      - name: Confirm we generated the version of the score we think we did
+        if: ${{ env.J40_VERSION_LABEL_STRING == 'beta' }}
+        run: |
+          curl "https://static-data-screeningtool.geoplatform.gov/data-versions/beta/data/score/downloadable/beta-data-documentation.zip" -s -f -I -o /dev/null && \
+          curl "https://static-data-screeningtool.geoplatform.gov/data-versions/beta/data/score/downloadable/beta-shapefile-codebook.zip" -s -f -I -o /dev/null
       - name: Set timezone for tippecanoe
         uses: szenius/set-timezone@v1.0
         with:

.github/workflows/data-checks.yml

@@ -22,9 +22,11 @@ jobs:
       # one execution of the tests per version listed above
       - uses: actions/checkout@v2
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v4
         with:
           python-version: ${{ matrix.python-version }}
+      - name: Upgrade wheel
+        run: pip install -U wheel
       - name: Print variables to help debug
         uses: hmarr/debug-action@v2
       - name: Load cached Poetry installation
@@ -39,6 +41,7 @@ jobs:
         run: poetry show -v
       - name: Install dependencies
         run: poetry install
-        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
+        # TODO: investigate why caching layer started failing.
+        # if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
       - name: Run tox
         run: poetry run tox

.github/workflows/deploy_be_staging.yml

@@ -25,19 +25,25 @@ jobs:
       - name: Print variables to help debug
         uses: hmarr/debug-action@v2
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v4
         with:
           python-version: ${{ matrix.python-version }}
+      - run: pip install -U wheel
       - name: Load cached Poetry installation
         id: cached-poetry-dependencies
         uses: actions/cache@v2
         with:
           path: ~/.cache/pypoetry/virtualenvs
           key: env-${{ runner.os }}-${{ matrix.python-version }}-${{ hashFiles('**/poetry.lock') }}-${{ hashFiles('.github/workflows/deploy_be_staging.yml') }}
       - name: Install poetry
-        uses: snok/install-poetry@v1
+        uses: snok/install-poetry@v1.3.3
       - name: Print Poetry settings
         run: poetry show -v
+      - name: Install GDAL/ogr2ogr
+        run: |
+          sudo apt-get update
+          sudo apt-get -y install gdal-bin
+          ogrinfo --version
       - name: Install dependencies
         run: poetry add s4cmd && poetry install
         if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
@@ -47,12 +53,21 @@ jobs:
           aws-access-key-id: ${{ secrets.DATA_DEV_AWS_ACCESS_KEY_ID }}
           aws-secret-access-key: ${{ secrets.DATA_DEV_AWS_SECRET_ACCESS_KEY }}
           aws-region: us-east-1
+      - name: Download census geo data for later user
+        run: |
+          poetry run python3 data_pipeline/application.py pull-census-data -s aws
       - name: Generate Score
         run: |
           poetry run python3 data_pipeline/application.py score-full-run
       - name: Generate Score Post
         run: |
-          poetry run python3 data_pipeline/application.py generate-score-post -s aws
+          poetry run python3 data_pipeline/application.py generate-score-post
+      - name: Generate Score Geo
+        run: |
+          poetry run python3 data_pipeline/application.py geo-score
+      - name: Run Smoketests
+        run: |
+          poetry run pytest data_pipeline/ -m smoketest
       - name: Deploy Score to Geoplatform AWS
         run: |
           poetry run s4cmd put ./data_pipeline/data/score/csv/ s3://justice40-data/data-pipeline-staging/${{env.PR_NUMBER}}/${{env.SHA_NUMBER}}/data/score/csv --recursive --force --API-ACL=public-read
@@ -63,19 +78,13 @@ jobs:
         with:
           # Deploy to S3 for the Staging URL
           message: |
-            ** Score Deployed! **   
-            Find it here: 
+            ** Score Deployed! **
+            Find it here:
             - Score Full usa.csv: https://justice40-data.s3.amazonaws.com/data-pipeline-staging/${{env.PR_NUMBER}}/${{env.SHA_NUMBER}}/data/score/csv/full/usa.csv
             - Download Zip Packet: https://justice40-data.s3.amazonaws.com/data-pipeline-staging/${{env.PR_NUMBER}}/${{env.SHA_NUMBER}}/data/score/downloadable/Screening_Tool_Data.zip
           repo-token: ${{ secrets.GITHUB_TOKEN }}
           repo-token-user-login: "github-actions[bot]"
           allow-repeats: false
-      - name: Install GDAL/ogr2ogr
-        run: |
-          sudo add-apt-repository ppa:ubuntugis/ppa
-          sudo apt-get update
-          sudo apt-get -y install gdal-bin
-          ogrinfo --version
       - name: Set timezone for tippecanoe
         uses: szenius/set-timezone@v1.0
         with:
@@ -93,9 +102,6 @@ jobs:
           mkdir -p /usr/local/bin
           cp tippecanoe /usr/local/bin/tippecanoe
           tippecanoe -v
-      - name: Generate Score Geo
-        run: |
-          poetry run python3 data_pipeline/application.py geo-score
       - name: Generate Tiles
         run: |
           poetry run python3 data_pipeline/application.py generate-map-tiles
@@ -110,8 +116,8 @@ jobs:
         with:
           # Deploy to S3 for the staging URL
           message: |
-            ** Map Deployed! **   
-            Map with Staging Backend: https://screeningtool.geoplatform.gov/en/?flags=stage_hash=${{env.PR_NUMBER}}/${{env.SHA_NUMBER}}
+            ** Map Deployed! **
+            Map with Staging Backend: https://screeningtool.geoplatform.gov/en?flags=stage_hash=${{env.PR_NUMBER}}/${{env.SHA_NUMBER}}
             Find tiles here: https://justice40-data.s3.amazonaws.com/data-pipeline-staging/${{env.PR_NUMBER}}/${{env.SHA_NUMBER}}/data/score/tiles
           repo-token: ${{ secrets.GITHUB_TOKEN }}
           repo-token-user-login: "github-actions[bot]"

.github/workflows/deploy_fe_staging.yml

@@ -108,9 +108,9 @@ jobs:
         uses: mshick/add-pr-comment@v1
         with:
           message: |
-            ** 👋  Attention translators!! 👋  **   
+            ** 👋  Attention translators!! 👋  **
             Copy changes have resulted in a new en.json file. Please download en.json file and send to translators: https://github.com/usds/justice40-tool/blob/${{env.COMMIT_HASH}}/client/src/intl/en.json
           repo-token: ${{ secrets.GITHUB_TOKEN }}
           repo-token-user-login: "github-actions[bot]" # The user.login for temporary GitHub tokens
           allow-repeats: true
-        if: contains(steps.changed-files.outputs.modified_files, 'client/src/intl/en.json')
+        if: contains(steps.changed-files.outputs.modified_files, 'client/src/intl/en.json')

.github/workflows/e2e.yml

@@ -10,7 +10,7 @@ on:
   schedule:
     # runs tests every day at 12am ET (4am UTC):
     - cron: '0 4 * * *'
-jobs: 
+jobs:
   nightly:
     runs-on: ubuntu-20.04
     env:
@@ -25,4 +25,4 @@ jobs:
           start: npm start
           wait-on: 'http://localhost:8000'
           # To run only specific spec/tests:
-          # spec: cypress/e2e/downloadPacket.spec.js
+          # spec: cypress/e2e/downloadPacket.spec.js

.github/workflows/generate-census.yml

@@ -1,5 +1,5 @@
 name: Generate Census
-on: 
+on:
   workflow_dispatch:
     inputs:
       confirm-action:

.github/workflows/main.yml

@@ -9,11 +9,11 @@ on:
   workflow_dispatch:
     inputs:
       logLevel:
-        description: 'Log level'     
+        description: 'Log level'
         required: true
         default: 'warning'
       tags:
-        description: 'Ping test'  
+        description: 'Ping test'
 jobs:
   sitePingCheck:
     name: Slack Notification

CODE_OF_CONDUCT-es.md

@@ -13,4 +13,3 @@ Los mantenedores del proyecto tienen el derecho y la obligación de eliminar, ed
 Los casos de abuso, acoso o de otro comportamiento inaceptable se pueden denunciar abriendo un problema o contactando con uno o más de los mantenedores del proyecto en justice40open@usds.gov.
 
 Este Código de conducta es una adaptación de la versión 1.0.0 del Convenio del colaborador ([Contributor Covenant](http://contributor-covenant.org), *en inglés*) disponible en el sitio http://contributor-covenant.org/version/1/0/0/ *(en inglés)*.
-

CODE_OF_CONDUCT.md

@@ -12,4 +12,4 @@ Project maintainers have the right and responsibility to remove, edit, or reject
 
 Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by opening an issue or contacting one or more of the project maintainers at justice40open@usds.gov.
 
-This Code of Conduct is adapted from the [Contributor Covenant](http://contributor-covenant.org), version 1.0.0, available at [http://contributor-covenant.org/version/1/0/0/](http://contributor-covenant.org/version/1/0/0/)
+This Code of Conduct is adapted from the [Contributor Covenant](http://contributor-covenant.org), version 1.0.0, available at [http://contributor-covenant.org/version/1/0/0/](http://contributor-covenant.org/version/1/0/0/)

COMMUNITY_GUIDELINES.md

@@ -32,4 +32,4 @@ When participating in Justice40 open source community conversations and spaces,
 - Take space and give space. We strive to create an equitable environment in which all are welcome and able to participate. We hope individuals feel comfortable voicing their opinions and providing contributions and will do our best to recognize and make space for individuals who may be struggling to find space here. Likewise, we expect individuals to recognize when they are taking up significant space and take a step back to allow room for others.
 - Be present when joining synchronous conversations such as our community chat. Why be here if you're not going to _be here_?
 - Be respectful.
-- Default to positive. Assume others' contributions are legitimate and valuable and that they are made with good intention.
+- Default to positive. Assume others' contributions are legitimate and valuable and that they are made with good intention.

CONTRIBUTING-es.md

@@ -43,4 +43,3 @@ Si desea colaborar con alguna parte del código base, bifurque el repositorio si
 * Al menos un revisor autorizado debe aprobar la confirmación (en [CODEOWNERS](https://github.com/usds/justice40-tool/tree/main/.github/CODEOWNERS), en inglés, consulte la lista más reciente de estos revisores).
 * Todas las verificaciones de estado obligatorias deben ser aprobadas.
 Si hay un desacuerdo importante entre los integrantes del equipo, se organizará una reunión con el fin de determinar el plan de acción para la solicitud de incorporación de cambios.
-

INSTALLATION.md

@@ -36,7 +36,7 @@ Homebrew is an easy way to manage software downloads on MacOS. You don't _have_
 
 You should regularly run `brew update` and `brew doctor` to make sure your packages are up to date and in good condition.
 
-### Install Node using NVM 
+### Install Node using NVM
 
 This will work for both MacOS and Win10. Follow instructions on this [link](https://medium.com/@nodesource/installing-node-js-tutorial-using-nvm-5c6ff5925dd8). Be sure to read through the whole doc to find the sections within each step relevant to you (e.g. if you're using Homebrew, when you get to Step 2 look for the section, "Install NVM with Homebrew").
 
@@ -54,7 +54,7 @@ You should then be able to switch to that version of node by:
 
 To validate you are using node 14, type:
 
-`node -v` 
+`node -v`
 
 This should return *Now using node 14.x.x (npm v6.x.x)*
 

LICENSE-es.md

@@ -28,4 +28,3 @@ Por estos u otros propósitos y motivos, y sin ninguna expectativa de otra consi
     c. El Afirmante excluye la responsabilidad de los derechos de compensación de otras personas que se puedan aplicar a la Obra o a cualquier uso de esta, incluidos, entre otros, los Derechos de Autor y Derechos Conexos de cualquier persona sobre la Obra. Además, el Afirmante excluye la responsabilidad de obtener los consentimientos o permisos u otros derechos necesarios que se exijan para cualquier uso de la Obra.
 
     d. El Afirmante entiende y reconoce que Creative Commons no es una parte en este documento y que no tiene ningún derecho u obligación con respecto a esta CC0 o al uso de la Obra.
-

README-es.md

@@ -2,7 +2,7 @@
 [![CC0 License](https://img.shields.io/badge/license-CCO--1.0-brightgreen)](https://github.com/usds/justice40-tool/blob/main/LICENSE.md)
 
 *[Read this in English!](README.md)*
- 
+
 Le damos la bienvenida a la comunidad de código abierto de Justice40. Este repositorio contiene el código, los procesos y la documentación que activa los datos y la tecnología de la Herramienta Justice40 para la Vigilancia del Clima y la Justicia Económica (CEJST, por sus siglas en inglés).
 
 ## Antecedentes
@@ -36,7 +36,7 @@ El equipo central usa el grupo para publicar la información más reciente sobre
 Las colaboraciones son siempre bien recibidas. Nos agradan las aportaciones en forma de conversación sobre los temas de este repositorio y las solicitudes para incorporación de cambios en la documentación y el código.
 En [CONTRIBUTING-es.md](CONTRIBUTING-es.md), consulte la manera de empezar a participar.
 
-## Instalación 
+## Instalación
 
 La instalación es una instalación típica de gatsby y los detalles se pueden encontrar en [INSTALLATION-es.md](INSTALLATION-es.md)