Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update CodeBuild images to Linux2 standard5.0 (node16 to node18) + Update Docker images to use AmazonLinux:2023 (node18 and Python3.9) #889

Merged
merged 11 commits into from
Dec 7, 2023
Merged

Update CodeBuild images to Linux2 standard5.0 (node16 to node18) + Update Docker images to use AmazonLinux:2023 (node18 and Python3.9) #889

merged 11 commits into from
Dec 7, 2023

Conversation

dlpzx
Copy link
Contributor

@dlpzx dlpzx commented Nov 23, 2023 •
edited
Loading

Feature or Bugfix

  • Bugfix

Detail

The purpose of this PR is to upgrade any compute resource that uses node16 to node18.

  • CodeBuild images: Amazon Linux 2 x86_64 standard:4.0 use node16 which is already deprecated. In this PR we update the CodeBuild images to use Amazon Linux 2 x86_64 standard:5.0 instead
  • Docker images: In this PR we replace AmazonLinux2 images by AmazonLinux2023, the next generation of Amazon Linux from Amazon Web Services. In AmazonLinux2023 the default Python version installed is 3.9. For this reason we also upgrade the Python version in this PR.

Relates

#782

Security

Please answer the questions below briefly where applicable, or write N/A. Based on
OWASP 10. N/A

  • Does this PR introduce or modify any input fields or queries - this includes
    fetching data from storage outside the application (e.g. a database, an S3 bucket)?
    • Is the input sanitized?
    • What precautions are you taking before deserializing the data you consume?
    • Is injection prevented by parametrizing queries?
    • Have you ensured no eval or similar functions are used?
  • Does this PR introduce any functionality or component that requires authorization?
    • How have you ensured it respects the existing AuthN/AuthZ mechanisms?
    • Are you logging failed auth attempts?
  • Are you using or adding any cryptographic features?
    • Do you use a standard proven implementations?
    • Are the used keys controlled by the customer? Where are they stored?
  • Are you introducing any new policies/roles/users?
    • Have you used the least-privilege principle? How?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@dlpzx
Copy link
Contributor Author

dlpzx commented Nov 23, 2023

At the moment it throws errors in the CICD pipeline because of DDK cli

@dlpzx dlpzx marked this pull request as draft November 24, 2023 07:26
@noah-paige
Copy link
Contributor

@dlpzx - thoughts on also specifying build_image=codebuild.LinuxBuildImage.AMAZON_LINUX_2_5 for the CodeBuild Jobs created by DataPipelines CDK Stack?

@dlpzx dlpzx marked this pull request as ready for review November 29, 2023 12:35
@dlpzx
Copy link
Contributor Author

dlpzx commented Nov 29, 2023

@noah-paige yes! Definitely, I did not want to interfere with the work of the DDK pipelines but I will do now. I also realized that we might be using deprecated versions of node in ECS. I'll review

@dlpzx dlpzx changed the title Update CodeBuild images to Linux2 standard5.0 Update CodeBuild images to Linux2 standard5.0 and Docker images to node18 Nov 29, 2023
@dlpzx
Copy link
Contributor Author

dlpzx commented Nov 29, 2023

I am currently testing the changes in AWS

@noah-paige
Copy link
Contributor

I deployed the changes before the last update and CICD Pipeline went through. I will pull latest changes and test again for CICD pipeline completing successfully, and pipeline creation still works

Will also check local deployment still works

@dlpzx
Copy link
Contributor Author

dlpzx commented Dec 1, 2023

Do not approve yet because there are some issues with using Linux5. In order to use node18 in the Docker images, we also need to upgrade the images used in the docker files. The new Amazon Linux 2023 is a bit different from Linux2, it is more minimalistic and does not include all packages, plus it replaces yum by dnf. There is no hurry, so in my opinion we should do this right and make sure that node is using node18 everywhere

@dlpzx dlpzx marked this pull request as draft December 1, 2023 07:40
@dlpzx dlpzx changed the title Update CodeBuild images to Linux2 standard5.0 and Docker images to node18 Update CodeBuild images to Linux2 standard5.0 (node16 to node18) + Update Docker images to use AmazonLinux:2023 (node18 and Python3.9) Dec 1, 2023
@dlpzx dlpzx marked this pull request as ready for review December 1, 2023 12:56
@dlpzx
Copy link
Contributor Author

dlpzx commented Dec 1, 2023 •
edited
Loading

Tested in AWS

  • CICD builds and registers ECS and Lambda images successfully
  • API calls (Lambda) work as expected
  • Creation of Dataset (ECS) works as expected
  • Frontend runs as expected
  • user guide runs as expedcted

Tested locally

  • docker-compose up creates all images and containers without issue
  • API calls and frontend appear as expected
  • Creation of Dataset (ECS) works as expected

Note: we need to update the GitHub pages and specify Python 3.9 as minimum

@dbalintx
Copy link
Contributor

dbalintx commented Dec 6, 2023

Deployed locally - images are built properly
Deployed to AWS - codebuild succeeds, images are built properly
Created dataset, clicked through the FE, checked the userguide - seems to be fine.

Left 2 nit comments, otherwise LGTM

@dlpzx dlpzx requested a review from dbalintx December 6, 2023 16:50
@dlpzx dlpzx requested a review from dbalintx December 6, 2023 17:11
@dlpzx dlpzx merged commit 5061ecb into main Dec 7, 2023
9 checks passed
@noah-paige noah-paige mentioned this pull request Dec 11, 2023
Merged
dlpzx pushed a commit that referenced this pull request Dec 12, 2023
@noah-paige
Fix DockerFile commands for vpc facing arch with updates to versions (#…
c7e2c85 
…907)

### Feature or Bugfix
<!-- please choose -->
- Bugfix

### Detail
- Fix docker image used on VPC Facing Deployments to successfully build
frontend + userguide images to host in ECS

### Relates
- #889

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
N/A
  - Is the input sanitized? N/A
- What precautions are you taking before deserializing the data you
consume? N/A
  - Is injection prevented by parametrizing queries? N/A
  - Have you ensured no `eval` or similar functions are used? Yes
- Does this PR introduce any functionality or component that requires
authorization? No
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
N/A
  - Are you logging failed auth attempts? N/A
- Are you using or adding any cryptographic features? N/A
  - Do you use a standard proven implementations? Yes
- Are the used keys controlled by the customer? Where are they stored?
N/A
- Are you introducing any new policies/roles/users? No
  - Have you used the least-privilege principle? How? N/A


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
@dlpzx dlpzx deleted the fix/update-codebuild-node-version branch December 18, 2023 15:15
@dlpzx dlpzx mentioned this pull request Feb 14, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dbalintx dbalintx dbalintx approved these changes

@nikpodsh nikpodsh nikpodsh approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dlpzx @noah-paige @dbalintx @nikpodsh