Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade data.all pipelines DDK version to DDK v2 #747

Closed
dlpzx opened this issue Sep 12, 2023 · 1 comment · Fixed by #866
Closed

Upgrade data.all pipelines DDK version to DDK v2 #747

dlpzx opened this issue Sep 12, 2023 · 1 comment · Fixed by #866
Assignees
@noah-paige
Labels
effort: low effort: medium priority: medium status: not-picked-yet At the moment we have not picked this item. Anyone can pick it up type: enhancement Feature enhacement
Milestone
v2.2.0

Comments

@dlpzx
Copy link
Contributor

dlpzx commented Sep 12, 2023

Is your idea related to a problem? Please describe.
Currently data.all pipelines use DDK version 1. We rely on ddk bootstrap commands, which is no longer used by DDK. DDK v2 allows us to use CDK bootstrap directly making it easier for us to work on our data pipelines.

Describe the solution you'd like
We would like data.all to upgrade to use the DDK v2. We have identified some places where we will need some changes:

  • in the CICD pipeline stacks that we deploy we need to subsitute any ddk deploy command by cdk deploy
  • same for the backend of data.all, replace ddk deploy by cdk deploy
  • change the app.py that we create to include the config_path to the ddk.json

P.S. Don't attach files. Please, prefer add code snippets directly in the message body.
@dlpzx dlpzx added type: enhancement Feature enhacement priority: medium status: not-picked-yet At the moment we have not picked this item. Anyone can pick it up effort: low effort: medium labels Sep 12, 2023
@dlpzx
Copy link
Contributor Author

dlpzx commented Sep 12, 2023

@malachi-constant

@dlpzx dlpzx mentioned this issue Oct 16, 2023
Merged
7 tasks
@anmolsgandhi anmolsgandhi added this to the v2.2.0 milestone Oct 16, 2023
@noah-paige noah-paige self-assigned this Nov 10, 2023
@noah-paige noah-paige mentioned this issue Nov 10, 2023
Merged
noah-paige added a commit that referenced this issue Nov 28, 2023
@noah-paige
Upgrade DDK and Resolve Data.all Pipelines (#866)
8ca73e6 
### Feature or Bugfix
<!-- please choose -->
- Enhancement

### Detail
- Upgrade DDK Core library to v1.3.0
- Remove calls to DDK CLI Library (deprecated)
- Edit `blueprints/` to work with CDK-native App and use DDK
Configurator

Also as part of this PR:
- Allowed the ability to update CDK Pipeline stacks (previously not
supported)

### Relates
- #747

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
@noah-paige noah-paige linked a pull request Nov 28, 2023 that will close this issue
Upgrade DDK and Resolve Data.all Pipelines #866
Merged
@dlpzx dlpzx closed this as completed Nov 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@noah-paige noah-paige

Labels
effort: low effort: medium priority: medium status: not-picked-yet At the moment we have not picked this item. Anyone can pick it up type: enhancement Feature enhacement
Projects
None yet
Milestone
v2.2.0
Development

Successfully merging a pull request may close this issue.

Upgrade DDK and Resolve Data.all Pipelines data-dot-all/dataall
3 participants
@noah-paige @dlpzx @anmolsgandhi