Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

modularization: Mlstudio modularization #448

Closed
wants to merge 81 commits into from
Closed

modularization: Mlstudio modularization #448

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
171a75c
Bump webpack from 5.75.0 to 5.76.1 in /frontend (#371)
dependabot[bot] Mar 15, 2023
4002963
Upgrade sqlalchemy 13.16 -> 1.3.24 and starlette 0.19.1 -> 0.25.0, ar…
dlpzx Mar 20, 2023
79f0e4c
Add dependency in dataset stack (#385)
dlpzx Mar 27, 2023
95c8619
feat: generate url with dynamically domain name for quicksight embede…
wolanlu Mar 28, 2023
f677371
fix: dev docker images base (#387)
AmrSaber Mar 28, 2023
e01e014
Added missing groupUri from get credentials (#391)
dlpzx Mar 30, 2023
9057116
388 race condition occurs when adding folder to shared items in share…
dlpzx Mar 30, 2023
d191b26
hotfix: Revert PR on custom url quicksight embedding sessions (#403)
dlpzx Apr 4, 2023
a17f12a
401 shared dbs worksheet list (#402)
noah-paige Apr 4, 2023
6460986
Fix sharing update (#404)
dlpzx Apr 4, 2023
219553f
V1.5.0 Features (#409)
dlpzx Apr 25, 2023
7ad099f
Bump flask from 2.0.3 to 2.3.2 in /backend (#439)
dependabot[bot] May 2, 2023
e4b3e73
Bump flask from 2.0.3 to 2.3.2 in /backend/dataall/cdkproxy (#438)
dependabot[bot] May 2, 2023
2a319ca
solve deployment bug #433 CloudFront logs does not enable ACL access …
akaitoua May 5, 2023
13a2fc0
Modify docker-compose yaml to read region and default region from env…
dlpzx May 10, 2023
e9ebb08
Bump pymdown-extensions from 8.1.1 to 10.0 in /documentation/userguid…
dependabot[bot] May 16, 2023
4ad8ce7
Bump starlette from 0.25.0 to 0.27.0 and upgrade fastapi (#460)
dlpzx May 17, 2023
ee4f34c
Fixes issue with existing cognito callbacks (#464)
gmuslia May 17, 2023
3b85ad2
Fix lambda/ECS IAM permissions for AOSS (#467)
kukushking May 22, 2023
3097a3a
465 - Update Aurora default Parameter Group to 'default.aurora-postgr…
rbernotas May 22, 2023
fd7da75
Bump requests from 2.27.1 to 2.31.0 in /backend (#469)
dependabot[bot] May 23, 2023
65ea17b
Bump requests from 2.27.1 to 2.31.0 in /backend/dataall/cdkproxy (#470)
dependabot[bot] May 23, 2023
b59cf9e
hotfix: Remove GitHub template option from data.all Pipelines (#472)
dlpzx May 23, 2023
3340610
fix: Upgrade aurora engine version to 11.16 (#471)
kimengu-david May 24, 2023
b441258
Updated CDK Version to fix issue with cdkproxy/ dataset stack creatio…
gmuslia May 24, 2023
d0ea832
update auth-at-edge semantic version to latest 2.1.5 (#480)
dlpzx May 25, 2023
2dc329f
sync modularization main with main (#395)
AmrSaber Mar 30, 2023
fbec440
backend: Pluginarchitecture (#359)
nikpodsh Mar 31, 2023
8bc1dee
frontend: simplify dev dockerfile (#396)
AmrSaber Apr 5, 2023
6a23c83
frontend: make styling consistent and remove dead code (#394)
AmrSaber Apr 5, 2023
c8ffe31
ignore styling commits in git blame (#405)
AmrSaber Apr 11, 2023
5e44e1b
Dataset Modularization pt.1 (#413)
nikpodsh Apr 24, 2023
1db7743
Datasets modularization pt.2 (#432)
nikpodsh May 2, 2023
433b5af
Datasets modularization pt.3 (#440)
nikpodsh May 4, 2023
ecaaf6e
Datasets modularization pt.4 (#441)
nikpodsh May 4, 2023
5558608
Datasets modularization pt.5 (#442)
nikpodsh May 10, 2023
7354367
Module dependencies (#447)
nikpodsh May 17, 2023
b5f289d
Worksheets modularization (#449)
dbalintx May 17, 2023
a6539ba
Added mlstudio module skeleton - api package except resolvers, skelet…
dlpzx May 8, 2023
7d716d4
mlstudio - moved permissions
dlpzx May 8, 2023
a2b27a2
ml studio - started to create db models, repositories, services
dlpzx May 8, 2023
bc15fae
Added create sagemaker studio service, added sm_studio_client
dlpzx May 9, 2023
8a7c33f
Renaming and advancing with db.repository, services and resolvers (list)
dlpzx May 9, 2023
2f193b8
Rename permissions to SGM_USER instead of SGM_NOTEBOOK
dlpzx May 9, 2023
4f23d38
More services and methods - presigned url...
dlpzx May 9, 2023
129f64a
Completed services-db-resolvers. Renamed to naming convention sagemak…
dlpzx May 10, 2023
5e32c85
Added cdk policies and stack
dlpzx May 10, 2023
6c47269
Linting
dlpzx May 10, 2023
c4367ed
Standardize naming for mlstudio in frontend
dlpzx May 11, 2023
7fd81cc
Remove unused query
dlpzx May 11, 2023
fa195a7
Cleanup, extract non class method in sagemaker client
dlpzx May 11, 2023
6163293
Cleanup, extract non class method in sagemaker client
dlpzx May 11, 2023
4054034
Replace mlStudioEnabled in environment parameters
dlpzx May 11, 2023
98d1933
Remove old files for MLStudio, left TODOs for v1.5 and migration scri…
dlpzx May 11, 2023
013fd61
Rebasing with modularization-main
dlpzx May 11, 2023
ec85577
Review comments
dlpzx May 16, 2023
aa3903a
Rebasing with modularization-main
dlpzx May 17, 2023
99a8f9a
First draft tests - needs migration of db permissions
dlpzx May 17, 2023
5ab0bcf
Fixing migration scripts
dlpzx May 17, 2023
2e60acf
Added mlstudio module skeleton - api package except resolvers, skelet…
dlpzx May 8, 2023
ba8b023
mlstudio - moved permissions
dlpzx May 8, 2023
df72352
ml studio - started to create db models, repositories, services
dlpzx May 8, 2023
78deecb
Added create sagemaker studio service, added sm_studio_client
dlpzx May 9, 2023
f39da0a
Renaming and advancing with db.repository, services and resolvers (list)
dlpzx May 9, 2023
8b39a99
Rename permissions to SGM_USER instead of SGM_NOTEBOOK
dlpzx May 9, 2023
5d817cb
More services and methods - presigned url...
dlpzx May 9, 2023
a6627f2
Completed services-db-resolvers. Renamed to naming convention sagemak…
dlpzx May 10, 2023
cf1c6c5
Linting
dlpzx May 10, 2023
48146b5
Cleanup, extract non class method in sagemaker client
dlpzx May 11, 2023
c8dc09f
Review comments
dlpzx May 16, 2023
b098e9f
Linting and minor errors
dlpzx May 17, 2023
4176228
Fixing migration scripts
dlpzx May 17, 2023
345baae
Finish SGM_STUDIO migration scripts, split migration scripts
dlpzx May 22, 2023
3c6cf8b
Typo in Makefile
dlpzx May 22, 2023
d96a3b3
Naming conventions
dlpzx May 22, 2023
4a163da
Renaming, fixed migration scripts and remove sagemaker_base
dlpzx May 22, 2023
4ce293a
mlstudio review
dlpzx May 25, 2023
b6def01
Rebasing modularization-main
dlpzx May 26, 2023
04c170f
Added environmentExtension for mlStudio
dlpzx May 26, 2023
a7b909d
Linting
dlpzx May 30, 2023
1da7496
Fixing init and stack imports mlstudio
dlpzx May 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ check-security: upgrade-pip install-backend install-cdkproxy

test:
export PYTHONPATH=./backend:/./tests && \
python -m pytest -v -ra tests/
python -m pytest -v -ra tests

coverage: upgrade-pip install-backend install-cdkproxy install-tests
export PYTHONPATH=./backend:/./tests && \
Expand Down
4 changes: 1 addition & 3 deletions backend/dataall/api/Objects/Environment/input_types.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@
gql.Argument('AwsAccountId', gql.NonNullableType(gql.String)),
gql.Argument('region', gql.NonNullableType(gql.String)),
gql.Argument('dashboardsEnabled', type=gql.Boolean),
gql.Argument('mlStudiosEnabled', type=gql.Boolean),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be deleted from schema as well

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should migrate mlStudiosEnabled to environment parameters. It's still used in Environment models, stacks and api.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is added in backend/migrations/versions/5fc49baecea4_add_enviromental_parameters.py:76 I thought of having all environment parameter migrations together, but we can discuss this or maybe I am missing something else

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Never mind, I got what you were referring to!

gql.Argument('pipelinesEnabled', type=gql.Boolean),
gql.Argument('warehousesEnabled', type=gql.Boolean),
gql.Argument('vpcId', gql.String),
Expand All @@ -53,7 +52,6 @@
gql.Argument('privateSubnetIds', gql.ArrayType(gql.String)),
gql.Argument('publicSubnetIds', gql.ArrayType(gql.String)),
gql.Argument('dashboardsEnabled', type=gql.Boolean),
gql.Argument('mlStudiosEnabled', type=gql.Boolean),
gql.Argument('pipelinesEnabled', type=gql.Boolean),
gql.Argument('warehousesEnabled', type=gql.Boolean),
gql.Argument('resourcePrefix', gql.String),
Expand Down Expand Up @@ -107,7 +105,7 @@ class EnvironmentSortField(GraphQLEnumMapper):
gql.Argument('term', gql.String),
gql.Argument('page', gql.Integer),
gql.Argument('pageSize', gql.Integer),
gql.Argument('uniqueShares', gql.Boolean)
gql.Argument('uniqueDatasets', gql.Boolean)
],
)

Expand Down
1 change: 1 addition & 0 deletions backend/dataall/api/Objects/Environment/resolvers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -522,6 +522,7 @@ def generate_environment_access_token(

def get_environment_stack(context: Context, source: models.Environment, **kwargs):
return stack_helper.get_stack_with_cfn_resources(
context=context,
targetUri=source.environmentUri,
environmentUri=source.environmentUri,
)
Expand Down
1 change: 0 additions & 1 deletion backend/dataall/api/Objects/Environment/schema.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,6 @@
),
gql.Field('validated', type=gql.Boolean),
gql.Field('dashboardsEnabled', type=gql.Boolean),
gql.Field('mlStudiosEnabled', type=gql.Boolean),
gql.Field('pipelinesEnabled', type=gql.Boolean),
gql.Field('warehousesEnabled', type=gql.Boolean),
gql.Field('roleCreated', type=gql.Boolean),
Expand Down
2 changes: 1 addition & 1 deletion backend/dataall/api/Objects/Glossary/registry.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,4 +65,4 @@ def reindex(cls, session, target_type: str, target_uri: str):
object_type="Dashboard",
model=models.Dashboard,
reindexer=DashboardIndexer
))
))
9 changes: 0 additions & 9 deletions backend/dataall/api/Objects/SagemakerStudio/__init__.py
View file
Open in desktop

This file was deleted.

27 changes: 0 additions & 27 deletions backend/dataall/api/Objects/SagemakerStudio/mutations.py
View file
Open in desktop

This file was deleted.

42 changes: 0 additions & 42 deletions backend/dataall/api/Objects/SagemakerStudio/queries.py
View file
Open in desktop

This file was deleted.

227 changes: 0 additions & 227 deletions backend/dataall/api/Objects/SagemakerStudio/resolvers.py
View file
Open in desktop

This file was deleted.

Loading