Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

modularization: Datasets modularization pt.5 #442

Merged
merged 124 commits into from
May 10, 2023

Commits on Apr 11, 2023

  1. Configuration menu
    Copy the full SHA
    3a5e0de View commit details
    Browse the repository at this point in the history
  2. Refactoring of datasets

    Moved dataset table column to modules
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    a50a02f View commit details
    Browse the repository at this point in the history
  3. Refactoring of datasets

    Renamed table column to the python's convention format
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    be14986 View commit details
    Browse the repository at this point in the history
  4. Refactoring of datasets

    Added dataset module to config.json
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    06f82ad View commit details
    Browse the repository at this point in the history
  5. Fixed leftover in loader

    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    38145ae View commit details
    Browse the repository at this point in the history
  6. Dataset refactoring

    Moved database table service
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    f0e146a View commit details
    Browse the repository at this point in the history
  7. Dataset refactoring

    Renamed DatasetTable to DatasetTableService to avoid collisions with models.DatasetTable
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    b039163 View commit details
    Browse the repository at this point in the history
  8. Dataset refactoring

    Moved DatasetTableColumn to modules
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    b7922ed View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    1771bca View commit details
    Browse the repository at this point in the history
  10. Renamed tasks to handlers

    Currently, only async handlers require dedicated loading. Long-running tasks (scheduled tasks) might not need to have a dedicated loading mode
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    3d1603f View commit details
    Browse the repository at this point in the history
  11. Dataset refactoring

    Extracted code from glue to glue_column_handler
    Added handlers importing for datasets
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    fb6b515 View commit details
    Browse the repository at this point in the history
  12. Dataset refactoring

    Extracted the code for dataset table handler
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    e3596a5 View commit details
    Browse the repository at this point in the history
  13. Dataset refactoring

    Extracted the long-running task for datasets
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    3af2ecf View commit details
    Browse the repository at this point in the history
  14. Dataset refactoring

    Extracted the subscription service into datasets
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    1a063b2 View commit details
    Browse the repository at this point in the history
  15. Dataset refactoring

    Extracted the handler to get table columns
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    b733714 View commit details
    Browse the repository at this point in the history
  16. Extracted feed registry

    Needed for migration for modules
    nikpodsh committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    2a4e2e0 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    c15d090 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2023

  1. Dataset refactoring

    Fixed tests and added new for dataset module
    nikpodsh committed Apr 12, 2023
    Configuration menu
    Copy the full SHA
    052a2b1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d984483 View commit details
    Browse the repository at this point in the history
  3. Split model registry into feed and glossaries

    Glossaries had different target types and had to be treated differently
    nikpodsh committed Apr 12, 2023
    Configuration menu
    Copy the full SHA
    dc0c935 View commit details
    Browse the repository at this point in the history
  4. Abstraction for glossaries

    Created API for glossaries to use modularization
    nikpodsh committed Apr 12, 2023
    Configuration menu
    Copy the full SHA
    727e353 View commit details
    Browse the repository at this point in the history
  5. Fixed leftovers

    nikpodsh committed Apr 12, 2023
    Configuration menu
    Copy the full SHA
    49fbb41 View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2023

  1. Datasets refactoring

    Added and fixed tests
    nikpodsh committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    7d029e7 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    be527eb View commit details
    Browse the repository at this point in the history
  3. Changed Feed type registration mechanism

    Moved FeedRegistry to gql since it's more appropriate place for this
    Started using registry to provide types
    Renaming and small fixes
    nikpodsh committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    3daf2aa View commit details
    Browse the repository at this point in the history
  4. Added TODO for future refactoring

    Solve circular dependecy for redshift. It should go away after the migration of redshift
    nikpodsh committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    db3bfd3 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    13b6e92 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    144dfea View commit details
    Browse the repository at this point in the history
  7. No need for Utils yet

    nikpodsh committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    d43b9b3 View commit details
    Browse the repository at this point in the history
  8. Fixed linting

    nikpodsh committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    39b244c View commit details
    Browse the repository at this point in the history

Commits on Apr 14, 2023

  1. Datasets refactoring

    Moving datasets profiling to datasets modules
    nikpodsh committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    cb3800a View commit details
    Browse the repository at this point in the history
  2. Datasets refactoring

    Renaming profiling
    nikpodsh committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    dd8e597 View commit details
    Browse the repository at this point in the history
  3. Datasets refactoring

    Renaming table_column_model to models to easier import other models
    nikpodsh committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    8ca7bea View commit details
    Browse the repository at this point in the history
  4. Datasets refactoring

    Moving DatasetProfilingRun model
    nikpodsh committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    e36ab3b View commit details
    Browse the repository at this point in the history
  5. Datasets refactoring

    Moving dataset profiling service and renaming it
    nikpodsh committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    31720c2 View commit details
    Browse the repository at this point in the history
  6. Datasets refactoring

    Extracted glue_profiling_handler
    nikpodsh committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    8a907df View commit details
    Browse the repository at this point in the history
  7. Datasets refactoring

    Deleted DatasetTableProfilingJob since could not find any usage of it
    nikpodsh committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    561da72 View commit details
    Browse the repository at this point in the history

Commits on Apr 17, 2023

  1. Datasets refactoring

    Moved dataset storage location into modules
    nikpodsh committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    73c8150 View commit details
    Browse the repository at this point in the history
  2. Datasets refactoring

    Renamed dataset storage location
    nikpodsh committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    56a3610 View commit details
    Browse the repository at this point in the history
  3. Datasets refactoring

    Returned the name to model after renaming the service
    nikpodsh committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    47a38cc View commit details
    Browse the repository at this point in the history
  4. Datasets refactoring

    Moved DatasetStorageLocation into modules
    nikpodsh committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    dbb5517 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    a3def13 View commit details
    Browse the repository at this point in the history
  6. Datasets refactoring

    Moved DatasetStorageLocation into dataset services
    nikpodsh committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    b256678 View commit details
    Browse the repository at this point in the history
  7. Datasets refactoring

    Extracted s3_location_handler
    nikpodsh committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    66b5ddb View commit details
    Browse the repository at this point in the history
  8. Datasets refactoring

    Moved the dataset stack into modules
    nikpodsh committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    352d824 View commit details
    Browse the repository at this point in the history
  9. Datasets refactoring

    Moved indexing into GlossaryRegistry
    nikpodsh committed Apr 17, 2023
    Configuration menu
    Copy the full SHA
    9934a9c View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2023

  1. Datasets refactoring

    Removed dead code
    nikpodsh committed Apr 18, 2023
    Configuration menu
    Copy the full SHA
    228c175 View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2023

  1. Datasets refactoring

    Extracted dataset share service
    nikpodsh committed Apr 19, 2023
    Configuration menu
    Copy the full SHA
    263d10c View commit details
    Browse the repository at this point in the history
  2. Datasets refactoring

    Solved broken reference
    nikpodsh committed Apr 19, 2023
    Configuration menu
    Copy the full SHA
    417e6e5 View commit details
    Browse the repository at this point in the history
  3. Introduced Indexers

    nikpodsh committed Apr 19, 2023
    Configuration menu
    Copy the full SHA
    7aaff5b View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    4e31b99 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    b772812 View commit details
    Browse the repository at this point in the history

Commits on Apr 20, 2023

  1. Configuration menu
    Copy the full SHA
    cd798e2 View commit details
    Browse the repository at this point in the history
  2. Renamed the service

    nikpodsh committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    b0e6a62 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    27c6d79 View commit details
    Browse the repository at this point in the history
  4. Created a dataset repository.

    There is a few of circular imports. It's a first attempt to solve it
    nikpodsh committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    0e730ac View commit details
    Browse the repository at this point in the history
  5. Moved DatasetTableIndexer

    nikpodsh committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    9ac7964 View commit details
    Browse the repository at this point in the history
  6. Fixed test mocking

    nikpodsh committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    a1825ba View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    d295485 View commit details
    Browse the repository at this point in the history
  8. Removed not used alarms

    nikpodsh committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    005a5e7 View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2023

  1. Configuration menu
    Copy the full SHA
    0fd7c02 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7030c82 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    ba45ca5 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    dc8ff72 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    c99ed58 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2023

  1. Migrated DatasetService

    nikpodsh committed Apr 24, 2023
    Configuration menu
    Copy the full SHA
    9ddbaf3 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ad845e7 View commit details
    Browse the repository at this point in the history
  3. Resolved code conflict

    nikpodsh committed Apr 24, 2023
    Configuration menu
    Copy the full SHA
    2ac3ae7 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    32be3ee View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    f95566c View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    d05196f View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2023

  1. Moved a SNS dataset handler

    nikpodsh committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    8b2accb View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    fca218f View commit details
    Browse the repository at this point in the history
  3. Merge branch 'datasets-mod-part3' into datasets-mod-part4

    # Conflicts:
    #	backend/dataall/modules/datasets/handlers/glue_column_handler.py
    #	tests/api/test_dataset_table.py
    nikpodsh committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    ef98aa0 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    ebac530 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    2cd19d2 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    741238d View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    81e7646 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    f5752e9 View commit details
    Browse the repository at this point in the history
  9. Refactored dataset handlers

    nikpodsh committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    ce0c4e0 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    50ff85a View commit details
    Browse the repository at this point in the history
  11. Got rid of datasets in votes

    nikpodsh committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    c4e9079 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2023

  1. Configuration menu
    Copy the full SHA
    1227ca3 View commit details
    Browse the repository at this point in the history
  2. Extracted dataset alarms

    nikpodsh committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    09f7de9 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d446ff0 View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2023

  1. Configuration menu
    Copy the full SHA
    1db0133 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    8d8d952 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d6ec387 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    788e35f View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    584a04b View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    4dd3a2b View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    95cbf81 View commit details
    Browse the repository at this point in the history
  8. Extracted data policy for dataset

    Did the same trick that was done for ServicePolicy
    nikpodsh committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    781bb08 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    1ec1235 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    9f2748e View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2023

  1. Configuration menu
    Copy the full SHA
    86f4529 View commit details
    Browse the repository at this point in the history
  2. Removed unused method

    nikpodsh committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    f4b1f67 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a52c145 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    19cc9aa View commit details
    Browse the repository at this point in the history
  5. Renamed files

    nikpodsh committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    cd218e2 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    3e24acb View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    b5eb774 View commit details
    Browse the repository at this point in the history

Commits on May 2, 2023

  1. Merge remote-tracking branch 'upstream/modularization-main' into data…

    …sets-mod-part3
    
    # Conflicts:
    #	backend/dataall/db/api/__init__.py
    #	backend/dataall/modules/datasets/__init__.py
    #	backend/dataall/modules/datasets/api/__init__.py
    #	backend/dataall/modules/datasets/db/models.py
    nikpodsh committed May 2, 2023
    Configuration menu
    Copy the full SHA
    2cd14e0 View commit details
    Browse the repository at this point in the history
  2. Merge branch 'datasets-mod-part3' into datasets-mod-part4

    # Conflicts:
    #	backend/dataall/modules/datasets/db/models.py
    nikpodsh committed May 2, 2023
    Configuration menu
    Copy the full SHA
    a3d9676 View commit details
    Browse the repository at this point in the history
  3. Merge branch 'datasets-mod-part4' into datasets-mod-part5

    # Conflicts:
    #	backend/dataall/modules/datasets/db/models.py
    nikpodsh committed May 2, 2023
    Configuration menu
    Copy the full SHA
    c0aa53a View commit details
    Browse the repository at this point in the history

Commits on May 3, 2023

  1. Fixed all tests

    nikpodsh committed May 3, 2023
    Configuration menu
    Copy the full SHA
    3ae1eca View commit details
    Browse the repository at this point in the history

Commits on May 4, 2023

  1. Review remarks

    nikpodsh committed May 4, 2023
    Configuration menu
    Copy the full SHA
    f382a68 View commit details
    Browse the repository at this point in the history
  2. Added TODO

    nikpodsh committed May 4, 2023
    Configuration menu
    Copy the full SHA
    532ff0d View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    afcae66 View commit details
    Browse the repository at this point in the history
  4. Merge branch 'datasets-mod-part4' into datasets-mod-part5

    # Conflicts:
    #	backend/dataall/modules/datasets/handlers/s3_location_handler.py
    #	backend/dataall/modules/datasets/indexers/dataset_indexer.py
    nikpodsh committed May 4, 2023
    Configuration menu
    Copy the full SHA
    abc301a View commit details
    Browse the repository at this point in the history
  5. After the merge of part3

    nikpodsh committed May 4, 2023
    Configuration menu
    Copy the full SHA
    61d4eb3 View commit details
    Browse the repository at this point in the history
  6. Merge branch 'datasets-mod-part4' into datasets-mod-part5

    # Conflicts:
    #	backend/dataall/modules/datasets/api/table/resolvers.py
    #	tests/searchproxy/test_indexers.py
    nikpodsh committed May 4, 2023
    Configuration menu
    Copy the full SHA
    1820bf2 View commit details
    Browse the repository at this point in the history
  7. Merge commit

    nikpodsh committed May 4, 2023
    Configuration menu
    Copy the full SHA
    9718e81 View commit details
    Browse the repository at this point in the history

Commits on May 5, 2023

  1. Fixed all tests

    nikpodsh committed May 5, 2023
    Configuration menu
    Copy the full SHA
    c2ac71c View commit details
    Browse the repository at this point in the history
  2. Review remarks

    nikpodsh committed May 5, 2023
    Configuration menu
    Copy the full SHA
    082fec9 View commit details
    Browse the repository at this point in the history

Commits on May 8, 2023

  1. Review remarks

    nikpodsh committed May 8, 2023
    Configuration menu
    Copy the full SHA
    5c67ef4 View commit details
    Browse the repository at this point in the history
  2. Review remarks

    nikpodsh committed May 8, 2023
    Configuration menu
    Copy the full SHA
    57f1e1f View commit details
    Browse the repository at this point in the history
  3. Review remarks

    nikpodsh committed May 8, 2023
    Configuration menu
    Copy the full SHA
    1ceb50e View commit details
    Browse the repository at this point in the history
  4. Moved dataset constants

    nikpodsh committed May 8, 2023
    Configuration menu
    Copy the full SHA
    0a5170e View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    19377d6 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2023

  1. Resolved cyclic import

    nikpodsh committed May 9, 2023
    Configuration menu
    Copy the full SHA
    ae0c611 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a2e27a8 View commit details
    Browse the repository at this point in the history
  3. Renamed the method

    nikpodsh committed May 9, 2023
    Configuration menu
    Copy the full SHA
    732f726 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    90d6429 View commit details
    Browse the repository at this point in the history