Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add neptune proxy #204

Merged
merged 4 commits into from
Oct 1, 2020
Merged

feat: add neptune proxy #204

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d882445
metadataservice: add neptune proxy
friendtocephalopods Sep 25, 2020
d61bd6b
Support roundtrip test segmentation by proxy.
friendtocephalopods Sep 28, 2020
a0e9014
feedback 2
friendtocephalopods Oct 1, 2020
d4e9fc0
feedback 3
friendtocephalopods Oct 1, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,3 +108,10 @@ We have Swagger documentation setup with OpenApi 3.0.2. This documentation is ge

## Code structure
Please visit [Code Structure](docs/structure.md) to read how different modules are structured in Amundsen Metadata service.

## Roundtrip tests
Roundtrip tests are a new feature - by implementing the abstract_proxy_tests and some test setup endpoints in the base_proxy, you can validate your proxy code against the actual data store. These tests do not run by default, but can be run by passing the `--roundtrip-[proxy]` argument. Note this requires
a fully-configured backend to test against.
```bash
$ python -m pytest --roundtrip-neptune .
```
6 changes: 5 additions & 1 deletion metadata_service/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@
import os
from typing import List, Dict, Optional, Set # noqa: F401
from metadata_service.entity.badge import Badge
from amundsen_gremlin.config import (
LocalGremlinConfig
)

# PROXY configuration keys
PROXY_HOST = 'PROXY_HOST'
Expand Down Expand Up @@ -77,7 +80,8 @@ class Config:
PROGRAMMATIC_DESCRIPTIONS_EXCLUDE_FILTERS = [] # type: list


class LocalConfig(Config):
# NB: If you're using the gremlin proxy, the appropriate GremlinConfig must be added to any other configs
class LocalConfig(LocalGremlinConfig, Config):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is subclass GremlinConfig for testing purpose? If that's the case, could we do sth like https://github.com/amundsen-io/amundsenfrontendlibrary/blob/ccfd2d6b82957fef347e956b243e4048c191fc0d/amundsen_application/config.py#L136 which create a test config in the file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, no, it inherits GremlinConfig so we don't need to litter the metadata service and databuilder config files with our gremlin endpoint information. The alternative is duplicating our gremlin-specific config in 3 different repos, which is unpleasant.

DEBUG = True
TESTING = False
LOG_LEVEL = 'DEBUG'
Expand Down
6 changes: 3 additions & 3 deletions metadata_service/proxy/atlas_proxy.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

from amundsen_common.models.dashboard import DashboardSummary
from amundsen_common.models.popular_table import PopularTable
from amundsen_common.models.table import Column, Statistics, Table, Tag, User, Reader, \
from amundsen_common.models.table import Column, Stat, Table, Tag, User, Reader, \
ProgrammaticDescription, ResourceReport
from amundsen_common.models.user import User as UserEntity
from atlasclient.client import Atlas
Expand Down Expand Up @@ -308,7 +308,7 @@ def _serialize_columns(self, *, entity: EntityUniqueAttribute) -> \
Union[List[Column], List]:
"""
Helper function to fetch the columns from entity and serialize them
using Column and Statistics model.
using Column and Stat model.
:param entity: EntityUniqueAttribute object,
along with relationshipAttributes
:return: A list of Column objects, if there are any columns available,
Expand Down Expand Up @@ -348,7 +348,7 @@ def _serialize_columns(self, *, entity: EntityUniqueAttribute) -> \
end_epoch = stats_attrs.get('end_epoch')

statistics.append(
Statistics(
Stat(
friendtocephalopods marked this conversation as resolved.
Show resolved Hide resolved
stat_type=stat_type,
stat_val=stat_val,
start_epoch=start_epoch,
Expand Down
303 changes: 0 additions & 303 deletions metadata_service/proxy/aws4authwebsocket/transport.py
View file
Open in desktop

This file was deleted.

6 changes: 3 additions & 3 deletions metadata_service/proxy/base_proxy.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

from amundsen_common.models.popular_table import PopularTable
from amundsen_common.models.table import Table
from amundsen_common.models.user import User as UserEntity
from amundsen_common.models.user import User
from amundsen_common.models.dashboard import DashboardSummary

from metadata_service.entity.dashboard_detail import DashboardDetail as DashboardDetailEntity
Expand All @@ -22,11 +22,11 @@ class BaseProxy(metaclass=ABCMeta):
"""

@abstractmethod
def get_user(self, *, id: str) -> Union[UserEntity, None]:
def get_user(self, *, id: str) -> Union[User, None]:
pass

@abstractmethod
def get_users(self) -> List[UserEntity]:
def get_users(self) -> List[User]:
pass

@abstractmethod
Expand Down
Loading