parent f6b21b3

author verdan <verdan.mahmood@gmail.com> 1620122095 +0200 committer verdan <verdan.mahmood@gmail.com> 1620776994 +0200 initial commit Fixes the lineage query, updates the common model more updates updates the look and feel removes commented code Signed-off-by: verdan <verdan.mahmood@gmail.com> passes the paramters to backend adds the zooming feature code refactor, addresses review comments Signed-off-by: verdan <verdan.mahmood@gmail.com> docs: update faq.md (#1038) Signed-off-by: mgorsk1 <gorskimariusz13@gmail.com> Modified methods to throw NOTFOUNDEXCEPTION instead RuntimeEror in neo4j_proxy (metadata_service) (#1028) Signed-off-by: aparna-cimpress <aparna.singhal@cimpress.com> perf: Update ModeDashboardLastModifiedTimestampExtractor to use Mode discovery api (#1040) * update ModeDashboardLastModifiedTimestampExtractor to use Mode discovery api Signed-off-by: Junda Yang <youngyjd@gmail.com> * add empty line at the end of the file Signed-off-by: Junda Yang <youngyjd@gmail.com> * update comment Signed-off-by: Junda Yang <youngyjd@gmail.com> * fix lint Signed-off-by: Junda Yang <youngyjd@gmail.com> * fix import Signed-off-by: Junda Yang <youngyjd@gmail.com> ci: fix metadata ci config (#1039) Signed-off-by: Dorian Johnson <2020@dorianj.net> perf: update ModeDashboardLastSuccessfulExecutionExtractor to discovery api (#1042) * update ModeDashboardLastSuccessfulExecutionExtractor to discovery api Signed-off-by: Junda Yang <youngyjd@gmail.com> * lint Signed-off-by: Junda Yang <youngyjd@gmail.com> * remove unused import Signed-off-by: Junda Yang <youngyjd@gmail.com> perf: Update ModeDashboardOwner extractor to use Mode discovery api (#1043) * update ModeDashboardOwnerExtractor to discovery api Signed-off-by: Junda Yang <youngyjd@gmail.com> * update import Signed-off-by: Junda Yang <youngyjd@gmail.com> update ModeDashboardQueryExtractor to discovery api (#1044) Signed-off-by: Junda Yang <youngyjd@gmail.com> chore: fix databuilder dep typo (#1046) Signed-off-by: feng-tao <fengtao04@gmail.com> build(deps): bump py from 1.5.3 to 1.10.0 in /databuilder (#1049) chore: Update ModeDashboardLastModifiedTimestampExtractor typing (#1048) update Mode dashboard extractor tests (#1047) fix: Temporarily fix potential 404 in mode dashboard executions extractor (#1051) * fix ModeDashboardExecutionsExtractor Signed-off-by: Junda Yang <youngyjd@gmail.com> * nit Signed-off-by: Junda Yang <youngyjd@gmail.com> test: refactor: update test and directory of ModeDashboardChartsBatchExtractor (#1052) Temporarily fix potential 404 in ModeDashboardUsageExtractor (#1053) feat: Add dbt extractor for table lineage + metadata (#1015) * converted dbt extractor to mono repo Signed-off-by: Grant Seward <grant@stemma.ai> * Flake8 formatting Signed-off-by: Grant Seward <grant@stemma.ai> * Updates per feedback from Convoy Signed-off-by: Grant Seward <grant@stemma.ai> * Commented sample logging Signed-off-by: Grant Seward <grant@stemma.ai> * Added dbt to main amundsen README Signed-off-by: Grant Seward <grant@stemma.ai> fix bug in rest_api_query (#1056) Signed-off-by: Junda Yang <youngyjd@gmail.com> feat: Add merge query results functionality to rest_api_query (#1058) * Add merge query results functionality Signed-off-by: Junda Yang <youngyjd@gmail.com> * fix mypy Signed-off-by: Junda Yang <youngyjd@gmail.com> * nit Signed-off-by: Junda Yang <youngyjd@gmail.com> chore: fix gh page build workflow (#1062) Signed-off-by: feng-tao <fengtao04@gmail.com> chore: add mailmap to associate the right user (#1061) Signed-off-by: feng-tao <fengtao04@gmail.com> fix: Update ModeDashboardExtractor to Mode discovery api (#1063) * Update ModeDashboardExtractor to Mode discovery api Signed-off-by: Junda Yang <youngyjd@gmail.com> * nit Signed-off-by: Junda Yang <youngyjd@gmail.com> chore: delete legacy ModeDashboardChartsExtractor (#1064) * remove-legacy-charts-extractor Signed-off-by: Junda Yang <youngyjd@gmail.com> * nit Signed-off-by: Junda Yang <youngyjd@gmail.com> bump up databuilder version (#1065) Signed-off-by: Junda Yang <youngyjd@gmail.com> fix in another PR Signed-off-by: verdan <verdan.mahmood@gmail.com> makes Verdan and Dorian the code owners (#1081) Signed-off-by: verdan <verdan.mahmood@gmail.com> Adds a new "parent" field in lineage model (#1080) * adds 'parent' field to help build the graph Signed-off-by: verdan <verdan.mahmood@gmail.com> * make the default to None Signed-off-by: verdan <verdan.mahmood@gmail.com> deploy_release: on release, publish to pypi/docker (#1079) Given a release with a title of `<module_name> <version_number>`, this will push that release to PyPi and docker. Note that this does not attempt to create the changelog or any of that stuff. That will need to be done by a separate action. This creates a nice modularity boundary, in that this action doesn't care where the release came from (manual or another automated step) Signed-off-by: Dorian Johnson <2020@dorianj.net> Fixes the test cases Signed-off-by: verdan <verdan.mahmood@gmail.com> Removes extra functionality Signed-off-by: verdan <verdan.mahmood@gmail.com> removes redundant flag Signed-off-by: verdan <verdan.mahmood@gmail.com>
amundsen-io · May 11, 2021 · ea8ed5a · ea8ed5a
1 parent f6b21b3
commit ea8ed5a
Show file tree

Hide file tree

Showing 74 changed files with 2,601 additions and 807 deletions.
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -13,9 +13,9 @@
 
 /amundsen-kube-helm/ @feng-tao @jornh @javamonkey79
 
-common/*.py        @feng-tao @jinhyukchang @allisonsuarez @verdan @bolkedebruin @mgorsk1
-databuilder/*.py   @feng-tao @jinhyukchang @allisonsuarez @dikshathakur3119
+common/*.py        @feng-tao @jinhyukchang @allisonsuarez @verdan @bolkedebruin @mgorsk1 @dorianj
+databuilder/*.py   @feng-tao @jinhyukchang @allisonsuarez @dikshathakur3119 @dorianj @verdan
 frontend/amundsen_application/static/    @Golodhros @ttannis @allisonsuarez @dikshathakur3119 @feng-tao @dorianj
-frontend/*.py    @ttannis @feng-tao @dikshathakur3119 @allisonsuarez @dorianj
+frontend/*.py    @ttannis @feng-tao @dikshathakur3119 @allisonsuarez @dorianj @verdan
 metadata/*.py    @feng-tao @jinhyukchang @allisonsuarez @dikshathakur3119 @verdan @bolkedebruin @mgorsk1
 search/*.py      @feng-tao @jinhyukchang @allisonsuarez @dikshathakur3119 @verdan @bolkedebruin @mgorsk1
diff --git a/.github/workflows/deploy_docs.yml b/.github/workflows/deploy_docs.yml
@@ -2,14 +2,14 @@ name: Publish docs via GitHub Pages
 on:
   push:
     branches:
-      - master
+      - main
 
 jobs:
   build:
     name: Deploy docs
     runs-on: ubuntu-latest
     steps:
-      - name: Checkout master
+      - name: Checkout main
         uses: actions/checkout@v1
       - name: Checkout submodules using a PAT
         run: |

diff --git a/.github/workflows/publish_release.yml → .github/workflows/deploy_release.yml b/.github/workflows/publish_release.yml → .github/workflows/deploy_release.yml
@@ -1,47 +1,40 @@
-name: Publish Release
+name: Publish Release to PyPi
 on:
-  push:
-    branches:
-      - master
-    paths:
-      # triggered when version is bumped
-      - "setup.py"
-      - "CHANGELOG.md"
-jobs:
   release:
+    types: [published]
+jobs:
+  gather-info:
     runs-on: ubuntu-latest
+    steps:
+      - name: Get module for release
+        id: module_folder
+        run: echo "::set-output name=module_folder::$(awk '{print $1}' << "${{ github.event.release.title }}")"
+    outputs:
+      module_folder: ${{ steps.module_folder.outputs.module_folder }}
+  push-to-pypi:
+    runs-on: ubuntu-latest
+    needs: gather-info
     steps:
       - name: Checkout code
         uses: actions/checkout@v2
       - name: Setup python 3.6
         uses: actions/setup-python@v1
         with:
           python-version: 3.6
-      - name: Get version
-        id: vers
-        run: |
-          pip install python-semantic-release
-          echo ::set-output name=version::$(semantic-release print-version --current)
-      - name: Create release
-        id: create_release
-        uses: actions/create-release@v1
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-        with:
-          tag_name: ${{ steps.vers.outputs.version }}
-          release_name: New Features & Bug Fixes
-          body_path: ./CHANGELOG.md
       - name: Add wheel dependency
         run: pip install wheel
       - name: Generate dist
         run: python setup.py sdist bdist_wheel
+        working-directory: ${{ needs.gather-info.outputs.module_folder }}
       - name: Publish to PyPI
         uses: pypa/gh-action-pypi-publish@master
         with:
           user: __token__
           password: ${{ secrets.pypi_password }}
+          packages_dir: ${{ needs.gather-info.outputs.module_folder }}/dist
   build-and-publish-docker-image:
     name: Build and publish docker image
+    needs: gather-info
     runs-on: ubuntu-18.04
     steps:
       - name: Checkout
@@ -51,5 +44,7 @@ jobs:
       - name: Publish to Registry for latest
         if: success()
         run: make build-push-image-latest
+        working-directory: ${{ needs.gather-info.outputs.module_folder }}
       - name: Publish to Registry for version
         run: make build-push-image-version
+        working-directory: ${{ needs.gather-info.outputs.module_folder }}
diff --git a/.github/workflows/metadata_pull_request.yml b/.github/workflows/metadata_pull_request.yml
@@ -24,4 +24,5 @@ jobs:
         working-directory: ./metadata
       - name: Codecov
         uses: codecov/codecov-action@v1
-        working-directory: ./metadata
+        with:
+          directory: ./metadata
diff --git a/.mailmap b/.mailmap
@@ -0,0 +1,170 @@
+ Aarni Alasaarela <aarni.alasaarela@gmail.com>
+ Adam Boscarino <aboscarino@devoted.com>
+ Adam Boscarino <ajboscarino@gmail.com>
+ Adam Weiss <amweiss@users.noreply.github.com>
+ Alagappan <Alagappan@users.noreply.github.com>
+ Alex Kompos <alexkompos@gmail.com>
+ Alex Levene <37639249+alevene@users.noreply.github.com>
+ Allison Doami <allison.ox@gmail.com>
+ Allison Suarez Miranda <22477579+allisonsuarez@users.noreply.github.com>
+ Alyssa <alran@users.noreply.github.com>
+ Anders Hafreager <andershaf@gmail.com>
+ Andrew Ciambrone <andrjc4@vt.edu>
+ Anna Elisabeth <anna.e.boyd@gmail.com>
+ Anurag870 <f2005870@gmail.com>
+ Arjun Landes <arjunlandes@gmail.com>
+ ArthurEgide <arthuregide@gmail.com>
+ Ashutosh Sanzgiri <sanzgiri@gmail.com>
+ Austin Gibbons <austin@periscope.io>
+ Ayush Chauhan <10010065+ayushchauhan0811@users.noreply.github.com>
+ Ayush Chauhan <57655135+ayush-san@users.noreply.github.com>
+ Ben Rifkind <benrifkind@users.noreply.github.com>
+ Bongso, Suriyanto <1008875+suriyanto@users.noreply.github.com>
+ Bruno Nirello <30416677+bruno-nirello@users.noreply.github.com>
+ Bryan Fox <39674+bryfox@users.noreply.github.com>
+ Carter Landis <c@carterlandis.com>
+ Craig Rosie <craigrosie@users.noreply.github.com>
+ Cristiano Perez <cristianoperez@gmail.com>
+ Damon <damon09273@gmail.com>
+ Daniel <dwon@lyft.com>
+ Daniel Mateus Pires <dmateusp@gmail.com>
+ Daniel Mateus Pires <dpires@earnestresearch.com>
+ Daniel Won <dwon@lyft.com>
+ Daren Sin <darensin01@gmail.com>
+ Dave Cameron <dave.bc+github@gmail.com>
+ Dave Cameron <dcameron@digitalocean.com>
+ David Manukian <53464203+davidmanukian@users.noreply.github.com>
+ Derek <dschaller@users.noreply.github.com>
+ Diksha Thakur <44571635+dikshathakur3119@users.noreply.github.com>
+ Dmitriy Kunitskiy <dkunitskiy@lyft.com>
+ Dorian Johnson <2019@dorianj.net>
+ Dorian Johnson <2020@dorianj.net>
+ Florent Bédécarrats <f.bedecarrats@gmail.com>
+ Gee <gshock94@gmail.com>
+ Gerard Toonstra <gtoonstra@users.noreply.github.com>
+ Guido Schmutz <guido.schmutz@trivadis.com>
+ Hugo Hobson <hugo.hobson@dazn.com>
+ Itay Levy <levyitay@gmail.com>
+ Jack Leow <jackgene@gmail.com>
+ Jacob Kim <me@jacobkim.io>
+ Jacob Kim <me@thejacobkim.com>
+ Jacob Scherffenberg <scherffenberg91@gmail.com>
+ Jakub Hettler <jakub.hettler@gmail.com>
+ James Davidheiser <jdavidheiser@users.noreply.github.com>
+ Jan Kyri <thenaturalist@users.noreply.github.com>
+ Jin Hyuk Chang <jinchang@lyft.com>
+ Jin Hyuk Chang <jinhyukchang.work@gmail.com>
+ Joao Correia <joao.correia@gmail.com>
+ John Cheng <ckljohn@gmail.com>
+ Jonas Brunsgaard <jonas.brunsgaard@gmail.com>
+ Jonathan Hehir <jonhehir@users.noreply.github.com>
+ Jonathan Vieira <joonathan.vieira@gmail.com>
+ Joseph Atkins-Turkish <jatkins-turkish@brex.com>
+ Joseph Atkins-Turkish <spacerat3004@gmail.com>
+ Josh Howard <joshthoward@gmail.com>
+ Josh Stanfield <p5k6@yahoo.com>
+ Joshua Hoskins <hoskins@squareup.com>
+ Junaid Effendi <junaidfnd@gmail.com>
+ Junda Yang <youngyjd@gmail.com>
+ Justin Kenny <63694689+JustinKenny@users.noreply.github.com>
+ Kent Murra <kmurra@gmail.com>
+ Lingkai Kong <lingkai.kong@databricks.com>
+ Louis Simoneau <simoneau.louis@gmail.com>
+ Luis Garcés-Erice <lga@zurich.ibm.com>
+ Lukas Tarasevicius <ltarasevicius@gmail.com>
+ Luke Lowery <lukelowery@users.noreply.github.com>
+ Lyman Gillispie <l@lygi.me>
+ Madison Bowden <52679885+madison-ookla@users.noreply.github.com>
+ Madison Swain-Bowden <52679885+madison-ookla@users.noreply.github.com>
+ Magnus Runesson <M.Runesson@gmail.com>
+ Marcos Iglesias <190833+Golodhros@users.noreply.github.com>
+ Mariusz Strzelecki <mariusz.strzelecki@getindata.com>
+ Mariusz Strzelecki <szczeles@gmail.com>
+ Mark Grover <grover.markgrover@gmail.com>
+ Mark Grover <mark@apache.org>
+ Markus Thurner <thoean@users.noreply.github.com>
+ Martin Traverso <mtraverso@gmail.com>
+ Mikhail Ivanov <mikhail.ivanov8@gmail.com>
+ Nanne <nanne.wielinga@ing.com>
+ Nanne <nanne@mycel.nl>
+ Nathan Lawrence <54294432+nathanlawrence-asana@users.noreply.github.com>
+ Neil Shah <shahneil88@gmail.com>
+ Nipun Agarwal <ibnipun10@gmail.com>
+ Norman Cheng <desource1990@yahoo.com>
+ Oliver Marshall <Akeboshiwind@users.noreply.github.com>
+ Pablo Torre <xctico@gmail.com>
+ Palash Das <palashkok3@gmail.com>
+ PaschalisDim <48329279+PaschalisDim@users.noreply.github.com>
+ Pathuri Sai Harish <saiharish97@gmail.com>
+ Paul Bergeron <paul.d.bergeron@gmail.com>
+ Pedro Gonçalves Rossi Rodrigues <pgrr@cin.ufpe.br>
+ Philippe Mizrahi <39501569+philippemizrahi@users.noreply.github.com>
+ Radhakrishna Pemmasani <pemmasanikrish@gmail.com>
+ Rafael A. Mejia <rafael@dataminded.net>
+ Robert Yi <rosiny@gmail.com>
+ Ryan Lieu <rlieu@lyft.com>
+ Sahithi Reddy Velma <46659774+sahithi03@users.noreply.github.com>
+ Sarah Harvey <worldwise001@users.noreply.github.com>
+ Sarthak Killedar <sarthak.killedar@gmail.com>
+ Saulius Grigaliunas <saulius@ninja.lt>
+ Shaun Elliott <javamonkey79@gmail.com>
+ Shenghu Yang <syang@lyft.com>
+ Shuichiro MAKIGAKI <shuichiro.makigaki@gmail.com>
+ Simon Brugman <sbrugman@users.noreply.github.com>
+ Stacey Watro <watro.stacey@gmail.com>
+ Tamika Tannis <ttannis@alum.mit.edu>
+ Tamika Tannis <ttannis@lyft.com>
+ Tao Feng <fengtao04@gmail.com>
+ Tao Feng <fengtao04@gmail.com> <tfeng@lyft.com>
+ Tao feng <fengtao04@gmail.com> <tfeng@lyft.com>
+ Travis Roesner <teqonix@gmail.com>
+ Van-Duyet Le <5009534+duyetdev@users.noreply.github.com>
+ Verdan Mahmood <verdan.mahmood@gmail.com>
+ Wonyeong Choi <ciwnyg0815@gmail.com>
+ Xuan <65048031+crazy-2020@users.noreply.github.com>
+ Yeshwanth Kumar <morpheyesh@gmail.com>
+ Youngil Choi <56865498+duddlf23@users.noreply.github.com>
+ Zack Wu <39970089+instazackwu@users.noreply.github.com>
+ abhinay04 <68522211+abhinay04@users.noreply.github.com>
+ allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>
+ aparna-cimpress <83464696+aparna-cimpress@users.noreply.github.com>
+ bluefa <chulyonga@gmail.com>
+ bogo96 <bogo96@icloud.com>
+ bolkedebruin <bolkedebruin@users.noreply.github.com>
+ christina stead <cstead@lyft.com>
+ christina stead <csteez@users.noreply.github.com>
+ damon09273@gmail.com <damon09273@gmail.com>
+ dechoma <dominik.choma@gmail.com>
+ dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
+ dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
+ dwarszawski <dwarszawski@users.noreply.github.com>
+ friendtocephalopods <52580251+friendtocephalopods@users.noreply.github.com>
+ giomerlin <56041379+giomerlin@users.noreply.github.com>
+ github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
+ hoskins <52580251+friendtocephalopods@users.noreply.github.com>
+ iamtodor <todor.ilya@gmail.com>
+ joeybaer <35610156+joeybaer@users.noreply.github.com>
+ jonhehir <jonhehir@users.noreply.github.com>
+ jornh <jornhansen@gmail.com>
+ lelandtran <lelandtran@ymail.com>
+ lexisantoro <43476866+lexisantoro@users.noreply.github.com>
+ louis <1160507+naiaden@users.noreply.github.com>
+ lyft-metaservice-3 <37090125+lyft-metaservice-3@users.noreply.github.com>
+ mborukhava <43936420+mborukhava@users.noreply.github.com>
+ mgorsk1 <gorskimariusz13@gmail.com>
+ minsu <o.minsu@gmail.com>
+ rogertangcn <rogertangcn@users.noreply.github.com>
+ rosejcday <16930240+rosejcday@users.noreply.github.com>
+ salilk957 <49253336+salilk957@users.noreply.github.com>
+ samshuster <sshuster@edmunds.com>
+ sewardgw <Sewardgw@gmail.com>
+ sewardgw <grant@stemma.ai>
+ sinkuladis <sink.vlad@gmail.com>
+ sshuster <sshuster@edmunds.com>
+ sterlins <stacy.sterling@gmail.com>
+ tianr.zhou <61992569+tianruzhou-db@users.noreply.github.com>
+ tianru zhou <61560998+UnbalancedShard@users.noreply.github.com>
+ verdan <verdan.mahmood@gmail.com>
+ zhmin <myzhmin@gmail.com>
+ Đặng Minh Dũng <dungdm93@live.com>
+ 之东 <0x4ec7@gmail.com>
diff --git a/README.md b/README.md
@@ -118,6 +118,7 @@ Please visit [Architecture](./docs/architecture.md#architecture) for Amundsen ar
 - [Apache Druid](https://druid.apache.org/)
 - [Apache Hive](https://hive.apache.org/)
 - CSV
+- [dbt](https://www.getdbt.com/)
 - [Delta Lake](https://delta.io/)
 - [Google BigQuery](https://cloud.google.com/bigquery)
 - [IBM DB2](https://www.ibm.com/analytics/db2)

diff --git a/common/amundsen_common/models/lineage.py b/common/amundsen_common/models/lineage.py
@@ -16,6 +16,7 @@ class LineageItem:
     source: str  # database this resource is from
     badges: Optional[List[Badge]] = None
     usage: Optional[int] = None  # statistic to sort lineage items by
+    parent: Optional[str] = None  # key of the parent entity, used to create the relationships in graph
 
 
 class LineageItemSchema(AttrsSchema):

diff --git a/common/setup.py b/common/setup.py
@@ -3,7 +3,7 @@
 
 from setuptools import find_packages, setup
 
-__version__ = '0.10.0'
+__version__ = '0.10.1'
 
 setup(
     name='amundsen-common',