feat: add certifiedby & certification details fields to the edit dataset columns fields #16454

pkdotson · 2021-08-25T23:54:55Z

SUMMARY

This Pr adds certifedby and certification details fields to the edit dataset columns and calculated columns fields.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Screen.Recording.2021-08-25.at.4.50.00.PM.mov

TESTING INSTRUCTIONS

Go to edit dataset and edit the certified by and certification details fields and saved modal. Modal should persist the fields on reload.

ADDITIONAL INFORMATION

[ x] Has associated issue: close: [editdataset]certify columns and calculated columns #16349
Required feature flags:
Changes UI
[ x] Includes DB Migration (follow approval process in SIP-59)
- [ x] Migration is atomic, supports rollback & is backwards-compatible
- [x ] Confirm DB migration upgrade and downgrade tested
- [x ] Runtime estimates and downtime expectations provided: 0.51s for 1m entries
Introduces new feature or API
Removes existing feature or API

etr2460

First off, thanks for extending this feature to apply to columns!

can you run @betodealmeida's migration perf testing script on this PR? unfortunately we're (at airbnb) still using MySQL and a column add will result in locking the table for us (and we have ~1.7 million rows in the table_colums table).

This PR will need a message in UPDATING.md, and folks on MySQL will either need to schedule downtime or use the percona toolkit (or similar) to perform the migration.

I'll review the rest of the JS/python once it's ready for review, but you'll want to add the documentation alongside here. I also don't know if a migration like this requires waiting for v1.4 or not... i hope not, but i don't remember the requirements. @eschutho could certainly comment

…dd-certified-columns

codecov · 2021-08-31T21:39:44Z

Codecov Report

Merging #16454 (490004c) into master (2a25e2d) will decrease coverage by 0.16%.
The diff coverage is 65.38%.

❗ Current head 490004c differs from pull request most recent head 92b2552. Consider uploading reports for the commit 92b2552 to get more accurate results

@@            Coverage Diff             @@
##           master   #16454      +/-   ##
==========================================
- Coverage   76.99%   76.82%   -0.17%     
==========================================
  Files        1007     1007              
  Lines       54180    54204      +24     
  Branches     7463     7473      +10     
==========================================
- Hits        41715    41644      -71     
- Misses      12225    12320      +95     
  Partials      240      240

Flag	Coverage Δ
hive	`?`
javascript	`71.27% <25.00%> (-0.04%)`	⬇️
mysql	`?`
postgres	`81.81% <100.00%> (+<0.01%)`	⬆️
presto	`81.70% <100.00%> (+0.04%)`	⬆️
python	`82.03% <100.00%> (-0.29%)`	⬇️
sqlite	`81.42% <100.00%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
.../src/explore/components/controls/SelectControl.jsx	`89.38% <ø> (ø)`
...ontend/src/views/CRUD/data/dataset/DatasetList.tsx	`65.92% <0.00%> (-3.49%)`	⬇️
superset/connectors/sqla/views.py	`64.01% <ø> (ø)`
superset/datasets/api.py	`91.92% <ø> (ø)`
...erset-frontend/src/datasource/DatasourceEditor.jsx	`73.57% <12.50%> (-1.25%)`	⬇️
...perset-frontend/src/datasource/DatasourceModal.tsx	`75.00% <71.42%> (+0.80%)`	⬆️
superset/connectors/sqla/models.py	`87.51% <100.00%> (-0.44%)`	⬇️
superset/datasets/schemas.py	`96.52% <100.00%> (+0.06%)`	⬆️
superset/models/helpers.py	`90.55% <100.00%> (+0.76%)`	⬆️
superset/db_engines/hive.py	`0.00% <0.00%> (-82.15%)`	⬇️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2a25e2d...92b2552. Read the comment docs.

etr2460

a few comments, but only one major one.

Does this also need a change in superset-ui so that everywhere you select columns the certification (or warning) icon is displayed?

UPDATING.md

superset-frontend/src/datasource/DatasourceModal.tsx

superset/connectors/sqla/models.py

geido

Code LGTM! Just some nits to follow our conventions

superset-frontend/src/datasource/DatasourceEditor.jsx

geido · 2021-09-09T11:32:42Z

/testenv up

github-actions · 2021-09-09T11:34:27Z

@geido Ephemeral environment spinning up at http://34.220.226.194:8080. Credentials are admin/admin. Please allow several minutes for bootstrapping and startup.

geido

@pkdotson below my feedback after manual testing:

Metrics use a tooltip for both fields. The tooltip is missing for both fields in Columns and Calculated columns. I think we should have it there too for consistency

The certified icon should show also in the Dataset panel. However, this is happening only for the Metrics currently

Co-authored-by: Geido <60598000+geido@users.noreply.github.com>

…dd-certified-columns

pkdotson · 2021-09-09T15:26:53Z

@pkdotson below my feedback after manual testing:

Metrics use a tooltip for both fields. The tooltip is missing for both fields in Columns and Calculated columns. I think we should have it there too for consistency

The certified icon should show also in the Dataset panel. However, this is happening only for the Metrics currently

Hey @geido the change has already been made in superset-ui. I just pulled in the latest code should have change. I'll spin up new test env.

github-actions · 2021-09-09T16:47:43Z

⚠️ @pkdotson Your base branch master has just also updated superset/migrations.

❗ Please consider rebasing your branch to avoid db migration conflicts.

pkdotson · 2021-09-09T16:56:52Z

/testenv up

pkdotson · 2021-09-15T20:48:45Z

/testenv up

github-actions · 2021-09-15T20:53:18Z

@pkdotson Ephemeral environment spinning up at http://52.12.101.216:8080. Credentials are admin/admin. Please allow several minutes for bootstrapping and startup.

etr2460

love it! thanks for adding the Mixin and reusing code

junlincc · 2021-09-15T22:28:19Z

@etr2460 thanks for the review. tested locally.

certification feature is added column and calculated columns ✅
certified items goes on top of the list in each section ✅
edited other uncertified fields did not affect the certified ones ✅
2 issues need to be addressed before merging (?)

icon alignment

when user click on the pencil icon from dataset crud view and open edit dataset modal, certification details are not showing.

junlincc

please see comment. 🙏

cc @jinghua-qa please add to our test case

…superset into add-certified-columns

pkdotson · 2021-09-20T20:05:21Z

/testenv up

pkdotson · 2021-09-20T20:26:18Z

Fixes for editing in datasetlist:
https://user-images.githubusercontent.com/17326228/134069819-150028c6-337b-4c03-a51a-395dfb01cc74.mov

fixes alignment issues:
https://user-images.githubusercontent.com/17326228/134070612-068431c4-9428-4e6c-8b0c-a14337c20266.mov

…dd-certified-columns

pkdotson · 2021-09-21T17:13:37Z

/testenv up

github-actions · 2021-09-21T17:17:46Z

@pkdotson Ephemeral environment spinning up at http://54.245.12.35:8080. Credentials are admin/admin. Please allow several minutes for bootstrapping and startup.

junlincc

Product sign-off! thanks for the improvement!

junlincc · 2021-09-22T21:34:56Z

Follow up tasks @pkdotson

add tooltip as @geido pointed out
2)certified calculated columns should go to the top of the list.

Screen.Recording.2021-09-22.at.2.29.28.PM.mov

let's address it in a separate pr.

github-actions · 2021-09-22T22:09:51Z

Ephemeral environment shutdown and build artifacts deleted.

…set columns fields (apache#16454) * add migration * add backend and frontend for certified * update migration with batch * fix integration test and update Updating.md * Update superset-frontend/src/datasource/DatasourceEditor.jsx Co-authored-by: Geido <60598000+geido@users.noreply.github.com> * Update superset-frontend/src/datasource/DatasourceEditor.jsx Co-authored-by: Geido <60598000+geido@users.noreply.github.com> * Update superset-frontend/src/datasource/DatasourceEditor.jsx Co-authored-by: Geido <60598000+geido@users.noreply.github.com> * change method name * add tooltip info * add mixin * merge heads * address comments * fix select label styles * add extra field * fix test? * add extra field to put schema Co-authored-by: Geido <60598000+geido@users.noreply.github.com>

pkdotson added 2 commits August 25, 2021 16:34

add migration

00a17ed

add backend and frontend for certified

75fdca9

superset-github-bot bot added the preset-io label Aug 25, 2021

pull-request-size bot added the size/L label Aug 25, 2021

junlincc requested a review from etr2460 August 26, 2021 02:24

etr2460 reviewed Aug 26, 2021

View reviewed changes

update migration with batch

75ad447

pkdotson requested a review from etr2460 August 30, 2021 22:02

pkdotson marked this pull request as ready for review August 31, 2021 16:18

pkdotson requested a review from a team as a code owner August 31, 2021 16:18

pkdotson requested review from betodealmeida and suddjian August 31, 2021 16:18

pkdotson added 2 commits August 31, 2021 13:50

fix integration test and update Updating.md

03ac0c7

Merge branch 'master' of https://github.com/preset-io/superset into a…

a0290ba

…dd-certified-columns

etr2460 reviewed Aug 31, 2021

View reviewed changes

UPDATING.md Outdated Show resolved Hide resolved

superset-frontend/src/datasource/DatasourceModal.tsx Outdated Show resolved Hide resolved

superset/connectors/sqla/models.py Outdated Show resolved Hide resolved

geido approved these changes Sep 9, 2021

View reviewed changes

superset-frontend/src/datasource/DatasourceEditor.jsx Outdated Show resolved Hide resolved

superset-frontend/src/datasource/DatasourceEditor.jsx Outdated Show resolved Hide resolved

superset-frontend/src/datasource/DatasourceEditor.jsx Outdated Show resolved Hide resolved

geido previously requested changes Sep 9, 2021

View reviewed changes

pkdotson and others added 5 commits September 9, 2021 08:08

Update superset-frontend/src/datasource/DatasourceEditor.jsx

ba2059e

Co-authored-by: Geido <60598000+geido@users.noreply.github.com>

Update superset-frontend/src/datasource/DatasourceEditor.jsx

b70d320

Co-authored-by: Geido <60598000+geido@users.noreply.github.com>

Update superset-frontend/src/datasource/DatasourceEditor.jsx

22e8ce2

Co-authored-by: Geido <60598000+geido@users.noreply.github.com>

Merge branch 'master' of https://github.com/preset-io/superset into a…

393dac2

…dd-certified-columns

change method name

5c292e0

pkdotson requested review from geido and etr2460 September 9, 2021 16:56

pkdotson added 2 commits September 15, 2021 12:16

merge heads

2d8264b

Merge branch 'master' into add-certified-columns

b9aecae

etr2460 approved these changes Sep 15, 2021

View reviewed changes

junlincc requested changes Sep 15, 2021

View reviewed changes

pkdotson added 3 commits September 20, 2021 08:56

address comments

aa1d81b

fix select label styles

96ef64c

Merge branch 'add-certified-columns' of https://github.com/preset-io/…

c994637

…superset into add-certified-columns

add extra field

29aa9bc

apache deleted a comment from github-actions bot Sep 20, 2021

pkdotson added 3 commits September 20, 2021 14:06

fix test?

a3d71bd

Merge branch 'master' of https://github.com/preset-io/superset into a…

a689c25

…dd-certified-columns

add extra field to put schema

92b2552

junlincc approved these changes Sep 22, 2021

View reviewed changes

pkdotson merged commit a198dbb into apache:master Sep 22, 2021

siegfriedweber mentioned this pull request Feb 9, 2022

Add Superset 1.4.1 stackabletech/docker-images#47

Closed

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.4.0 labels Mar 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add certifiedby & certification details fields to the edit dataset columns fields #16454

feat: add certifiedby & certification details fields to the edit dataset columns fields #16454

pkdotson commented Aug 25, 2021 •

edited

Loading

etr2460 left a comment •

edited

Loading

codecov bot commented Aug 31, 2021 •

edited

Loading

etr2460 left a comment

geido left a comment

geido commented Sep 9, 2021

github-actions bot commented Sep 9, 2021

geido left a comment •

edited

Loading

pkdotson commented Sep 9, 2021

github-actions bot commented Sep 9, 2021

pkdotson commented Sep 9, 2021

pkdotson commented Sep 15, 2021

github-actions bot commented Sep 15, 2021

etr2460 left a comment

junlincc commented Sep 15, 2021 •

edited

Loading

junlincc left a comment

pkdotson commented Sep 20, 2021

pkdotson commented Sep 20, 2021

pkdotson commented Sep 21, 2021

github-actions bot commented Sep 21, 2021

junlincc left a comment

junlincc commented Sep 22, 2021

github-actions bot commented Sep 22, 2021

feat: add certifiedby & certification details fields to the edit dataset columns fields #16454

feat: add certifiedby & certification details fields to the edit dataset columns fields #16454

Conversation

pkdotson commented Aug 25, 2021 • edited Loading

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

etr2460 left a comment • edited Loading

Choose a reason for hiding this comment

codecov bot commented Aug 31, 2021 • edited Loading

Codecov Report

etr2460 left a comment

Choose a reason for hiding this comment

geido left a comment

Choose a reason for hiding this comment

geido commented Sep 9, 2021

github-actions bot commented Sep 9, 2021

geido left a comment • edited Loading

Choose a reason for hiding this comment

pkdotson commented Sep 9, 2021

github-actions bot commented Sep 9, 2021

pkdotson commented Sep 9, 2021

pkdotson commented Sep 15, 2021

github-actions bot commented Sep 15, 2021

etr2460 left a comment

Choose a reason for hiding this comment

junlincc commented Sep 15, 2021 • edited Loading

junlincc left a comment

Choose a reason for hiding this comment

pkdotson commented Sep 20, 2021

pkdotson commented Sep 20, 2021

pkdotson commented Sep 21, 2021

github-actions bot commented Sep 21, 2021

junlincc left a comment

Choose a reason for hiding this comment

junlincc commented Sep 22, 2021

github-actions bot commented Sep 22, 2021

pkdotson commented Aug 25, 2021 •

edited

Loading

etr2460 left a comment •

edited

Loading

codecov bot commented Aug 31, 2021 •

edited

Loading

geido left a comment •

edited

Loading

junlincc commented Sep 15, 2021 •

edited

Loading