Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add separate endpoint to fetch function names for autocomplete #12840

Merged
merged 8 commits into from
Feb 3, 2021

Conversation

betodealmeida
Copy link
Member

@betodealmeida betodealmeida commented Jan 29, 2021

SUMMARY

Currently some of the /api/v1/database/ API responses include the list of functions supported by the DB. This is currently only implemented in Presto and Hive, and both implementations need to query the 3rd party DB in order to get that list. If the DB is down or overloaded, this might time out or take too long to complete, affecting users who are trying to see the list of databases or using SQL Lab.

Ideally we want to separate API calls that need to talk to 3rd party DBs (like Hive or Presto), from the calls that talk only to the main Superset DB. This PR changes the database response to not include the list of function names. This has been moved to a separate endpoint that is called asynchronously wherever needed (AFAIK only for autocomplete in SQL Lab).

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A

TEST PLAN

I changed superset.db_engine_specs.base.BaseEngineSpec.get_function_names to return some fake data. Loaded SQL Lab and verified that they show up in the autocomplete, both when loading an existing tab as well as when creating a new one.

ADDITIONAL INFORMATION

  • Has associated issue:
  • Changes UI
  • Requires DB Migration.
  • Confirm DB Migration upgrade and downgrade tested.
  • Introduces new feature or API
  • Removes existing feature or API

@betodealmeida betodealmeida changed the title WIP feat: add separate endpoint to fetch function names for autocomplete Jan 29, 2021
@pull-request-size pull-request-size bot added size/L and removed size/M labels Feb 1, 2021
@betodealmeida betodealmeida marked this pull request as ready for review February 1, 2021 18:55
@codecov-io
Copy link

codecov-io commented Feb 1, 2021

Codecov Report

Merging #12840 (150b55e) into master (a0e05a5) will decrease coverage by 16.21%.
The diff coverage is 80.00%.

Impacted file tree graph

@@             Coverage Diff             @@
##           master   #12840       +/-   ##
===========================================
- Coverage   67.00%   50.78%   -16.22%     
===========================================
  Files        1022      477      -545     
  Lines       50105    17211    -32894     
  Branches     5191     4444      -747     
===========================================
- Hits        33572     8741    -24831     
+ Misses      16402     8470     -7932     
+ Partials      131        0      -131     
Flag Coverage Δ
cypress 50.78% <80.00%> (-0.14%) ⬇️
javascript ?
python ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...erset-frontend/src/SqlLab/components/SqlEditor.jsx 54.46% <ø> (-1.62%) ⬇️
...rontend/src/SqlLab/components/SqlEditorLeftBar.jsx 50.00% <0.00%> (-2.95%) ⬇️
...et-frontend/src/SqlLab/reducers/getInitialState.js 50.00% <ø> (ø)
superset-frontend/src/SqlLab/actions/sqlLab.js 23.89% <83.33%> (-40.01%) ⬇️
...rontend/src/SqlLab/components/AceEditorWrapper.tsx 57.53% <100.00%> (-3.23%) ⬇️
superset-frontend/src/SqlLab/reducers/sqlLab.js 17.62% <100.00%> (-22.46%) ⬇️
...uperset-frontend/src/dashboard/util/dnd-reorder.js 0.00% <0.00%> (-100.00%) ⬇️
...rset-frontend/src/dashboard/util/getEmptyLayout.js 0.00% <0.00%> (-100.00%) ⬇️
...dashboard/components/resizable/ResizableHandle.jsx 0.00% <0.00%> (-100.00%) ⬇️
...dashboard/components/nativeFilters/ScopingTree.tsx 6.25% <0.00%> (-93.75%) ⬇️
... and 894 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a0e05a5...150b55e. Read the comment docs.

Copy link
Member

@etr2460 etr2460 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this makes sense to me.

one thing you should check is if these function names also show up in the adhoc metric creation popover or in the virtual dataset editor. these are 2 other places where i'd imagine the function names would be wanted for autocomplete

superset-frontend/src/SqlLab/actions/sqlLab.js Outdated Show resolved Hide resolved
class ImportV1DatabaseExtraSchema(Schema):
metadata_params = fields.Dict(keys=fields.Str(), values=fields.Raw())
engine_params = fields.Dict(keys=fields.Str(), values=fields.Raw())
metadata_cache_timeout = fields.Dict(keys=fields.Str(), values=fields.Integer())
schemas_allowed_for_csv_upload = fields.List(fields.String)
schemas_allowed_for_csv_upload = fields.List(fields.String())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this change be here? Also how is fields.Str() different from fields.String()?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be String() here. Str is an alias of String (see bottom of https://marshmallow.readthedocs.io/en/stable/_modules/marshmallow/fields.html).

@betodealmeida
Copy link
Member Author

@etr2460 the other places where user can type SQL don't have autocomplete for function names. I created a ticket for this: #12899

@betodealmeida betodealmeida merged commit ab3f4bd into apache:master Feb 3, 2021
amitmiran137 pushed a commit to nielsen-oss/superset that referenced this pull request Feb 3, 2021
* master: (23 commits)
  feat(explore): clear search on dataset change (apache#12909)
  chore: remove SIP-38 feature flag (apache#12894)
  fix: Config for dataset health check (apache#12906)
  fix(chart): allow null for most query object props (apache#12905)
  feat: add separate endpoint to fetch function names for autocomplete (apache#12840)
  chore: add required review on master (apache#12694)
  fix: comment typo (apache#12898)
  Migrates Radio component from Bootstrap to AntD. (apache#12738)
  fix: allow users to reset their passwords (apache#12886)
  fix(explore): missing select when groupby without metrics (apache#12890)
  refactor: dbapi exception mapping for dbapi's (apache#12869)
  feat(style-theme): add support for custom superset themes (apache#12858)
  chore(lint): fix pre-commit error (apache#12884)
  refactor(color-schemes): refactor setting of color schemes (apache#12857)
  feat(native-filters): Add defaultValue for Native filters modal (apache#12199)
  feat(release): add github token to changelog script (apache#12872)
  fix(menu): always show settings dropdown (apache#12877)
  Migrates Label component from Bootstrap to AntD. (apache#12774)
  [Helm] Automate datasource import (apache#10771)
  build: Skip loading example data from configs in CI (apache#12610)
  ...
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.2.0 labels Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/L 🚢 1.2.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants