Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generalize switch between different datasources #1078

Merged
merged 33 commits into from
Sep 21, 2016
Merged

Generalize switch between different datasources #1078

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
d73e9a0
Generalize switch between different datasources.
Sep 8, 2016
383ffe4
Fix previous migration since slice model changed
Sep 8, 2016
22caae3
Fix warm up cache and other small stuff
Sep 8, 2016
44d0e6f
Adding modules and datasources through config
Sep 9, 2016
81ffb89
Replace tabs w/ spaces
Sep 9, 2016
1f60b8d
Fix other style issues
Sep 9, 2016
9d82efa
Change add method for SliceModelView to pick the first non-empty ds
Sep 9, 2016
b96628f
Remove tests on slice add redirect
Sep 9, 2016
2991bbd
Change way of db migration
Sep 12, 2016
4587349
Fix styling
Sep 12, 2016
36bd979
Merge branch 'master' into src_reg
Sep 12, 2016
88c91a3
Merge branch 'master' into src_reg
Sep 13, 2016
b6ed8fc
Merge branch 'master' into src_reg
Sep 14, 2016
68e054a
Merge branch 'master' into src_reg
Sep 15, 2016
3068bf2
Fix create slice
Sep 15, 2016
eda117b
Small fixes
Sep 15, 2016
a30722c
Fix code climate check
Sep 15, 2016
c24de2f
Merge branch 'master' into src_reg
Sep 16, 2016
c053a4b
Adding notes on how to create new datasource in CONTRIBUTING.md
Sep 16, 2016
5e7e9fb
Merge branch 'master' into src_reg
Sep 16, 2016
092caaa
Fix last merge
Sep 16, 2016
12f149d
Merge branch 'master' into src_reg
Sep 19, 2016
c7a41e9
A commit just to trigger travis build again
Sep 19, 2016
efe185e
Merge branch 'master' into src_reg
Sep 20, 2016
ab60779
Add migration to merge two heads
Sep 20, 2016
328cbe7
Fix codeclimate
Sep 20, 2016
9da4358
Simplify source_registry
Sep 20, 2016
49e8369
Fix codeclimate
Sep 20, 2016
d4b2976
Remove all getter methods
Sep 20, 2016
05e80ef
Merge branch 'master' into src_reg
Sep 20, 2016
cad85ae
Merge branch 'master' into src_reg
Sep 20, 2016
a386f60
Merge branch 'master' into src_reg
Sep 20, 2016
0e4f221
Merge branch 'master' into src_reg
Sep 21, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -251,3 +251,20 @@ You can then translate the strings gathered in files located under
to take effect, they need to be compiled using this command:

fabmanager babel-compile --target caravel/translations/


## Adding new datasources

1. Create Models and Views for the datasource, add them under caravel folder, like a new my_models.py
with models for cluster, datasources, columns and metrics and my_views.py with clustermodelview
and datasourcemodelview.

2. Create db migration files for the new models

3. Specify this variable to add the datasource model and from which module it is from in config.py:

For example:

`ADDITIONAL_MODULE_DS_MAP = {'caravel.my_models': ['MyDatasource', 'MyOtherDatasource']}`

This means it'll register MyDatasource and MyOtherDatasource in caravel.my_models module in the source registry.
5 changes: 4 additions & 1 deletion caravel/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
from flask_appbuilder.baseviews import expose
from flask_cache import Cache
from flask_migrate import Migrate
from caravel import source_registry
from werkzeug.contrib.fixers import ProxyFix


Expand Down Expand Up @@ -95,5 +96,7 @@ def index(self):

sm = appbuilder.sm

src_registry = source_registry.SourceRegistry()

get_session = appbuilder.get_session
from caravel import config, views # noqa
from caravel import views, config # noqa
8 changes: 8 additions & 0 deletions caravel/bin/caravel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,14 @@ config = app.config

manager = Manager(app)
manager.add_command('db', MigrateCommand)
module_datasource_map = config.get("DEFAULT_MODULE_DS_MAP")
module_datasource_map.update(config.get("ADDITIONAL_MODULE_DS_MAP"))

datasources = {}
for module in module_datasource_map:
datasources[module] = __import__(module, fromlist=module_datasource_map[module])

utils.register_sources(datasources, module_datasource_map, caravel.src_registry)


@manager.option(
Expand Down
7 changes: 7 additions & 0 deletions caravel/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,13 @@

DRUID_DATA_SOURCE_BLACKLIST = []

# --------------------------------------------------
# Modules and datasources to be registered
# --------------------------------------------------
DEFAULT_MODULE_DS_MAP = {'caravel.models': ['DruidDatasource', 'SqlaTable']}
ADDITIONAL_MODULE_DS_MAP = {}


"""
1) http://docs.python-guide.org/en/latest/writing/logging/
2) https://docs.python.org/2/library/logging.config.html
Expand Down
56 changes: 28 additions & 28 deletions caravel/data/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ def load_energy():
slice_name="Energy Sankey",
viz_type='sankey',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=textwrap.dedent("""\
{
"collapsed_fieldsets": "",
Expand Down Expand Up @@ -105,7 +105,7 @@ def load_energy():
slice_name="Energy Force Layout",
viz_type='directed_force',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=textwrap.dedent("""\
{
"charge": "-500",
Expand Down Expand Up @@ -136,7 +136,7 @@ def load_energy():
slice_name="Heatmap",
viz_type='heatmap',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=textwrap.dedent("""\
{
"all_columns_x": "source",
Expand Down Expand Up @@ -224,7 +224,7 @@ def load_world_bank_health_n_pop():
slice_name="Region Filter",
viz_type='filter_box',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
viz_type='filter_box',
Expand All @@ -233,7 +233,7 @@ def load_world_bank_health_n_pop():
slice_name="World's Population",
viz_type='big_number',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
since='2000',
Expand All @@ -245,7 +245,7 @@ def load_world_bank_health_n_pop():
slice_name="Most Populated Countries",
viz_type='table',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
viz_type='table',
Expand All @@ -255,7 +255,7 @@ def load_world_bank_health_n_pop():
slice_name="Growth Rate",
viz_type='line',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
viz_type='line',
Expand All @@ -267,7 +267,7 @@ def load_world_bank_health_n_pop():
slice_name="% Rural",
viz_type='world_map',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
viz_type='world_map',
Expand All @@ -277,7 +277,7 @@ def load_world_bank_health_n_pop():
slice_name="Life Expectancy VS Rural %",
viz_type='bubble',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
viz_type='bubble',
Expand All @@ -298,7 +298,7 @@ def load_world_bank_health_n_pop():
slice_name="Rural Breakdown",
viz_type='sunburst',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
viz_type='sunburst',
Expand All @@ -310,7 +310,7 @@ def load_world_bank_health_n_pop():
slice_name="World's Pop Growth",
viz_type='area',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
since="1960-01-01",
Expand All @@ -321,7 +321,7 @@ def load_world_bank_health_n_pop():
slice_name="Box plot",
viz_type='box_plot',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
since="1960-01-01",
Expand All @@ -333,7 +333,7 @@ def load_world_bank_health_n_pop():
slice_name="Treemap",
viz_type='treemap',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
since="1960-01-01",
Expand All @@ -345,7 +345,7 @@ def load_world_bank_health_n_pop():
slice_name="Parallel Coordinates",
viz_type='para',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
since="2011-01-01",
Expand Down Expand Up @@ -615,7 +615,7 @@ def load_birth_names():
slice_name="Girls",
viz_type='table',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
groupby=['name'],
Expand All @@ -625,7 +625,7 @@ def load_birth_names():
slice_name="Boys",
viz_type='table',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
groupby=['name'],
Expand All @@ -636,7 +636,7 @@ def load_birth_names():
slice_name="Participants",
viz_type='big_number',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
viz_type="big_number", granularity="ds",
Expand All @@ -645,15 +645,15 @@ def load_birth_names():
slice_name="Genders",
viz_type='pie',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
viz_type="pie", groupby=['gender'])),
Slice(
slice_name="Genders by State",
viz_type='dist_bar',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
flt_eq_1="other", viz_type="dist_bar",
Expand All @@ -663,7 +663,7 @@ def load_birth_names():
slice_name="Trends",
viz_type='line',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
viz_type="line", groupby=['name'],
Expand All @@ -672,7 +672,7 @@ def load_birth_names():
slice_name="Title",
viz_type='markup',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
viz_type="markup", markup_type="html",
Expand All @@ -690,7 +690,7 @@ def load_birth_names():
slice_name="Name Cloud",
viz_type='word_cloud',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
viz_type="word_cloud", size_from="10",
Expand All @@ -700,7 +700,7 @@ def load_birth_names():
slice_name="Pivot Table",
viz_type='pivot_table',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
viz_type="pivot_table", metrics=['sum__num'],
Expand All @@ -709,7 +709,7 @@ def load_birth_names():
slice_name="Number of Girls",
viz_type='big_number_total',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(
defaults,
viz_type="big_number_total", granularity="ds",
Expand Down Expand Up @@ -862,7 +862,7 @@ def load_unicode_test_data():
slice_name="Unicode Cloud",
viz_type='word_cloud',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(slice_data),
)
merge_slice(slc)
Expand Down Expand Up @@ -935,7 +935,7 @@ def load_random_time_series_data():
slice_name="Calendar Heatmap",
viz_type='cal_heatmap',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(slice_data),
)
merge_slice(slc)
Expand Down Expand Up @@ -1005,7 +1005,7 @@ def load_long_lat_data():
slice_name="Mapbox Long/Lat",
viz_type='mapbox',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(slice_data),
)
merge_slice(slc)
Expand Down Expand Up @@ -1084,7 +1084,7 @@ def load_multiformat_time_series_data():
slice_name="Calendar Heatmap multiformat" + str(i),
viz_type='cal_heatmap',
datasource_type='table',
table=tbl,
datasource_id=tbl.id,
params=get_slice_json(slice_data),
)
merge_slice(slc)
25 changes: 22 additions & 3 deletions caravel/migrations/versions/27ae655e4247_make_creator_owners.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,34 @@
down_revision = 'd8bc074f7aad'

from alembic import op
from caravel import db, models
from caravel import db
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relationship
from flask_appbuilder import Model
from sqlalchemy import (
Column, Integer, ForeignKey, Table)

Base = declarative_base()


class Slice(Base):
"""Declarative class to do query in upgrade"""
__tablename__ = 'slices'
id = Column(Integer, primary_key=True)


class Dashboard(Base):
"""Declarative class to do query in upgrade"""
__tablename__ = 'dashboards'
id = Column(Integer, primary_key=True)


def upgrade():
bind = op.get_bind()
session = db.Session(bind=bind)

objects = session.query(models.Slice).all()
objects += session.query(models.Dashboard).all()
objects = session.query(Slice).all()
objects += session.query(Dashboard).all()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

created_by does not exist as a column in the slice or dashboard tables causing an exception to be raised two lines below this if objects is not empty.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to modify existing migration files? Existing environments would not rerun this script right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@the-dcruz could you please open a new issue with the stacktrace please?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@the-dcruz I added a column datasource_id in Slice and Dashboard. If it uses models.Slice and models.Dashboard the session.query would fail because it couldn't find this column at this migration point. It would be helpful if you can share the stacktrace. Thank you

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@the-dcruz I run into some problem as well and I think it's what you have. I think the tests run on empty databases that's why I didn't catch it the first time. When there's existing data, you'd see the error. I've created a pull request for the fix. Can you take a look to see if it fixes what you have? #1262

for obj in objects:
if obj.created_by and obj.created_by not in obj.owners:
obj.owners.append(obj.created_by)
Expand Down
Loading