RFC - Customizations

[0x0ece]

Hi,

I'm proposing an initial plugins system.

The idea is to have a plugins directory containing mostly empty files, which are already "properly linked" in the code. This way, one can tweak Superset, but also easily upgrade to newer versions, with low-to-none merge conflicts. (This is heavily inspired by Airflow.)

At shopkick we're using superset since 0.10, did a lot of internal small patches, mostly personalizing the UI, and every time there's an update re-merging is a pretty painful operation.

With this current system, we're currently personalizing:

• bootstrap variables, notably the brand color
• other css
• colors, very important for our internal/core metrics
• additional personalized visualizations (probably too personalized to be added to the main branch)
• views, with their own template, including static pages, additional api calls, etc.
• generic python extensions, e.g. a security class that auto-register users if the email domain is the correct one (again, probably too personalized to be added in the main branch)

You can see an example of the plugins system in action here:
https://github.com/shopkick/superset/tree/sk_0.15.2/superset/plugins

To reiterate, the main purpose is to have empty files somewhere that one can personalize as she likes, without incurring in merge conflicts when a new version is released. In principle every single piece of Superset can be extended in a similar fashion, but I'm not a fan of too general implementations, I'd grow the list of supported features as the community requires them.

Please LMK what do you think, E.

[xrmx]

On the python side i really don't like the let's import everything from a file approach. Airflow take on handling this looks quite a bit more sophisticated to me https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py

[xrmx]

BTW we now have something similar for custom datasources, see 5a0e06e and fc921d6

[alanmcruickshank]

Could be tricky in this iteration, but plugins for time grains would be a great addition for our use case.

On Fri, 20 Jan 2017, 20:37 Riccardo Magliocchetti, ***@***.***> wrote: BTW we now have something similar for custom datasources, see 5a0e06e <5a0e06e> and fc921d6 <fc921d6> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2011 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEdFuC9bMltG80DSLbxLTU0_NFXEkfanks5rURrugaJpZM4LpsP6> .

--

…

_______ Alan Cruickshank E: alanmcruickshank@gmail.com T: +44 770 6851825

[0x0ece]

Sounds good to me, we can think of config vars where one defines the list of things to import.
Any comment on the js part? That's the most annoying imo.

[mahmoud]

@xrmx I think the difference in approaches can be traced to semantics. Airflow has more of a plugin system which offers extension for an installed instance. @0x0ece's feature may more properly be called a "customization" framework, suitable for those maintaining (and developing on) forks. Personally, I appreciate the functionality and simplicity of the approach.

@alanmcruickshank The import * antipattern is mostly referring to cases where the code goes on to use imported components by name. Programmatic import/usage like this is far less offensive, though I agree a full plugin system would have a more refined approach.

[0x0ece]

I agree with @mahmoud, that's a good point - I should rename the directory to custom instead of plugins. This is not intended, as in Airflow, to install Superset and load arbitrary code, but to recompile Superset with your code inside. The main reason being of course js.

[xrmx]

@mahmoud i don't think the python side of this PR is about a plugin system as it's not about making the code extensible. It's just a shortcut to maintain forks easily.

@0x0ece thanks for sharing your customization, a few random thoughts:

• the usa and europe map can use some code reuse and a generic way to express a subset of the world may be interesting for everyone
• the custom fab security should imho be a different package and just a configuration for your installation, not something you'll be shipping with superset
• for the templates you should use something like this i think https://buxty.com/b/2012/05/custom-template-folders-with-flask/
• the color configuration changes seems legit (and would make a lot of people happy)

[0x0ece]

All right, let's recap then:

• I'll rename plugins into custom, so the meaning is more clear
• I'll avoid import * in favor of a config à la data sources for views and viz

I'll wait for a comment on the js from the FE people, e.g. @ascott, and post an update asap.

As for my specific example:

• we can certainly extract some stuff from the maps and improve the world map, but this will be an independent PR (if anyone wants to jump on it, feel free!)
• custom security and templates can be placed wherever one likes, not necessarily inside the custom -- for me it's a simplification so I can manage everything in a single place
• in short, no action wrt to this PR

[0x0ece]

@alanmcruickshank BTW, we were using something around these lines: #1188
but don't have the bandwidth now to port look into that and how it evolved.

[mistercrunch]

This is all pretty reasonable, though the codebase may get shuffled around quite a bit still (before 1.0) and I'd prefer not having to worry about keeping this "plugins interface" working while refactoring.

For instance, we probably want to refactor the visualizations so that each one gets a folder with there related .py, .js and .css. Similarly, I was planning on refactoring the "connectors" (druid and sqlalchemy) out of models.py and view.py.

If we hook up the plugins as defined in this PR, it will make that refactoring harder if not impossible....

[0x0ece]

All right, I'm closing this PR then. But please think of a way for simple contributions/personalization in the main refactor. :)