Add a Redash extension system based on Python entrypoints.

[jezdez]

This is along the lines of what Flask does for CLI plugins:

http://flask.pocoo.org/docs/0.12/cli/#cli-plugins

The API allows specifying Python callbacks that receive the
Redash Flask app as the only argument and allow extending
the Redash process with the usual Flask API as needed. This
does not cover front-end specific extensions (yet).

[arikfr]

Something like this was on mind for sometime now. At first I was thinking to use it for query runners, to move them to separate packages. But it can be great for extending the app in general, as you did here.

You're welcome to push this upstream.

[jezdez]

Yay!

[arikfr]

Next thing is to find a good way to extend the UI, so you can move your custom stuff to extensions.

It easy enough to add new components and pages with the current setup (as it takes all pages/components from a specific folder and loads them). But the bigger thing is to be able to override existing UI elements (or extend them).

It might be possible with Angular's DI, but I prefer not to rely on it, as I hope one day to move away from Angular.

[jezdez]

@arikfr Yeah, I was thinking of adding support for another redash.static entrypoint next to the redash.extensions one which would be called as part of the webpack build process. The entrypoint callbacks would return lists of filepaths, that are then copied into a temporary location that webpack will be configured to pick up during the build process.

I think that's easy enough to accomplish. Would you only limit the folders to copy files to components and pages? Or just do a merge of the client/app directory with whatever static extension would return?

I realize that this mixes Python code with front-end stuff, and may be the wrong way to do this. Can you think of a JS-native way to extend the client?

[jezdez]

@arikfr Oh, BTW, in case you saw the second commit to this PR, here's the first use case to use the extension API, for a simple thing with just one entrypoint for now to fix #37: https://github.com/mozilla/redash-stmo

[jezdez]

@arikfr Opened getredash#2354 with just the extension bit.

[bealdav]

as I hope one day to move away from Angular.

@arikfr please could you tell more on this topic : abstract front end layer, vue.js ?

Thanks

[arikfr]

I realize that this mixes Python code with front-end stuff, and may be the wrong way to do this. Can you think of a JS-native way to extend the client?

I guess there is a similar way to leverage npm to deliver frontend extensions, but as I think about this maybe entrypoints is the right way to go. The benefit of using entrypoints for this is that we have a single way of building extensions, and that a single extension can bring in its own backend code and frontend code.

Or maybe support both ways for frontend developers?

Anyway, for entrypoints, the way it could work is that we will have a script that iterates over the entrypoints and returns additional paths to use in the webpack build process. I'm not even sure we will need to copy them, might be able to use as is.

Need to think about this a bit more and see how it will work with the webpack build process to make sure it creates a good enough workflow for deployment and development.

---

As a side note, some of the features in dockerflow like detection of missing migrations and structured logging are useful upstream. I wonder if we should use dockerflow as a whole in upsrteam or just copy those parts...

[jezdez]

@washort I've updated the PR to only install redash-stmo, since the extension PR was merged upstream.

[jezdez]

I guess there is a similar way to leverage npm to deliver frontend extensions, but as I think about this maybe entrypoints is the right way to go. The benefit of using entrypoints for this is that we have a single way of building extensions, and that a single extension can bring in its own backend code and frontend code.

Yeah, entrypoints are a good way to deliver extensions to Python-language-specific libraries, which I think any Redash extension by nature would have to be. Unless you think there would be front-end-only extensions, too? A good example where the lack of super clear guidelines for frontend developers has been Jupyter in the past years IMO, since it only recently gained the ability to easily develop frontend extensions. It still requires the full Python packaging story now, which may prevent Jupyter to gain momentum outside the traditionally Python-centric data community. I’m not sure if Redash wants to go in the same direction. Rest assured that the extension API is useful in itself if you ever want to consider breaking out some of the heavier parts of the Redash backend like the query runners.

Or maybe support both ways for frontend developers?

I think it would make sense to support both if you consider allowing 3rd party changes to the frontend via injected Javascript. I know the npmjs community has many examples where there are pluggable frameworks, so the methodology is probably just a matter of researching the most common one? A requisite to do any of that though is the ability to install additional packages during deployment in the first place. @robotblake basically said that it would be best to be able to write an own Dockerfile file that extends from the Redash Dockerfile and then are able to install npm or Python packages for whatever need. The actual extension setup would then happen via a registry pattern (e.g. entrypoints for Python packages, whatever similar for Node packages) and trigger a) inclusion in the webpack build process and b) the callback during the Flask startup. But I realize that this may not easily be portable to other ways of deploying Redash, which I’m sure you know a lot more about. There are a few ways how various people/projects have tried to shoeshorn JavaScript package management into Python tooling, but AFAIK no solution has seen wide adoption. Personally I would caution you to try to solve this problem in Redash and instead go with the language native solution. In case our redash-stmo Python library for example, it’s simple enough to just release our (potential) front-end extensions as an additional redash-stmo NPM package.

Anyway, for entrypoints, the way it could work is that we will have a script that iterates over the entrypoints and returns additional paths to use in the webpack build process. I'm not even sure we will need to copy them, might be able to use as is.

Ah, yeah, that seems easy enough to write, assuming webpack has a hook where the script can be called from. That’s why I only suggested to copy files, since I don’t know the webpack API very well.

Need to think about this a bit more and see how it will work with the webpack build process to make sure it creates a good enough workflow for deployment and development. As a side note, some of the features in dockerflow like detection of missing migrations and structured logging are useful upstream. I wonder if we should use dockerflow as a whole in upsrteam or just copy those parts…

That’s a good question, hm. In general Dockerflow is an internal thing, or at least I’m not aware of anyone adoption it outside Mozilla. I’m hesitant mostly since I’m not sure if it would be a dead-end in case we were to decide to drop it again for something better after a while. @robotblake, do you think it would make sense for Dockerflow to be adopted outside?

…

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[washort]

r+