Extend jdbc functionality #185
Conversation
jomar83
commented
Jul 26, 2015
jomar83
commented
- Adjust web ui to allow definition of jdbc connections
- Use host column for jdbc connection url
- Use connection models extra for storing jdbc driver path and classname
- Use connections extra_dejson to create the jdbc connection
- Use DbApiHook as a base for JdbcHook
- Make JdbcOperator consistent with other Db ops by using run
…tions Conflicts: airflow/www/app.py
|
Example of creating a connection:
|
|
Maybe a little clarification on this. The reason we extended this further was due to the fact, that adhoc queries to jdbc wouldn't work. In short, predefining jdbc connections wasn't possible, because there was no way to define jdbc driver location and jdbc driver classname. Now that's possible and everything works including adhoc queries. |
airflow/hooks/jdbc_hook.py
Outdated
There was a problem hiding this comment.
I think there's no need for an __init__ anymore as DbApiHook should work just fine.
You can move the get_connection call into get_conn and pass the info straight into the connect call.
|
About the UI customization, it's a nice flask-admin cutomization that optimizes for user experience but moves away from vanilla CRUD. I'm not sure if I want to optimize for user experience or ease of maintenance though. We could just document how to use the extras key/values very clearly and get rid of a bunch jdbc custom logic that requires knowledge of flask_admin/javascript/jinja to understand/evolve. On the other hand it looks really nice and fits the JDBC use case. What do you think? |
|
I think it's a difficult decision. That said, if you want to get as many people/companies to use airflow as possible, it's probably worth doing a little UI customisation to get a bigger audience to use it. Usability will go a long way when it comes to that. Good documentation will mitigate this to a certain extend, but I think overall the changes that were necessary weren't that intrusive. Still, your argument is valid and it's a tradeoff worth discussing. Let's see if someone else has an opinion on the matter. In the meantime I'll fix the other things. |
|
The design is based on the goal of making the setup of JDBC Drivers as simple as possible (via the webinterface and not by diving into the code). |
|
Totally, thanks for taking the time to bend the form into doing this, it also sets a precedent on how to do it if we want to replicate it for other hooks/connections. |
|
|
Extend jdbc functionality
|
Fixing a bug and generalizing the idea here: |
3 participants
