Set effective user of (web)hdfs hooks using connection config, optionally overridden by constructor #942
Conversation
|
I like the idea, but I what about using the username from the Connection object instead if/when it is defined? You could just set the proxy user in the HDFSHook constructor. |
|
well, that makes sense. I missed that. |
xadhoom
force-pushed
the
hdfs_proxy_user
branch
from
32f27f1
to
afb9ed8
Compare
|
PR updated. Now if login is set into the connection, it is used as proxy user. I preferred to have also a way to override it in the constructor, since it can be useful in certain situations, where you have to deal with different proxy users and defining an hdfs connection for each one can be "too static". |
xadhoom
force-pushed
the
hdfs_proxy_user
branch
5 times, most recently
from
bd4b251
to
c132866
Compare
… WebHDFSHook, optionally allow to override it in the constructor. Add simple init test for WebHDFS.
xadhoom
force-pushed
the
hdfs_proxy_user
branch
from
bc5bae0
to
9a3a755
Compare
xadhoom
changed the title
|
PR updated to include webhdfs proxy user support. |
| client.content('/') | ||
| proxy_user = self.proxy_user or nn.login | ||
| client = InsecureClient(connection_str, user=proxy_user) | ||
| client.status('/') |
There was a problem hiding this comment.
I'm not familiar with the lib, what is this change about?
There was a problem hiding this comment.
As I understand, the content method is called to check if the namenode is valid, but it fails also on current airflow, if the task is run as any user != hdfs, while the status method will not, so I think that calling status is better for checking the namenode validity.
The other change is for adding the "doAs" also to the hdfs client.
|
Awesome. Thanks for this contribution. |
Set effective user of (web)hdfs hooks using connection config, optionally overridden by constructor
|
Can confirm. Content was a way to check the validity of namenode. Thanks for this! |
Sometimes is needed to do hdfs operations on behalf of other users (like creating dirs or setting permissions), so this PR adds a simple method to do that (optionally, without breaking compatibility).