Airflow s3 connection test intermittent failures

[alexander-semenets]

Airflow S3 connection test fails periodically with next error:

`Documentation:	Fails if container contains item one or more times.
20181005 11:26:17.997 / 20181005 11:26:17.997 / 00:00:00.000
`
Airflow worker log has the next error:
`HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Fri, 05 Oct 2018 09:18:23 GMT', 'Content-Length': '357'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets "airflow-credentials-s3-conn" is forbidden: User "system:serviceaccount:company-a:airflow-company-a-airflow-worker" cannot get secrets in the namespace "company-a"","reason":"Forbidden","details":{"name":"airflow-credentials-s3-conn","kind":"secrets"},"code":403}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/legion_airflow/hooks/k8s_base_hook.py", line 32, in get_connection

return cls._get_conn_from_k8s(conn_id)

File "/usr/local/lib/python3.6/dist-packages/legion_airflow/hooks/k8s_base_hook.py", line 51, in _get_conn_from_k8s
k8s_namespace=os.environ['NAMESPACE']
File "/usr/local/lib/python3.6/dist-packages/legion/k8s/properties.py", line 99, in retrive
instance.load()
File "/usr/local/lib/python3.6/dist-packages/legion/k8s/properties.py", line 347, in load
self._read_k8s_resource_exception_handler(load_exception)
File "/usr/local/lib/python3.6/dist-packages/legion/k8s/properties.py", line 716, in _read_k8s_resource_exception_handler
.format(self.k8s_name, self.k8s_namespace_or_default, exception))
Exception: Cannot read secret 'airflow-credentials-s3-conn' in namespace 'company-a': (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Fri, 05 Oct 2018 09:18:23 GMT', 'Content-Length': '357'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets "airflow-credentials-s3-conn" is forbidden: User "system:serviceaccount:company-a:airflow-company-a-airflow-worker" cannot get secrets in the namespace "company-a"","reason":"Forbidden","details":{"name":"airflow-credentials-s3-conn","kind":"secrets"},"code":403}
Stack (most recent call last):
File "/usr/local/bin/airflow", line 27, in <module>
args.func(args)
File "/usr/local/lib/python3.6/dist-packages/airflow/bin/cli.py", line 438, in run
handler.close()
File "/usr/local/lib/python3.6/dist-packages/airflow/utils/log/s3_task_handler.py", line 77, in close
self.s3_write(log, remote_loc)
File "/usr/local/lib/python3.6/dist-packages/airflow/utils/log/s3_task_handler.py", line 150, in s3_write`

[aliaksandr-d]

s3_task_handler.py fails because it uses s3_hook that doesn't take credentials from Kube2IAM. It means that kube2iam doesn't work on Airflow.

[ablatov]

@aliaksandr-d still have this issue in develop
https://cc.epm.kharlamov.biz/jenkins/job/Deploy_Legion_Release/607/robot/report/log.html#s1-s3

[aliaksandr-d]

https://jenkins.cc.epm.kharlamov.biz/jenkins/job/Deploy_Legion_Release/631/robot/report/log.html

[kirillmakhonin]

Closed by #560