Proxy settings

[svamet]

hi there,
i dont know if i am right here, but i have not found anything according to my problem in the web.
i have to use elasticsearch in python from behind a proxy server. how can i pass down the proxy setting to elasticsearch.
i tried something like that without success.

es = Elasticsearch([es_url], _proxy = "http://proxyurl:port", _proxy_headers = { 'basic_auth': 'USERNAME:PASSWORD' })
res = es.search(index=index, body=request, search_type="count")

any help would be very nice.
thanks

[honzakral]

Hi,

unfortunately our connection classes don't support proxy settings out of the box. The solution is to create your own and set it directly via urllib3 or requests.

Another, easier, option would be to just set the env variables (mentioned in http://www.python-requests.org/en/latest/user/advanced/#proxies) when using requests.

[svamet]

thanks for your fast answer.
sry this maybe goes too far, but im new to python and elasticsearch and any help would be nice..

how can i set my connection class via requests?
something like this wont work:

import requests
from elasticsearch import Elasticsearch

proxies = {'https': 'http://user:pw@proxy.org:port'}
es = Elasticsearch([requests.get(es_url, proxies = proxies)])
res = es.search(index=index, body=request, search_type="count")

thanks

[svamet]

or do you mean something like?:

            proxies = {'https': 'http://user:pw@proxy.org:port'}

            s = requests.Session()
            s.proxies = proxies

            es = Elasticsearch([es_url], connection_class = s)
            res = es.search(index=index, body=request, search_type="count")
            return res

Sorry i am realy desperate for a solution to this problem..

[honzakral]

this should work:

from elasticsearch import RequestsHttpConnection

class MyConnection(RequestsHttpConnection):
    def __init__(*args, **kwargs):
        proxies = kwargs.pop('proxies', {})
        super(MyConnection, self).__init__(*args, **kwargs)
        self.session.proxies = proxies

es = Elasticsearch([es_url], connection_class=MyConnection, proxies = {'https': 'http://user:pw@proxy.org:port'})


print(es.info())

[svamet]

Thank you very much!! It works perfect!!

[ahmadkarim]

It gives an error "name error" self not defined

[ahmadkarim]

i want to do the same thing as svamet , but i think its because i am using python3.4 i am getting this error, but i cant get around it.

[honzakral]

Sorry, my bad, it should be:

from elasticsearch import RequestsHttpConnection

class MyConnection(RequestsHttpConnection):
    def __init__(self, *args, **kwargs):
        proxies = kwargs.pop('proxies', {})
        super(MyConnection, self).__init__(*args, **kwargs)
        self.session.proxies = proxies

es = Elasticsearch([es_url], connection_class=MyConnection, proxies = {'https': 'http://user:pw@proxy.org:port'})

print(es.info())

I forgot the self in __init__.

[ahmadkarim]

Thankyou very much!! i should have figured it out :P

[ahmadkarim]

After setting up the proxy as you devised now when i do:
print ( es.info() )
I receive the following error:

C:\Python34>python.exe C:\Users\akh2rt\Desktop\PythonStuff\PythonGit\Kibana
\ES.py
GET http://xxxx.com:443/ [status:400 request:0.04
6s]
Traceback (most recent call last):
  File "C:\Users\akh2rt\Desktop\PythonStuff\PythonGit\Kibana\ES.py", line 7
8, in <module>
    print(  es.info() )
  File "C:\Python34\lib\site-packages\elasticsearch\client\utils.py", line
69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "C:\Python34\lib\site-packages\elasticsearch\client\__init__.py", li
ne 219, in info
    _, data = self.transport.perform_request('GET', '/', params=params)
  File "C:\Python34\lib\site-packages\elasticsearch\transport.py", line 329
, in perform_request
    status, headers, data = connection.perform_request(method, url, params,
 body, ignore=ignore, timeout=timeout)
  File "C:\Python34\lib\site-packages\elasticsearch\connection\http_request
s.py", line 84, in perform_request
    self._raise_error(response.status_code, raw_data)
  File "C:\Python34\lib\site-packages\elasticsearch\connection\base.py", li
ne 108, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, err
or_message, additional_info)
elasticsearch.exceptions.RequestError: TransportError(400, '<html>\r\n<head
><title>400 The plain HTTP request was sent to HTTPS port</title></head>\r\
n<body bgcolor="white">\r\n<center><h1>400 Bad Request</h1></center>\r\n<ce
nter>The plain HTTP request was sent to HTTPS port</center>\r\n<hr><center>
nginx/1.9.6</center>\r\n</body>\r\n</html>\r\n')`

[honzakral]

You are trying to speak http to https port, could it be that you forgot use_ssl=True in your configuration?

[ahmadkarim]

yeah i thought that too but when i use_ssl= True then i get a lot of errors that latest being

raise ConnectionError('N/A', str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(HTTPSConnectionPo
ol(host='xxxx.com', port=443): Max retries exceed
ed with url: / (Caused by NewConnectionError('<requests.packages.urllib3.co
nnection.VerifiedHTTPSConnection object at 0x000000000348FE80>: Failed to e
stablish a new connection: [Errno 11004] getaddrinfo failed',))) caused by:
 ConnectionError(HTTPSConnectionPool(host='xxxxx.c
om', port=443): Max retries exceeded with url: / (Caused by NewConnectionEr
ror('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object a
t 0x000000000348FE80>: Failed to establish a new connection: [Errno 11004]
getaddrinfo failed',)))

[honzakral]

and can you connect to the target just using requests? It looks like the host cannot resolve the address of the target or proxy.

Either way I am afraid this is out of scope of this library. First step would be to make your environment work with just plain requests from python using your proxy, then I can help you. Until then there is nothing I can do unfortunately.

[ahmadkarim]

I can connect to an *http * server over a proxy and every thing works fine but when i do this:
r = requests.get('https://www.google.com',proxies=proxies, verify= False)
i get:

10, in <module>
    r = requests.get('https://www.google.com',proxies=proxies, verify= Fals
e)
  File "C:\Python34\lib\requests\api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Python34\lib\requests\api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Python34\lib\requests\sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python34\lib\requests\sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python34\lib\requests\adapters.py", line 437, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.google.c
om', port=443): Max retries exceeded with url: / (Caused by NewConnectionEr
ror('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object a
t 0x0000000003384780>: Failed to establish a new connection: [Errno 11004]
getaddrinfo failed',))

but when i access a local https server on the intranet then request.get works fine without the proxy