ERROR: Creating a client session outside of coroutine

[Archelyst]

I'm using elasticsearch-py-async in a web application. The client gets instantiated during startup and is shared across requests. The whole startup process is synchronous, hence, there is no loop involved. As a result I get an "ERROR: Creating a client session outside of coroutine".

My understanding is, that during the creation of the client no connection / session should be opened. How am I supposed to do this correctly?

[honzakral]

You shouldn't have to instantiate the client inside a coroutine - the example code in README also doesn't do that. What could be happening is that you requested sniff_on_start which, as the name describes, tries to connect to the cluster on start to do the sniffing. Are you doing that by any chance?

If not, could you please share the code that you are using? Thanks!

[Archelyst]

My code looks like this:

from elasticsearch_async import AsyncElasticsearch
from elasticsearch_dsl.serializer import AttrJSONSerializer

from sanic import Sanic

def create_app(config_overrides=None):
    app = Sanic('gaia')
    # ... do some config
    app.es_client = AsyncElasticsearch(app.config.INDEX_URL, serializer=AttrJSONSerializer())

Looking into the code I find:

class AIOHttpConnection(Connection):
    def __init__(self, host='localhost', port=9200, http_auth=None,
            use_ssl=False, verify_certs=False, ca_certs=None, client_cert=None,
            client_key=None, loop=None, **kwargs):
        ...
        self.session = aiohttp.ClientSession(
            auth=http_auth,
            connector=aiohttp.TCPConnector(
                loop=self.loop,
                verify_ssl=verify_certs,
                conn_timeout=self.timeout,

            )
        )

That's run during the creation of the connection which happens during the creation of the transport (in elasticsearch-py) which happens during the creation of the client - no matter the configuration.

[Archelyst]

Full stacktrace:

2017-01-09 11:27:05,439: ERROR: Creating a client session outside of coroutine
client_session: <aiohttp.client.ClientSession object at 0x7fdf6e3766a0>
source_traceback: Object created at (most recent call last):
File "/srv/app/backend/run.py", line 9, in
app = create_app()
File "/srv/app/backend/mediadb/app.py", line 36, in create_app
app.es_client = AsyncElasticsearch(app.config.INDEX_URL, serializer=AttrJSONSerializer())
File "/usr/local/lib/python3.5/dist-packages/elasticsearch_async/init.py", line 8, in init
super().init(hosts, transport_class=transport_class, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/client/init.py", line 171, in init
self.transport = transport_class(_normalize_hosts(hosts), **kwargs)
File "/usr/local/lib/python3.5/dist-packages/elasticsearch_async/transport.py", line 19, in init
super().init(hosts, connection_class=connection_class, sniff_on_start=False, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/transport.py", line 112, in init
self.set_connections(hosts)
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/transport.py", line 167, in set_connections
connections = list(zip(connections, hosts))
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/transport.py", line 164, in _create_connection
return self.connection_class(**kwargs)
File "/usr/local/lib/python3.5/dist-packages/elasticsearch_async/connection.py", line 32, in init
conn_timeout=self.timeout,

[honzakral]

This is very weird because this code works for me just fine:

import asyncio
from elasticsearch_async import AsyncElasticsearch

client = AsyncElasticsearch(hosts=['localhost'])

@asyncio.coroutine
def print_info():
    info = yield from client.info()
    print(info)

loop = asyncio.get_event_loop()
loop.run_until_complete(print_info())
loop.close()
client.transport.close()

[honzakral]

From looking at the code (0) it looks like the message should have been a warning. Are you doing any configuration in your code to promote warning to errors?

0 - http://aiohttp.readthedocs.io/en/stable/_modules/aiohttp/client.html

[Archelyst]

Ya, it's a warning, it still works. But I'm afraid I'll run into strange bugs sometime, don't want to ignore that. The stacktrace is printed when enabling debug mode for asyncio

[Archelyst]

And your code gives me the same warning.

Plus, I first entered the wrong hostname, which (of course) raised an exception. During handling that this error occurred.

File "/usr/local/lib/python3.5/dist-packages/elasticsearch_async/connection.py", line 69, in perform_request
self.log_request_fail(method, url, body, self.loop.time() - start, exception=e)
TypeError: log_request_fail() missing 1 required positional argument: 'duration'

Should I open a ticket for that?

[honzakral]

Ah, I see. I am unfortunately not that familiar with aiohttp or asyncio to answer that. From my point of view it should be fine since we don't open any connection (with the exception for sniff_on_start) until a request is made.

Maybe @asvetlov could help?

[honzakral]

oh, the missing duration is the result of elasticsearch-async not being compatible with elasticsearch-py 5.0. I will update the code and make a release. Thanks for the report!

[sethmlarson]

This issue is fixed in elasticsearch v7.8.0, see #81 for more information.