AWX has problem to connect to Redis

[filipprosovsky]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.

Bug Summary

After upgrading the AWX-operator to 1.1.1 and AWX included is 21.10.0 with removed aio redis package, we are getting errors in connection.py. See below:

future: <Task finished name='Task-3' coro=<Connection.disconnect() done, defined at /var/lib/awx/venv/awx/lib64/python3.9/site-packages/redis/asyncio/connection.py:828> exception=RuntimeError('Event loop is closed')> Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/redis/asyncio/connection.py", line 837, in disconnect self._writer.close() # type: ignore[union-attr] File "/usr/lib64/python3.9/asyncio/streams.py", line 353, in close return self._transport.close() File "/usr/lib64/python3.9/asyncio/selector_events.py", line 698, in close self._loop.call_soon(self._call_connection_lost, None) File "/usr/lib64/python3.9/asyncio/base_events.py", line 751, in call_soon self._check_closed() File "/usr/lib64/python3.9/asyncio/base_events.py", line 515, in _check_closed raise RuntimeError('Event loop is closed')

AWX version

21.10.0

Select the relevant components

• UI
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

Chrome

Steps to reproduce

Deploy 1.1.1 of AWX-operator, sync dynamic inventory.

Expected results

Inventory sync

Actual results

Job failed and error is present in log.

Additional information

No response

[pankaj-pal]

I can confirm, I have the same issue. Just brought up awx-operator and awx.

future: <Task finished name='Task-3' coro=<Connection.disconnect() done, defined at /var/lib/awx/venv/awx/lib64/python3.9/site-packages/redis/asyncio/connection.py:828> exception=RuntimeError('Event loop is closed')>
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/redis/asyncio/connection.py", line 837, in disconnect
    self._writer.close()  # type: ignore[union-attr]
  File "/usr/lib64/python3.9/asyncio/streams.py", line 353, in close
    return self._transport.close()
  File "/usr/lib64/python3.9/asyncio/selector_events.py", line 698, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 751, in call_soon
    self._check_closed()
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
2022-12-08 13:34:04,686 ERROR    [-] awx.main.utils.reload supervisorctl restart awx-rsyslogd errored with exit code `7`, stdout:
b'tower-processes:awx-rsyslogd: stopped\ntower-processes:awx-rsyslogd: ERROR (abnormal termination)'stderr:
b''
Task exception was never retrieved

Judging by the similar timing of the issue.. python just releases the latest patch ..may it broke something?

[filipprosovsky]

Looks like a redis-py issue, similar redis-py issue
AWX creates the pool same way redis_conn = await redis.asyncio.create_redis_pool(settings.BROKER_URL) line of pool creation

[anxstj]

I experience the same issue with my "docker development environment". AWX 21.10.0.

[hungpr0]

Same issue on K3D setup.
I needed to scale down to replicas=1 to workaround the issue. awx-web container keeps crashing because of this.

[jsalgado78]

Same issue. It's a new issue in AWX 21.10.0

[skattaa]

same here , but very weird seems to be .. I have tried operator 1.1.1 with AWX 21.10.0 and works fine with a new fresh database . when I have migrated the old postgres data to the new database, this problem started to show up !

[Cartman25]

The same issue with AWX 21.10.0 & AWX 21.10.1 with an existing Database.
UI is working for now, but this error message keeps looping in awx-task pod.

[shanemcd]

django/channels_redis#332 @fosterseth found this

[filipprosovsky]

Found 2 commits which could cause this. As removing channels and replacing it for daphne could affect functioning of CHANNEL LAYER CORE
There is core used instead of pubsub as proposed in issue @fosterseth found.
Commits: This1 and This2

Could this be fixed with replacing core to pubsub?

[relrod]

@shanemcd In light of that issue, should we downgrade channels-redis for now?

[pankaj-pal]

Is there a workaround for this issue in meanwhile?

[filipprosovsky]

@relrod I see that PR is merged, thanks a lot! When can we expect a new release to come out, do we have any info?

[relrod]

@filipprosovsky I've just cut a release with this fix.

[filipprosovsky]

@relrod deployed and working! Thanks a lot, can be closed

[HGS9761]

Hello,

Perhaps not the smartest question.
What do I need to do to have the issue fixed in my environment?

Regards Hans-Peter

[anxstj]

Hello @HGS9761,

you need to update to 21.10.2. (Or backport the above-mentioned PR to your environment.)

[HGS9761]

Thanks
Well that is beyond my expertise I am afraid.
But do I understand correctly that the error can be ignored?