AUTH issue with version 3.4.0 #1274

sileht · 2020-01-31T09:39:21Z

Version

redis-py 3.4.0
redis_version: 5.0.4 (redis-cloud on heroku)

Platform:

python 3.8 on heroku

Description:

After upgrading from 3.3.11 to 3.4.0 the authentication won't work anymore:

  File "/app/mergifyio/cache.py", line 127, in get
    value = self.redis.get(key)
  File "/app/.heroku/python/lib/python3.8/site-packages/redis/client.py", line 1585, in get
    return self.execute_command('GET', name)
  File "/app/.heroku/python/lib/python3.8/site-packages/sentry_sdk/integrations/redis.py", line 68, in sentry_patched_execute_command
    return old_execute_command(self, name, *args, **kwargs)
  File "/app/.heroku/python/lib/python3.8/site-packages/redis/client.py", line 881, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "/app/.heroku/python/lib/python3.8/site-packages/redis/connection.py", line 1178, in get_connection
    connection.connect()
  File "/app/.heroku/python/lib/python3.8/site-packages/redis/connection.py", line 559, in connect
    self.on_connect()
  File "/app/.heroku/python/lib/python3.8/site-packages/redis/connection.py", line 633, in on_connect
    if nativestr(self.read_response()) != 'OK':
  File "/app/.heroku/python/lib/python3.8/site-packages/redis/connection.py", line 739, in read_response
    raise response
redis.exceptions.ResponseError: wrong number of arguments for 'AUTH' command

Downgrading have workaround our issue.

The url passed to redis client looks like:

redis://rediscloud:<password>@redis-yyyyy.xxx.us-east-z-z.ec2.cloud.redislabs.com:12345

The text was updated successfully, but these errors were encountered:

RadoRado · 2020-01-31T12:17:55Z

Can confirm. Did a Heroku deploy, which installed the latest version of redis and got the following error:

Redis ResponseError: wrong number of arguments for 'auth' command

andymccurdy · 2020-01-31T20:24:31Z

Your connection URL has a username specified, which prior to Redis 6 is technically incorrect. Redis pre-6.0 had no concept of usernames. redis-py prior to 3.4.0 simply ignored the username component.

redis-py 3.4.0 supports ACL authentication and obviously can no longer ignore the username component of a connection URL. I've created a patch in e39ae45 that attempts to detect this condition and falls back to old-style authentication when a username is supplied but the Redis server doesn't support it.

zyv · 2020-02-01T11:06:39Z

I've upgraded an getting the following traceback with Celery / Kombu:

2020-02-01T11:04:31.823281+00:00 app[worker.1]: [2020-02-01 11:04:31,822: INFO/MainProcess] Connected to redis://h:**@xxx.eu-west-1.compute.amazonaws.com:26099//
2020-02-01T11:04:31.842580+00:00 app[worker.1]: [2020-02-01 11:04:31,842: INFO/MainProcess] mingle: searching for neighbors
2020-02-01T11:04:31.899939+00:00 app[worker.1]: [2020-02-01 11:04:31,896: CRITICAL/MainProcess] Unrecoverable error: TypeError("unhashable type: 'Redis'")
2020-02-01T11:04:31.899942+00:00 app[worker.1]: Traceback (most recent call last):
2020-02-01T11:04:31.899943+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.8/site-packages/celery/worker/worker.py", line 205, in start
2020-02-01T11:04:31.899945+00:00 app[worker.1]:     self.blueprint.start(self)
2020-02-01T11:04:31.899946+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.8/site-packages/celery/bootsteps.py", line 119, in start
2020-02-01T11:04:31.899948+00:00 app[worker.1]:     step.start(parent)
2020-02-01T11:04:31.899949+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.8/site-packages/celery/bootsteps.py", line 369, in start
2020-02-01T11:04:31.899950+00:00 app[worker.1]:     return self.obj.start()
2020-02-01T11:04:31.899951+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 318, in start
2020-02-01T11:04:31.899953+00:00 app[worker.1]:     blueprint.start(self)
2020-02-01T11:04:31.899954+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.8/site-packages/celery/bootsteps.py", line 119, in start
2020-02-01T11:04:31.899955+00:00 app[worker.1]:     step.start(parent)
2020-02-01T11:04:31.899957+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.8/site-packages/celery/worker/consumer/mingle.py", line 40, in start
2020-02-01T11:04:31.899958+00:00 app[worker.1]:     self.sync(c)
2020-02-01T11:04:31.899959+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.8/site-packages/celery/worker/consumer/mingle.py", line 44, in sync
2020-02-01T11:04:31.899961+00:00 app[worker.1]:     replies = self.send_hello(c)
2020-02-01T11:04:31.899963+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.8/site-packages/celery/worker/consumer/mingle.py", line 57, in send_hello
2020-02-01T11:04:31.899964+00:00 app[worker.1]:     replies = inspect.hello(c.hostname, our_revoked._data) or {}
2020-02-01T11:04:31.899965+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.8/site-packages/celery/app/control.py", line 154, in hello
2020-02-01T11:04:31.899966+00:00 app[worker.1]:     return self._request('hello', from_node=from_node, revoked=revoked)
2020-02-01T11:04:31.899968+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.8/site-packages/celery/app/control.py", line 98, in _request
2020-02-01T11:04:31.899969+00:00 app[worker.1]:     return self._prepare(self.app.control.broadcast(
2020-02-01T11:04:31.899971+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.8/site-packages/celery/app/control.py", line 475, in broadcast
2020-02-01T11:04:31.899972+00:00 app[worker.1]:     return self.mailbox(conn)._broadcast(
2020-02-01T11:04:31.899973+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.8/site-packages/kombu/pidbox.py", line 349, in _broadcast
2020-02-01T11:04:31.899975+00:00 app[worker.1]:     return self._collect(reply_ticket, limit=limit,
2020-02-01T11:04:31.899976+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.8/site-packages/kombu/pidbox.py", line 391, in _collect
2020-02-01T11:04:31.899977+00:00 app[worker.1]:     self.connection.drain_events(timeout=timeout)
2020-02-01T11:04:31.899979+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.8/site-packages/kombu/connection.py", line 323, in drain_events
2020-02-01T11:04:31.899980+00:00 app[worker.1]:     return self.transport.drain_events(self.connection, **kwargs)
2020-02-01T11:04:31.899981+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 963, in drain_events
2020-02-01T11:04:31.899983+00:00 app[worker.1]:     get(self._deliver, timeout=timeout)
2020-02-01T11:04:31.899984+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.8/site-packages/kombu/transport/redis.py", line 374, in get
2020-02-01T11:04:31.899985+00:00 app[worker.1]:     self._register_BRPOP(channel)
2020-02-01T11:04:31.899987+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.8/site-packages/kombu/transport/redis.py", line 311, in _register_BRPOP
2020-02-01T11:04:31.899988+00:00 app[worker.1]:     if not self._client_registered(channel, channel.client, 'BRPOP'):
2020-02-01T11:04:31.899989+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.8/site-packages/kombu/transport/redis.py", line 306, in _client_registered
2020-02-01T11:04:31.899990+00:00 app[worker.1]:     (channel, client, cmd) in self._chan_to_sock)
2020-02-01T11:04:31.899992+00:00 app[worker.1]: TypeError: unhashable type: 'Redis'

Is it the same problem, or a different one? Is a new release coming?

zyv · 2020-02-01T11:08:00Z

Ah, OK - it seems it's a different problem :( #1277 / #1275

sileht · 2020-02-03T09:03:37Z

Looks 3.4.1 still have the issue.

andymccurdy · 2020-02-03T15:34:48Z

@sileht if you’re still having this issue I’ll need some more info. 3.4.1 seems to have fixed this issue for others.

sileht · 2020-02-03T15:44:35Z

yeah, the backtrace on sentry is a lit bit different:

auth_response = self.read_response() raises:

ResponseError("wrong number of arguments for 'AUTH' command")

and it looks like it's not catched by the:

except AuthenticationWrongNumberOfArgsError:

I will try to find the complete trace in the logs.

sileht · 2020-02-03T15:47:58Z

  File "/app/mergifyio/cache.py", line 127, in get
    value = self.redis.get(key)
  File "/app/.heroku/python/lib/python3.8/site-packages/redis/client.py", line 1579, in get
    return self.execute_command('GET', name)
  File "/app/.heroku/python/lib/python3.8/site-packages/sentry_sdk/integrations/redis.py", line 68, in sentry_patched_execute_command
    return old_execute_command(self, name, *args, **kwargs)
  File "/app/.heroku/python/lib/python3.8/site-packages/redis/client.py", line 875, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "/app/.heroku/python/lib/python3.8/site-packages/redis/connection.py", line 1185, in get_connection
    connection.connect()
  File "/app/.heroku/python/lib/python3.8/site-packages/redis/connection.py", line 561, in connect
    self.on_connect()
  File "/app/.heroku/python/lib/python3.8/site-packages/redis/connection.py", line 637, in on_connect
    auth_response = self.read_response()
  File "/app/.heroku/python/lib/python3.8/site-packages/redis/connection.py", line 752, in read_response
    raise response
redis.exceptions.ResponseError: wrong number of arguments for 'AUTH' command

sileht · 2020-02-03T15:48:26Z

I noted that the case here is not the same: https://github.com/andymccurdy/redis-py/blob/master/redis/connection.py#L139

'auth' vs 'AUTH'

is that related ?

andymccurdy · 2020-02-03T16:00:57Z

Yes, that seems to be the problem. The 3.4.1 fallback is looking for that specific error from the server and that comparison is case sensitive.

The easiest thing you could do right now is to remove the username component from your Redis connection string. Redis pre version 6 has no concept of usernames. I’m not sure why heroku is supplying one to you, but redis-py pre 3.4 would simply ignore it.

I’ll consider making the error message checking case insensitive.

itamarhaber · 2020-02-03T17:56:19Z

@andymccurdy what do you think about trying to call HELLO 2 before AUTH? If HELLO succeeds, you know you're on v6 or greater.

andymccurdy · 2020-02-12T23:21:22Z

@itamarhaber I'm hesitant to call HELLO 2 because I don't know what will happen in Redis v7 when RESP2 is slated to be deprecated. At that point I assumeHELLO 2 will fail, which will break this again. As ugly as it is, I can easily trap both "AUTH" and "auth" easily.

Ref #1274

andymccurdy · 2020-02-12T23:29:19Z

@sileht I added support for both 'auth' and 'AUTH' strings in master.

sileht · 2020-02-13T06:10:41Z

Great thx !

atleta · 2020-02-25T04:48:34Z

I started to see this same error after upgrading from 3.3.x to 3.4.1. However, I do NOT have a username in the connection string (I'm using redis 3.2.6). My connection string looks like this: 'redis://<password>@cache.infra.ourdomain.com:6379/0'

Seems like it may be related to the fix of the bug above.

andymccurdy · 2020-02-25T05:43:32Z

@atleta Your connection string does have a username. A connection string that has no username but does have a password looks like: redis://:<password>@host:port (notice the colon before the password).

shawnrobb · 2020-03-04T19:36:31Z

I was having the same issue with 3.4.1 deployed on Heroku. I'm using the RedisCloud add-on which automatically adds the REDISCLOUD_URL config variable. The connection string includes 'rediscloud' as the username. I removed the username and it started working again. Thanks for the info.

amureki · 2020-03-06T15:59:30Z

@shawnrobb I am also using rediscloud and heroku redis addons on my servers and I have to warn you here:
Addons may have a maintenance operations, which may end up in changing servers, for example I am noticing this in my logs:

heroku-redis: Update CACHE by heroku-redis

I am not yet sure how this is internally done, but I can imagine they can put back credentials with username and password, as it happens the moment you provision a new addon.
This means, if you dropped username part, but then addon reprovisioned a new instance for you, this might end up not well. :)

I did not have a chance to check this, but I guess it is worth noting here.

Cheers,
Rust

vanities · 2020-03-27T20:35:52Z

I was having the same issue with 3.4.1 deployed on Heroku. I'm using the RedisCloud add-on which automatically adds the REDISCLOUD_URL config variable. The connection string includes 'rediscloud' as the username. I removed the username and it started working again. Thanks for the info.

This helped us on our heroku issue today, not sure if why it all of a sudden popped up. Thanks for the info!

danjac mentioned this issue Jan 31, 2020

Thumbnail tag:redis.exceptions.ResponseError: wrong number of arguments for 'auth' command jazzband/sorl-thumbnail#621

Open

andymccurdy closed this as completed in e39ae45 Jan 31, 2020

zyv mentioned this issue Feb 1, 2020

3.4.0 breaks connecting with connection URL #1278

Closed

andymccurdy added a commit that referenced this issue Feb 12, 2020

expand AUTH fallback support to pre-v6 Redis servers.

b298975

Ref #1274

noisecapella pushed a commit to mitodl/micromasters that referenced this issue Mar 5, 2020

Downgrade redis to workaround redis/redis-py#1274 (comment)

17ec2b1

noisecapella mentioned this issue Mar 5, 2020

Downgrade redis to workaround mitodl/micromasters#4538

Merged

noisecapella pushed a commit to mitodl/micromasters that referenced this issue Mar 6, 2020

Downgrade redis to workaround redis/redis-py#1274 (comment) (#4538)

ea68a69

odlbot mentioned this issue Mar 6, 2020

Release 0.160.1 mitodl/micromasters#4539

Merged

2 tasks

jthurne mentioned this issue Jun 15, 2020

Don't pass the username when connecting to Redis < v6 even if a username is provided IBM-Cloud/redli#27

Closed

dependabot bot mentioned this issue Mar 17, 2021

Bump redis from 3.4.1 to 3.5.3 in /requirements tchellomello/awx#8

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AUTH issue with version 3.4.0 #1274

AUTH issue with version 3.4.0 #1274

sileht commented Jan 31, 2020 •

edited

Loading

RadoRado commented Jan 31, 2020

andymccurdy commented Jan 31, 2020

zyv commented Feb 1, 2020

zyv commented Feb 1, 2020 •

edited

Loading

sileht commented Feb 3, 2020

andymccurdy commented Feb 3, 2020

sileht commented Feb 3, 2020

sileht commented Feb 3, 2020

sileht commented Feb 3, 2020 •

edited

Loading

andymccurdy commented Feb 3, 2020

itamarhaber commented Feb 3, 2020

andymccurdy commented Feb 12, 2020

andymccurdy commented Feb 12, 2020

sileht commented Feb 13, 2020

atleta commented Feb 25, 2020

andymccurdy commented Feb 25, 2020 •

edited

Loading

shawnrobb commented Mar 4, 2020

amureki commented Mar 6, 2020

vanities commented Mar 27, 2020

AUTH issue with version 3.4.0 #1274

AUTH issue with version 3.4.0 #1274

Comments

sileht commented Jan 31, 2020 • edited Loading

RadoRado commented Jan 31, 2020

andymccurdy commented Jan 31, 2020

zyv commented Feb 1, 2020

zyv commented Feb 1, 2020 • edited Loading

sileht commented Feb 3, 2020

andymccurdy commented Feb 3, 2020

sileht commented Feb 3, 2020

sileht commented Feb 3, 2020

sileht commented Feb 3, 2020 • edited Loading

andymccurdy commented Feb 3, 2020

itamarhaber commented Feb 3, 2020

andymccurdy commented Feb 12, 2020

andymccurdy commented Feb 12, 2020

sileht commented Feb 13, 2020

atleta commented Feb 25, 2020

andymccurdy commented Feb 25, 2020 • edited Loading

shawnrobb commented Mar 4, 2020

amureki commented Mar 6, 2020

vanities commented Mar 27, 2020

sileht commented Jan 31, 2020 •

edited

Loading

zyv commented Feb 1, 2020 •

edited

Loading

sileht commented Feb 3, 2020 •

edited

Loading

andymccurdy commented Feb 25, 2020 •

edited

Loading