[BUG] asynq client failed when use command stats

[duyhungtnn]

Describe the bug

• After connect asynq with our Redis server, run a command asynq stats. It take so long to complete and end up with

ryan@ ~ asynq stats

Using config file: /Users/ryan/.asynq.yml

UNKNOWN: UNKNOWN: redis command error: MEMORY USAGE failed: redis: nil

To Reproduce
Steps to reproduce the behavior (Code snippets if applicable):
0. Setup redis with docker-compose

version: '3.1'

services:
  redis:
    container_name: abc_redis
    image: redis:6.2.5-alpine
    command: redis-server --requirepass ${REDIS_PASSWORD}
    environment:
      - ALLOW_EMPTY_PASSWORD=no
      - REDIS_DISABLE_COMMANDS=FLUSHDB,FLUSHALL
    ports:
      - '6379:6379'
    volumes:
      - './data:/data'

networks:
  default:
    external:
      name: abc

1. Setup background processing ...
2. Enqueue tasks ...
3. See Error ...

Expected behavior

• Need to see the stats of the system

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: [Ubuntu 18.4]
• Version of asynq package [v0.18.2]

Additional context

• The worker and scheduler work as normal
  

• The data on Redis is okay too
  

[crossworth]

Hello, I have encoutered this problem as well, but using asynqmon, I could not yet determine why (for this reason I havent create a issue or pr), but I know that I can cause it by enqueuing a lot of tasks and processing they very fast (no code, no printing, just a empty handler).

Probably the MEMORY USAGE command is returning nil and since we dont handle it we get this error message.

I will investigate it.

[hibiken]

@duyhungtnn thank you for reporting this!

Could you double check the CLI version by running asynq version?

[hibiken]

The error message in the screenshot is coming from here:

asynq/internal/rdb/inspect.go

Line 197 in 95c90a5

return 0, errors.E(op, errors.Unknown, &errors.RedisCommandError{Command: "memory usage", Err: err})

My initial guess is that the for loop for the scan command is taking too long (

asynq/internal/rdb/inspect.go

Line 183 in 95c90a5

for {

) since we recently changed each task to have its only redis key.
Since redis only executes one command at a time, since other clients are sending commands to redis while this for loop is executing scan commands to collect all keys, keys may have been deleted (i.e. task processed successfully), you'll see the key not found error when memeorUsage command is invoked.

If this above analysis is correct, then we should make this operation into a Lua script and run the script with EXECUTE so that all of this operation becomes atomic (i.e. other clients commands won't be executing between multiple calls to scan and memoryusage). But now that we can have a lot more redis-keys than before (each task has its own key), maybe getting memory usage within a short timeframe is not feasible anymore.

[duyhungtnn]

@duyhungtnn thank you for reporting this!

Could you double check the CLI version by running asynq version?

@hibiken yeah my asynq is asynq version 0.18.2

@crossworth I am facing that issue with asynqmon too. The web dashboard display nothing when viewing the queue info.
Yes, I was using a scheduler to trigger a single task every 2h. On that single task will enqueue 5000 tasks to feed the workers. It may be the cause.

[hibiken]

@duyhungtnn The fix is in 👍
Let me know if you continue to see issues with new versions (asynq: v0.18.3, asynqmon v0.2.2)