Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] WARN: recoverer: could not move task to archive: NOT FOUND #274

Closed
TopScrew opened this issue May 20, 2021 · 8 comments
Closed

[BUG] WARN: recoverer: could not move task to archive: NOT FOUND #274

TopScrew opened this issue May 20, 2021 · 8 comments
Assignees
Labels
bug Something isn't working

Comments

@TopScrew
Copy link

Describe the bug
WARN: recoverer: could not move task to archive: NOT FOUND

Expected behavior
image

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

  • OS: Linux
  • Version of asynq v0.17.1
@TopScrew TopScrew added the bug Something isn't working label May 20, 2021
@TopScrew TopScrew changed the title [BUG] Description of the bug [BUG] WARN: recoverer: could not move task to archive: NOT FOUND May 20, 2021
@hibiken
Copy link
Owner

hibiken commented May 20, 2021

@TopScrew Thank you for reporting this issue.

Could you give me more details so that I can reproduce this bug?

@hibiken
Copy link
Owner

hibiken commented Jun 6, 2021

Hi @TopScrew, are you still seeing the issue? Or were you able to figure it out?

@hibiken
Copy link
Owner

hibiken commented Jun 9, 2021

Closing this for now since no follow up.
Feel free to reopen the issue if anyone is encountering the same bug.

@hibiken hibiken closed this as completed Jun 9, 2021
@TheDhejavu
Copy link

same issue here: This bug is hard to re-create, i could only think of two things

  • When the redis configuration was changed to delete LRU
  • When redis storage was full

Screen Shot 2023-03-22 at 11 52 00 AM

@yeqown
Copy link
Contributor

yeqown commented Jul 30, 2023

image

I got this issue too, I debug and monitor the Redis command, and I got the above screenshot, and refer to the archiveCmd lua code:

// KEYS[1] -> asynq:{<qname>}:t:<task_id>
// KEYS[2] -> asynq:{<qname>}:active
// KEYS[3] -> asynq:{<qname>}:lease
// KEYS[4] -> asynq:{<qname>}:archived
// KEYS[5] -> asynq:{<qname>}:processed:<yyyy-mm-dd>
// KEYS[6] -> asynq:{<qname>}:failed:<yyyy-mm-dd>
// KEYS[7] -> asynq:{<qname>}:processed
// KEYS[8] -> asynq:{<qname>}:failed
// -------
// ARGV[1] -> task ID
// ARGV[2] -> updated base.TaskMessage value
// ARGV[3] -> died_at UNIX timestamp
// ARGV[4] -> cutoff timestamp (e.g., 90 days ago)
// ARGV[5] -> max number of tasks in archive (e.g., 100)
// ARGV[6] -> stats expiration timestamp
// ARGV[7] -> max int64 value
var archiveCmd = redis.NewScript(`
if redis.call("LREM", KEYS[2], 0, ARGV[1]) == 0 then
  return redis.error_reply("NOT FOUND")
end
if redis.call("ZREM", KEYS[3], ARGV[1]) == 0 then
  return redis.error_reply("NOT FOUND")
end
redis.call("ZADD", KEYS[4], ARGV[3], ARGV[1])
redis.call("ZREMRANGEBYSCORE", KEYS[4], "-inf", ARGV[4])
redis.call("ZREMRANGEBYRANK", KEYS[4], 0, -ARGV[5])
redis.call("HSET", KEYS[1], "msg", ARGV[2], "state", "archived")
local n = redis.call("INCR", KEYS[5])
if tonumber(n) == 1 then
	redis.call("EXPIREAT", KEYS[5], ARGV[6])
end
local m = redis.call("INCR", KEYS[6])
if tonumber(m) == 1 then
	redis.call("EXPIREAT", KEYS[6], ARGV[6])
end
local total = redis.call("GET", KEYS[7])
if tonumber(total) == tonumber(ARGV[7]) then
   	redis.call("SET", KEYS[7], 1)
   	redis.call("SET", KEYS[8], 1)
else
  	redis.call("INCR", KEYS[7])
   	redis.call("INCR", KEYS[8])
end
return redis.status_reply("OK")`)

Conclusion is the qname (queue name) is empty causes this output, but why the qname is empty, I'm still following the source code, since I'm new guy as for asynq source code.

@yeqown
Copy link
Contributor

yeqown commented Jul 30, 2023

By following the calling stack, I found recoverLeaseExpiredTasks call archiveCmd, and here provide an ZERO value of TaskMessage it's queueName is empty.

image

More deeply reading the source code, github.com/hibiken/asynq@v0.24.1/internal/rdb/rdb.go:1293 ListLeaseExpired calls base.DecodeMessage, and it returns a ZERO value even though data parameter is an empty byte slice.

image

@bingtianyiyan
Copy link

1706671082956 also have this problem

@kamikazechaser
Copy link
Collaborator

@bingtianyiyan What version of asynq are you using? Try using @master and see if the issue still persists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

6 participants