If we can't update_attributes on a queue row, set state to error #14365
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jrafanie
commented
Mar 17, 2017
| @@ -35,7 +35,7 @@ def thresholds_exceeded? | |||
| def get_message_via_drb | |||
| loop do | |||
| begin | |||
| msg_id, lock_version = @worker_monitor_drb.get_queue_message(@worker.pid) | |||
| msg_id, lock_version = worker_monitor_drb.get_queue_message(@worker.pid) | |||
There was a problem hiding this comment.
We already have a attr_reader on the @worker_monitor_drb so i'm using it here so it's easier to test.
https://bugzilla.redhat.com/show_bug.cgi?id=1429747 In the reported bug, we had a Rails 4.2 era class [1] serialized in the args column of a miq_queue row. This class was removed in rails 5.0.0 [2], so we'd be unable to deserialize the column with: ArgumentError: undefined class/module ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Integer If we're unable to update_attributes because a column can't be deserialized, the message can't be handled by a worker, the worker dies, and the message remains in the miq_queue for another worker to try and also fail on. Instead, if update_attributes fails, we can try to set just the state column to 'error'. In this way, the server will not try to dispatch the same queue multiple times. We clear errored messages at server boot, so we can clean them up then. [1] ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Integer [2] rails/rails@aafee23
jrafanie
force-pushed
the
dont_retry_failed_message
branch
from
f9b439b
to
e21d1b9
Compare
|
@Fryguy Please review |
|
Some comments on commit jrafanie@e21d1b9 spec/models/miq_queue_worker_base/runner_spec.rb
|
|
Checked commit jrafanie@e21d1b9 with ruby 2.2.6, rubocop 0.47.1, and haml-lint 0.20.0 app/models/miq_queue_worker_base/runner.rb
|
Fryguy
reviewed
Mar 20, 2017
| @@ -64,6 +64,7 @@ def get_message_via_drb | |||
| _log.debug("#{log_prefix} #{MiqQueue.format_short_log_msg(msg)} stale, retrying...") | |||
| next | |||
| rescue => err | |||
| msg.update_column(:state, MiqQueue::STATUS_ERROR) | |||
There was a problem hiding this comment.
What if this fails? Say, if the database is down? should we do some kind of rescue nil here?
There was a problem hiding this comment.
I don't think it is needed.
If we have got here, the database was running a second ago. If it is no longer running, we cannot do anything sensible at that point, thus we can fail horribly.
There was a problem hiding this comment.
So @Fryguy I was debating if I should just delete the row if this fails but I really can't understand how this line could throw an error if we retrieved the msg on line 45. We're bypassing all other columns and validations. If we fail here, the only recourse is to raise. Who knows if a delete of the row would work either.
|
Ok, this is ready to go and is also for backport. Thanks for the review! |
jrafanie
added a commit
to jrafanie/manageiq
that referenced
this pull request
Mar 21, 2017
https://bugzilla.redhat.com/show_bug.cgi?id=1434454 The PostgreSQL::OID::Integer class was removed in: rails/rails@aafee23 It's possible that old Rails 4.2 versions of objects could have been serialized in the MiqQueue in the args column and we won't be able to deserialize them with Rails 5+, so we need to remove these rows. Related to ManageIQ#14365 Related to https://bugzilla.redhat.com/show_bug.cgi?id=1429747
simaishi
pushed a commit
that referenced
this pull request
Mar 28, 2017
If we can't update_attributes on a queue row, set state to error (cherry picked from commit 90cd1b0) https://bugzilla.redhat.com/show_bug.cgi?id=1436854
|
Euwe backport details: |
Fryguy
pushed a commit
to ManageIQ/manageiq-schema
that referenced
this pull request
Jun 20, 2017
https://bugzilla.redhat.com/show_bug.cgi?id=1434454 The PostgreSQL::OID::Integer class was removed in: rails/rails@aafee23 It's possible that old Rails 4.2 versions of objects could have been serialized in the MiqQueue in the args column and we won't be able to deserialize them with Rails 5+, so we need to remove these rows. Related to ManageIQ/manageiq#14365 Related to https://bugzilla.redhat.com/show_bug.cgi?id=1429747 (transferred from ManageIQ/manageiq@9795934)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://bugzilla.redhat.com/show_bug.cgi?id=1429747
In the reported bug, we had a Rails 4.2 era class [1] serialized in the args
column of a miq_queue row. This class was removed in rails 5.0.0 [2],
so we'd be unable to deserialize the column with:
ArgumentError: undefined class/module ActiveRecord::ConnectionAdapters::PostgreSQL::OID::IntegerIf we're unable to update_attributes because a column can't be
deserialized, the message can't be handled by a worker, the worker dies,
and the message remains in the miq_queue for another worker to try and
also fail on.
Instead, if update_attributes fails, we can try to set just the state
column to 'error'. In this way, the server will not try to dispatch the
same queue multiple times. We clear errored messages at server boot, so
we can clean them up then.
[1] ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Integer
[2] rails/rails@aafee23