Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add conversion_host_id to miq_request_task #281

Conversation

ghost
Copy link

@ghost ghost commented Oct 1, 2018

The ServiceTemplateTransformationPlanTask class relies on a conversion host to actually run the migration. The conversion host is modeled in ConversionHost class. From a database point of view, the ServiceTemplateTransformationPlanTask class is stored in miq_request_tasks table, because it's a sub-sub-class of MiqRequestTask. This PR adds a conversion_host_id column to miq_request_tasks to set the relationship.

Associated RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1634029

@carbonin
Copy link
Member

carbonin commented Oct 1, 2018

It looks like this information is currently being stored in the ServiceTemplateTransformationPlanTask#options column. I think this PR should also include a data migration to remove that value and put it in the new column for all previous tasks.

@ghost
Copy link
Author

ghost commented Oct 1, 2018

@carbonin That's even more complex. Today, the information stored in options[:transformation_host_id] is the id of a host (class Host). The PR stores the conversion host (class ConversionHost) id. A conversion host has a resource which is either a Host or a Vm, stored as resource_type. So current migration should create a ConversionHost with a resource relationship to a resource_type object with id resource_id, if it doesn't exist. Then the conversion_host_id can be set.

@agrare what are the possible values for resource_type ? Host and Vm ?

@agrare
Copy link
Member

agrare commented Oct 1, 2018

@fdupont-redhat if options[:transformation_host_id] can only ever contain a Host object then this should create a ConversionHost record for every distinct options[:transformation_host_id] and then conversion_host.resource = Host.find(options[:transformation_host_id]).

It is a little more complicated than just Host or Vm because the hosts will be STI'd e.g. ManageIQ::Providers::Redhat::InfraManager::Host.

@ghost ghost force-pushed the v2v_add_conversion_host_id_to_transformation_task branch from 3713049 to 659be65 Compare October 1, 2018 21:35
@ghost
Copy link
Author

ghost commented Oct 1, 2018

@miq-bot add-label gaprindashvili/no, schema, transformation

@miq-bot
Copy link
Member

miq-bot commented Oct 2, 2018

@fdupont-redhat 'agrare, carbonin' is an invalid reviewer, ignoring...

@ghost
Copy link
Author

ghost commented Oct 2, 2018

@miq-bot add-reviewer agrare
@miq-bot add-reviewer carbonin

def up
add_column :miq_request_tasks, :conversion_host_id, :bigint

ServiceTemplateTransformationPlanTask.all.reject { |task| task.options[:transformation_host_id].nil? }.each do |task|
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few things here ...
First, I don't think this query will work the way you're intending it to. STI is disabled so I think you would be better off using the base class and specifying the type in a where clause.

Also, instead of using reject here, can we make this a guard clause in the each block? That will simplify this and prevent us from iterating through the list of tasks twice.

So something like this:

MiqRequestTask.where(:type => "ServiceTemplateTransformationPlanTask").each do |task|
  next unless task.options[:transformation_host_id]
  ...
end

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the tip. Updated.

end

def down
ServiceTemplateTransformationPlanTask.all.select { |task| task.conversion_host.present? }.each do |task|
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, use the base class and you don't need to loop through the tasks twice.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.


class Host < ActiveRecord::Base
self.inheritance_column = :_type_disabled
has_many :tags, :class_name => "AddConversionHostIdToMiqRequestTasks::Tag"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this relation needed?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remnant of tests. Removed.


class ConversionHost < ActiveRecord::Base
self.inheritance_column = :_type_disabled
has_many :service_template_transformation_plan_tasks, :class_name => "AddConversionHostIdToMiqRequestTasks::MiqRequestTask"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relation isn't being used in the migration, why define it here?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. Removed.

belongs_to :conversion_host, :class_name => "AddConversionHostIdToMiqRequestTasks::ConversionHost"
end

class ServiceTemplateTransformationPlanTask < MiqRequestTask
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can be removed ... See my other comments about STI and using the base class for querying.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

add_column :miq_request_tasks, :conversion_host_id, :bigint

ServiceTemplateTransformationPlanTask.all.reject { |task| task.options[:transformation_host_id].nil? }.each do |task|
host = Host.find_by(:id => task.options[:transformation_host_id])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can use find rather than find_by. Also this entire block would be way easier to read if task.options[:transformation_host_id] was stored in a local variable (ideally before the guard clause).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.


ServiceTemplateTransformationPlanTask.all.reject { |task| task.options[:transformation_host_id].nil? }.each do |task|
host = Host.find_by(:id => task.options[:transformation_host_id])
task.conversion_host = ConversionHost.where(:id => task.options[:transformation_host_id]).first_or_create do |ch|
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

find_or_create_by! would be better over .where(:id => ...).first_or_create

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, if options[:transformation_host_id] is a reference to the Host record id then it can't also be a reference to the ConversionHost id.

This where just looks wrong.

Copy link
Member

@agrare agrare Oct 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah good catch, @fdupont-redhat did you mean to ConversionHost.where(:id => host.conversion_host.try(:id)) or something?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed for find_or_create_by!.
The where was wrong. Now it checks the resource_id attribute against the host_id.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should check for the type also then, just checking for the id and not the type

add_column :miq_request_tasks, :conversion_host_id, :bigint

ServiceTemplateTransformationPlanTask.all.reject { |task| task.options[:transformation_host_id].nil? }.each do |task|
host = Host.find_by(:id => task.options[:transformation_host_id])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should probably throw in a next if host.nil? since you're using host.name, host.type, ... below just in case the host was deleted and the task options weren't updated which given they aren't kept in sync by refresh is a distinct possibility.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, if that's a possibility @agrare then we should also keep the find_by (find raises if the record doesn't exist).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added test on host.nil?

task.conversion_host = ConversionHost.where(:id => task.options[:transformation_host_id]).first_or_create do |ch|
ch.name = host.name
ch.resource_type = host.type
ch.resource_id = host.id
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be able to just do ch.resource = host here

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When using find_or_create_by! the resource_id attribute is already set, so now I simply add the resource_type.

ch.name = host.name
ch.resource_type = host.type
ch.resource_id = host.id
ch.address = host.ipaddress
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're going to delegate ipaddress to the resource per #242 (comment) then we should leave this blank in the conversion_host record. That way if the host ip address changes it will be automatically picked up.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch.

it "creates conversion host" do
task = task_stub.create!
host = host_stub.create!
conversion_host = conversion_host_stub.create!(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can just do .create!(:resource => host) no need to split up the id/type

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you creating a ConversionHost in a spec that should be testing that a conversion host is created?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question @carbonin. Probably, because I'm dumb 😄
Removed and spec updated.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@agrare It fails if I use :resource => host. The error message is:

ActiveModel::UnknownAttributeError:
       unknown attribute 'resource' for AddConversionHostIdToMiqRequestTasks::ConversionHost.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You probably need to add the relation to the stubbed model

:resource_id => host.id,
:resource_type => host.type
)
task.options = { :transformation_host_id => conversion_host.id }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can pass :options => {:trans....} into the create!() and then you don't need the extra save

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

:resource_type => host.type
)
conversion_host.save
task.conversion_host = conversion_host
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise just pass :conversion_host in to the .create!

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

migrate

task.reload
expect(task.attributes).not_to include('conversion_host_id')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to check that ActiveRecord deleted the column, if that is broken we have bigger issues

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

:resource_id => host.id,
:resource_type => host.type
)
task.options = { :transformation_host_id => conversion_host.id }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, shouldn't this be a Host reference, not a conversion_host reference?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@ghost ghost force-pushed the v2v_add_conversion_host_id_to_transformation_task branch from eb6b3cb to 7d45a4e Compare October 2, 2018 15:20

MiqRequestTask.where(:type => 'ServiceTemplateTransformationPlanTask').each do |task|
host_id = task.options[:transformation_host_id]
if host_id
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it is a style thing but I usually prefer next if host_id.nil? over if host_id and then everything else indented. Reminds me of the java only one return style which brings back bad memories 😄

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Me too. But code climate complains about cognitive complexity when using next if. Hadn't met that error before and I use quite some next if or next unless in Automate.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't worry about code climate, ha and rubocop complained for not using next to skip
Can't win :)

@ghost ghost force-pushed the v2v_add_conversion_host_id_to_transformation_task branch from 7d45a4e to 04ca79a Compare October 2, 2018 15:30
task.options[:transformation_host_id] = task.conversion_host.resource.id
task.save!
task.conversion_host.id
end
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should .uniq also since there will be many tasks with the same conversion host. You can do end.uniq.compact or ConversionHost.destroy(conversion_host_ids.uniq.compact) your call

@@ -35,11 +34,12 @@ def up

def down
conversion_host_ids = MiqRequestTask.where(:type => 'ServiceTemplateTransformationPlanTask').map do |task|
return if task.conversion_host.nil?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be next if

end
task.save!
end
end
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the #up method looks good to me now

Copy link
Member

@agrare agrare left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@carbonin can you do another once-over?

Copy link
Member

@carbonin carbonin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking much better, just a few comments on the specs.

let(:conversion_host_stub) { migration_stub(:ConversionHost) }

migration_context :up do
it "when host doesn't exist" do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, but it would be great if this wrote out what we were testing for.

For example, it "doesn't set the conversion host when the host object doesn't exist"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

expect(task.conversion_host).to be_nil
end

it "when host exist" do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, it "references the existing Host in the ConversionHost record"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

task.reload

expect(task.options).to eq(:dummy_key => 'dummy_value')
expect(AddConversionHostIdToMiqRequestTasks::ConversionHost.find_by(:resource_id => conversion_host_id)).to be_nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be conversion_host_stub.find_by... right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

task.reload

expect(task.options).to eq(:dummy_key => 'dummy_value')
expect(AddConversionHostIdToMiqRequestTasks::ConversionHost.find_by(:resource => host)).not_to be_nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, I think this should use the stub.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


expect(task.options).to eq(:dummy_key => 'dummy_value')
expect(AddConversionHostIdToMiqRequestTasks::ConversionHost.find_by(:resource => host)).not_to be_nil
expect(task.conversion_host).to eq(AddConversionHostIdToMiqRequestTasks::ConversionHost.find_by(:resource => host))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@carbonin carbonin self-assigned this Oct 2, 2018
@miq-bot
Copy link
Member

miq-bot commented Oct 2, 2018

Checked commits fabiendupont/manageiq-schema@659be65~...7c3eb63 with ruby 2.3.3, rubocop 0.52.1, haml-lint 0.20.0, and yamllint 1.10.0
2 files checked, 0 offenses detected
Everything looks fine. 🍰

@carbonin carbonin merged commit eb7e077 into ManageIQ:master Oct 2, 2018
@carbonin carbonin added this to the Sprint 96 Ending Oct 8, 2018 milestone Oct 2, 2018
Copy link
Member

@bdunne bdunne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aside from the code comments, the BZ is marked with a Target release of 5.10 (based on hammer branch) and we've already passed the schema freeze for hammer. I hope this PR isn't expected to be backported to hammer.


class ConversionHost < ActiveRecord::Base
self.inheritance_column = :_type_disabled
belongs_to :resource, :polymorphic => true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you can use polymorphic relations with the stub models. If you do, you'll get something like: :resource_type => MigrationClass::StubModel which won't be found in production.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah you're right I thought in production it would use the type from the record but appears not

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, who wants to take on changing this? Or should we write a follow up migration to alter all the broken resources to the "production ready" name?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can, should be able to just edit this migration?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're going to update the .where to .in_my_region.find_each anyway

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think Jason would say we need a new migration since developers may have already run this one...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay #284 data migration to fix the resource_type

def up
add_column :miq_request_tasks, :conversion_host_id, :bigint

MiqRequestTask.where(:type => 'ServiceTemplateTransformationPlanTask').each do |task|
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually we prefer find_each to load records in batches.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this should probably be scoped with in_my_region to avoid running the migration on more records than necessary.

next unless host_id
host = Host.find_by(:id => host_id)
if host.present?
task.conversion_host = ConversionHost.find_or_create_by!(:resource => host) do |ch|
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ConversionHost.find_or_initialize_by(:resource_type => "Host", :resource_id => host.id).update_attributes!(.....)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see why we should use find_or_initialize_by + update_attributes over what's here, but yeah it does look like this will break because the host class will be incorrect.

def down
conversion_host_ids = MiqRequestTask.where(:type => 'ServiceTemplateTransformationPlanTask').map do |task|
next if task.conversion_host.nil?
task.options[:transformation_host_id] = task.conversion_host.resource.id
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe task.conversion_host.resource_id to avoid loading resource (which I think would fail anyway)?

:type => 'ServiceTemplateTransformationPlanTask',
:options => { :dummy_key => 'dummy_value', :transformation_host_id => conversion_host_id }
)
host.destroy!
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be able to generate a record id (in the current region) without creating and destroying a record

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure how you could do that without querying the sequences. I think it would have been nice to create the host in a before block, but other than that I think this is fine.

end

def down
conversion_host_ids = MiqRequestTask.where(:type => 'ServiceTemplateTransformationPlanTask').map do |task|
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should also be scoped with in_my_region

task.reload

expect(task.options).to eq(:dummy_key => 'dummy_value')
expect(conversion_host_stub.find_by(:resource => host)).not_to be_nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this eq(something)?

agrare added a commit to agrare/manageiq-schema that referenced this pull request Oct 2, 2018
Fix an invalid polymorphic resource_type from
ManageIQ#281 which resulted in
the type being "AddConversionHostIdToMiqRequestTasks::Host" instead of
"Host".
agrare added a commit to agrare/manageiq-schema that referenced this pull request Oct 2, 2018
Fix an invalid polymorphic resource_type from
ManageIQ#281 which resulted in
the type being "AddConversionHostIdToMiqRequestTasks::Host" instead of
"Host".
agrare added a commit to agrare/manageiq-schema that referenced this pull request Oct 2, 2018
Fix an invalid polymorphic resource_type from
ManageIQ#281 which resulted in
the type being "AddConversionHostIdToMiqRequestTasks::Host" instead of
"Host".
agrare added a commit to agrare/manageiq-schema that referenced this pull request Oct 2, 2018
Fix an invalid polymorphic resource_type from
ManageIQ#281 which resulted in
the type being "AddConversionHostIdToMiqRequestTasks::Host" instead of
"Host".
simaishi pushed a commit that referenced this pull request Oct 3, 2018
…d_to_transformation_task

Add conversion_host_id to miq_request_task

(cherry picked from commit eb7e077)

https://bugzilla.redhat.com/show_bug.cgi?id=1634029
@simaishi
Copy link
Contributor

simaishi commented Oct 3, 2018

Backport approved by Engineering/QE.

Hammer backport details:

$ git log -1
commit 58bd0c05dd61a8f5597f9d49f1e4fedb39cb88ec
Author: Nick Carboni <ncarboni@redhat.com>
Date:   Tue Oct 2 14:02:07 2018 -0400

    Merge pull request #281 from fdupont-redhat/v2v_add_conversion_host_id_to_transformation_task
    
    Add conversion_host_id to miq_request_task
    
    (cherry picked from commit eb7e077f462b9ab5e1c652f8220a34d22e4c8633)
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1634029

@ghost ghost deleted the v2v_add_conversion_host_id_to_transformation_task branch October 16, 2018 09:45
@Fryguy Fryguy added v2v and removed transformation labels Feb 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants