Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ems event collection refresh #105

Closed
wants to merge 9 commits into from
Closed

Ems event collection refresh #105

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
2fff138
Remove single event payload refresh
Ladas Jan 13, 2017
4580c11
Add EmsEventCollection target
Ladas Jan 13, 2017
17865c7
Add ems_event_collection collection wrapper
Ladas Jan 13, 2017
3bf18a0
Add attr_readers to inventory for each source of data
Ladas Jan 13, 2017
2d8ed9e
Group all ems events and parse them all
Ladas Jan 13, 2017
0dcfd49
Add EmsEventCollection target to factory
Ladas Jan 13, 2017
8768acb
Enhance Inventory::Targets base class
Ladas Jan 13, 2017
463b461
Move the array filtering to parser side
Ladas Jan 13, 2017
9c9273b
Fix rubocop issues
Ladas Jan 20, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions app/models/manageiq/providers/amazon/cloud_manager/refresher.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,14 @@ def parse_targeted_inventory(ems, _target, inventory)
hashes, = Benchmark.realtime_block(:parse_inventory) do
if refresher_options.try(:[], :inventory_object_refresh)
ManageIQ::Providers::Amazon::CloudManager::RefreshParserInventoryObject.new(inventory).populate_inventory_collections

if inventory.kind_of?(ManageIQ::Providers::Amazon::Inventory::Targets::EmsEventCollection)
# For EmsEventCollection, we want to parse everything, regardless whether it belongs to Cloud or Network
# Manager
ManageIQ::Providers::Amazon::NetworkManager::RefreshParserInventoryObject.new(inventory).populate_inventory_collections
Copy link
Contributor

@gberginc gberginc Jan 16, 2017 •
edited by chessbyte
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is where we'll have to populate other refresh parsers as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Events are going only through a cloud manager now, so no need.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yeah, sorry, I answered to something else it seems. Yes. In here, it should parse all data, if this was an event based refresh.

end

inventory.inventory_collections
else
ManageIQ::Providers::Amazon::CloudManager::RefreshParser.ems_inv_to_hashes(ems, refresher_options)
end
Expand All @@ -35,6 +43,25 @@ def parse_targeted_inventory(ems, _target, inventory)
hashes
end

def preprocess_targets
@targets_by_ems_id.each do |ems_id, targets|
if targets.any? { |t| t.kind_of?(ExtManagementSystem) }
ems = @ems_by_ems_id[ems_id]
targets_for_log = targets.map { |t| "#{t.class} [#{t.name}] id [#{t.id}] " }
_log.info "Defaulting to full refresh for EMS: [#{ems.name}], id: [#{ems.id}], from targets: #{targets_for_log}" if targets.length > 1
end

# We want all targets of class EmsEvent to be merged into one target, so they can be refreshed together, otherwise
# we could be missing some crosslinks in the refreshed data
ems_event_targets, other_targets = targets.partition { |x| x.kind_of?(EmsEvent) }
ems_event_collection = ManageIQ::Providers::Amazon::Inventory::EmsEventCollection.new(ems_event_targets)

@targets_by_ems_id[ems_id] = other_targets << ems_event_collection
end

super
end

# TODO(lsmola) NetworkManager, remove this once we have a full representation of the NetworkManager.
# NetworkManager should refresh base on its own conditions
def save_inventory(ems, target, inventory_collections)
Expand Down
133 changes: 77 additions & 56 deletions app/models/manageiq/providers/amazon/inventory.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,21 @@ class ManageIQ::Providers::Amazon::Inventory

attr_reader :ems, :target, :inventory_collections, :options

attr_reader :instances, :instances_refs, :instances_deleted
attr_reader :flavors, :flavors_refs, :flavors_deleted
attr_reader :availability_zones, :availability_zones_refs, :availability_zones_deleted
attr_reader :key_pairs, :key_pairs_refs, :key_pairs_deleted
attr_reader :private_images, :private_images_refs, :private_images_deleted
attr_reader :shared_images, :shared_images_refs, :shared_images_deleted
attr_reader :public_images, :public_images_refs, :public_images_deleted
attr_reader :cloud_networks, :cloud_networks_refs, :cloud_networks_deleted
attr_reader :cloud_subnets, :cloud_subnets_refs, :cloud_subnets_deleted
attr_reader :security_groups, :security_groups_refs, :security_groups_deleted
attr_reader :floating_ips, :floating_ips_refs, :floating_ips_deleted
attr_reader :network_ports, :network_ports_refs, :network_ports_deleted
attr_reader :load_balancers, :load_balancers_refs, :load_balancers_deleted
attr_reader :stacks, :stacks_refs, :stacks_deleted

def initialize(ems, target)
@ems = ems
@target = target
Expand All @@ -13,9 +28,59 @@ def initialize(ems, target)

@known_flavors = Set.new

initialize_inventory_sources
initialize_inventory_collections
end

def initialize_inventory_sources
@instances = []
@instances_refs = []
@instances_deleted = []
@flavors = []
@flavors_refs = []
@flavors_deleted = []
@availability_zones = []
@availability_zones_refs = []
@availability_zones_deleted = []
@key_pairs = []
@key_pairs_refs = []
@key_pairs_deleted = []
@private_images = []
@private_images_refs = []
@private_images_deleted = []
@shared_images = []
@shared_images_refs = []
@shared_images_deleted = []
@public_images = []
@public_images_refs = []
@public_images_deleted = []
@cloud_networks = []
@cloud_networks_refs = []
@cloud_networks_deleted = []
@cloud_subnets = []
@cloud_subnets_refs = []
@cloud_subnets_deleted = []
@security_groups = []
@security_groups_refs = []
@security_groups_deleted = []
@floating_ips = []
@floating_ips_refs = []
@floating_ips_deleted = []
@network_ports = []
@network_ports_refs = []
@network_ports_deleted = []
@load_balancers = []
@stacks = [] # We lack events for stacks
@stacks_refs = [] # We lack events for stacks
# Nested resources
@stack_resources = {} # We lack events for stacks
@stack_resources_refs = {} # We lack events for stacks
@stack_template = {} # We lack events for stacks
@stack_template_refs = {} # We lack events for stacks
@health_check_members = {}
@health_check_members_refs = {}
end

def aws_ec2
@aws_ec2 ||= ems.connect
end
Expand All @@ -32,71 +97,27 @@ def aws_s3
@aws_s3 ||= ems.connect(:service => :S3)
end

def instances
[]
end

def flavors
[]
end

def availability_zones
[]
end

def key_pairs
[]
end

def private_images
[]
end

def shared_images
[]
end

def public_images
[]
end

def stacks
[]
end

def stack_resources(_stack_name)
[]
end

def stack_template(_stack_name)
[]
end

def cloud_networks
[]
end

def cloud_subnets
[]
def stack_resources(stack_name)
@stack_resources.try(:[], stack_name) || []
end

def security_groups
[]
def stack_resources_refs(stack_name)
@stack_resources_refs.try(:[], stack_name) || []
end

def network_ports
[]
def stack_template(stack_name)
@stack_template.try(:[], stack_name) || []
end

def load_balancers
[]
def stack_template_refs(stack_name)
@stack_template_refs.try(:[], stack_name) || []
end

def health_check_members(_load_balancer_name)
[]
def health_check_members(load_balancer_name)
@health_check_members.try(:[], load_balancer_name) || []
end

def floating_ips
[]
def health_check_members_refs(load_balancer_name)
@health_check_members_refs.try(:[], load_balancer_name) || []
end
end
25 changes: 25 additions & 0 deletions app/models/manageiq/providers/amazon/inventory/ems_event_collection.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
class ManageIQ::Providers::Amazon::Inventory::EmsEventCollection
attr_accessor :targets

def initialize(targets)
@targets = targets
end

def all_related_ems_events(ems)
# We want all EmsEvents around the time of collected EmsEvents, and to the future since AWS send all related events
# at once
EmsEvent.where(:ems_id => ems.id).where(:timestamp => datetime_min..DateTime::Infinity.new).order("timestamp DESC")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any nicer way for checking timestamp >= datetime_min?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, using arel_table

end

def datetime_min
targets.collect(&:timestamp).min - 3.minutes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest adding a comment why 3 mins are used here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's a first draft condition to find related events, usually the similar events should have a CloudTrail id, but it's not true for some reason, when deploying through stack.

end

def name
"Collection of EmsEvent targets with name: #{targets.collect(&:name)}"
end

def id
"Collection of EmsEvent targets with id: #{targets.collect(&:id)}"
end
end
19 changes: 6 additions & 13 deletions app/models/manageiq/providers/amazon/inventory/factory.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,7 @@
class ManageIQ::Providers::Amazon::Inventory::Factory
class << self
def inventory(ems, target)
if target.kind_of?(EmsEvent)
event_target(ems, target) || target(ems, ems)
else
target(ems, target)
end
target(ems, target)
end

def target(ems, target)
Expand All @@ -18,15 +14,12 @@ def target(ems, target)
ManageIQ::Providers::Amazon::Inventory::Targets::StorageManager::Ebs.new(ems, target)
when ManageIQ::Providers::Amazon::StorageManager::S3
ManageIQ::Providers::Amazon::Inventory::Targets::StorageManager::S3.new(ems, target)
when ManageIQ::Providers::Amazon::Inventory::EmsEventCollection
# TODO(lsmola) we need to scan if we've recognized all events and fallback to full refresh if not
ManageIQ::Providers::Amazon::Inventory::Targets::EmsEventCollection.new(ems, target)
else
ManageIQ::Providers::Amazon::Inventory::Targets::CloudManager.new(ems, target)
end
end

def event_target(ems, target)
case target[:full_data]["configurationItem"]["resourceType"]
when "AWS::EC2::Instance"
ManageIQ::Providers::Amazon::Inventory::Targets::EventPayloadVm.new(ems, target)
# Fallback to ems refresh
ManageIQ::Providers::Amazon::Inventory::Targets::CloudManager.new(ems, ems)
end
end
end
Expand Down
20 changes: 14 additions & 6 deletions app/models/manageiq/providers/amazon/inventory/targets.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
class ManageIQ::Providers::Amazon::Inventory::Targets < ManageIQ::Providers::Amazon::Inventory
require_nested :CloudManager
require_nested :NetworkManager
require_nested :EmsEventCollection

include ManageIQ::Providers::Amazon::Inventory::InventoryCollectionDefaultInitData

protected
Expand All @@ -9,9 +13,9 @@ def initialize_inventory_collections

def add_inventory_collection(inventory_collection_data, key = nil)
model_class, data = inventory_collection_data
data[:parent] ||= ems
data[:parent] ||= ems
key ||= data[:association]

key ||= data[:association]
inventory_collections[key] = ::ManagerRefresh::InventoryCollection.new(model_class, data)
end

Expand All @@ -23,9 +27,8 @@ def add_inventory_collections(inventory_collections, inventory_collections_data

def add_remaining_inventory_collections(inventory_collections_data)
# Get names of all inventory collections defined in InventoryCollectionDefaultInitData
all_inventory_collections = ManageIQ::Providers::Amazon::Inventory::InventoryCollectionDefaultInitData.
public_instance_methods.grep(/.+_init_data/).map { |x| x.to_s.gsub("_init_data", "") }

all_inventory_collections = ManageIQ::Providers::Amazon::Inventory::InventoryCollectionDefaultInitData
.public_instance_methods.grep(/.+_init_data/).map { |x| x.to_s.gsub("_init_data", "") }
# Get names of all defined inventory_collections
defined_inventory_collections = inventory_collections.keys.map(&:to_s)

Expand All @@ -35,7 +38,12 @@ def add_remaining_inventory_collections(inventory_collections_data)
end

def event_payload(event)
transform_keys(event[:full_data]["configurationItem"]["configuration"])
transform_keys((event.full_data || []).fetch_path("configurationItem", "configuration"))
end

def event_deleted_payload(event)
transform_keys((event.full_data || []).fetch_path("configurationItemDiff", "changedProperties", "Configuration",
"previousValue"))
end

def transform_keys(value)
Expand Down
Loading