Instantiate Container Template #10737

zakiva · 2016-08-24T15:03:20Z

This PR is built on #10159 - Persist Container Templates.
Changes included:

Process a Container Template.
Create objects from Container Template.
Rollback feature in case of error.
Create new project.

zakiva · 2016-08-24T15:20:43Z

@simon3z @zeari @moolitayer Please review

moolitayer · 2016-08-28T08:37:48Z

app/models/container_template.rb

+
+    ext_management_system.with_provider_connection do |client|
+      kubeclient = Marshal.load(Marshal.dump(client))
+      kubeclient.api_endpoint.path = "/api"


This is somewhat hacky since it has assumptions on kubeclient internal state.
you can use the ext_management_system method that does not use a block to get two clients and do the clean up yourself.

also see this

In the future it would be helpful if we could provide a forwarding client, just like the oc client lets us create objects in OpenShift and k8s alike cc @abonas @simon3z.

moolitayer · 2016-08-28T09:31:03Z

Looks good overall , container_template.rb needs some attention

moolitayer · 2016-08-28T09:35:09Z

app/models/container_template.rb

+    temp[:metadata][:name] = name
+    temp[:metadata][:namespace] = container_project.name
+    temp[:objects] = objects
+    temp[:parameters] = params


container_template_parameters?

we would like to allow the user to edit the template parameters and not just use them as they are (to be added later on)

its up to you but this might be nicer as:

{ :metadata => { :name => name :namespace => container_project.name } :objects => object :parameters => params }

zakiva · 2016-08-28T10:09:59Z

@moolitayer thanks, I've made the changes please take another look

zeari · 2016-08-29T09:18:10Z

app/models/container_template.rb

+        begin
+          kubeclient.send("create_#{obj[:kind].underscore}", obj)
+        rescue KubeException => e
+          $log.error("Unexpected Exception during creating object: #{e}")


s/during/while/g

It doesn't bother me a lot but I wonder if the code here could be smarter:

Can we know beforehand if the template should be processed through openshift or kubernetes clients?

Should we try the other client if the KubeException is something like a connection timeout?

@simon3z

simon3z · 2016-08-31T11:57:27Z

@miq-bot assign zakiva

miq-bot · 2016-09-14T22:16:31Z

<pr_mergeability_checker />This pull request is not mergeable. Please rebase and repush.

moolitayer · 2016-09-20T12:18:08Z

app/models/container_template.rb

+      obj = obj.symbolize_keys
+      obj[:metadata][:namespace] = project
+      create_entity = "create_#{obj[:kind].underscore}"
+      client = openshift_client.respond_to?(create_entity) ? openshift_client : kubeclient


if you are creating objects from different api groups you would have to create a new kubeclient for that group.
Knowing what your target group is available through the apiVersion of the object:

apiVersion: batch/v1 kind: Job

zakiva · 2016-10-10T09:41:37Z

@moolitayer please review

zakiva · 2016-10-10T10:43:37Z

dropping wip cc @simon3z @zeari
@miq-bot remove_label wip

bzwei · 2017-01-03T16:48:31Z

@zakiva I think the build-in rollback implementation (choice No. 1) is fine.

simon3z · 2017-01-04T10:34:11Z

@zakiva @zeari I think we should make the required changes to kubeclient ASAP (since it's a dependency and we want to get that out soon).

zakiva · 2017-01-08T15:13:22Z

@zakiva In the code I saw you have container_project and project. My understanding is that container_project is stored in VMDB table container_projects. What about project? Does it have one-to-one relationship with the template instantiation? Does it get stored in a VMDB table?

@bzwei Please see my comments in the code.

The create_objects method should return @created_objects array. Each created object contains some identifiers that can be used to find the object in VMDB after it is eventually collected through the refresh process. Since the creation of objects triggers a provider refresh, automate can wait until all objects are found in VMDB. Does this make sense?

@zakiva @zeari I think we should make the required changes to kubeclient ASAP (since it's a dependency and we want to get that out soon).

Please note that not all created objects will necessarily be saved in VMDB because we don't collect all kinds of objects.
create_objects returns an array of the created objects (hashes), unless any exception was raised.
Each object has kind, namespace and name that can be used in order to find it in VMDB, if we generally collect this kind of objects.
The only thing I think is missing is a mapping between openshift/k8s kind to miq kind. @simon3z I can simply add a static mapping of these kinds unless we already have it somewere (didn't find any). Also, What kubeclient changes did you refer to? for my understanding we already have everything we need, please let me know if I'm missing something.

zakiva · 2017-01-08T15:20:27Z

app/models/container_template.rb

@@ -12,4 +12,55 @@ class ContainerTemplate < ApplicationRecord
  serialize :objects, Array

  acts_as_miq_taggable
+
+  def instantiate(params, project = nil)


@bzwei project is the container project name in which the template will be instantiated. This container project is saved in the VMDB as you mentioed, here we just need to use its name. I can change this var name to project_name, it would probably be more accurate.

zakiva · 2017-01-08T15:24:45Z

app/models/manageiq/providers/openshift/container_manager.rb

@@ -23,4 +23,10 @@ def self.event_monitor_class
  def supported_auth_attributes
    %w(userid password auth_key)
  end
+
+  def create_project(project)


@bzwei project here is a hash that represents the project to be created in openshift and then collected and saved in VMDB table container_projects.

simon3z · 2017-01-09T14:02:43Z

Also, What kubeclient changes did you refer to? for my understanding we already have everything we need, please let me know if I'm missing something.

@zakiva I explained to @zeari what we need and why, and I gave him a demo on bluejeans (generateName).

zeari · 2017-01-09T17:29:53Z

I explained to @zeari what we need and why, and I gave him a demo on bluejeans (generateName).

@simon3z We get the generate name from process_template and we are able to find the entity using that info so we might not need the kubeclient changes. (correct me if im wrong @zakiva )

simon3z · 2017-01-09T17:57:30Z

@simon3z We get the generate name from process_template and we are able to find the entity using that info so we might not need the kubeclient changes. (correct me if im wrong @zakiva )

@zeari we tested that together last time:

oc process -n openshift -v HAWKULAR_METRICS_HOSTNAME=example.com \
    metrics-deployer-template | jq -C .items[].metadata

{
  "generateName": "metrics-deployer-",
  "labels": {
    "component": "deployer",
    "metrics-infra": "deployer",
    "provider": "openshift"
  }
}

Out of the process command we see the defined generateName but not the name of the entity which is generated at creation time.

zakiva · 2017-01-09T18:33:10Z

@zeari we tested that together last time:

oc process -n openshift -v HAWKULAR_METRICS_HOSTNAME=example.com
metrics-deployer-template | jq -C .items[].metadata
{
"generateName": "metrics-deployer-",
"labels": {
"component": "deployer",
"metrics-infra": "deployer",
"provider": "openshift"
}
}
Out of the process command we see the defined generateName but not the name of the entity which is generated at creation time.

@simon3z @zeari that's correct of course, but we do have the actual name in @created_objects. Kubeclient already returns openshift response which conatins this field. So we can return this array to the automation side, only thing that I see here missing is the kind mapping (as said before)?

zakiva · 2017-01-10T09:59:32Z

app/models/container_template.rb

@@ -12,4 +12,69 @@ class ContainerTemplate < ApplicationRecord
  serialize :objects, Array

  acts_as_miq_taggable
+
+  MIQ_ENTITY_MAPPING = {


@simon3z I added this mapping (only for objects we collect & can instantiate) so the automation side will be able to know in which table to look for the objects.
We may consider adding a more general mapping to reuse it later on in different places.
cc @moolitayer

bzwei · 2017-01-12T20:02:27Z

app/models/manageiq/providers/openshift/container_manager.rb

@@ -23,4 +23,10 @@ def self.event_monitor_class
  def supported_auth_attributes
    %w(userid password auth_key)
  end
+
+  def create_project(project)
+    connect.create_project_request project


prefer connect.create_project_request(project)

bzwei · 2017-01-12T20:04:31Z

With current implementation it should provide sufficient information to start the automate provisioning design/implementation 👍

miq-bot · 2017-01-17T17:22:19Z

Checked commit zakiva@37e6ece with ruby 2.2.6, rubocop 0.46.0, and haml-lint 0.19.0
4 files checked, 0 offenses detected
Everything looks good. 👍

simaishi · 2017-01-18T03:17:07Z

@zakiva is there a BZ for this? If not, please create one. Thanks!

simon3z · 2017-01-18T08:38:03Z

@bzwei @gmcculloug @zakiva my impression is that we don't need this in euwe. Is that right? (So we don't need to create a BZ as asked by @simaishi and we can drop the euwe label).

simon3z · 2017-01-18T10:05:23Z

@zakiva please send another PR adding a comment to rollback_objects stating that the rollback cannot catch children objects created during the instantiation and therefore those objects will remain around.

bzwei · 2017-01-18T15:15:35Z

@simaishi This is for future release together with the automate provisioning work (to be developed). Like @simon3z said, it should be dropped from euwe.

simaishi · 2017-01-18T15:23:23Z

@simon3z @bzwei thank you, changed to euwe/no

zakiva changed the title ~~Instantiate template~~ [WIP] Instantiate Container Template Aug 24, 2016

miq-bot added the sql migration label Aug 24, 2016

chessbyte added wip providers/containers labels Aug 24, 2016

chessbyte assigned simon3z Aug 24, 2016

zakiva force-pushed the instantiate_template branch from 3bdb4ee to 9299c15 Compare August 24, 2016 15:50

moolitayer reviewed Aug 28, 2016
View reviewed changes

zakiva force-pushed the instantiate_template branch from 9299c15 to cb708fd Compare August 28, 2016 09:59

zeari reviewed Aug 29, 2016
View reviewed changes

miq-bot assigned zakiva and unassigned simon3z Aug 31, 2016

zakiva force-pushed the instantiate_template branch 2 times, most recently from 478d908 to b483b2d Compare September 6, 2016 15:07

zakiva force-pushed the instantiate_template branch 3 times, most recently from 941daba to ed67fcb Compare September 20, 2016 11:35

moolitayer reviewed Sep 20, 2016

View reviewed changes

zakiva force-pushed the instantiate_template branch 2 times, most recently from 3a9cfde to 6e317ab Compare September 27, 2016 16:07

zakiva force-pushed the instantiate_template branch 2 times, most recently from e5f44e5 to 5d15fb6 Compare October 10, 2016 09:40

zakiva changed the title ~~[WIP] Instantiate Container Template~~ Instantiate Container Template Oct 10, 2016

zakiva commented Jan 8, 2017

View reviewed changes

zakiva force-pushed the instantiate_template branch 2 times, most recently from e110b86 to 2130627 Compare January 10, 2017 09:55

zakiva commented Jan 10, 2017

View reviewed changes

bzwei reviewed Jan 12, 2017

View reviewed changes

zakiva force-pushed the instantiate_template branch from 2130627 to 475ab1c Compare January 17, 2017 17:07

gmcculloug approved these changes Jan 17, 2017

View reviewed changes

zakiva force-pushed the instantiate_template branch from 475ab1c to 37e6ece Compare January 17, 2017 17:16

Instantiate Container Template

37e6ece

gmcculloug merged commit 054ac9c into ManageIQ:master Jan 17, 2017

gmcculloug added this to the Sprint 53 Ending Jan 30, 2017 milestone Jan 17, 2017

simaishi added the bugzilla needed label Jan 18, 2017

simaishi added euwe/no and removed bugzilla needed euwe/yes labels Jan 18, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Instantiate Container Template #10737

Instantiate Container Template #10737

zakiva commented Aug 24, 2016 •

edited

Loading

zakiva commented Aug 24, 2016

moolitayer Aug 28, 2016

moolitayer commented Aug 28, 2016

moolitayer Aug 28, 2016

zakiva Aug 28, 2016

zeari Aug 29, 2016

zakiva commented Aug 28, 2016

zeari Aug 29, 2016 •

edited

Loading

simon3z commented Aug 31, 2016

miq-bot commented Sep 14, 2016

moolitayer Sep 20, 2016

zakiva commented Oct 10, 2016

zakiva commented Oct 10, 2016

bzwei commented Jan 3, 2017

simon3z commented Jan 4, 2017

zakiva commented Jan 8, 2017 •

edited

Loading

zakiva Jan 8, 2017

zakiva Jan 8, 2017

simon3z commented Jan 9, 2017

zeari commented Jan 9, 2017 •

edited

Loading

simon3z commented Jan 9, 2017

zakiva commented Jan 9, 2017 •

edited

Loading

zakiva Jan 10, 2017

bzwei Jan 12, 2017

zakiva Jan 17, 2017

bzwei commented Jan 12, 2017

miq-bot commented Jan 17, 2017

simaishi commented Jan 18, 2017

simon3z commented Jan 18, 2017

simon3z commented Jan 18, 2017

bzwei commented Jan 18, 2017

simaishi commented Jan 18, 2017

Instantiate Container Template #10737

Instantiate Container Template #10737

Conversation

zakiva commented Aug 24, 2016 • edited Loading

zakiva commented Aug 24, 2016

Choose a reason for hiding this comment

moolitayer commented Aug 28, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zakiva commented Aug 28, 2016

zeari Aug 29, 2016 • edited Loading

Choose a reason for hiding this comment

simon3z commented Aug 31, 2016

miq-bot commented Sep 14, 2016

Choose a reason for hiding this comment

zakiva commented Oct 10, 2016

zakiva commented Oct 10, 2016

bzwei commented Jan 3, 2017

simon3z commented Jan 4, 2017

zakiva commented Jan 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simon3z commented Jan 9, 2017

zeari commented Jan 9, 2017 • edited Loading

simon3z commented Jan 9, 2017

zakiva commented Jan 9, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bzwei commented Jan 12, 2017

miq-bot commented Jan 17, 2017

simaishi commented Jan 18, 2017

simon3z commented Jan 18, 2017

simon3z commented Jan 18, 2017

bzwei commented Jan 18, 2017

simaishi commented Jan 18, 2017

zakiva commented Aug 24, 2016 •

edited

Loading

zeari Aug 29, 2016 •

edited

Loading

zakiva commented Jan 8, 2017 •

edited

Loading

zeari commented Jan 9, 2017 •

edited

Loading

zakiva commented Jan 9, 2017 •

edited

Loading