Expose plugin ansible content consolidation as a rake task #17407

carbonin · 2018-05-10T21:58:49Z

This will allow us to pull the set of ansible content provided by our plugins together at build time to provide the data to build an rpm.

This rpm can then be used in the embedded ansible container image to provide plugin ansible content when embedded ansible does not share a file system with the manageiq application at runtime.

jrafanie · 2018-05-14T11:40:53Z

Looks good so far. I'm guessing that since we gate ensure_plugin_playbooks_project_seeded for containers, we don't have to worry about implementing create_local_playbook_repo yet, right? If so, that gives us a place to implement it when we want to. It looks very clean. 👍

This only applies to the appliance version of embedded ansible so it makes more sense for the logic to live here rather than in the worker which should only be responsible for creating the repo in the provider. https://www.pivotaltracker.com/story/show/157313841

https://www.pivotaltracker.com/story/show/157313841

This task will be used to create an rpm for providing plugin ansible content to the embedded ansible container image https://www.pivotaltracker.com/story/show/157313841

carbonin · 2018-05-14T14:12:12Z

we don't have to worry about implementing create_local_playbook_repo yet

I think this will still fail for ManageIQ, but it was already failing before this change for other reasons.
I haven't decided if I should try to address this here or wait until I tackle making the playbook seeding work for DockerEmbeddedAnsible.

carbonin · 2018-05-14T18:56:50Z

Removing WIP after testing this out on an appliance 👍

Fryguy · 2018-05-14T18:59:51Z

lib/embedded_ansible/appliance_embedded_ansible.rb

+      Rugged::Commit.create(repo, options)
+    end
+
+    FileUtils.chown_R('awx', 'awx', playbook_repo_path)


Why does awx own the repo? I know that was there before, but that doesn't make any sense to me.

@jrafanie ? I know the tower processes run as the awx user, but if it's just copying off the files it seems like it wouldn't need to own the directory.

The assumption was the awx user would be reading this git repo's path when we posted this local path for the new project. If it has permission to read in all situations, then this can be removed.

Fryguy · 2018-05-14T19:01:09Z

lib/tasks/evm.rake

+  task :write_plugin_ansible_content => :environment do
+    dest_dir = ENV["ANSIBLE_CONTENT_DIR"] || Rails.root.join("tmp", "ansible_content")
+    EmbeddedAnsible.consolidate_plugin_playbooks(dest_dir)
+  end


So, this just copies the content but doesn't set up git?

Fryguy · 2018-05-14T19:02:29Z

lib/embedded_ansible/appliance_embedded_ansible.rb

+      options[:author]     = options[:committer] = { :email => "system@localhost", :name => "System", :time => Time.now.utc }
+      options[:message]    = "Initial Commit"
+      options[:parents]    = []
+      options[:update_ref] = 'HEAD'


I'm surprised there isn't a master branch. Does awx/tower only reference the HEAD commit anyway?

There is a master branch after everything is set up. Is that just the default behavior?

[root@localhost ~]# cd /var/lib/awx_consolidated_source/ [root@localhost awx_consolidated_source]# git status # On branch master nothing to commit, working directory clean [root@localhost awx_consolidated_source]# git log commit 882ea0dbf707c366de82655bd1b74376ccf1d712 Author: System <system@localhost> Date: Mon May 14 19:06:53 2018 +0000 Initial Commit [root@localhost awx_consolidated_source]#

Yeah, this is probably using the default behavior.

Fryguy · 2018-05-14T19:03:43Z

Looks good @carbonin. Most of my questions are on the existing implementation, so no need to hold up this PR for those.

The only one I'm really curious about is #17407 (review)

carbonin · 2018-05-17T18:46:25Z

@jrafanie did you get a chance to take a look at this? I think this one is ready to go.

jrafanie · 2018-05-17T20:12:45Z

@carbonin still failing

  1) EmbeddedAnsibleWorker ObjectManagement concern #ensure_initial_objects skips playbook seeding for containers
     Failure/Error: ensure_plugin_playbooks_project_seeded(provider, connection)
     
       (#<EmbeddedAnsibleWorker id: 63000000000044, guid: "22a4969b-8085-4992-8c28-49f7e5eb48c0", status: "ready", started_on: nil, stopped_on: nil, last_heartbeat: nil, pid: 17806, queue_name: nil, type: "EmbeddedAnsibleWorker", percent_memory: nil, percent_cpu: nil, cpu_time: nil, os_priority: nil, memory_usage: nil, memory_size: nil, uri: nil, miq_server_id: nil, sql_spid: nil, proportional_set_size: nil, unique_set_size: nil>).ensure_plugin_playbooks_project_seeded(#<ManageIQ::Providers::EmbeddedAnsible::Provider id: 63000000000012, type: "ManageIQ::Providers::Embe...3000000000236, created_at: "2018-05-17 18:58:58", updated_at: "2018-05-17 18:58:58", tenant_id: nil>, #<Double "AnsibleAPIConnection">)
           expected: 0 times with any arguments
           received: 1 time with arguments: (#<ManageIQ::Providers::EmbeddedAnsible::Provider id: 63000000000012, type: "ManageIQ::Providers::Embe...3000000000236, created_at: "2018-05-17 18:58:58", updated_at: "2018-05-17 18:58:58", tenant_id: nil>, #<Double "AnsibleAPIConnection">)
     # ./app/models/embedded_ansible_worker/object_management.rb:9:in `ensure_initial_objects'
     # ./spec/models/embedded_ansible_worker_spec.rb:53:in `block (4 levels) in <top (required)>'

Can that test just be deleted now that containers aren't skipped? ^

…repo This is currently the case for DockerEmbeddedAnsible and ContainerEmbeddedAnsible so this will allow those cases to continue without the default project

miq-bot · 2018-05-17T20:25:57Z

Checked commits carbonin/manageiq@8c32282~...f8568b9 with ruby 2.3.3, rubocop 0.52.1, haml-lint 0.20.0, and yamllint 1.10.0
7 files checked, 0 offenses detected
Everything looks fine. 🏆

jrafanie · 2018-05-21T18:50:16Z

carbonin · 2018-09-13T13:22:49Z

Changing this to gaprindashvili/yes as not having this is breaking DockerEmbeddedAnsible with AWX.

The error looks like this:

[----] I, [2018-09-13T09:21:35.143849 #11824:866b3b0]  INFO -- : MIQ(EmbeddedAnsibleWorker::Runner#setup_ansible) Finished starting embedded ansible service.
[----] E, [2018-09-13T09:21:36.854678 #11824:866b3b0] ERROR -- : [ArgumentError]: can't find user for awx  Method:[block in method_missing]
[----] E, [2018-09-13T09:21:36.854772 #11824:866b3b0] ERROR -- : /usr/local/lib/ruby/2.3.0/fileutils.rb:1115:in `getpwnam'
/usr/local/lib/ruby/2.3.0/fileutils.rb:1115:in `fu_get_uid'
/usr/local/lib/ruby/2.3.0/fileutils.rb:1086:in `chown_R'
/var/www/miq/vmdb/app/models/embedded_ansible_worker/object_management.rb:94:in `chown_playbooks_tempdir'
/var/www/miq/vmdb/app/models/embedded_ansible_worker/object_management.rb:65:in `ensure_plugin_playbooks_project_seeded'
/var/www/miq/vmdb/app/models/embedded_ansible_worker/object_management.rb:11:in `ensure_initial_objects'
/var/www/miq/vmdb/app/models/embedded_ansible_worker/runner.rb:52:in `update_embedded_ansible_provider'
/var/www/miq/vmdb/app/models/embedded_ansible_worker/runner.rb:13:in `do_before_work_loop'
/var/www/miq/vmdb/app/models/embedded_ansible_worker/runner.rb:5:in `prepare'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:126:in `start'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:22:in `start_worker'
/var/www/miq/vmdb/app/models/embedded_ansible_worker.rb:20:in `block in start_monitor_thread'

carbonin added enhancement wip core/workers gaprindashvili/no labels May 10, 2018

carbonin assigned jrafanie May 10, 2018

carbonin requested a review from Fryguy May 10, 2018 21:58

carbonin added 3 commits May 14, 2018 10:09

Extract playbook consolidation logic to EmbeddedAnsible

7e090f5

https://www.pivotaltracker.com/story/show/157313841

Create a rake task for writing plugin ansible content to a directory

9b6fe1e

This task will be used to create an rpm for providing plugin ansible content to the embedded ansible container image https://www.pivotaltracker.com/story/show/157313841

carbonin force-pushed the container_playbook_seeding branch from a52eede to 9b6fe1e Compare May 14, 2018 14:10

carbonin changed the title ~~[WIP] Expose plugin ansible content consolidation as a rake task~~ Expose plugin ansible content consolidation as a rake task May 14, 2018

carbonin removed the wip label May 14, 2018

Fryguy reviewed May 14, 2018

View reviewed changes

Don't fail if the EmbeddedAnsible instance can't create the playbook …

f8568b9

…repo This is currently the case for DockerEmbeddedAnsible and ContainerEmbeddedAnsible so this will allow those cases to continue without the default project

carbonin force-pushed the container_playbook_seeding branch from db7c6f6 to f8568b9 Compare May 17, 2018 20:24

jrafanie merged commit 428d092 into ManageIQ:master May 21, 2018

jrafanie added this to the Sprint 86 Ending May 21, 2018 milestone May 21, 2018

carbonin added gaprindashvili/yes and removed gaprindashvili/no labels Sep 13, 2018

This was referenced Nov 16, 2018

Loop enabling embedded ansible #17738

Closed

Can't enable Embedded Ansible #17600

Closed

carbonin deleted the container_playbook_seeding branch August 16, 2019 17:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expose plugin ansible content consolidation as a rake task #17407

Expose plugin ansible content consolidation as a rake task #17407

carbonin commented May 10, 2018 •

edited

Loading

jrafanie commented May 14, 2018

carbonin commented May 14, 2018

carbonin commented May 14, 2018

Fryguy May 14, 2018

carbonin May 14, 2018

jrafanie May 14, 2018

Fryguy May 14, 2018

carbonin May 14, 2018

Fryguy May 14, 2018

carbonin May 14, 2018

jrafanie May 14, 2018

Fryguy commented May 14, 2018

carbonin commented May 17, 2018

jrafanie commented May 17, 2018

miq-bot commented May 17, 2018

jrafanie commented May 21, 2018

carbonin commented Sep 13, 2018

Expose plugin ansible content consolidation as a rake task #17407

Expose plugin ansible content consolidation as a rake task #17407

Conversation

carbonin commented May 10, 2018 • edited Loading

jrafanie commented May 14, 2018

carbonin commented May 14, 2018

carbonin commented May 14, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fryguy commented May 14, 2018

carbonin commented May 17, 2018

jrafanie commented May 17, 2018

miq-bot commented May 17, 2018

jrafanie commented May 21, 2018

carbonin commented Sep 13, 2018

carbonin commented May 10, 2018 •

edited

Loading