Can't enable Embedded Ansible

[smopihub]

Hi!

I try enable Embedded Ansible with no success:

tail -f evm.log | grep -i "EmbeddedAnsible"

[----] I, [2018-06-18T07:56:59.619206 #2156:b7dd9c] INFO -- : MIQ(DockerEmbeddedAnsible#start) Waiting for Ansible container to respond
[----] I, [2018-06-18T07:57:00.951231 #2156:b7dd9c] INFO -- : MIQ(DockerEmbeddedAnsible#start) Waiting for Ansible container to respond
[----] I, [2018-06-18T07:57:08.503775 #2156:b7dd9c] INFO -- : MIQ(EmbeddedAnsibleWorker::Runner#setup_ansible) Finished starting embedded ansible service.
[----] I, [2018-06-18T07:57:16.150046 #2156:b7dd9c] INFO -- : MIQ(EmbeddedAnsibleWorker#log_status) [Embedded Ansible Worker] Worker ID [69], PID [], GUID [aee9e33b-605f-4503-b2fb-ab87da354287], Last Heartbeat [2018-06-18 11:56:27 UTC], Process Info: Memory Usage [], Memory Size [], Proportional Set Size: [], Unique Set Size: [], Memory % [], CPU Time [], CPU % [], Priority []
[----] E, [2018-06-18T07:57:16.150310 #2156:b7dd9c] ERROR -- : MIQ(EmbeddedAnsibleWorker::Runner) ID [69] PID [2156] GUID [aee9e33b-605f-4503-b2fb-ab87da354287] {"detail":"A server error has occurred."} Worker exiting.
[----] I, [2018-06-18T07:57:16.150401 #2156:b7dd9c] INFO -- : MIQ(EmbeddedAnsibleWorker#start_monitor_thread) SystemExit received, exiting monitoring Thread
[----] I, [2018-06-18T07:57:16.218982 #2156:b65134] INFO -- : MIQ(MiqServer#clean_worker_records) SQL Record for Worker [EmbeddedAnsibleWorker] with ID: [69], PID: [], GUID: [aee9e33b-605f-4503-b2fb-ab87da354287], Status: [aborted] is being deleted
[----] I, [2018-06-18T07:57:31.809991 #2156:b65134] INFO -- : MIQ(EmbeddedAnsibleWorker.sync_workers) Workers are being synchronized: Current #: [0], Desired #: [1]
[----] I, [2018-06-18T07:57:31.923288 #2156:b65134] INFO -- : MIQ(MiqQueue.put) Message id: [310], id: [], Zone: [default], Role: [], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqEvent.raise_evm_event], Timeout: [600], Priority: [100], State: [ready], Deliver On: [], Data: [], Args: [["MiqServer", 1], "evm_worker_start", {:event_details=>"Worker started: ID [70], PID [], GUID [f6dce890-8b16-48ef-9de1-3c3ad1da60d1]", :type=>"EmbeddedAnsibleWorker"}]
[----] I, [2018-06-18T07:57:31.923415 #2156:b65134] INFO -- : MIQ(EmbeddedAnsibleWorker#start) Worker started: ID [70], PID [], GUID [f6dce890-8b16-48ef-9de1-3c3ad1da60d1]
[----] I, [2018-06-18T07:57:32.098069 #2156:75e7700] INFO -- : MIQ(EmbeddedAnsibleWorker::Runner#sync_config) ID [70], PID [2156], GUID [f6dce890-8b16-48ef-9de1-3c3ad1da60d1], Zone [default], Active Roles [automate,database_operations,database_owner,embedded_ansible,ems_inventory,ems_operations,event,reporting,scheduler,smartstate,user_interface,web_services,websocket], Assigned Roles [automate,database_operations,database_owner,embedded_ansible,ems_inventory,ems_operations,event,reporting,scheduler,smartstate,user_interface,web_services,websocket], Configuration:
[----] I, [2018-06-18T07:57:32.559610 #2156:75e7700] INFO -- : MIQ(EmbeddedAnsibleWorker::Runner#setup_ansible) Starting embedded ansible service ...
[----] I, [2018-06-18T07:57:34.660730 #3188:b65134] INFO -- : MIQ(MiqGenericWorker::Runner#get_message_via_drb) Message id: [310], MiqWorker id: [38], Zone: [default], Role: [], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqEvent.raise_evm_event], Timeout: [600], Priority: [100], State: [dequeue], Deliver On: [], Data: [], Args: [["MiqServer", 1], "evm_worker_start", {:event_details=>"Worker started: ID [70], PID [], GUID [f6dce890-8b16-48ef-9de1-3c3ad1da60d1]", :type=>"EmbeddedAnsibleWorker"}], Dequeued in: [2.744226555] seconds
[----] I, [2018-06-18T07:57:34.691883 #3188:b65134] INFO -- : MIQ(MiqQueue.put) Message id: [311], id: [], Zone: [default], Role: [automate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqAeEngine.deliver], Timeout: [3600], Priority: [20], State: [ready], Deliver On: [], Data: [], Args: [{:object_type=>"MiqServer", :object_id=>1, :attrs=>{:event_type=>"evm_worker_start", :event_details=>"Worker started: ID [70], PID [], GUID [f6dce890-8b16-48ef-9de1-3c3ad1da60d1]", :type=>"EmbeddedAnsibleWorker", "MiqEvent::miq_event"=>80, :miq_event_id=>80, "EventStream::event_stream"=>80, :event_stream_id=>80}, :instance_name=>"Event", :user_id=>1, :miq_group_id=>2, :tenant_id=>1, :automate_message=>nil}]
[----] I, [2018-06-18T07:57:38.762874 #23404:b65134] INFO -- : MIQ(MiqPriorityWorker::Runner#get_message_via_drb) Message id: [311], MiqWorker id: [58], Zone: [default], Role: [automate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqAeEngine.deliver], Timeout: [3600], Priority: [20], State: [dequeue], Deliver On: [], Data: [], Args: [{:object_type=>"MiqServer", :object_id=>1, :attrs=>{:event_type=>"evm_worker_start", :event_details=>"Worker started: ID [70], PID [], GUID [f6dce890-8b16-48ef-9de1-3c3ad1da60d1]", :type=>"EmbeddedAnsibleWorker", "MiqEvent::miq_event"=>80, :miq_event_id=>80, "EventStream::event_stream"=>80, :event_stream_id=>80}, :instance_name=>"Event", :user_id=>1, :miq_group_id=>2, :tenant_id=>1, :automate_message=>nil}], Dequeued in: [4.077434976] seconds
[----] I, [2018-06-18T07:57:38.772352 #23404:b65134] INFO -- : MIQ(MiqAeEngine.deliver) Delivering {:event_type=>"evm_worker_start", :event_details=>"Worker started: ID [70], PID [], GUID [f6dce890-8b16-48ef-9de1-3c3ad1da60d1]", :type=>"EmbeddedAnsibleWorker", "MiqEvent::miq_event"=>80, :miq_event_id=>80, "EventStream::event_stream"=>80, :event_stream_id=>80} for object [MiqServer.1] with state [] to Automate
[----] I, [2018-06-18T07:57:48.996727 #2156:b65134] INFO -- : MIQ(MiqServer#worker_set_message) Worker [EmbeddedAnsibleWorker] with ID: [70], PID: [], GUID: [f6dce890-8b16-48ef-9de1-3c3ad1da60d1] is being requested to sync_config
[----] I, [2018-06-18T07:58:03.624793 #2156:75e7700] INFO -- : MIQ(DockerEmbeddedAnsible#start) Waiting for Ansible container to respond
[----] I, [2018-06-18T07:58:05.055296 #2156:75e7700] INFO -- : MIQ(DockerEmbeddedAnsible#start) Waiting for Ansible container to respond

All help is welcome!

Kind Regards!

[smopihub]

[root@manageiq ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e249619df4cb ansible/awx_task:latest "/tini -- /bin/sh ..." 2 minutes ago Up 2 minutes 8052/tcp awx_task
91f2dcf273ef ansible/awx_web:latest "/tini -- /bin/sh ..." 2 minutes ago Up 2 minutes 0.0.0.0:54321->8052/tcp awx_web
f50e2db8bf1f memcached:alpine "docker-entrypoint..." 2 minutes ago Up 2 minutes 11211/tcp memcached
1509723120b0 rabbitmq:3 "docker-entrypoint..." 2 minutes ago Up 2 minutes 4369/tcp, 5671-5672/tcp, 25672/tcp rabbitmq

but after a few moment docker ps shows:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

[ghost]

I'm not sure it is related but with yesterday's build (manageiq-ovirt-gaprindashvili-20180704-afd03d7488.qc2), I have a similar behavior: ManageIQ continuously tries to start AWX and fails. In the awx_web container logs, I have the following stack trace right before the container dies:

2018-07-05 15:33:28,504 ERROR    django.request Internal Server Error: /api/v1/hosts/
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/usr/lib/python2.7/site-packages/awx/wsgi.py", line 65, in _legacy_get_response
    return super(AWXWSGIHandler, self)._legacy_get_response(request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/utils/decorators.py", line 185, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/generics.py", line 299, in dispatch
    return super(APIView, self).dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 494, in dispatch
    response = self.handle_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 454, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 491, in dispatch
    response = handler(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/generics.py", line 244, in post
    return self.create(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/mixins.py", line 21, in create
    self.perform_create(serializer)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/mixins.py", line 26, in perform_create
    serializer.save()
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/serializers.py", line 214, in save
    self.instance = self.create(validated_data)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/serializers.py", line 917, in create
    instance = ModelClass.objects.create(**validated_data)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py", line 394, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/lib/python2.7/site-packages/awx/main/models/inventory.py", line 700, in save
    super(Host, self).save(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/main/models/base.py", line 275, in save
    super(PrimordialModel, self).save(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/main/models/base.py", line 164, in save
    super(CreatedModifiedModel, self).save(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/base.py", line 808, in save
    force_update=force_update, update_fields=update_fields)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/base.py", line 848, in save_base
    update_fields=update_fields, raw=raw, using=using,
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 193, in send
    for receiver in self._live_receivers(sender)
  File "/usr/lib/python2.7/site-packages/awx/main/signals.py", line 146, in emit_update_inventory_on_created_or_deleted
    update_inventory_computed_fields.delay(inventory.id, True)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/task.py", line 461, in delay
    return self.apply_async(args, kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/task.py", line 573, in apply_async
    **dict(self._get_exec_options(), **options)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/base.py", line 345, in send_task
    options = router.route(options, name, args, kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/routes.py", line 50, in route
    route = self.lookup_route(task, args, kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/routes.py", line 78, in lookup_route
    return _first_route(self.routes, task, args, kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/utils/functional.py", line 233, in _matcher
    answer = getattr(maybe_evaluate(obj), method)(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/main/utils/ha.py", line 58, in route_for_task
    (changed, instance) = Instance.objects.get_or_register()
  File "/usr/lib/python2.7/site-packages/awx/main/managers.py", line 106, in get_or_register
    return (False, self.me())
  File "/usr/lib/python2.7/site-packages/awx/main/managers.py", line 88, in me
    raise RuntimeError("No instance found with the current cluster host id")
RuntimeError: No instance found with the current cluster host id

And right after the container dies, I have the same error message as @smopihub :

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Any idea ?

[skunkr]

Hi,
As i've posted here: #17738 , I am geeting errors comming out from awx_task container.
Then docker daemon restarts in a continous loop...

[carbonin]

I think this was solved in the Hammer RC-1 build.

I believe one of the main fixes that allows this to work in Hammer is #17407 which has been requested for Gaprindashvili backport.

[carbonin]

I think this is tracking the same issue as #17738 going to close this as a duplicate.