Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UF Containers broken for all 9.x Images #557

Closed
mag-mkorn opened this issue Sep 27, 2022 · 16 comments
Closed

UF Containers broken for all 9.x Images #557

mag-mkorn opened this issue Sep 27, 2022 · 16 comments
Assignees

Comments

@mag-mkorn
Copy link

Hi folks,

apparently all the UF Images for Version 9.x seem to be broken.

The ansible playbooks always get stuck at task Get Splunk status.

I tried to troubleshoot the problem, but wasn't able to find the problem, yet. Here are my findings so far:

  • The Warning "Attempting to revert the SPLUNK_HOME ownership" is written to user-seed.conf. It's not really nice, but doesn't seem to be the problem.
  • When playbook execution is paused before the failing task, I can manually execute the status command without any problems. Playbook execution can be continued after the manual execution as well and runs without problems.
  • When the playbook runs into the problematic state and I run the status command afterwards, splunk tries to perform an upgrade and fails.
  • The failing task performs some, but not all the actions that happen during the manual command execution. Certificates get created in both cases, but the failing task does not remove the ftr file.

I don't know why the same command fails when it is executed by ansible. All could image is that some environment variables could differ. I didn't check this, yet.

chuck-confluent referenced this issue in splunk/splunk-ansible Oct 25, 2022
When a Pod resets, a new pod comes up and uses the same persistent
`etc` and `var` locations. Pod reset causes the file permissions to
change. Open permissions on `var/lib/splunk/kvstore/mongo/splunk.key`
causing the Mongod 4.2.17 to come up during the migration path, that
way the mongod is marked to start with 3.6.x, and that fails forever
as the existing mongod db is already 4.2.x compatiable.
@alishamayor alishamayor self-assigned this Nov 4, 2022
@chipzzz
Copy link

chipzzz commented Dec 5, 2022

@alishamayor , has this been tested before it has been approved? Because it broke everything. splunk/splunk-ansible@81b552d#commitcomment-87802600

@Adam2Marsh
Copy link

I've raised this to splunk via there support portal; case ref is 3200123

Will update this once I hear back.

@Adam2Marsh
Copy link

I got the following response; it's a known issue


This is a known Docker bug, its currently working in progress with Expected fix: 9.1.0
As of now the workaround is as follows

Workaround:

  • Remove: /opt/splunkforwarder/etc/myinstall/splunkd.xml file then start Splunk

Or this can be confgured:

  • Add the following to the compose file:
    splunk:
    tty: true

https://community.splunk.com/t5/Installation/Upgrading-Universal-Forwarder-8-x-x-to-9-x-x-does-not-work/m-p/619919_

I've asked for a date when version 9.1.0 will be released.

@gergelydavidwork
Copy link

I got the following response; it's a known issue

This is a known Docker bug, its currently working in progress with Expected fix: 9.1.0 As of now the workaround is as follows

Workaround:

  • Remove: /opt/splunkforwarder/etc/myinstall/splunkd.xml file then start Splunk

Or this can be confgured:

  • Add the following to the compose file:
    splunk:
    tty: true

https://community.splunk.com/t5/Installation/Upgrading-Universal-Forwarder-8-x-x-to-9-x-x-does-not-work/m-p/619919_

I've asked for a date when version 9.1.0 will be released.

Hi @Adam2Marsh ,
Did you get any response maybe when this will be released?
Thank you!

@Adam2Marsh
Copy link

Hey @gergelydavidwork

This was fixed in 9.0.5; I haven't tested it myself but assured it is.

@gergelydavidwork
Copy link

gergelydavidwork commented Jun 8, 2023

Thanks @Adam2Marsh, but I have tried 9.0.5 and it is NOT fixed. That's why I asked if you got any response for the release date of 9.1.0

@tnedeczki
Copy link

I can't believe why it's taking so much time to fix this bug, which has been going on since September. Instead, you have to use 8.2.X versions which are quite risky from a security point of view.

The whole ansible process gets stuck when it comes to a task where it needs to call ./splunk.

For example:

If you go inside the container and kill of those process, you will get the following error message:

Error calling execve(): No such file or directory
Error launching  command: Invalid argument

@Adam2Marsh
Copy link

Hey All,

Re-raising back to splunk now... as I've doubled check and yep you still need the workaround to make 9.0.5

@adityapinglesf
Copy link
Contributor

adityapinglesf commented Jul 13, 2023

9.0.5 was updated yesterday. I was able to get the UF running with the following command -

 docker run -p 8000:8000 -p 8089:8089 -e "SPLUNK_PASSWORD=trypass@" -e "SPLUNK_START_ARGS=--accept-license" --name uf-1 -it splunk/universalforwarder:9.0.5

Can you try now with the newly released image?

@tnedeczki
Copy link

Hi @adityapinglesf

Thanks, it is working now.

@Iammusa18
Copy link

Iammusa18 commented Jul 13, 2023

Hi,
@adityapinglesf @tnedeczki while it is encouraging to hear its working for you... i have ran the exact same command but 9.0.5 it still NOT working and it is stuck on below and never progresses beyond that...So no, 9.0.5 does not seem fixed at all. If someone has got this to work, please share any workaround you've employed. cheers

@Adam2Marsh can you please suggest how that workaround can be run as part of a Dockerfile etc..?

Thursday 13 July 2023  16:49:19 +0000 (0:00:00.062)       0:00:08.499 *********

TASK [splunk_common : include_tasks] *************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_splunk_status.yml for localhost
ok: [localhost]
Thursday 13 July 2023  16:49:19 +0000 (0:00:00.242)       0:00:08.742 *********

@splunk when will this be fixed? It does not make sense to release newer versions if your offical docker image does not support or is incompatible with said version? Thanks

@bdalpe
Copy link

bdalpe commented Jul 13, 2023

@Iammusa18 double check you're not running an old cached version of the 9.0.5 image. A newer version was published with the same tag. Docker will not pull a new image if one is cached locally.

Run docker pull splunk/universalforwarder:9.0.5 to make sure it is up-to-date.

@tnedeczki
Copy link

Hi, @adityapinglesf @tnedeczki while it is encouraging to hear its working for you... i have ran the exact same command but 9.0.5 it still NOT working and it is stuck on below and never progresses beyond that...So no, 9.0.5 does not seem fixed at all. If someone has got this to work, please share any workaround you've employed. cheers

@Adam2Marsh can you please suggest how that workaround can be run as part of a Dockerfile etc..?

Thursday 13 July 2023  16:49:19 +0000 (0:00:00.062)       0:00:08.499 *********

TASK [splunk_common : include_tasks] *************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_splunk_status.yml for localhost
ok: [localhost]
Thursday 13 July 2023  16:49:19 +0000 (0:00:00.242)       0:00:08.742 *********

@splunk when will this be fixed? It does not make sense to release newer versions if your offical docker image does not support or is incompatible with said version? Thanks

yes, I ran into the same problem, but as @bdalpe mentions, you need to refresh the image.

➜  docker run -p 8000:8000 -p 8089:8089 -e "SPLUNK_PASSWORD=trypass@" -e "SPLUNK_START_ARGS=--accept-license" --name uf-1 -it splunk/universalforwarder:9.0.5
[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the controller starting with Ansible 2.12. Current version: 3.7.16 (default, Jul 13 2023, 00:11:50) [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)].
This feature will be removed from ansible-core in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: COMMAND_WARNINGS option, the command warnings feature is being removed. This feature will be removed from ansible-core in version 2.14. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: [defaults]callback_whitelist option, normalizing names to new standard, use callbacks_enabled instead. This feature will be removed from ansible-core in version 2.15. Deprecation
warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [Run default Splunk provisioning] **************************************************************************************************************************************************************************
Friday 14 July 2023  07:57:32 +0000 (0:00:00.322)       0:00:00.322 ***********

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:35 +0000 (0:00:03.532)       0:00:03.855 ***********
Friday 14 July 2023  07:57:35 +0000 (0:00:00.136)       0:00:03.992 ***********

TASK [Provision role] *******************************************************************************************************************************************************************************************
Friday 14 July 2023  07:57:36 +0000 (0:00:00.230)       0:00:04.222 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_facts.yml for localhost
Friday 14 July 2023  07:57:36 +0000 (0:00:00.151)       0:00:04.374 ***********

TASK [splunk_common : Set privilege escalation user] ************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:36 +0000 (0:00:00.177)       0:00:04.551 ***********

TASK [splunk_common : Check for scloud] *************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:37 +0000 (0:00:01.093)       0:00:05.644 ***********
Friday 14 July 2023  07:57:37 +0000 (0:00:00.096)       0:00:05.741 ***********
Friday 14 July 2023  07:57:37 +0000 (0:00:00.064)       0:00:05.806 ***********

TASK [splunk_common : Check for existing installation] **********************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:38 +0000 (0:00:00.679)       0:00:06.486 ***********

TASK [splunk_common : Set splunk install fact] ******************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:38 +0000 (0:00:00.162)       0:00:06.648 ***********

TASK [splunk_common : Check for existing splunk secret] *********************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:39 +0000 (0:00:00.613)       0:00:07.262 ***********

TASK [splunk_common : Set first run fact] ***********************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:39 +0000 (0:00:00.129)       0:00:07.391 ***********

TASK [splunk_common : Set splunk_build_type fact] ***************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_facts_build_type.yml for localhost
Friday 14 July 2023  07:57:39 +0000 (0:00:00.131)       0:00:07.523 ***********
Friday 14 July 2023  07:57:39 +0000 (0:00:00.077)       0:00:07.601 ***********
Friday 14 July 2023  07:57:39 +0000 (0:00:00.087)       0:00:07.688 ***********
Friday 14 July 2023  07:57:39 +0000 (0:00:00.084)       0:00:07.773 ***********
Friday 14 July 2023  07:57:39 +0000 (0:00:00.115)       0:00:07.889 ***********
Friday 14 July 2023  07:57:40 +0000 (0:00:00.134)       0:00:08.023 ***********

TASK [splunk_common : Set target version fact] ******************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_facts_target_version.yml for localhost
Friday 14 July 2023  07:57:40 +0000 (0:00:00.214)       0:00:08.238 ***********
Friday 14 July 2023  07:57:40 +0000 (0:00:00.073)       0:00:08.312 ***********
Friday 14 July 2023  07:57:40 +0000 (0:00:00.072)       0:00:08.384 ***********
Friday 14 July 2023  07:57:40 +0000 (0:00:00.100)       0:00:08.485 ***********

TASK [splunk_common : Find manifests] ***************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:41 +0000 (0:00:00.974)       0:00:09.460 ***********

TASK [splunk_common : Set current version fact] *****************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:41 +0000 (0:00:00.211)       0:00:09.672 ***********

TASK [splunk_common : Setting upgrade fact] *********************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:41 +0000 (0:00:00.345)       0:00:10.017 ***********

TASK [splunk_common : Setting indexer cluster fact from config] *************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:42 +0000 (0:00:00.235)       0:00:10.253 ***********

TASK [splunk_common : Setting search head cluster fact from config] *********************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:42 +0000 (0:00:00.191)       0:00:10.444 ***********
Friday 14 July 2023  07:57:42 +0000 (0:00:00.109)       0:00:10.554 ***********
Friday 14 July 2023  07:57:42 +0000 (0:00:00.167)       0:00:10.722 ***********
Friday 14 July 2023  07:57:42 +0000 (0:00:00.130)       0:00:10.852 ***********

TASK [splunk_common : Detect service name] **********************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_facts_service_name.yml for localhost
Friday 14 July 2023  07:57:43 +0000 (0:00:00.268)       0:00:11.121 ***********

TASK [splunk_common : Setting service_name fact from config] ****************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:43 +0000 (0:00:00.216)       0:00:11.337 ***********
Friday 14 July 2023  07:57:43 +0000 (0:00:00.116)       0:00:11.454 ***********
Friday 14 July 2023  07:57:43 +0000 (0:00:00.141)       0:00:11.596 ***********
Friday 14 July 2023  07:57:43 +0000 (0:00:00.179)       0:00:11.776 ***********
Friday 14 July 2023  07:57:43 +0000 (0:00:00.236)       0:00:12.013 ***********
Friday 14 July 2023  07:57:44 +0000 (0:00:00.144)       0:00:12.157 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/change_splunk_directory_owner.yml for localhost
Friday 14 July 2023  07:57:44 +0000 (0:00:00.299)       0:00:12.456 ***********

TASK [splunk_common : Update Splunk directory owner] ************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:45 +0000 (0:00:01.288)       0:00:13.745 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/update_etc.yml for localhost
Friday 14 July 2023  07:57:46 +0000 (0:00:00.342)       0:00:14.087 ***********

TASK [splunk_common : Check if /sbin/updateetc.sh exists] *******************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:46 +0000 (0:00:00.810)       0:00:14.897 ***********

TASK [splunk_common : Update /opt/splunk/etc] *******************************************************************************************************************************************************************
changed: [localhost]
Friday 14 July 2023  07:57:48 +0000 (0:00:01.144)       0:00:16.042 ***********
Friday 14 July 2023  07:57:48 +0000 (0:00:00.124)       0:00:16.167 ***********
Friday 14 July 2023  07:57:48 +0000 (0:00:00.186)       0:00:16.354 ***********
Friday 14 July 2023  07:57:48 +0000 (0:00:00.117)       0:00:16.472 ***********
Friday 14 July 2023  07:57:48 +0000 (0:00:00.145)       0:00:16.618 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/remove_first_login.yml for localhost
Friday 14 July 2023  07:57:48 +0000 (0:00:00.153)       0:00:16.771 ***********

TASK [splunk_common : Create .ui_login] *************************************************************************************************************************************************************************
changed: [localhost]
Friday 14 July 2023  07:57:49 +0000 (0:00:00.838)       0:00:17.610 ***********
Friday 14 July 2023  07:57:49 +0000 (0:00:00.173)       0:00:17.784 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/set_splunk_secret.yml for localhost
Friday 14 July 2023  07:57:50 +0000 (0:00:00.265)       0:00:18.049 ***********
Friday 14 July 2023  07:57:50 +0000 (0:00:00.135)       0:00:18.185 ***********
Friday 14 July 2023  07:57:50 +0000 (0:00:00.121)       0:00:18.306 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/enable_admin_auth.yml for localhost
Friday 14 July 2023  07:57:50 +0000 (0:00:00.233)       0:00:18.540 ***********

TASK [splunk_common : Hash the password] ************************************************************************************************************************************************************************
changed: [localhost]
Friday 14 July 2023  07:57:52 +0000 (0:00:02.238)       0:00:20.778 ***********

TASK [splunk_common : Generate user-seed.conf (Linux)] **********************************************************************************************************************************************************
changed: [localhost] => (item=USERNAME)
changed: [localhost] => (item=HASHED_PASSWORD)
Friday 14 July 2023  07:57:54 +0000 (0:00:01.885)       0:00:22.664 ***********
Friday 14 July 2023  07:57:54 +0000 (0:00:00.203)       0:00:22.867 ***********
Friday 14 July 2023  07:57:55 +0000 (0:00:00.203)       0:00:23.071 ***********
Friday 14 July 2023  07:57:55 +0000 (0:00:00.131)       0:00:23.203 ***********
Friday 14 July 2023  07:57:55 +0000 (0:00:00.138)       0:00:23.341 ***********
Friday 14 July 2023  07:57:55 +0000 (0:00:00.070)       0:00:23.411 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/pre_splunk_start_commands.yml for localhost
Friday 14 July 2023  07:57:55 +0000 (0:00:00.187)       0:00:23.599 ***********
Friday 14 July 2023  07:57:55 +0000 (0:00:00.103)       0:00:23.703 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/enable_s2s.yml for localhost
Friday 14 July 2023  07:57:55 +0000 (0:00:00.223)       0:00:23.926 ***********
Friday 14 July 2023  07:57:56 +0000 (0:00:00.239)       0:00:24.166 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/s2s/configure_splunktcp.yml for localhost
Friday 14 July 2023  07:57:56 +0000 (0:00:00.396)       0:00:24.562 ***********

TASK [splunk_common : Enable splunktcp input] *******************************************************************************************************************************************************************
changed: [localhost]
Friday 14 July 2023  07:57:57 +0000 (0:00:00.813)       0:00:25.376 ***********

TASK [splunk_common : Remove splunktcp-ssl input] ***************************************************************************************************************************************************************
changed: [localhost]
Friday 14 July 2023  07:57:58 +0000 (0:00:00.749)       0:00:26.125 ***********

TASK [splunk_common : Remove input SSL settings] ****************************************************************************************************************************************************************
changed: [localhost]
Friday 14 July 2023  07:57:58 +0000 (0:00:00.759)       0:00:26.885 ***********

TASK [splunk_common : Reset root CA] ****************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:59 +0000 (0:00:00.708)       0:00:27.594 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/trigger_restart.yml for localhost
Friday 14 July 2023  07:57:59 +0000 (0:00:00.151)       0:00:27.746 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_splunk_status.yml for localhost
Friday 14 July 2023  07:57:59 +0000 (0:00:00.202)       0:00:27.948 ***********

TASK [splunk_common : Restrict permissions on splunk.key for Status] ********************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/restrict_permissions.yml for localhost => (item=/opt/splunkforwarder/var/lib/splunk/kvstore/mongo/splunk.key)
Friday 14 July 2023  07:58:00 +0000 (0:00:00.541)       0:00:28.490 ***********

TASK [splunk_common : Check if /opt/splunkforwarder/var/lib/splunk/kvstore/mongo/splunk.key exists] *************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:01 +0000 (0:00:01.144)       0:00:29.634 ***********
Friday 14 July 2023  07:58:01 +0000 (0:00:00.073)       0:00:29.708 ***********

TASK [splunk_common : Get Splunk status] ************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:02 +0000 (0:00:00.792)       0:00:30.501 ***********

TASK [splunk_common : Trigger restart] **************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:03 +0000 (0:00:00.605)       0:00:31.106 ***********
Friday 14 July 2023  07:58:03 +0000 (0:00:00.078)       0:00:31.185 ***********
Friday 14 July 2023  07:58:03 +0000 (0:00:00.108)       0:00:31.294 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/set_mgmt_port.yml for localhost
Friday 14 July 2023  07:58:03 +0000 (0:00:00.156)       0:00:31.451 ***********

TASK [splunk_common : Set mgmt port] ****************************************************************************************************************************************************************************
changed: [localhost]
Friday 14 July 2023  07:58:04 +0000 (0:00:00.750)       0:00:32.201 ***********
Friday 14 July 2023  07:58:04 +0000 (0:00:00.064)       0:00:32.266 ***********
Friday 14 July 2023  07:58:04 +0000 (0:00:00.099)       0:00:32.365 ***********
Friday 14 July 2023  07:58:04 +0000 (0:00:00.103)       0:00:32.469 ***********
Friday 14 July 2023  07:58:04 +0000 (0:00:00.125)       0:00:32.595 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/enable_splunkd_ssl.yml for localhost
Friday 14 July 2023  07:58:04 +0000 (0:00:00.227)       0:00:32.823 ***********

TASK [splunk_common : Enable Splunkd SSL] ***********************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:05 +0000 (0:00:00.704)       0:00:33.527 ***********
Friday 14 July 2023  07:58:05 +0000 (0:00:00.064)       0:00:33.592 ***********
Friday 14 July 2023  07:58:05 +0000 (0:00:00.065)       0:00:33.657 ***********
Friday 14 July 2023  07:58:05 +0000 (0:00:00.066)       0:00:33.724 ***********
Friday 14 July 2023  07:58:05 +0000 (0:00:00.057)       0:00:33.782 ***********
Friday 14 July 2023  07:58:05 +0000 (0:00:00.134)       0:00:33.916 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/enable_forwarding.yml for localhost
Friday 14 July 2023  07:58:06 +0000 (0:00:00.251)       0:00:34.168 ***********
Friday 14 July 2023  07:58:06 +0000 (0:00:00.138)       0:00:34.306 ***********
Friday 14 July 2023  07:58:06 +0000 (0:00:00.168)       0:00:34.475 ***********
Friday 14 July 2023  07:58:06 +0000 (0:00:00.162)       0:00:34.637 ***********

TASK [splunk_common : Disable indexing on the current node] *****************************************************************************************************************************************************
changed: [localhost]
Friday 14 July 2023  07:58:07 +0000 (0:00:00.588)       0:00:35.226 ***********
Friday 14 July 2023  07:58:07 +0000 (0:00:00.114)       0:00:35.341 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/trigger_restart.yml for localhost
Friday 14 July 2023  07:58:07 +0000 (0:00:00.120)       0:00:35.461 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_splunk_status.yml for localhost
Friday 14 July 2023  07:58:07 +0000 (0:00:00.109)       0:00:35.571 ***********

TASK [splunk_common : Restrict permissions on splunk.key for Status] ********************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/restrict_permissions.yml for localhost => (item=/opt/splunkforwarder/var/lib/splunk/kvstore/mongo/splunk.key)
Friday 14 July 2023  07:58:07 +0000 (0:00:00.112)       0:00:35.683 ***********

TASK [splunk_common : Check if /opt/splunkforwarder/var/lib/splunk/kvstore/mongo/splunk.key exists] *************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:08 +0000 (0:00:00.756)       0:00:36.439 ***********
Friday 14 July 2023  07:58:08 +0000 (0:00:00.065)       0:00:36.505 ***********

TASK [splunk_common : Get Splunk status] ************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:09 +0000 (0:00:00.575)       0:00:37.080 ***********

TASK [splunk_common : Trigger restart] **************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:09 +0000 (0:00:00.525)       0:00:37.606 ***********
Friday 14 July 2023  07:58:09 +0000 (0:00:00.052)       0:00:37.659 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/start_splunk.yml for localhost
Friday 14 July 2023  07:58:09 +0000 (0:00:00.182)       0:00:37.841 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_splunk_status.yml for localhost
Friday 14 July 2023  07:58:09 +0000 (0:00:00.095)       0:00:37.937 ***********

TASK [splunk_common : Restrict permissions on splunk.key for Status] ********************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/restrict_permissions.yml for localhost => (item=/opt/splunkforwarder/var/lib/splunk/kvstore/mongo/splunk.key)
Friday 14 July 2023  07:58:10 +0000 (0:00:00.094)       0:00:38.032 ***********

TASK [splunk_common : Check if /opt/splunkforwarder/var/lib/splunk/kvstore/mongo/splunk.key exists] *************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:10 +0000 (0:00:00.588)       0:00:38.620 ***********
Friday 14 July 2023  07:58:10 +0000 (0:00:00.075)       0:00:38.696 ***********

TASK [splunk_common : Get Splunk status] ************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:11 +0000 (0:00:00.608)       0:00:39.304 ***********

TASK [splunk_common : Cleanup Splunk runtime files] *************************************************************************************************************************************************************
ok: [localhost] => (item=/opt/splunkforwarder/var/run/splunk/splunkd.pid)
ok: [localhost] => (item=/opt/splunkforwarder/var/lib/splunk/kvstore/mongo/mongod.lock)
Friday 14 July 2023  07:58:12 +0000 (0:00:01.279)       0:00:40.584 ***********

TASK [splunk_common : Restrict permissions on splunk.key] *******************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/restrict_permissions.yml for localhost => (item=/opt/splunkforwarder/var/lib/splunk/kvstore/mongo/splunk.key)
Friday 14 July 2023  07:58:12 +0000 (0:00:00.113)       0:00:40.697 ***********

TASK [splunk_common : Check if /opt/splunkforwarder/var/lib/splunk/kvstore/mongo/splunk.key exists] *************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:13 +0000 (0:00:00.462)       0:00:41.160 ***********
Friday 14 July 2023  07:58:13 +0000 (0:00:00.054)       0:00:41.214 ***********

TASK [splunk_common : Start Splunk via CLI] *********************************************************************************************************************************************************************
changed: [localhost]
Friday 14 July 2023  07:58:17 +0000 (0:00:04.358)       0:00:45.573 ***********
Friday 14 July 2023  07:58:17 +0000 (0:00:00.097)       0:00:45.670 ***********
Friday 14 July 2023  07:58:17 +0000 (0:00:00.113)       0:00:45.783 ***********

TASK [splunk_common : Wait for splunkd management port] *********************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:19 +0000 (0:00:01.757)       0:00:47.541 ***********
Friday 14 July 2023  07:58:19 +0000 (0:00:00.030)       0:00:47.572 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/set_certificate_prefix.yml for localhost
Friday 14 July 2023  07:58:19 +0000 (0:00:00.416)       0:00:47.989 ***********

TASK [splunk_common : Test basic https endpoint] ****************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:21 +0000 (0:00:02.005)       0:00:49.994 ***********

TASK [splunk_common : Set url prefix for future REST calls] *****************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:22 +0000 (0:00:00.116)       0:00:50.111 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/clean_user_seed.yml for localhost
Friday 14 July 2023  07:58:22 +0000 (0:00:00.195)       0:00:50.306 ***********

TASK [splunk_common : Remove user-seed.conf] ********************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:22 +0000 (0:00:00.592)       0:00:50.898 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/add_splunk_license.yml for localhost
Friday 14 July 2023  07:58:23 +0000 (0:00:00.209)       0:00:51.108 ***********

TASK [splunk_common : Initialize licenses array] ****************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:23 +0000 (0:00:00.141)       0:00:51.249 ***********

TASK [splunk_common : Determine available licenses] *************************************************************************************************************************************************************
ok: [localhost] => (item=splunk.lic)
Friday 14 July 2023  07:58:23 +0000 (0:00:00.161)       0:00:51.411 ***********

TASK [splunk_common : Apply licenses] ***************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/apply_licenses.yml for localhost => (item=splunk.lic)
Friday 14 July 2023  07:58:23 +0000 (0:00:00.202)       0:00:51.614 ***********
Friday 14 July 2023  07:58:23 +0000 (0:00:00.204)       0:00:51.818 ***********
Friday 14 July 2023  07:58:24 +0000 (0:00:00.240)       0:00:52.059 ***********
Friday 14 July 2023  07:58:24 +0000 (0:00:00.099)       0:00:52.159 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/licenses/add_license.yml for localhost
Friday 14 July 2023  07:58:24 +0000 (0:00:00.178)       0:00:52.338 ***********
Friday 14 July 2023  07:58:24 +0000 (0:00:00.097)       0:00:52.435 ***********

TASK [splunk_common : Ensure license path] **********************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:25 +0000 (0:00:00.793)       0:00:53.229 ***********
Friday 14 July 2023  07:58:25 +0000 (0:00:00.077)       0:00:53.306 ***********
Friday 14 July 2023  07:58:25 +0000 (0:00:00.065)       0:00:53.371 ***********
Friday 14 July 2023  07:58:25 +0000 (0:00:00.024)       0:00:53.395 ***********
Friday 14 July 2023  07:58:25 +0000 (0:00:00.126)       0:00:53.521 ***********

TASK [splunk_universal_forwarder : include_tasks] ***************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_universal_forwarder/tasks/../../../roles/splunk_common/tasks/set_as_hec_receiver.yml for localhost
Friday 14 July 2023  07:58:25 +0000 (0:00:00.109)       0:00:53.631 ***********

TASK [splunk_universal_forwarder : Setup global HEC] ************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:27 +0000 (0:00:01.751)       0:00:55.382 ***********

TASK [splunk_universal_forwarder : Get existing HEC token] ******************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:28 +0000 (0:00:01.300)       0:00:56.682 ***********
Friday 14 July 2023  07:58:28 +0000 (0:00:00.072)       0:00:56.755 ***********
Friday 14 July 2023  07:58:28 +0000 (0:00:00.094)       0:00:56.850 ***********
Friday 14 July 2023  07:58:28 +0000 (0:00:00.092)       0:00:56.942 ***********
Friday 14 July 2023  07:58:28 +0000 (0:00:00.031)       0:00:56.974 ***********
Friday 14 July 2023  07:58:29 +0000 (0:00:00.071)       0:00:57.046 ***********
Friday 14 July 2023  07:58:29 +0000 (0:00:00.057)       0:00:57.103 ***********
Friday 14 July 2023  07:58:29 +0000 (0:00:00.057)       0:00:57.161 ***********
Friday 14 July 2023  07:58:29 +0000 (0:00:00.083)       0:00:57.244 ***********

TASK [splunk_universal_forwarder : include_tasks] ***************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_universal_forwarder/tasks/../../../roles/splunk_common/tasks/check_for_required_restarts.yml for localhost
Friday 14 July 2023  07:58:29 +0000 (0:00:00.108)       0:00:57.352 ***********

TASK [splunk_universal_forwarder : Check for required restarts] *************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:30 +0000 (0:00:01.416)       0:00:58.769 ***********
Friday 14 July 2023  07:58:30 +0000 (0:00:00.125)       0:00:58.895 ***********

TASK [Check all instances for required restarts] ****************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/check_for_required_restarts.yml for localhost
Friday 14 July 2023  07:58:30 +0000 (0:00:00.122)       0:00:59.017 ***********

TASK [Check for required restarts] ******************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:32 +0000 (0:00:01.295)       0:01:00.312 ***********

PLAY RECAP ******************************************************************************************************************************************************************************************************
localhost                  : ok=81   changed=10   unreachable=0    failed=0    skipped=70   rescued=0    ignored=0

Friday 14 July 2023  07:58:32 +0000 (0:00:00.060)       0:01:00.373 ***********
===============================================================================
splunk_common : Start Splunk via CLI --------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4.36s
Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 3.53s
splunk_common : Hash the password ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2.24s
splunk_common : Test basic https endpoint ---------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.01s
splunk_common : Generate user-seed.conf (Linux) ---------------------------------------------------------------------------------------------------------------------------------------------------------- 1.89s
splunk_common : Wait for splunkd management port --------------------------------------------------------------------------------------------------------------------------------------------------------- 1.76s
splunk_universal_forwarder : Setup global HEC ------------------------------------------------------------------------------------------------------------------------------------------------------------ 1.75s
splunk_universal_forwarder : Check for required restarts ------------------------------------------------------------------------------------------------------------------------------------------------- 1.42s
splunk_universal_forwarder : Get existing HEC token ------------------------------------------------------------------------------------------------------------------------------------------------------ 1.30s
Check for required restarts ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 1.30s
splunk_common : Update Splunk directory owner ------------------------------------------------------------------------------------------------------------------------------------------------------------ 1.29s
splunk_common : Cleanup Splunk runtime files ------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.28s
splunk_common : Update /opt/splunk/etc ------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.14s
splunk_common : Check if /opt/splunkforwarder/var/lib/splunk/kvstore/mongo/splunk.key exists ------------------------------------------------------------------------------------------------------------- 1.14s
splunk_common : Check for scloud ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.09s
splunk_common : Find manifests --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.97s
splunk_common : Create .ui_login ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.84s
splunk_common : Enable splunktcp input ------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.81s
splunk_common : Check if /sbin/updateetc.sh exists ------------------------------------------------------------------------------------------------------------------------------------------------------- 0.81s
splunk_common : Ensure license path ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.79s
===============================================================================

Ansible playbook complete, will begin streaming var/log/splunk/splunkd_stderr.log

@mag-mkorn
Copy link
Author

Thanks, the current 9.0.5 Image finally works again. I'll close the issue now.

@adityapinglesf
Copy link
Contributor

Hi, @adityapinglesf @tnedeczki while it is encouraging to hear its working for you... i have ran the exact same command but 9.0.5 it still NOT working and it is stuck on below and never progresses beyond that...So no, 9.0.5 does not seem fixed at all. If someone has got this to work, please share any workaround you've employed. cheers

@Adam2Marsh can you please suggest how that workaround can be run as part of a Dockerfile etc..?

Thursday 13 July 2023  16:49:19 +0000 (0:00:00.062)       0:00:08.499 *********

TASK [splunk_common : include_tasks] *************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_splunk_status.yml for localhost
ok: [localhost]
Thursday 13 July 2023  16:49:19 +0000 (0:00:00.242)       0:00:08.742 *********

@splunk when will this be fixed? It does not make sense to release newer versions if your offical docker image does not support or is incompatible with said version? Thanks

As suggested please try re-fetching the image 🙂

@Iammusa18
Copy link

Yeah my bad. Re-fetching image has worked!! Silly me. thanks all

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants