Upgrade 7.2.6 -> 7.3.0 fails

[mkinsley]

Expect
I should be able to upgrade from 7.2.6 to 7.3.0 tags

Observed
Upgrades fail with permission denied when performing stat on {{ splunk_home}}/etc/auth/splunk.secret

Repro Steps

1. docker-compose up -d
2. docker-compose logs splunkenterprise --follow
3. Wait for splunkweb to be available
4. change image tag in compose to ":7.3.0" or ":edge"
5. add env var SPLUNK_UPGRADE="true"
6. docker-compose up -d
7. docker-compose logs splunkenterprise

Docker Compose File

# docker run splunk/enterprise:7.0.3
# Options on how to review the EULA and accept it: 
# 1. docker run -it splunk/enterprisetrial:7.0.3
# 2. Add the following environment variable: SPLUNK_START_ARGS=--accept-license
# e.g., docker run -e "SPLUNK_START_ARGS=--accept-license" splunk/enterprisetrial 

# Support for Docker Compose v3, https://docs.docker.com/compose/overview/
version: '3'
volumes:
  opt-splunk-etc:
  opt-splunk-var:
services:
  splunkenterprise:
    #build: .
    hostname: splunkenterprise
    image: splunk/splunk:7.2.6
    image: splunk/splunk:7.3.0
    #image: splunk/splunk:edge
    environment:
      SPLUNK_START_ARGS: --accept-license --answer-yes
      DEBUG: "true"
      ANSIBLE_EXTRA_FLAGS: "-vvvv"
      SPLUNK_PASSWORD: 'icanhazpasswd'
      # SPLUNK_UPGRADE: "true"
      SPLUNK_ENABLE_LISTEN: 9997
      SPLUNK_ADD: tcp 1514
    volumes:
      - opt-splunk-etc:/opt/splunk/etc
      - opt-splunk-var:/opt/splunk/var
    ports:
      - "8000:8000"
      - "9997:9997"
      - "8088:8088"
      - "1514:1514"

[nwang92]

This should be fixed in the latest splunk/splunk:edge images after splunk/splunk-ansible#211 was merged. Closing this, but feel free to reopen.

[johan1252]

I am still seeing this issue when upgrading from 7.2.4 to splunk:7.3, splunk:7.3.1 or splunk:edge.

[root@ip-10-129-2-126 ec2-user]# docker logs 3460619d9414

PLAY [Run default Splunk provisioning] *****************************************
Thursday 22 August 2019  20:51:34 +0000 (0:00:00.026)       0:00:00.026 ******* 

TASK [Gathering Facts] *********************************************************
ok: [localhost]
Thursday 22 August 2019  20:51:36 +0000 (0:00:01.543)       0:00:01.569 ******* 
Thursday 22 August 2019  20:51:36 +0000 (0:00:00.024)       0:00:01.594 ******* 
Thursday 22 August 2019  20:51:36 +0000 (0:00:00.022)       0:00:01.617 ******* 
Thursday 22 August 2019  20:51:36 +0000 (0:00:00.092)       0:00:01.710 ******* 
included: /opt/ansible/roles/splunk_common/tasks/get_facts.yml for localhost
Thursday 22 August 2019  20:51:36 +0000 (0:00:00.041)       0:00:01.751 ******* 

TASK [splunk_common : Set privilege escalation user] ***************************
ok: [localhost]
Thursday 22 August 2019  20:51:36 +0000 (0:00:00.023)       0:00:01.774 ******* 

TASK [splunk_common : Check for existing installation] *************************
ok: [localhost]
Thursday 22 August 2019  20:51:36 +0000 (0:00:00.219)       0:00:01.994 ******* 

TASK [splunk_common : Set splunk install fact] *********************************
ok: [localhost]
Thursday 22 August 2019  20:51:36 +0000 (0:00:00.024)       0:00:02.019 ******* 

TASK [splunk_common : Check for existing splunk secret] ************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Permission denied"}

PLAY RECAP *********************************************************************
localhost                  : ok=5    changed=0    unreachable=0    failed=1    skipped=2    rescued=0    ignored=0   

Thursday 22 August 2019  20:51:36 +0000 (0:00:00.086)       0:00:02.105 ******* 
=============================================================================== 
Gathering Facts --------------------------------------------------------- 1.54s
splunk_common : Check for existing installation ------------------------- 0.22s
Provision role ---------------------------------------------------------- 0.09s
splunk_common : Check for existing splunk secret ------------------------ 0.09s
splunk_common : include_tasks ------------------------------------------- 0.04s
Determine captaincy ----------------------------------------------------- 0.02s
splunk_common : Set splunk install fact --------------------------------- 0.02s
splunk_common : Set privilege escalation user --------------------------- 0.02s
Execute pre-setup playbooks --------------------------------------------- 0.02s
splunkd is not running.

@nwang92 Anything that I can do to figure out why this is happening still?

[bb03]

@johan1252 Can you give us your reproduction steps?

[johan1252]

@bb03
Stopped container running splunk:7.2.4.
I launched the new 7.3.1 docker container like so, where /splunk-data was the same volume used with old container running 7.2.4.

docker run -d -p 8000:8000 -p 9997:9997 -e 'SPLUNK_START_ARGS=--accept-license --answer-yes' -e 'SPLUNK_PASSWORD=<admin password>' -v /splunk-data:/opt/splunk/etc -v /splunk-data:/opt/splunk/var splunk/splunk:7.3.1

Then to get the above log output using docker logs

[johan1252]

Was able to get around the issue by doing chmod 777 auth/ ; chmod 777 auth/splunk.secret.

[stephenmuss]

I've also had this issue upgrading from 7.2.4 to anything above 7.2.4.

Using Kubernetes statefulsets. After updating the image tag in the statefulsets and redeploying the recreated pods immediately fail with this error.

I'm also not very comfortable with the workaround above with chmod 777.

Currently the /opt/splunk/etc/auth directory looks like this

drwx------.  7 splunk splunk  4096 Feb  6  2019 auth

The /opt/splunk/etc/auth/splunk.secret looks like this.

-r--------. 1 splunk splunk  255 Mar  4  2019 splunk.secret

So far as I can tell this should be enough for the Splunk user that ansible assumes to be able to check for the file's existence. However, I continue to see the error

TASK [splunk_common : Check for existing splunk secret] ************************                                                                                                                         
fatal: [localhost]: FAILED! => {                                                                                                                                                                         
    "changed": false                                                                                                                                                                                     
}                                                                                                                                                                                                        
                                                                                                                                                                                                         
MSG:                                                                                                                                                                                                     
                                                                                                                                                                                                         
Permission denied

[dmpopoff]

the issue was reproduced then run 8.0.0 over 7.2.4 (stop 7.2.4, rm 7.2.4 container, run 8.0.0)
error is the same. chmod 777 to auth folder and splunk.secret file helps to workaround but I think this is not appropriate.

here is the error message:
Wednesday 27 November 2019 12:13:25 +0300 (0:00:00.042) 0:00:03.629 ****
TASK [splunk_common : Check for existing splunk secret] ************************
fatal: [localhost]: FAILED! => { "changed": false } MSG: Permission denied
PLAY RECAP *********************************************************************
localhost : ok=5 changed=0 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0

[dmpopoff]

Why issue is marked as "Closed"? Where is no decision, only workaround. @nwang92 @bb03 @jmeixensperger What are right permissions should be?

[jonnyyu]

the fix mentioned above was overridden by another change to use 'splunk.user'

splunk/splunk-ansible@4ab2caa

but the second change actually cause problem. In old splunk image, user id of splunk is 999. But in new splunk image, the ansible user id is 999, the splunk user id is 41812.

[nwang92]

Sorry I seem to have stopped getting emails about this issue until the last comment. Let me take a look at the 7.2.x to 8.x upgrade path.

Regarding @jonnyyu, the old Splunk image is not compatible with the new Splunk image: https://github.com/splunk/docker-splunk/blob/develop/docs/INTRODUCTION.md#history