splunk.service: Refusing to accept PID outside of service control group #185
Comments
Reproducing the error with Chef-Splunk Kitchen test with EC2-driverI can run the chef-splunk kitchen test with dokken successfully without reproducing the error. If I swap the dokken driver with the ec2 driver and add a very simple Inspec test I can reproduce our error in the The Inspec test: describe service('splunk') do
it { should be_installed }
it { should be_enabled }
it { should be_running }
endThe result: Investigating on the instance: Shell outputkitchen.ymlkitchen.yml |
|
@JonoRicci can you show what your systemd unit file looks like? If you're calling the client recipe directly, you may not be setting up the splunk auth attributes. There is logic in the default recipe that reads the splunk admin user/pass from a data bag or from chef-vault. |
|
you need to have this in an encrypted data bag or chef-vault item: |
4 tasks
|
Hi @haidangwa, I've created PR #186 that adds some inspec tests to chef-splunk for the client suite that shows the issue we are seeing without our wrapper cookbook. The output of verify on ubuntu-2004 is as follows: To me the cause seems with the starting of splunk to accept the license, if I login to the docker container and stop splunk with Edit: Made it clearer we see these issues directly with chef-splunk and added summary to the verify output. |
|
@jjm Have you accepted the license? There is one way and only one way to accept the license: https://github.com/chef-cookbooks/chef-splunk#license-acceptance |
|
@haidangwa Yes, it's done by this line of the EDIT: Linked to chef license acceptance, not splunk. |
|
The failure mode can be seem at https://github.com/chef-cookbooks/chef-splunk/pull/186/checks?check_run_id=1260401551 too. |
haidangwa
added a commit
that referenced
this issue
Oct 17, 2020
* a startup issue was resolved for SplunkForwarder installations with an improved
systemd unit file (fix below)
* Adds Inspec tests to verify from SplunkForwarder starts (thanks, @jjm)
- Fixes Issue [#187](#187)
* the systemd unit file is now relegated to the `splunk enable boot-start` command to manage
- Adds Inspec tests and sets the verifier in Test Kitchen
Signed-off-by: Dang H. Nguyen <dang.nguyen@disney.com>
haidangwa
added a commit
that referenced
this issue
Oct 18, 2020
* a startup issue was resolved for SplunkForwarder installations with an improved
systemd unit file (fix below)
* Adds Inspec tests to verify from SplunkForwarder starts (thanks, @jjm)
- Fixes Issue [#187](#187)
* the systemd unit file is now relegated to the `splunk enable boot-start` command to manage
- Adds Inspec tests and sets the verifier in Test Kitchen for some test suites; some are still in serverspec
- Render the user-seed.conf with a file resource rather than a template
- The default recipe no longer includes the disable recipe; to disable splunk, add `recipe[chef-splunk::disabled]` to a run list explicitly
- Disabling splunk will no longer uninstall Splunk Enterprise nor the Splunk Universal Forwarder
- Adds `#SecretsHelper` to aid with secrets rotation and maintaining idempotency for handling Splunk's hashed secret values
- Improved guards to prevent `service[splunk]` restart/start when it should be disabled.
Signed-off-by: Dang H. Nguyen <dang.nguyen@disney.com>
|
Happening the same here. In my case, Recipe: chef-splunk::service
* service[splunk] action restart
- restart service service[splunk]
Running handlers:
Running handlers complete
Chef Infra Client finished, 22/44 resources updated in 36 secondsbut after that if I run kitchen verify: System Package splunkforwarder
✔ is expected to be installed
Service splunk
✔ is expected to be installed
✔ is expected to be enabled
× is expected to be running
expected that `Service splunk` is running
Test Summary: 3 successful, 1 failure, 0 skippedThe interesting thing is what root@default-ubuntu-1804:/# ps -aux | grep splunk
root 839 1.6 1.0 294276 80752 ? Sl 20:28 0:00 splunkd -p 8089 restart
root 840 0.0 0.1 87852 13584 ? Ss 20:28 0:00 [splunkd pid=839] splunkd -p 8089 restart [process-runner]
root 965 0.0 0.0 11460 1028 pts/0 S+ 20:29 0:00 grep --color=auto splunkit seems to me that the problem is in the restart of the service. If I kill those processes and I converge again, then everything is fine: System Package splunkforwarder
✔ is expected to be installed
Service splunk
✔ is expected to be installed
✔ is expected to be enabled
✔ is expected to be running
Test Summary: 4 successful, 0 failures, 0 skipped |
haidangwa
added a commit
that referenced
this issue
Oct 20, 2020
* - Fixes Issue [#185](#185) * a startup issue was resolved for SplunkForwarder installations with an improved systemd unit file (fix below) * Adds Inspec tests to verify from SplunkForwarder starts (thanks, @jjm) - Fixes Issue [#187](#187) * the systemd unit file is now relegated to the `splunk enable boot-start` command to manage - Adds Inspec tests and sets the verifier in Test Kitchen for some test suites; some are still in serverspec - Render the user-seed.conf with a file resource rather than a template - The default recipe no longer includes the disable recipe; to disable splunk, add `recipe[chef-splunk::disabled]` to a run list explicitly - Disabling splunk will no longer uninstall Splunk Enterprise nor the Splunk Universal Forwarder - Adds `#SecretsHelper` to aid with secrets rotation and maintaining idempotency for handling Splunk's hashed secret values - Improved guards to prevent `service[splunk]` restart/start when it should be disabled. Signed-off-by: Dang H. Nguyen <dang.nguyen@disney.com> * fix some chefspecs Signed-off-by: Dang H. Nguyen <dang.nguyen@disney.com> * updates test matrix in Github actions ci workflow Signed-off-by: Dang H. Nguyen <dang.nguyen@disney.com> * fixes this error condition when executing the `#splunk_login_successful?` helper method: ``` Errno::ENOENT ------------- No such file or directory - /opt/splunk/bin/splunk ``` Signed-off-by: Dang H. Nguyen <dang.nguyen@disney.com> * fixes a typo: the question is "should_not" not "shuold_not" Signed-off-by: Dang H. Nguyen <dang.nguyen@disney.com> * - uses `#splunk_secret_inspect` in the search head clustering server.conf.erb - updates inspec - disables Splunk's file locking verification on startup during Test Kitchen runs Signed-off-by: Dang H. Nguyen <dang.nguyen@disney.com> * fixes inspec tests for uninstall_forwarder and server-cluster-master suites Signed-off-by: Dang H. Nguyen <dang.nguyen@disney.com> * fixes inspec tests Signed-off-by: Dang H. Nguyen <dang.nguyen@disney.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Myself and @jjm have encountered the following problem and would be very grateful for any assistance.
Expected Behaviour
I want to install the splunk universal forwarder in my AWS EC2 environment.
I am using a wrapper cookbook which only determines the host OS and passes through a private appropriate installation URL to the chef-splunk cookbook. In my wrapper cookbook I am calling the
chef-splunk::clientrecipe directly.Actual Behaviour
On Ubuntu 16.04, 18.04 and 20.04 (using the latest images via the ec2-driver) my Kitchen Test in my wrapper cookbook fails to converge.
Below is the error output from Ubuntu 20.04.
Error output
Further investigation reveals:
systemctl status splunk.service
Details
chef-splunkcookbook version: 6.3.0Workaround
I have a manual workaround:
kitchen loginkillthe splunk process.service splunk startThis successfully launches the splunk service:
Shell output
You will notice the
splunk.service: Failed to parse PID from file /opt/splunkforwarder/var/run/splunk/splunkd.pid: Invalid argumentis still present even on a successful start.This leads me to be unsure whether the PID is the root error or a red herring in this case.
Stack trace
Stack trace
The text was updated successfully, but these errors were encountered: