Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BREAKING: Ability to download from download.splunk.com #150

Merged
merged 5 commits into from
Oct 3, 2017
Merged

BREAKING: Ability to download from download.splunk.com #150

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e21b53c
Updated code to start using the *new* download path being followed by…
TraGicCode Oct 3, 2017
f1d8ca1
Add docker nodesets we want to run under travis-ci
TraGicCode Oct 3, 2017
af3c081
Add code to actually allow this module to work properly while inside …
TraGicCode Oct 3, 2017
e3d9b6e
Since there is a bug where splunk doesn't uninstall cleanly currently…
TraGicCode Oct 3, 2017
a933014
Fix a bug in the forwarder where it is not actually idempotent. This…
TraGicCode Oct 3, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .sync.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
---
.travis.yml:
secure: "RrAmWtM6eTjZZzD954AIgR179Pqp14lzHhd/C9cpKbTPynLncuim08CEvjmq+7pgAy9XDg1d02x3udfZt4btR1sBdyNRpNN3yUhWptmGU61HRJdiZq+nSLQkIYsqXanhk+O3NndFojO58WRD01dkWEc6HRHjlivuYNxDXmMkg9k="
docker_sets:
- set: docker/centos-6
- set: docker/centos-7
- set: docker/debian-7
- set: docker/debian-8
- set: docker/ubuntu-14.04
- set: docker/ubuntu-16.04
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you split the new nodesets at least into an own commit? Or a separate PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure thing!

spec/spec_helper.rb:
spec_overrides:
- "require 'splunk_data.rb'"
Expand Down
36 changes: 36 additions & 0 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,42 @@ script:
matrix:
fast_finish: true
include:
- rvm: 2.4.1
bundler_args: --without development
dist: trusty
env: PUPPET_INSTALL_TYPE=agent BEAKER_debug=true BEAKER_set=docker/centos-6 CHECK=beaker
services: docker
sudo: required
- rvm: 2.4.1
bundler_args: --without development
dist: trusty
env: PUPPET_INSTALL_TYPE=agent BEAKER_debug=true BEAKER_set=docker/centos-7 CHECK=beaker
services: docker
sudo: required
- rvm: 2.4.1
bundler_args: --without development
dist: trusty
env: PUPPET_INSTALL_TYPE=agent BEAKER_debug=true BEAKER_set=docker/debian-7 CHECK=beaker
services: docker
sudo: required
- rvm: 2.4.1
bundler_args: --without development
dist: trusty
env: PUPPET_INSTALL_TYPE=agent BEAKER_debug=true BEAKER_set=docker/debian-8 CHECK=beaker
services: docker
sudo: required
- rvm: 2.4.1
bundler_args: --without development
dist: trusty
env: PUPPET_INSTALL_TYPE=agent BEAKER_debug=true BEAKER_set=docker/ubuntu-14.04 CHECK=beaker
services: docker
sudo: required
- rvm: 2.4.1
bundler_args: --without development
dist: trusty
env: PUPPET_INSTALL_TYPE=agent BEAKER_debug=true BEAKER_set=docker/ubuntu-16.04 CHECK=beaker
services: docker
sudo: required
- rvm: 2.1.9
bundler_args: --without system_tests development
env: PUPPET_VERSION="~> 4.0" CHECK=test
Expand Down
54 changes: 31 additions & 23 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,32 +70,40 @@ The files must be placed according to directory structure example given below.

The expected directory structure is:

`-- files
|-- splunk
| `-- $platform
| `-- splunk-${version}-${build}-${additl}
`-- universalforwarder
`-- $platform
`-- splunkforwarder-${version}-${build}-${additl}
$root_url/
└── products/
├── universalforwarder/
│ └── releases/
| └── $version/
| └── $platform/
| └── splunkforwarder-${version}-${build}-${additl}
└── splunk/
└── releases/
└── $version/
└── $platform/
└── splunk-${version}-${build}-${additl}

A semi-populated example files directory might then contain:

`-- files
|-- splunk
| `-- linux
| |-- splunk-6.3.3-f44afce176d0-linux-2.6-amd64.deb
| |-- splunk-6.3.3-f44afce176d0-linux-2.6-intel.deb
| `-- splunk-6.3.3-f44afce176d0-linux-2.6-x86_64.rpm
`-- universalforwarder
|-- linux
| |-- splunkforwarder-6.3.3-f44afce176d0-linux-2.6-amd64.deb
| |-- splunkforwarder-6.3.3-f44afce176d0-linux-2.6-intel.deb
| `-- splunkforwarder-6.3.3-f44afce176d0-linux-2.6-x86_64.rpm
|-- solaris
| `-- splunkforwarder-6.3.3-f44afce176d0-solaris-9-intel.pkg
`-- windows
|-- splunkforwarder-6.3.3-f44afce176d0-x64-release.msi
`-- splunkforwarder-6.3.3-f44afce176d0-x86-release.msi
$root_url/
└── products/
├── universalforwarder/
│ └── releases/
| └── 7.0.0/
| ├── linux/
| | ├── splunkforwarder-7.0.0-c8a78efdd40f-linux-2.6-amd64.deb
| | ├── splunkforwarder-7.0.0-c8a78efdd40f-linux-2.6-intel.deb
| | └── splunkforwarder-7.0.0-c8a78efdd40f-linux-2.6-x86_64.rpm
| ├── solaris/
| └── windows/
| └── splunkforwarder-7.0.0-c8a78efdd40f-x64-release.msi
└── splunk/
└── releases/
└── 7.0.0/
└── linux/
├── splunk-7.0.0-c8a78efdd40f-linux-2.6-amd64.deb
├── splunk-7.0.0-c8a78efdd40f-linux-2.6-intel.deb
└── splunk-7.0.0-c8a78efdd40f-linux-2.6-x86_64.rpm

Second, you will need to supply the `splunk::params` class with three critical
pieces of information.
Expand Down
2 changes: 1 addition & 1 deletion manifests/forwarder.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,7 @@
group => $splunk_user,
mode => $facts['kernel'] ? {
'windows' => undef,
default => '0644',
default => '0600',
}
}

Expand Down
15 changes: 15 additions & 0 deletions manifests/init.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,21 @@
tag => 'splunk_server',
}

if $facts['virtual'] == 'docker' {
ini_setting { 'OPTIMISTIC_ABOUT_FILE_LOCKING':
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know if this is always needed in any Docker container? Or is it just required for the acceptance tests to work in Travis?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both. I don't know the exact details but if you google splunk docker unusable filesystem This is pretty much the solution everyone provides. Even the chef-splunk repo did this but this was for their acceptance test. I Also could not find exact details on why/how this actually works. If you want to try this out locally under docker remove that line and watch it fail with the above error message.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I saw the comment before. Just wondering whether it's best to do this in the module or in the acceptance test; I think a case could possibly be made either way, but seems kind of magical, esp. since there's no parameter to control it.

It sounds like if you setup the volumes correctly, it may not be necessary to set this?
https://hub.docker.com/r/alexanderm/docker-splunk/

To work around the unusable filesystem issue you need to mount in a volume from the host for the /opt/splunk/var/lib
For example I started docker like this:
docker run -v /data/splunk-test:/opt/splunk/var/lib -p 8000:8000 -it alexanderm/docker-splunk bash

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wyardley I saw that but that isn't "Applicable" i believe in our case. When running docker acceptance tests we are not taking a folder on the host and mapping it to a folder in the docker container. Make sense?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yeah, I thought this change was in the manifest itself, not in the test. I agree that this is a reasonable workaround in the acceptance test itself.

ensure => present,
section => '',
setting => 'OPTIMISTIC_ABOUT_FILE_LOCKING',
value => '1',
path => '/opt/splunk/etc/splunk-launch.conf',
}

Package[$package_name]
-> Ini_setting['OPTIMISTIC_ABOUT_FILE_LOCKING']
-> Exec <| tag == 'splunk_server' |>
}


splunk_input { 'default_host':
section => 'default',
setting => 'host',
Expand Down
79 changes: 45 additions & 34 deletions manifests/params.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,32 +33,43 @@
# the puppet/archive module supports. This includes both puppet:// and
# http://. The expected directory structure is:
#
# `-- $root_url
# |-- splunk
# | `-- $platform
# | `-- splunk-${version}-${build}-${additl}
# `-- universalforwarder
# `-- $platform
# `-- splunkforwarder-${version}-${build}-${additl}
#
# $root_url/
# └── products/
# ├── universalforwarder/
# │ └── releases/
# | └── $version/
# | └── $platform/
# | └── splunkforwarder-${version}-${build}-${additl}
# └── splunk/
# └── releases/
# └── $version/
# └── $platform/
# └── splunk-${version}-${build}-${additl}
#
#
# A semi-populated example src_root then contain:
#
# `-- $root_url
# |-- splunk
# | `-- linux
# | |-- splunk-4.3.2-123586-linux-2.6-amd64.deb
# | |-- splunk-4.3.2-123586-linux-2.6-intel.deb
# | `-- splunk-4.3.2-123586-linux-2.6-x86_64.rpm
# `-- universalforwarder
# |-- linux
# | |-- splunkforwarder-4.3.2-123586-linux-2.6-amd64.deb
# | |-- splunkforwarder-4.3.2-123586-linux-2.6-intel.deb
# | `-- splunkforwarder-4.3.2-123586-linux-2.6-x86_64.rpm
# |-- solaris
# | `-- splunkforwarder-4.3.2-123586-solaris-10-intel.pkg
# `-- windows
# |-- splunkforwarder-4.3.2-123586-x64-release.msi
# `-- splunkforwarder-4.3.2-123586-x86-release.msi
# $root_url/
# └── products/
# ├── universalforwarder/
# │ └── releases/
# | └── 7.0.0/
# | ├── linux/
# | | ├── splunkforwarder-7.0.0-c8a78efdd40f-linux-2.6-amd64.deb
# | | ├── splunkforwarder-7.0.0-c8a78efdd40f-linux-2.6-intel.deb
# | | └── splunkforwarder-7.0.0-c8a78efdd40f-linux-2.6-x86_64.rpm
# | ├── solaris/
# | └── windows/
# | └── splunkforwarder-7.0.0-c8a78efdd40f-x64-release.msi
# └── splunk/
# └── releases/
# └── 7.0.0/
# └── linux/
# ├── splunk-7.0.0-c8a78efdd40f-linux-2.6-amd64.deb
# ├── splunk-7.0.0-c8a78efdd40f-linux-2.6-intel.deb
# └── splunk-7.0.0-c8a78efdd40f-linux-2.6-x86_64.rpm
#
#
# Actions:
#
Expand All @@ -67,9 +78,9 @@
# Requires: nothing
#
class splunk::params (
$version = '6.3.3',
$build = 'f44afce176d0',
$src_root = 'puppet:///modules/splunk',
$version = '7.0.0',
$build = 'c8a78efdd40f',
$src_root = 'https://download.splunk.com',
$splunkd_port = '8089',
$logging_port = '9997',
$server = 'splunk',
Expand Down Expand Up @@ -101,36 +112,36 @@
case $::kernel {
'Linux': {
$path_delimiter = '/'
$forwarder_src_subdir = 'universalforwarder/linux'
$forwarder_src_subdir = 'linux'
$forwarder_service = [ 'splunk' ]
$password_config_file = "${forwarder_dir}/etc/passwd"
$secret_file = "${forwarder_dir}/etc/splunk.secret"
$forwarder_confdir = "${forwarder_dir}/etc"
$server_src_subdir = 'splunk/linux'
$server_src_subdir = 'linux'
$server_service = [ 'splunk', 'splunkd', 'splunkweb' ]
$server_confdir = "${server_dir}/etc"
$forwarder_install_options = undef
}
'SunOS': {
$path_delimiter = '/'
$forwarder_src_subdir = 'universalforwarder/solaris'
$forwarder_src_subdir = 'solaris'
$forwarder_service = [ 'splunk' ]
$password_config_file = "${forwarder_dir}/etc/passwd"
$secret_file = "${forwarder_dir}/etc/splunk.secret"
$forwarder_confdir = "${forwarder_dir}/etc"
$server_src_subdir = 'splunk/solaris'
$server_src_subdir = 'solaris'
$server_service = [ 'splunk', 'splunkd', 'splunkweb' ]
$server_confdir = "${server_dir}/etc"
$forwarder_install_options = undef
}
'Windows': {
$path_delimiter = '\\'
$forwarder_src_subdir = 'universalforwarder/windows'
$forwarder_src_subdir = 'windows'
$password_config_file = 'C:/Program Files/SplunkUniversalForwarder/etc/passwd'
$secret_file = 'C:/Program Files/SplunkUniversalForwarder/etc/splunk.secret'
$forwarder_service = [ 'SplunkForwarder' ] # UNKNOWN
$forwarder_confdir = "${forwarder_dir}/etc"
$server_src_subdir = 'splunk/windows'
$server_src_subdir = 'windows'
$server_service = [ 'Splunkd', 'SplunkWeb' ] # UNKNOWN
$server_confdir = "${server_dir}/etc"
$forwarder_install_options = [
Expand Down Expand Up @@ -237,8 +248,8 @@
$server_src_pkg = "splunk-${package_suffix}"

$server_pkg_ensure = 'installed'
$server_pkg_src = "${src_root}/${server_src_subdir}/${server_src_pkg}"
$forwarder_pkg_src = "${src_root}/${forwarder_src_subdir}/${forwarder_src_pkg}"
$server_pkg_src = "${src_root}/products/splunk/releases/${version}/${server_src_subdir}/${server_src_pkg}"
$forwarder_pkg_src = "${src_root}/products/universalforwarder/releases/${version}/${forwarder_src_subdir}/${forwarder_src_pkg}"
$create_password = true

$forwarder_pkg_ensure = 'installed'
Expand Down
6 changes: 5 additions & 1 deletion spec/acceptance/splunk_forwarder_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,11 @@
# Using puppet_apply as a helper
it 'works idempotently with no errors' do
pp = <<-EOS
class { '::splunk::forwarder': }
class { '::splunk::params':
}
class { '::splunk::forwarder':
splunkd_port => 8090,
}
EOS

# Run it twice and test for idempotency
Expand Down