Enable ca/cert management

.config/ansible-lint.yml

@@ -6,10 +6,10 @@ warn_list:
   - experimental  # all rules tagged as experimental
   - key-order[task]  # Ensure specific order of keys in mappings.
   - name[casing]
+  - 'risky-shell-pipe'
 skip_list:
   - '106'
   - 'command-instead-of-module'
-  - 'risky-shell-pipe'
   - 'role-name'
   - 'line-length'
   - 'fqcn-builtins'

README.md

@@ -39,6 +39,10 @@ You may want the following Ansible roles installed. There other ways to achieve
 
 ## Usage
 
+Make sure all hosts that should be configured are part of your playbook. (See below for details on groups etc.). The collection is built to first collect all facts from all hosts (including those only running beats) and then use facts like hostnames or ip addresses to connect the tools to each other.
+
+You will want to have reliable DNS resolution or enter all hosts of the stack into your systems hosts files.
+
 ### Default Passwords 
 
 Default Passwords  can be seen during generation, or found later in `/usr/share/elasticsearch/initial_passwords`

docs/role-beats.md

@@ -102,7 +102,9 @@ If you want to use this role with your own TLS certificates, use these variables
 * *beats_ca_dir*: Path to custom CA certificates and keys (default: none - if not set will be filled with different values depending on which Stack variant is used)
 * *beats_tls_key*: Path to the keyfiles (default: `{{ beats_ca_dir }}/{{ ansible_hostname }}.key`)
 * *beats_tls_cert*: Path to the certificate (default: `{{ beats_ca_dir }}/{{ ansible_hostname }}.crt`)
-* *beats_tls_key_passphrase*: Passphrase of the keyfile (default: `ChangeMe`)
+* *beats_tls_key_passphrase*: Passphrase of the keyfile (default: `BeatsChangeMe`)
+* *beats_cert_expiration_buffer*: Ansible will renew the beats certificate if its validity is shorter than this value, which should be number of days. (default: 30)
+* *beats_cert_will_expire_soon*: Set it to true to renew beats certificate (default: `false`), Or run the playbook with `--tags renew_beats_cert` to do that.
 * *beats_tls_cacert*: Path to the CA.crt (default: `{{ beats_ca_dir }}/ca.crt`)
 
 ## Usage

docs/role-elasticsearch.md

@@ -5,7 +5,7 @@ ELASTICSEARCH
 
 This role installs manages Elasticsearch on your hosts. Optionally it can configure Elastics Security components, too.
 
-If you use the role to set up security you can use its CA to create certificates for Logstash and Kibana, too.
+If you use the role to set up security you, can use its CA to create certificates for Logstash and Kibana, too. When you enable security, the role will check the expiration date of the CA and all certificates every Ansible run and renew the one, which will expire soon default before 30 days.
 
 Please note that setting `elasticsearch_bootstrap_pw` as variable will only take effect when initialising Elasticsearch. Changes after starting elasticsearch for the first time will not change the bootstrap password for the instance and will lead to breaking tests.
 
@@ -15,6 +15,12 @@ Role Variables
 * *elasticsearch_enable*: Start and enable Elasticsearch (default: `true`)
 * *elasticsearch_heap*: Heapsize for Elasticsearch. (Half of free memory on host. Maximum 30GB. (default: Half of hosts memory. Min 1GB, Max 30GB)
 * *elasticsearch_ca*: Set to the inventory hostname of the host that should house the CA for certificates for inter-node communication. (default: First node in the `elasticsearch` host group)
+* *elastic_ca_pass*: Password for Elasticsearch CA (default: `PleaseChangeMe`)
+* *elastic_ca_expiration_buffer*: Ansible will renew the CA if its validity is shorter than this value, which should be number of days. (default: 30)
+* *elastic_ca_will_expire_soon*: Set it to true to renew the CA and the certificate of all Elastic Stack components (default: `fasle`), Or run the playbook with `--tags renew_ca` to do that.
+* *elasticsearch_tls_key_passphrase*: Passphrase for elasticsearch certificates (default: `PleaseChangeMeIndividually`)
+* *elasticsearch_cert_expiration_buffer*: Ansible will renew the elasticsearch certificate if its validity is shorter than this value, which should be number of days. (default: 30)
+* *elasticsearch_cert_will_expire_soon*: Set it to true to renew elasticsearch certificate (default: `fasle`), Or run the playbook with `--tags renew_elasticsearch_cert` to do that.
 * *elasticsearch_datapath*: Path where Elasticsearch will store it's data. (default: `/var/lib/elasticsearch` - the packages default)
 * *elasticsearch_create_datapath*: Create the path for data to store if it doesn't exist. (default: `false` - only useful if you change `elasticsearch_datapath`)
 * *elasticsearch_fs_repo*: List of paths that should be registered as repository for snapshots (only filesystem supported so far). (default: none) Remember, that every node needs access to the same share under the same path.

docs/role-kibana.md

@@ -21,6 +21,9 @@ Role Variables
 These variables are identical over all our elastic related roles, hence the different naming scheme.
 
 * *elastic_stack_full_stack*: Use `ansible-role-elasticsearch` as well (default: `false`)
+* *kibana_tls_key_passphrase*: Passphrase for kibana certificates (default: `PleaseChangeMe`)
+* *kibana_cert_expiration_buffer*: Ansible will renew the kibana certificate if its validity is shorter than this value, which should be number of days. (default: 30)
+* *kibana_cert_will_expire_soon*: Set it to true to renew kibana certificate (default: `fasle`), Or run the playbook with `--tags renew_kibana_cert` to do that.
 * *elasticsearch_ca*: Set to the inventory hostname of the host that should house the CA for certificates for inter-node communication. (default: First node in the `elasticsearch` host group)
 * *elastic_ca_dir*: Directory where on the Elasticsearch CA host certificates are stored. This is only useful in connection with out other Elastic Stack related roles. (default: `/opt/es-ca`)
 * *elastic_ca_pass*: Password for Elasticsearch CA (default: `PleaseChangeMe`)

docs/role-logstash.md

@@ -47,21 +47,27 @@ Aside from `logstash.yml` we can manage Logstashs pipelines.
 
 * *logstash_manage_pipelines*: Manage `pipelines.yml` (default: `true`)
 * *logstash_no_pipelines*: Don't manage pipelines at all (default: `false`)
-* *logstash_pipelines*: List of pipelines with optional URL to repo (see docs/pipelines.md for details)
+* *logstash_pipelines*: List of pipelines with optional URL to repo (see [pipelines documentation](file:///roles/logstash/docs/pipelines.md) for details)
 * *logstash_global_ecs*: Set ECS compatibilty mode (default: none. Possible values: `disabled` or `v1`)
 * *logstash_elasticsearch_output*: Enable default pipeline to Elasticsearch (default: `true`)
 * *logstash_ident*: Add a field identifying the node that processed an event (default: `true`)
 * *logstash_ident_field_name*: Name of the identifying the instance (default: `"[netways][instance]"`)
 * *logstash_beats_input*: Enable default pipeline with `beats` input (default: `true`)
 * *logstash_beats_input_congestion*: Optional congestion threshold for the beats input pipeline
 * *logstash_beats_tls*: Activate TLS for the beats input pipeline (default: none but `true` with full stack setup if not set)
-* *logstash_tls_key_passphrase*: Passphrase for Logstash certificates (default: `ChangeMe`)
+* *logstash_tls_key_passphrase*: Passphrase for Logstash certificates (default: `LogstashChangeMe`)
+* *elastic_ca_pass*: Password for Elasticsearch CA (default: `PleaseChangeMe`)
+* *logstash_cert_expiration_buffer*: Ansible will renew the Logstash certificate if its validity is shorter than this value, which should be number of days. (default: 30)
+* *logstash_cert_will_expire_soon*: Set it to true to renew logstash certificate (default: `fasle`), Or run the playbook with `--tags renew_logstash_cert` to do that.
 * *logstash_elasticsearch*: Address of Elasticsearch instance for default output (default: list of Elasticsearch nodes from `elasticsearch` role or `localhost` when used standalone)
 * *logstash_security*: Enable X-Security (No default set, but will be activated when in full stack mode)
 * *logstash_user*: Name of the user to connect to Elasticsearch (Default: `logstash_writer`)
 * *logstash_password*: Password of Elasticsearch user (Default: `password`)
 * *logstash_user_indices*: Indices the user has access to (Default: `'"ecs-logstash*", "logstash*", "logs*"'`)
 * *logstash_reset_writer_role*: Reset user and role with every run: (Default: `true`)
+* *logstash_validate_after_inactivity*: How long should logstash wait, before starting a new connection and leave the old one with elasticsearch, when the connection with elasticsearch get lost: (Default: `300`).
+* *logstash_queue_type*: What kind of queue should Logstash use per default: (Default: `persisted`, alternative: `memory`)
+* *logstash_queue_max_bytes*: The total capacity of ansible-forwarder queue in number of bytes: (Default: `2gb`)
 
 
 

molecule/elasticstack_default/molecule.yml

@@ -6,6 +6,9 @@ driver:
 platforms:
   - name: elasticsearch-cluster1
     groups:
+      - beats
+      - logstash
+      - kibana
       - elasticsearch
     image: "geerlingguy/docker-${MOLECULE_DISTRO:-centos7}-ansible:latest"
     command: ${MOLECULE_DOCKER_COMMAND:-""}
@@ -16,6 +19,9 @@ platforms:
     pre_build_image: true
   - name: elasticsearch-cluster2
     groups:
+      - beats
+      - logstash
+      - kibana
       - elasticsearch
     image: "geerlingguy/docker-${MOLECULE_DISTRO:-centos7}-ansible:latest"
     command: ${MOLECULE_DOCKER_COMMAND:-""}

roles/beats/defaults/main.yml

@@ -18,7 +18,7 @@ beats_manage_unzip: true
 beats_tls_key: "{{ beats_ca_dir }}/{{ inventory_hostname }}-beats.key"
 beats_tls_cert: "{{ beats_ca_dir }}/{{ inventory_hostname }}-beats.crt"
 beats_tls_cacert: "{{ beats_ca_dir }}/ca.crt"
-beats_tls_key_passphrase: ChangeMe
+beats_tls_key_passphrase: BeatsChangeMe
 
 # Filebeat specific #
 
@@ -68,6 +68,8 @@ elastic_security: true
 elastic_ca_dir: /opt/es-ca
 elastic_ca_pass: PleaseChangeMe
 elastic_initial_passwords: /usr/share/elasticsearch/initial_passwords
+beats_cert_expiration_buffer: 30
+beats_cert_will_expire_soon: false
 
 # Variables for debugging and development
 

roles/beats/handlers/main.yml

@@ -5,16 +5,22 @@
   service:
     name: filebeat
     state: restarted
-  when: filebeat_enable | bool
+  when:
+    - beats_filebeat | bool
+    - filebeat_enable | bool
 
 - name: Restart Auditbeat
   service:
     name: auditbeat
     state: restarted
-  when: auditbeat_enable | bool
+  when:
+    - beats_auditbeat | bool
+    - auditbeat_enable | bool
 
 - name: Restart Metricbeat
   service:
     name: metricbeat
     state: restarted
-  when: metricbeat_enable | bool
+  when:
+  - beats_metricbeat | bool
+  - metricbeat_enable | bool

roles/beats/tasks/beats-security.yml

@@ -4,6 +4,123 @@
   package:
     name: unzip
   when: beats_manage_unzip | bool
+  tags:
+    - renew_ca
+    - renew_beats_cert
+
+- name: Ensure beats certificate exists
+  stat:
+    path: "/etc/beats/certs/{{ inventory_hostname }}-beats.crt"
+  register: beats_cert_exists
+
+- name: Get the beats certificate expiration date
+  shell: >-
+    if test -v BASH; then set -o pipefail; fi;
+    openssl x509
+    -in '/etc/beats/certs/{{ inventory_hostname }}-beats.crt'
+    -noout -enddate |
+    awk -F'=' '{print $2}'
+  register: beats_cert_expiration_date
+  args:
+    executable: /bin/bash
+  changed_when: false
+  when: beats_cert_exists.stat.exists | bool
+
+- name: Set the beats certificate expiration date in days
+  set_fact:
+    beats_cert_expiration_days: "{{ ((beats_cert_expiration_date.stdout | to_datetime('%b %d %H:%M:%S %Y %Z')) - ( ansible_date_time.date | to_datetime('%Y-%m-%d'))).days }}"
+  when: beats_cert_expiration_date.skipped is not defined
+
+- name: Set beats certificate will expire soon to true
+  set_fact:
+    beats_cert_will_expire_soon: true
+  when: beats_cert_expiration_days is defined and beats_cert_expiration_days | int <= beats_cert_expiration_buffer | int
+
+- name: Print the beats certificate renew message
+  debug:
+    msg: |
+      Your beats certificate will expire in {{ beats_cert_expiration_days }}.
+      Ansible will renew it.
+  when: beats_cert_expiration_days is defined and beats_cert_expiration_days | int <= beats_cert_expiration_buffer | int
+
+- name: Backup beats certs then remove
+  block:
+    - name: Check if cert directory exists
+      stat:
+        path: /etc/beats/certs
+      register: beats_check_cert_path
+
+    - name: Move cert directory
+      copy:
+        src: /etc/beats/certs
+        dest: "/etc/beats/certs_{{ ansible_date_time.iso8601_micro }}"
+        mode: preserve
+        remote_src: true
+      when: beats_check_cert_path.stat.exists
+      register: beats_move_cert_directory
+
+    - name: Remove cert directory
+      file:
+        path: /etc/beats/certs
+        state: absent
+      when: beats_move_cert_directory.changed
+  when: "'renew_beats_cert' in ansible_run_tags or 'renew_ca' in ansible_run_tags or beats_cert_will_expire_soon | bool"
+  tags:
+    - renew_ca
+    - renew_beats_cert
+
+- name: Backup beats certs on elasticsearch_ca host then remove
+  block:
+    - name: Check if cert file exists
+      stat:
+        path: "{{ elastic_ca_dir }}/{{ ansible_hostname }}-beats.zip"
+      register: beats_check_cert_file
+
+    - name: Move cert file
+      copy:
+        src: "{{ elastic_ca_dir }}/{{ ansible_hostname }}-beats.zip"
+        dest: "{{ elastic_ca_dir }}/{{ ansible_hostname }}-beats.zip_{{ ansible_date_time.iso8601_micro }}"
+        mode: preserve
+        remote_src: true
+      when: beats_check_cert_file.stat.exists
+      register: beats_move_cert_file
+
+    - name: Remove cert file
+      file:
+        path: "{{ elastic_ca_dir }}/{{ ansible_hostname }}-beats.zip"
+        state: absent
+      when: beats_move_cert_file.changed
+  when: "'renew_beats_cert' in ansible_run_tags or 'renew_ca' in ansible_run_tags or beats_cert_will_expire_soon | bool"
+  delegate_to: "{{ elasticsearch_ca }}"
+  tags:
+    - renew_ca
+    - renew_beats_cert
+
+- name: Backup beats zip file on localhost then remove
+  block:
+    - name: Check the existance of cert on localhost
+      stat:
+        path: /tmp/{{ ansible_hostname }}-beats.zip
+      register: beats_check_temporary_cert
+
+    - name: Move temporary zip file
+      copy:
+        src: /tmp/{{ ansible_hostname }}-beats.zip
+        dest: "/tmp/{{ ansible_hostname }}-beats.zip_{{ ansible_date_time.iso8601_micro }}"
+        mode: preserve
+      when: beats_check_temporary_cert.stat.exists
+      register: beats_move_cert_file
+
+    - name: Remove temporary cert file
+      file:
+        path: /tmp/{{ ansible_hostname }}-beats.zip
+        state: absent
+      when: beats_move_cert_file.changed
+  delegate_to: localhost
+  when: "'renew_beats_cert' in ansible_run_tags or 'renew_ca' in ansible_run_tags or beats_cert_will_expire_soon | bool"
+  tags:
+    - renew_ca
+    - renew_beats_cert
 
 - name: Create certificate directory
   file:
@@ -12,6 +129,10 @@
     owner: root
     group: root
     mode: 0700
+  tags:
+    - certificates
+    - renew_ca
+    - renew_beats_cert
 
 - name: Create individual certificates for Beats
   command: >
@@ -27,6 +148,10 @@
   delegate_to: "{{ elasticsearch_ca }}"
   args:
     creates: "{{ elastic_ca_dir }}/{{ ansible_hostname }}-beats.zip"
+  tags:
+    - certificates
+    - renew_ca
+    - renew_beats_cert
 
 - name: Fetch certificate from ca host to master
   fetch:
@@ -36,6 +161,8 @@
   delegate_to: "{{ elasticsearch_ca }}"
   tags:
     - certificates
+    - renew_ca
+    - renew_beats_cert
 
 - name: Copy the certificate to actual node
   unarchive:
@@ -46,6 +173,8 @@
     mode: 0640
   tags:
     - certificates
+    - renew_ca
+    - renew_beats_cert
 
 - name: Copy certificate locally
   copy:
@@ -55,8 +184,14 @@
     group: root
     mode: 0640
     remote_src: true
+  notify:
+    - Restart Filebeat
+    - Restart Auditbeat
+    - Restart Metricbeat
   tags:
     - certificates
+    - renew_ca
+    - renew_beats_cert
 
 - name: Copy key locally
   copy:
@@ -66,8 +201,14 @@
     group: root
     mode: 0640
     remote_src: true
+  notify:
+    - Restart Filebeat
+    - Restart Auditbeat
+    - Restart Metricbeat
   tags:
     - certificates
+    - renew_ca
+    - renew_beats_cert
 
 - name: Fetch ca certificate from ca host to master
   fetch:
@@ -77,6 +218,8 @@
   delegate_to: "{{ elasticsearch_ca }}"
   tags:
     - certificates
+    - renew_ca
+    - renew_beats_cert
 
 - name: Copy the ca certificate to actual node
   copy:
@@ -85,13 +228,20 @@
     owner: root
     group: root
     mode: 0640
+  notify:
+    - Restart Filebeat
+    - Restart Auditbeat
+    - Restart Metricbeat
   tags:
     - certificates
+    - renew_ca
+    - renew_beats_cert
 
 - name: Fetch Beats password
   shell: >
     grep "PASSWORD elastic" {{ elastic_initial_passwords }} |
     awk {' print $4 '}
   register: beats_writer_password
   changed_when: false
+  no_log: true
   delegate_to: "{{ elasticsearch_ca }}"