Packer VPC filter (#635)

* GitHub Actions - Rebuilt documentation.

* Need to check if is_local is defined in webserver meta dependencies. (#522)

* Ce dev refactor pr 1.x (#518)

* Making it easier to test with provision-target and ce-dev.

* Moving the provision forcing var back to plays so _init has it.

* Adding defaults vars and test script extra options.

* Adding a web server test to CI.

* examples string needs to be in quotes.

* Making sure is_local and _ce_provision_force_play are available to the _init role.

* Adding SSH keys to the provision user.

* Adding a --force to the test script.

* Explicitly adding vars to role.

* Fixing _init behaviour and adding SSH key for web role.

* Setting default PHP version to 7.4.

* Looking up the generated ce-dev SSH key instead of hard-coding one.

* We cannot run the ssh_server role locally, so excluding for tests of webserver role.

* Trying to remove user_root.yml in case it's breaking CI.

* Adding a verbose mode to the test script.

* Exposing the command in the test script.

* Trying hard-coded keys again.

* Changing location of data dir for test containers.

* Putting vars back and restricting CI to the 'web' example.

* Adding backup handling to ldap_server. (#525)

* Adding backup handling to ldap_server.

* Improving SSL docs and handling perms for openldap and letsencrypt.

* Cron user must be specified with file.

* Running as root, do not need a 'sudo' in this cron.

* Allowing 'gitLab' to disable Prometheus. (#530)

* Allowing 'gitLab' to disable Prometheus.

* Booleans to use in jinja2 as strings must be cast as strings.

* GitHub Actions - Rebuilt documentation. (#526)

Co-authored-by: Code Enigma CI <sysadm@codeenigma.com>

* Prometheus pr 1.x (#533)

* Allowing 'gitLab' to disable Prometheus.

* Booleans to use in jinja2 as strings must be cast as strings.

* Tidying up CI and adding a GitLab test.

* Fixing CI job description.

* Add private files support for Drupal in Nginx. (#535)

* Prometheus pr 1.x (#539)

* Allowing 'gitLab' to disable Prometheus.

* Booleans to use in jinja2 as strings must be cast as strings.

* Tidying up CI and adding a GitLab test.

* Fixing CI job description.

* Adding a firewall config preset to open port 80 for LetsEncrypt.

* Removing our unused ClamAV roles and adding a Galaxy role to common base. (#541)

* Revert "Moving OSSEC pkill to use process_manager role instead. (#258)" (#544)

This reverts commit 73c7bd0.

* Moving key servers to a variable so we can set them. (#555)

* Moving key servers to a variable so we can set them.

* Allowing us to disable sending keys completely.

* Oops, doubled up on existing functionality.

* Fixing var name.

* Adding a reboot option to the patching role. (#557)

* Add minimal support for Aurora RDS instances (#567)

* Attempt to create an RDS read replica.

* Use new task to create Aurora RDS instances.

* Try and fix linting issues.

* Don't pass max_storage variable for Aurora instances.

* Remove more storage related vars from Aurora RDS instance creation task.

* Add profile and region to read replica creation.

* Try creating the Aurora read replica another way.

* Add some debug info.

* Work around the silly registering of variables in Ansible.

* Rename an RDS CloudWatch task for Aurora DBs and remove RDS debug info.

* Add some Aurora info to aws_rds README file.

* Use reader instead of replica for Aurora readers.

* Remove db_cluster_identifier variable from non-Aurora RDS task.

* Gpg servers fix pr 1.x (#571)

* Moving key servers to a variable so we can set them.

* Allowing us to disable sending keys completely.

* Oops, doubled up on existing functionality.

* Fixing var name.

* Using a pipe to grep with 'command' cannot work, refactoring.

* Making CI use the meta deploy role to test gitlab.

* We mustn't assume AWS servers for deploy and controller.

* Support termination protection in EC2. (#573)

* Support termination protection in EC2.

* Fixing CI vars.

* Fixing CI vars.

* Fix managed SSL key perms and the variable used for the private key. (#575)

* Ec2 subnet lookup pr 1.x (#583)

* First pass at EC2 subnet detection.

* Touching subnet file to ensure it exists.

* Trying a different approach, file module didn't work.

* Switching back to file module.

* We need to create the directory for new servers too.

* Bad variable name.

* Ec2 subnet lookup pr 1.x (#589)

* First pass at EC2 subnet detection.

* Touching subnet file to ensure it exists.

* Trying a different approach, file module didn't work.

* Switching back to file module.

* We need to create the directory for new servers too.

* Bad variable name.

* Changing subnet lookup order to check for defined subnet first.

* Fixing gitlab-runner overriders so upgrades do not break the runner. (#586)

* Fixing gitlab-runner overriders so upgrades do not break the runner.

* Fixing override file template.

* Hopefully fixing CI.

* Making sure the service directory exists.

* We cannot use the deploy meta role in CI because of LDAP.

* Changing dir perms and adding a force.

* Gitlab runner service override pr 1.x (#591)

* Fixing gitlab-runner overriders so upgrades do not break the runner.

* Fixing override file template.

* Hopefully fixing CI.

* Making sure the service directory exists.

* We cannot use the deploy meta role in CI because of LDAP.

* Changing dir perms and adding a force.

* Debugging gitlab-runner directory creation issues in CI.

* Fixing linting error.

* Removing verbosity again but leaving 'stat' command in.

* Pass db_cluster_identifier for RDS instance during ASG build (#600)

* Pass RDS db_cluster_identifier, if present, during an ASG build.

* Use correct variable name for RDS db_cluster_identifier.

* Add a commented variable to ASG role for db_cluster_identifier so it's documented.

* Also pass in the aurora_reader var from the ASG role when including the aws_rds role. (#605)

* Removing obsolete MySQL config option log_syslog from template. (#607)

* GitHub Actions - Rebuilt documentation. (#536)

Co-authored-by: Code Enigma CI <sysadm@codeenigma.com>

* Consistent default region pr 1.x (#611)

* Moving all region settings to _aws_region var and adding README update.

* Documentation update.

* No need for region, IAM SAML setup is global, (#617)

* Support ebs encryption pr 1.x (#609)

* Adding volume encryption and type options plus a bit more flexibility on EBS control for EC2.

* Setting more sane default instance sizes.

* Adding more EBS options for ASGs.

* Setting encryption to match AMI settings.

* Setting encryption to match AMI settings.

* We also need to dynamically set the ASGs own encrypt_boot var.

* We need to merge the new branch changes before we can rebuild the docs.

* Fixing merge command in CI.

* Not sure toc.sh is actually executing.

* Refactoring encrypt EBS flags to avoid detected loop condition in vars.

* Safer CI, only adds .md files.

* Trying to figure out CI logic for building docs.

* Trying to figure out CI logic for building docs.

* Trying to figure out CI logic for building docs.

* Trying adding a git pull.

* Setting git pull config options.

* Reordering things.

* Adding --allow-unrelated-histories to the git pull.

* Trying a feature branch approach.

* Forcing the GitHub action to fetch all git history.

* Bad whitespace, naughty whitespace.

* Trying a different PR action.

* Do not merge the branch in, we only want the markdown changes.

* Keeping the documentation branch clean.

* We need to push a detached HEAD.

* Do we need the checkout at all?

* Adding a docs pull.

* Allow install|update scripts in Drupal8+ (#599)

* Add some flexibility to Packer (#633)

* Add ability to pass on-error and force to Packer.

* Add new Packer options to the ASG role as well.

* Packer build options need to be declared before the file that is being built.

* Allow Packer ssh_username to be set.

* Making PHP >= 8.0 compatible (#634)

* Add ability to set vpc_filter and subnet AZ for Packer builds.

* Add fqcn-builtins to .ansible-lint warn_list for now.

* GitHub Actions seemingly ignores warn_list.

Co-authored-by: Code Enigma CI <sysadm@codeenigma.com>
Co-authored-by: Greg Harvey <greg.harvey@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Dionisio <dionisiofernandez83@gmail.com>

Author: 5 people

.ansible-lint

@@ -11,3 +11,4 @@ skip_list:
   - var-naming
   - no-jinja-nesting
   - command-instead-of-module
+  - fqcn-builtins

roles/aws/aws_ami/defaults/main.yml

@@ -10,6 +10,10 @@ aws_ami:
   owner: "136693071363" # Global AWS account ID of owner, defaults to Debian official
   ssh_username: "admin"
   encrypt_boot: false
+  #vpc_filter: "example" # If defined, Packer will search for a VPC with the `Name` tag of the value given. vpc_id takes precednece over this if both are defined. This also assumes the VPC is not the default and has a CIDR block of /16.
+  vpc_filter: ""
+  #subnet_filter_az: "a" # If vpc_id and/or vpc_filter are defined, subnet_filter_az MUST be defined and must match an AZ that has public networking.
+  subnet_filter_az: ""
   playbook_file: "{{ playbook_dir }}/base-playbook.yml" # Path to a playbook used to provision the image.
   force: false # Forces a builder to run when artifacts from a previous build prevent a build from running. May be necessary if on_error is 'abort'
   # on_error can be one of:

roles/aws/aws_ami/templates/packer.json.j2

@@ -15,17 +15,30 @@
       "associate_public_ip_address": true,
       "encrypt_boot": "{{ aws_ami.encrypt_boot }}",
       "source_ami_filter": {
-      "filters": {
-        "virtualization-type": "{{ aws_ami.virtualization_type }}",
-        "name": "{{ aws_ami.name_filter }}",
-        "root-device-type": "{{ aws_ami.root_device_type }}"
+        "filters": {
+          "virtualization-type": "{{ aws_ami.virtualization_type }}",
+          "name": "{{ aws_ami.name_filter }}",
+          "root-device-type": "{{ aws_ami.root_device_type }}"
+        },
+        "owners": ["{{ aws_ami.owner }}"],
+        "most_recent": true
       },
-      "owners": ["{{ aws_ami.owner }}"],
-      "most_recent": true
-    },
-    "tags": 
-      {
-      "Name": "{{ aws_ami.ami_name }}"
+      {% if aws_ami.vpc_filter is defined and aws_ami.vpc_filter | length %}
+      "vpc_filter": {
+        "filters": {
+          "tag:Name": "{{ aws_ami.vpc_filter }}",
+          "isDefault": "false",
+          "cidr": "/16"
+        }
+      },
+      "subnet_filter": {
+        "filters": {
+          "tag:Name": "{{ _infra_name }}-{{ _env_type }}-{{ aws_ami.subnet_filter_az }}"
+        }
+      },
+      {% endif %}
+      "tags": {
+        "Name": "{{ aws_ami.ami_name }}"
       }
     }
   ],

roles/aws/aws_ec2_autoscale_cluster/defaults/main.yml

@@ -25,6 +25,9 @@ aws_ec2_autoscale_cluster:
   ami_playbook_file: "{{ playbook_dir }}/ami.yml"
   packer_on_error: cleanup # see aws_ami for details
   packer_force: false # see aws_ami for details
+  packer_vpc:
+    vpc_filter: "" # see aws_ami for details
+    subnet_filter_az: "" # see aws_ami for details
   ami_refresh: true # Whether to build a new AMI or not.
   asg_refresh: true # Whether to build a new ASG or not.
   asg_cloudwatch_policy_scale_up_name: "{{ _env_type }}-scale-up-policy"

roles/aws/aws_ec2_autoscale_cluster/tasks/main.yml

@@ -127,6 +127,8 @@
       encrypt_boot: "{{ aws_ec2_autoscale_cluster.encrypt_boot }}"
       playbook_file: "{{ aws_ec2_autoscale_cluster.ami_playbook_file }}"
       on_error: "{{ aws_ec2_autoscale_cluster.packer_on_error }}"
+      vpc_filter: "{{ aws_ec2_autoscale_cluster.packer_vpc.vpc_filter | default('') }}"
+      subnet_filter_az: "{{ aws_ec2_autoscale_cluster.packer_vpc.subnet_filter_az | default('') }}"
       force: "{{ aws_ec2_autoscale_cluster.packer_force }}"
       operation: create
   when: