Allow branch switching pr 1.x

.github/workflows/README.md

@@ -1,26 +1,74 @@
 # GitHub Actions
-
 This document describes the current CI for the ce-dev project. The CI is written for GitHub Actions, the built in GitHub system for orchestration and automation. We operate three separate GitHub Actions workflows.
 
-## ce-dev-build
+## ce-dev-build-dev
+Builds base and controller images using the contents of the `devel` branch on push. The built images get pushed to Docker Hub and tagged with `devel`. This is very useful for pushing test containers to try out locally.
+
+### Testing project images
+To test a `devel` container on a project, firstly fetch the container image:
+
+```
+docker pull codeenigma/ce-dev-controller-1.x:devel
+```
+
+Then edit your project's `ce-dev.compose.yml` file, changing the image for each applicable service to use the `devel` tag instead of the `latest` tag, for example:
+
+```yaml
+    image: codeenigma/ce-dev-1.x:devel
+```
+
+Finally, destroy and recreate your `docker compose` file and containers:
+
+```bash
+ce-dev destroy
+ce-dev init -t ce-dev.compose.yml
+ce-dev start
+ce-dev provision
+ce-dev deploy
+```
+
+### Testing an experimental controller
+To test the `devel` version of the `ce-dev-controller` image locally you can do the following:
+
+```bash
+docker pull codeenigma/ce-dev-controller-1.x:devel
+# Replace the 'latest' tag with your 'devel' image
+docker tag codeenigma/ce-dev-controller-1.x:devel codeenigma/ce-dev-controller-1.x:latest
+docker kill ce_dev_controller
+# Then in any ce-dev project
+ce-dev start
+ce-dev provision
+```
 
-This builds the images and binaries, pushes the Docker containers to Docker Hub, pushes the binaries to a GitHub release and builds and commits the ce-dev documentation. It runs when someone accepts a PR for or pushes to the `1.x` branch and has added a valid tag in the format `1.*`.
+If you need to hop branches of `ce-provision` or `ce-deploy` on the controller to try things you can hop on to the container and merge the test branch into `1.x` like so:
+
+```bash
+docker exec -it ce_dev_controller bash
+su ce-dev
+cd ~/ce-provision # or ~/ce-deploy
+git merge origin my_test_branch
+```
+
+Then use `ce-dev provision` and `ce-dev deploy` in the usual way.
 
 ### Known issues
+* The golang release needs keeping up to date (there's no 'latest' download we can use)
+* `mkcert` is built from source, initially because of a bad release but now it's like that we might as well keep it that way
+
+## ce-dev-build
+This builds the images and binaries, pushes the Docker containers to Docker Hub, pushes the binaries to a GitHub release and builds and commits the ce-dev documentation. It runs when someone accepts a PR for or pushes to the `1.x` branch and has added a valid tag in the format `1.*`. These images are tagged `latest` in the Docker repository.
 
+### Known issues
 * The golang release needs keeping up to date (there's no 'latest' download we can use)
-* mkcert is built from source, initially because of a bad release but now it's like that we might as well keep it that way
+* `mkcert` is built from source, initially because of a bad release but now it's like that we might as well keep it that way
 * If we start developing in a new version branch we will need to update the `on: push:` YAML in this workflow to allow the new tags or it won't execute
 
 ## ce-dev-lint
-
 This runs ESLint over the JavaScript code in the `src` directory. It runs on every pull request. It uses the ESLint config defined in `./.eslintrc` so if you need to change its behaviour, edit that file.
 
 ## ce-dev-test
-
 This builds the images and tests the ce-dev stack. It runs on every pull request and daily at 6:30.
 
 ### Known issues
-
 * The golang release needs keeping up to date (there's no 'latest' download we can use)
-* mkcert is built from source, initially because of a bad release but now it's like that we might as well keep it that way
+* `mkcert` is built from source, initially because of a bad release but now it's like that we might as well keep it that way

.github/workflows/ce-dev-PR-test.yml

@@ -30,5 +30,6 @@ jobs:
           rm -Rf mkcert
       - name: Build and test
         run: |
-          /bin/sh docker-images/export.sh latest
+          /bin/sh docker-images/export.sh --version latest --image-name ce-dev
+          /bin/sh docker-images/export.sh --version latest --image-name ce-dev-controller
           /bin/sh templates/prebuild.sh

.github/workflows/ce-dev-build-dev.yml

@@ -8,7 +8,7 @@ on:
 
 jobs:
   build:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
       - name: Install modules
@@ -39,4 +39,5 @@ jobs:
       - name: Build and push Docker images
         run: |
           echo "${{ secrets.DOCKER_PASSWORD }}" | docker login -u "${{ secrets.DOCKER_USERNAME }}" --password-stdin
-          /bin/sh docker-images/export.sh devel --push
+          /bin/sh docker-images/export.sh --version devel --image-name ce-dev --base-image debian:bookworm-slim --dockerfile-path base-devel --push
+          /bin/sh docker-images/export.sh --version devel --image-name ce-dev-controller --base-image debian:bookworm-slim --dockerfile-path controller-devel --push

.github/workflows/ce-dev-build.yml

@@ -42,7 +42,8 @@ jobs:
       - name: Build and push Docker images
         run: |
           echo "${{ secrets.DOCKER_PASSWORD }}" | docker login -u "${{ secrets.DOCKER_USERNAME }}" --password-stdin
-          /bin/sh docker-images/export.sh latest --push
+          /bin/sh docker-images/export.sh --version latest --image-name ce-dev --dockerfile-path base --push
+          /bin/sh docker-images/export.sh --version latest --image-name ce-dev-controller --dockerfile-path controller --push
           echo "${{ secrets.DOCKER_PASSWORD }}" | docker login -u "${{ secrets.DOCKER_USERNAME }}" --password-stdin
           /bin/sh templates/prebuild.sh --push
        # Uses https://github.com/marketplace/actions/gh-release

.github/workflows/ce-dev-test.yml

@@ -34,5 +34,6 @@ jobs:
           rm -Rf mkcert
       - name: Build and test
         run: |
-          /bin/sh docker-images/export.sh latest
+          /bin/sh docker-images/export.sh --version latest --image-name ce-dev
+          /bin/sh docker-images/export.sh --version latest --image-name ce-dev-controller
           /bin/sh templates/prebuild.sh

.gitignore

@@ -6,3 +6,4 @@ lib
 tmp
 node_modules
 oclif.manifest.json
+.vscode/*

contribute/README.md

@@ -1,5 +1,8 @@
 # Contribute to ce-dev
 
+## Using test images
+See [the GitHub Actions README](https://github.com/codeenigma/ce-dev/blob/1.x/.github/workflows/README.md) for information on using `devel` tagged images pushed by the CI.
+
 ## Release instructions
 
 1. [Create a pull request](https://github.com/codeenigma/ce-dev/compare) to the `1.x` branch.
@@ -21,7 +24,6 @@
 [GitHub Actions](https://github.com/codeenigma/ce-dev/actions) should now take care of the release for you.
 
 ## Rolling back a release
-
 If you need to pull a release, follow these instructions:
 
 1. Locally in your terminal on the `1.x` branch update the tags, e.g. `git pull --tags`

docker-images/base-devel/Dockerfile

@@ -0,0 +1,103 @@
+FROM debian:bookworm-slim as unison
+RUN \
+  set -x && \
+  export DEBIAN_FRONTEND=noninteractive && \
+  apt-get update && \
+  apt-get install -y -o Dpkg::Options::="--force-confnew" \
+  p7zip-full \
+  build-essential \
+  wget \
+  ocaml-native-compilers && \
+  cd /tmp/ && \
+  wget https://github.com/bcpierce00/unison/archive/v2.53.3.tar.gz && \
+  tar -xzvf v2.53.3.tar.gz && \
+  cd /tmp/unison-2.53.3 && \
+  make
+
+FROM debian:bookworm-slim
+
+RUN \
+  set -x && \
+  export DEBIAN_FRONTEND=noninteractive && \
+  apt-get update && \
+  apt-get dist-upgrade -y -o Dpkg::Options::="--force-confnew" && \
+  apt-get install -y -o Dpkg::Options::="--force-confnew" \
+  anacron \
+  apt-transport-https \
+  apt-utils \
+  aptitude \
+  bash \
+  binutils \
+  cron \
+  curl \
+  dirmngr \ 
+  gnupg \
+  rsync \
+  openssh-server \
+  postfix \
+  procmail \
+  python3-apt \
+  python3-dev \
+  python3-pycurl \
+  python3-pip \
+  python3-venv \
+  rsyslog \
+  sudo \
+  systemd \
+  systemd-sysv \
+  unzip \
+  vim \
+  wget && \
+  apt-get clean  && \
+  update-alternatives --install /usr/bin/python python /usr/bin/python3 1 && \
+  rm -rf \
+  /var/lib/apt/lists/* \
+  /var/log/* \
+  /tmp/*
+
+RUN \
+  echo 'UseDNS no' >> /etc/ssh/sshd_config && \
+  mkdir -p /var/run/sshd && \
+  rm /usr/sbin/policy-rc.d
+
+RUN \
+  set -x && \
+  export DEBIAN_FRONTEND=noninteractive && \
+  useradd -s /bin/bash ce-dev && \
+  echo ce-dev:ce-dev | chpasswd -m && \
+  install -m 755 -o ce-dev -g ce-dev -d /home/ce-dev && \
+  install -m 700 -o ce-dev -g ce-dev -d /home/ce-dev/.ssh && \
+  echo root:ce-dev | chpasswd -m && \
+  echo 'ce-dev ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/ce-dev && \
+  chmod 0440 /etc/sudoers.d/ce-dev && \
+  rm -rf /tmp/*
+
+RUN \
+  rm -f \
+  /etc/machine-id \
+  /var/lib/dbus/machine-id
+
+COPY --from=unison /tmp/unison-2.53.3/src/unison /usr/local/bin/
+COPY --from=unison /tmp/unison-2.53.3/src/unison-fsmonitor /usr/local/bin/
+COPY ./ce-dev-ownership.sh /opt/
+COPY ./ce-dev-ssh.sh /opt/
+COPY ./unison.sh /opt/
+COPY ./unison-startup.sh /opt/
+COPY ./procmailrc /etc/procmailrc
+
+RUN \
+  wget https://github.com/FiloSottile/mkcert/releases/download/v1.4.4/mkcert-v1.4.4-linux-amd64 -O /usr/local/bin/mkcert && \
+  mkdir -p /home/ce-dev/deploy/live.local /home/ce-dev/.composer/cache /home/ce-dev/.nvm/versions/node /home/ce-dev/.local/share/mkcert && \
+  chown -R ce-dev:ce-dev /home/ce-dev && \
+  chmod +x /usr/local/bin/*
+
+RUN \
+  systemctl mask --   \
+  dev-hugepages.mount \
+  sys-fs-fuse-connections.mount
+
+ENV container docker
+STOPSIGNAL SIGRTMIN+3
+VOLUME [ "/sys/fs/cgroup", "/run", "/run/lock", "/tmp" ]
+
+ENTRYPOINT ["/sbin/init"]

docker-images/base-devel/ce-dev-ownership.sh

@@ -0,0 +1,33 @@
+#!/bin/sh
+
+# Ensure user numeric uid/gid matches.
+# @param $1
+# User id.
+# @param $2
+# Group id.
+ensure_user_ids(){
+  OLD_UID="$(id -u ce-dev)"
+  OLD_GID="$(id -g ce-dev)"
+  if [ "$OLD_UID" = "$1" ] && [ "$OLD_GID" = "$2" ]; then
+    return
+  fi
+  if [ "$OLD_UID" != "$1" ]; then
+    usermod -u "$1" ce-dev
+    chown -R --from="$OLD_UID" "$1" /var
+    echo "User ID changed to $1."
+  fi
+  if [ "$OLD_GID" != "$2" ]; then
+    groupmod -g "$2" ce-dev
+    chown -R --from=":$OLD_GID" ":$2" /var
+    echo "Group ID changed to $2."
+  fi
+  if [ -d /.x-ce-dev ]; then
+      chown -R ce-dev:ce-dev /.x-ce-dev
+  fi
+  chown -R ce-dev:ce-dev /home/ce-dev
+}
+
+# Match ids with host user.
+if [ -n "$1" ] && [ -n "$2" ]; then 
+    ensure_user_ids "$1" "$2"
+fi

docker-images/base-devel/ce-dev-ssh.sh

@@ -0,0 +1,15 @@
+#!/bin/sh
+
+# Generate ssh key pair.
+ensure_ssh_key(){
+  rm -rf /home/ce-dev/.ssh/*
+  ssh-keygen -t rsa -b 4096 -N "" -f /home/ce-dev/.ssh/id_rsa
+  cp /home/ce-dev/.ssh/id_rsa.pub /home/ce-dev/.ssh/authorized_keys
+  touch /home/ce-dev/.ssh/config
+  chmod 600 /home/ce-dev/.ssh/id_rsa
+  chmod 600 /home/ce-dev/.ssh/id_rsa.pub
+  chmod 600 /home/ce-dev/.ssh/authorized_keys
+  chown -R ce-dev:ce-dev /home/ce-dev/.ssh
+}
+
+ensure_ssh_key

docker-images/base-devel/procmailrc

@@ -0,0 +1,3 @@
+ORGMAIL=/dev/null
+DEFAULT=${ORGMAIL}
+MAILDIR=${ORGMAIL}

docker-images/base-devel/unison-startup.sh

@@ -0,0 +1,3 @@
+#!/bin/sh
+
+nohup /bin/sh /opt/unison.sh "$@" </dev/null >/dev/null 2>&1 &

docker-images/base-devel/unison.sh

@@ -0,0 +1,6 @@
+#!/bin/sh
+
+mkdir -p "/tmp/$1"
+while true; do
+  flock -w 30 /tmp"$1"/unison.lock /usr/local/bin/unison -copythreshold 100000 -owner -group -batch -repeat watch -fastercheckUNSAFE -prefer "$1" "$@" || exit 1
+done

docker-images/base/Dockerfile

@@ -40,6 +40,7 @@ RUN \
   python3-dev \
   python3-pycurl \
   python3-pip \
+  python3-venv \
   rsyslog \
   sudo \
   systemd \
@@ -99,4 +100,4 @@ ENV container docker
 STOPSIGNAL SIGRTMIN+3
 VOLUME [ "/sys/fs/cgroup", "/run", "/run/lock", "/tmp" ]
 
-ENTRYPOINT ["/sbin/init"]
+ENTRYPOINT ["/sbin/init"]

docker-images/controller-ci/Dockerfile

@@ -14,9 +14,8 @@ RUN \
   apt-get update && \
   apt-get dist-upgrade -y -o Dpkg::Options::="--force-confnew" && \
   apt-get install -y -o Dpkg::Options::="--force-confnew" \
-  git ca-certificates git-lfs && \
+  git ca-certificates git-lfs ansible python3-boto3 && \
   apt-get clean  && \
-  pip3 install ansible boto3 && \
   git lfs install --skip-repo && \
   update-alternatives --install /usr/bin/python python /usr/bin/python3 1 && \
   useradd -m controller && \
@@ -54,7 +53,7 @@ RUN \
   set -x && \
   export DEBIAN_FRONTEND=noninteractive && \
   apt-get update && \
-  su - ce-dev -c "/usr/local/bin/ansible-playbook --extra-vars=\"{ansible_common_remote_group: controller}\" /home/ce-dev/ce-provision/provision.yml" && \
+  su - ce-dev -c "/usr/bin/ansible-playbook --extra-vars=\"{ansible_common_remote_group: controller}\" /home/ce-dev/ce-provision/provision.yml" && \
   rm /home/ce-dev/ce-provision/provision.yml && \
   apt-get clean  && \
   rm -rf \

docker-images/controller-ci/provision.yml

@@ -10,6 +10,8 @@
     - is_local: true
     - _env_type: utility
     - ce_provision:
+        venv_path: "/home/{{ _ce_provision.username }}/ansible"
+        venv_command: /usr/bin/python3 -m venv
         own_repository: https://github.com/codeenigma/ce-provision.git
         own_repository_branch: 1.x
         own_repository_skip_checkout: false
@@ -19,9 +21,21 @@
         username: controller
         local_dir: /home/controller/ce-provision
         groups: []
+        contrib_roles:
+          - directory: wazuh
+            repo: https://github.com/wazuh/wazuh-ansible.git
+            branch: stable
+          - directory: systemd_timers
+            repo: https://github.com/vlcty/ansible-systemd-timers.git
+            branch: master
         galaxy_custom_requirements_file: ""
         galaxy_roles_directory: "/home/{{ _ce_provision.username }}/.ansible/roles"
         upgrade_galaxy:
           enabled: false
+    - ce_ansible:
+        upgrade:
+          enabled: false
+        linters:
+          enabled: true
   roles:
     - ce_provision