File Permissions not preserved when using workingDir

[BreckanM]

Describe the issue

When trying to use the cytopia/ansible docker container, i need to copy SSH priv key into the workingDir and reference it explicitly so that the passwordless ssh connection to remote host will work.

ssh public key has already copied to the remote machine successfully.

Note: i did change the Docker volumes to mounts in my compose.yml file :

 kestra:
    image: kestra/kestra:latest
    restart: unless-stopped
    pull_policy: always
    user: "root"
    command: server standalone
    volumes:
      - ./kestra-data:/app/storage
      - /var/run/docker.sock:/var/run/docker.sock
      - ./tmp/kestra-wd:/tmp/kestra-wd:rw

Kestra Flow:

id: ansible_packages
namespace: homelab.ansible
description: Call Ansible playbook via Cytopia Docker Container

tasks:
  - id: ansible_task_file_load
    type: io.kestra.plugin.ansible.cli.AnsibleCLI
    inputFiles:
      inventory.ini: "{{ read('inv_lt.ini')}}" 
      playbook.yml: "{{ read('packages.yml')}}"
      kestra_id_rsa: "{{ read('kestra_id_rsa')}}"
      
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
      pullPolicy: IF_NOT_PRESENT
      volumes:
        - /tmp/kestra-wd:/app
    containerImage: cytopia/ansible:latest-tools
    env:
      "ANSIBLE_HOST_KEY_CHECKING": "false"
      "ANSIBLE_PRIVATE_KEY_FILE": "kestra_id_rsa"
    commands:
      - ansible-playbook -i inventory.ini playbook.yml

the error in log from Ansible in Docker is that permissions are too wide

fatal: [LinuxTestHost]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '192.168.0.99' (ED25519) to the list of known hosts.\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\n@ WARNING: UNPROTECTED PRIVATE KEY FILE! @\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\nPermissions 0644 for '/tmp/kestra-wd/tmp/6rPK2pHakCy8L8id7oEfK4/kestra_id_rsa' are too open.\r\nIt is required that your private key files are NOT accessible by others.\r\nThis private key will be ignored.\r\nLoad key \"/tmp/kestra-wd/tmp/6rPK2pHakCy8L8id7oEfK4/kestra_id_rsa\": bad permissions\r\nkestra@192.168.0.99: Permission denied (publickey,password).", "unreachable": true}

Source Folder ( namespaceFiles )

xxx@awx:~/kestra/kestra-data/homelab/ansible/_files$ ls -la
total 60
drwxr-xr-x 2 root root 4096 Oct 21 09:27 .
drwxr-xr-x 3 root root 4096 Oct 18 01:25 ..
-rw-r--r-- 1 root root  246 Oct 21 22:07 inv_lt.ini
-rw-r--r-- 1 root root  190 Oct 18 05:53 inv_tsr.ini
-rw-r--r-- 1 root root   35 Oct 18 03:41 inventory2.ini
-rw------- 1 root root 1811 Oct 21 00:06 kestra_id_rsa
-rw------- 1 root root  392 Oct 21 00:07 kestra_id_rsa.pub

workingDir status during execution of flow

xxx@awx:~/kestra/tmp/kestra-wd/tmp# ls -laR
.:
total 12
drwxr-xr-x 3 root root 4096 Oct 21 22:18 .
drwxr-xr-x 3 root root 4096 Oct 17 12:01 ..
drwxr-xr-x 3 root root 4096 Oct 21 22:18 6rPK2pHakCy8L8id7oEfK4

./6rPK2pHakCy8L8id7oEfK4:
total 24
drwxr-xr-x 3 root root 4096 Oct 21 22:18 .
drwxr-xr-x 3 root root 4096 Oct 21 22:18 ..
drwxr-xr-x 2 root root 4096 Oct 21 22:18 2ShD0iqFmas5MbcR9KhcX2
-rw-r--r-- 1 root root  246 Oct 21 22:18 inventory.ini
-rw-r--r-- 1 root root 1811 Oct 21 22:18 kestra_id_rsa
-rw-r--r-- 1 root root  356 Oct 21 22:18 playbook.yml

It appears that when the files are pre-staged by Kestra (aka copied into the temp workingDir), permissions are not preserved.
Specifically 0600 changes to 0644 for the private key.

Is there a way to explicitly set permissions on the files in the workingDir ?

Environment

• Kestra Version: 0.19.3
• Operating System (OS/Docker/Kubernetes): Ubuntu 24.04.1 LTS, Docker 27.2.1
• Java Version (if you don't run kestra in Docker):

[Ben8t]

Can't say about the setting permission in workingDir (will wait developer review).
However, maybe you can test to add a chmod 0600 on the ssh key in the commands before running ansible ?

[BreckanM]

good thought - i changed my flow to this:

tasks:
  - id: workingDir
    type: io.kestra.plugin.core.flow.WorkingDirectory
    namespaceFiles: 
      enabled:  true
      include:
        - inv_lt.ini
        - packages.yml
        - kestra_id_rsa
    tasks:
      - id: fix_permissions
        type: io.kestra.plugin.scripts.shell.Commands
        commands:
          - 'chmod 600 {{ workingDir }}/kestra_id_rsa'
          - 'ls -laR'
      - id: ansible_task_file_load
        type: io.kestra.plugin.ansible.cli.AnsibleCLI
        taskRunner:
          type: io.kestra.plugin.scripts.runner.docker.Docker
          pullPolicy: IF_NOT_PRESENT
          volumes:
            - /tmp/kestra-wd:/app
        containerImage: cytopia/ansible:latest-tools
        env:
          "ANSIBLE_HOST_KEY_CHECKING": "false"
          "ANSIBLE_PRIVATE_KEY_FILE": "kestra_id_rsa"
        commands:
          # - ansible-playbook -i inventory.ini playbook.yml
          - ansible-playbook -i inv_lt.ini packages.yml

the ls -laR output shows that the permission does get changed to 0600 successfully, but even with the parent WorkingDirectory (which i thought would preserve the workingDir for the duration of the flow), the permission reverts to 0644 by the time ansible executes, and i get the same error as before :

fatal: [LinuxTestHost]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '192.168.0.99' (ED25519) to the list of known hosts.\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\n@ WARNING: UNPROTECTED PRIVATE KEY FILE! @\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\nPermissions 0644 for 'kestra_id_rsa' are too open.\r\nIt is required that your private key files are NOT accessible by others.\r\nThis private key will be ignored.\r\nLoad key \"kestra_id_rsa\": bad permissions\r\nkestra@192.168.0.99: Permission denied (publickey,password).", "unreachable": true}

[BreckanM]

in fact, regardless of the original owner or permissions, the files that end up in {{ workingDir }} end up 0644 and owned by root.

[loicmathieu]

@BreckanM this is due by the fact you run Docker engine as root, so the files created by the Docker engine (we use a volume mount) will be created as root.

[BreckanM]

thanks @loicmathieu.

totally understand that copying the files into workingdir will make them owned by root.

from what i can see the process is :

1. copy files from namespaceFiles location ( /kestra-data/<namespace>/_files ) to workingDir location ( /tmp/kestra-wd/tmp/<executionID> , resetting owner to root:root and explicitly setting permission mask to 0644

2. the permissions in workingDir seem to revert to 0644 between tasks under the directive

  - id: workingDir
    type: io.kestra.plugin.core.flow.WorkingDirectory

3. the workingDir would then be volume mapped into the Cytopia instance

Concept :
would it be viable to map a completely new mount from host linux to Kestra Docker, then mount that into Cytopia and reference the SSHPUBKEY explicitly from a different path (ie outside of any workingDir functionality) ?

eg :
parent linux host : ~/kestra/mysshkeys
kestra compose.yml :

<snip>
  kestra:
    volumes:
      - ~/kestra/mysshkeys : /sshkeys
<snip>

kestra flow:

<snip>
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
      pullPolicy: IF_NOT_PRESENT
      volumes:
        - /tmp/kestra-wd:/app
        - /sshkeys:/sshkeys
    containerImage: cytopia/ansible:latest-tools
    env:
      "ANSIBLE_HOST_KEY_CHECKING": "false"
      "ANSIBLE_SSH_PRIVATE_KEY_FILE": "/sshkeys/kestra_id_rsa"
<snip>

[BreckanM]

Just for clarity - I have a workaround that does now work, but there are some details that need to be understood:

Cytopia/ansible doesnt run 'docker-in-docker' - it runs on the parent OS not inside Kestra, so its 'docker-beside-docker'
Also, the volume bind for the docker taskrunner flow configuration doesnt/cant understand relative paths from the host OS (so cannot use ./sshkeys or ~/kestra/sshkeys. you must use /full/path/on/host/sshkeys, or whatever you happen to use)

So my full workaround is:

1. use ssh-keygen to create your desired pub/priv keys for your desired connecting username for use by ansible playbooks
2. copy the private key to somewhere relevant to your kestra docker deployment
   (no need to bind anything in the compose.yml for kestra)
3. configure the kestra flow

  taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
      pullPolicy: IF_NOT_PRESENT
      volumes:
        - /tmp/kestra-wd:/app
        - /path/on/host/sshkeys:/path/on/cytopia/sshkeys
    containerImage: cytopia/ansible:latest-tools
    env:
      "ANSIBLE_HOST_KEY_CHECKING": "false"
      "ANSIBLE_SSH_PRIVATE_KEY_FILE": "/path/on/cytopia/sshkeys/kestra_id_rsa"
    commands:
      - ansible-playbook -i inventory.ini playbook.yml

Note:

• in the Cytopia container, the SSH keys will be ansible:anisble
• as its a bind, not a copy, the permissions appear to stay as per original files (eg 0644 for pub keys, 0600 for priv keys)

Playbook will now connect using SSHKEY successfully.