Add r10k's purge_level, revert unmanaged/change files

[baurmatt]

It would be great if the r10k feature purge_level could be added. Currently $someone could manually edit the deployed module/environment without g10k noticing.

That said, we're looking for a feature which verifies on each g10k run that all files are in the state the Puppetfile/control repository describes them.

[baurmatt]

Just checked the debug log and it turns out that something like that is already happening, but only for some directories:

2017/11/20 17:10:33 DEBUG resolvePuppetfile(): Removing unmanaged file /etc/puppetlabs/code/environments/production/modules/external

This is a folder each of our environments has which should contain all external repositories. The folder structure looks like this:

• manifests
  • site.pp
• modules
  • internal_repo_1 (git repo)
  • internal_repo_2 (git repo)
  • internal_repo_3 (git repo)
  • internal_repo_4 (git repo)
  • external (folder)
    • stdlib (git repo)
    • mysql (git repo)
    • ...
• hieradata

The module path is: modules/external:modules/internal_repo1/modules:modules/internal_repo_2/modules:modules/internal_repo_3/modules:modules/internal_repo_4/modules

Can you please explain how the purging works? :)

[xorpaul]

g10k checks your moduledir directory inside your Puppet environment if there are folders/modules which are not managed inside the respective Puppetfile.

This is also the reason I added the local modules support:

#75

g10k would not detect if a file or a subdirectory inside a module directory was changed though. I don't know if that is enough for your use-case.

[baurmatt]

Thanks for the explanation! :) I think the purging should always happen after the deployment of the module, currently it seems to happen before the modules get deployed which leads to the following experience:

root@mbaur-g10k:~# for i in {1..5}; do echo "Run ${i}"; ./g10k -config g10k.yaml -branch pp-4170; ls -la /etc/puppetlabs/code/environments/pp-4170/modules/; echo "---------------------";done
Run 1
Resolving Git modules (43/43) 1.712s [====================================================================] 100%
Synced g10k.yaml with branch pp-4170 with 49 git repositories and 0 Forge modules in 11.0s with git (4.4s sync, I/O 21.4s) and Forge (0.0s query+download, I/O 0.0s) using 50 resolv
and 20 extract workers
total 28
drwxr-xr-x  7 root root 4096 Nov 20 17:46 .
drwxr-xr-x  8 root root 4096 Nov 20 17:46 ..
drwxr-xr-x  5 root root 4096 Nov 20 17:46 internal_repo_1
drwxr-xr-x 40 root root 4096 Nov 20 17:46 external
drwxr-xr-x  3 root root 4096 Nov 20 17:46 internal_repo_2
drwxr-xr-x  3 root root 4096 Nov 20 17:46 internal_repo_3
drwxr-xr-x  3 root root 4096 Nov 20 17:46 internal_repo_4
---------------------
Run 2
Resolving Git modules (43/43) 1.937s [====================================================================] 100%
Synced g10k.yaml with branch pp-4170 with 49 git repositories and 0 Forge modules in 6.6s with git (4.6s sync, I/O 1.0s) and Forge (0.0s query+download, I/O 0.0s) using 50 resolv and 20 extract workers
total 24
drwxr-xr-x 6 root root 4096 Nov 20 17:46 .
drwxr-xr-x 8 root root 4096 Nov 20 17:46 ..
drwxr-xr-x 5 root root 4096 Nov 20 17:46 internal_repo_1
drwxr-xr-x 3 root root 4096 Nov 20 17:46 internal_repo_2
drwxr-xr-x 3 root root 4096 Nov 20 17:46 internal_repo_3
drwxr-xr-x 3 root root 4096 Nov 20 17:46 internal_repo_4
---------------------
Run 3
Resolving Git modules (43/43) 1.948s [====================================================================] 100%
Synced g10k.yaml with branch pp-4170 with 49 git repositories and 0 Forge modules in 5.9s with git (4.6s sync, I/O 8.5s) and Forge (0.0s query+download, I/O 0.0s) using 50 resolv and 20 extract workers
total 28
drwxr-xr-x  7 root root 4096 Nov 20 17:46 .
drwxr-xr-x  8 root root 4096 Nov 20 17:46 ..
drwxr-xr-x  5 root root 4096 Nov 20 17:46 internal_repo_1
drwxr-xr-x 40 root root 4096 Nov 20 17:46 external
drwxr-xr-x  3 root root 4096 Nov 20 17:46 internal_repo_2
drwxr-xr-x  3 root root 4096 Nov 20 17:46 internal_repo_3
drwxr-xr-x  3 root root 4096 Nov 20 17:46 internal_repo_4
---------------------
Run 4
Resolving Git modules (43/43) 1.694s [====================================================================] 100%
Synced g10k.yaml with branch pp-4170 with 49 git repositories and 0 Forge modules in 5.1s with git (4.5s sync, I/O 0.0s) and Forge (0.0s query+download, I/O 0.0s) using 50 resolv and 20 extract workers
total 24
drwxr-xr-x 6 root root 4096 Nov 20 17:46 .
drwxr-xr-x 8 root root 4096 Nov 20 17:46 ..
drwxr-xr-x 5 root root 4096 Nov 20 17:46 internal_repo_1
drwxr-xr-x 3 root root 4096 Nov 20 17:46 internal_repo_2
drwxr-xr-x 3 root root 4096 Nov 20 17:46 internal_repo_3
drwxr-xr-x 3 root root 4096 Nov 20 17:46 internal_repo_4
---------------------
Run 5
Resolving Git modules (43/43) 1.474s [====================================================================] 100%
Synced g10k.yaml with branch pp-4170 with 49 git repositories and 0 Forge modules in 5.8s with git (4.5s sync, I/O 9.3s) and Forge (0.0s query+download, I/O 0.0s) using 50 resolv and 20 extract workers
total 28
drwxr-xr-x  7 root root 4096 Nov 20 17:46 .
drwxr-xr-x  8 root root 4096 Nov 20 17:46 ..
drwxr-xr-x  5 root root 4096 Nov 20 17:46 internal_repo_1
drwxr-xr-x 40 root root 4096 Nov 20 17:46 external
drwxr-xr-x  3 root root 4096 Nov 20 17:46 internal_repo_2
drwxr-xr-x  3 root root 4096 Nov 20 17:46 internal_repo_3
drwxr-xr-x  3 root root 4096 Nov 20 17:46 internal_repo_4
---------------------

[xorpaul]

The removal of unmanaged directories inside the moduledir should happen after all Puppetfile modules are deployed: https://github.com/xorpaul/g10k/blob/master/puppetfile.go#L305

Is the external folder part of the control repository branch in your example?

[baurmatt]

No, neither the modules nor the external folder are part of the control repository.

I just guessed by the test output i posted above that the cleanup happens before the deployment. Not sure what is going on here.

[xorpaul]

Do you have the environment variable g10k_cachdir set to /etc/puppetlabs/code/environments/pp-4170/modules/external by any chance?

[baurmatt]

Nope, i only use the configuration file:

root@mbaur-g10k:~# env | grep -i g10k
root@mbaur-g10k:~# cat g10k.yaml
---
:cachedir: /opt/puppetlabs/puppet/cache/g10k
:sources:
  syseleven:
    basedir: /etc/puppetlabs/code/environments
    remote: git@gitserver.example.org:control.git

[xorpaul]

So the external module is part of your Puppetfile and it's missing every now and then?

[baurmatt]

No, external is just a folder which contains upstream modules as described #80 (comment)

[baurmatt]

You should be able to reproduce this problem with https://github.com/baurmatt/g10k-77-control

[xorpaul]

The problem was that I did not consider the install_path Git module attribute correctly while detecting what is unmanaged inside the moduledir and that it can also be a subdirectory inside the moduledir.

I'm now checking if the already existing directory is a parent directory of the install_path Git module 394c5f6#diff-df78fe67e2c9bffdb3673a5fd14d2bb7R306

Please check out v0.4.4

[baurmatt]

Awesome, thank you very much! :) I can't reproduce the problem described in #80 (comment).

The problem I described at the beginning of the issue (unchanged/modified files/folder within the modules aren't reverted) still exists. Should i open a new issue for that?

[xorpaul]

Yes, please open a new ticket if the problem still exists.