Salt provisioner fails to upload minion config

[piohhmy]

Ran into a problem in 1.7.3 doing a vagrant up using a trusty64 box with a salt provisioner specifying a custom minion_config. This is the error:

Copying salt minion config to /etc/salt
Failed to upload a file to the guest VM via SCP due to a permissions
error. This is normally because the SSH user doesn't have permission
to write to the destination location. Alternately, the user running
Vagrant on the host machine may not have permission to read the file.

Source: /Users/danny/dev/se/one-offs/vagrant/minion.yaml
Dest: /etc/salt/minion

It appears that the vagrant user does not have permission by default to scp to /etc/salt/ so provisioning fails. I confirmed it is possible to vagrant ssh into the machine and change the permissions/owner on /etc/salt/ and then the provisioning will work OK.

It appears 74d2206 may have been the breaking change as it introduced the uploading to /etc/salt.

[SoftCreatR]

I can confirm this. We're currently experiencing this issue, too.

[joshuacherry]

I can confirm this as well. I'm using Vagrant 1.7.3 with VirtualBox 5.0. As a workaround I've been using a shell provisioner with the following to make sure salt in installed and the config is copied over:

#!/usr/bin/env bash

echo "Setup saltstack PPA"
sudo apt-get install software-properties-common python-software-properties -y >> /dev/null
sudo add-apt-repository ppa:saltstack/salt -y >> /dev/null
sudo apt-get update >> /dev/null

echo "Installing salt-minion"
sudo apt-get install salt-minion -y >> /dev/null
sudo cp /vagrant/salt/configs/ubuntu.conf /etc/salt/minion

echo "Restart salt-minion"
sudo salt-minion -d
sudo service salt-minion restart

[nmadhok]

I can confirm this issue. As a workaround, I specified config.ssh.username = "root" and config.ssh.password = "vagrant" in my Vagrant file

[clamor]

Same issue with provider vmware_desktop.

[samjacobclift]

I've got this issue too

[amchoukir]

I can confirm the issue too

[sonoranDesert]

This also started happening to me after upgrading to Vagrant 1.7.4.

[dwickwire]

Same experience here using vagrant 1.7.3 and 1.7.4 on both Virtual Box 4.3 and 5 using OSX Yosemite.

[aaazalea]

@sephlaire could you explain your workaround in more detail?

[joshuacherry]

@Jake223
I have 3 relevant files that I am using to provision the VM. Vagrantfile, server.yaml, and ubuntu_bootstrap.sh which are below. The Vagrantfile is using vm.provision "shell" to call the shell script which installs salt and copies over my minion config.

Vagrantfile

# -*- mode: ruby -*-
# # vi: set ft=ruby :

# Specify minimum Vagrant version and Vagrant API version
Vagrant.require_version ">= 1.6.0"
VAGRANTFILE_API_VERSION = "2"

# Require YAML module
require 'yaml'

# Read YAML file with box details
servers = YAML.load_file('servers.yaml')

# Create boxes
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  # Iterate through entries in YAML file
  servers.each do |servers|
    config.vm.define servers["name"] do |srv|
      srv.vm.box = servers["box"]
      srv.vm.network "private_network", ip: servers["ip"]
      srv.vm.hostname = servers["hostname"]
      srv.vm.provision "shell", path: servers["bootstrap"] unless servers["bootstrap"].nil?
      srv.vm.network "forwarded_port", guest: servers["guest_port"], host: servers["host_port"] unless servers["guest_port"].nil? || servers["host_port"].nil?
      srv.vm.synced_folder servers["sync-local"], servers["sync-remote"] unless servers["sync-local"].nil? || servers["sync-remote"].nil?
      srv.vm.provider :virtualbox do |vb|
        vb.name = servers["name"]
        vb.memory = servers["ram"]
      end
      unless servers["minion_config"].nil?
       # srv.vm.provision "file", source: servers["minion_config"], destination: "/etc/salt/minion"
        srv.vm.provision :salt do |salt|
          salt.run_highstate = true
          salt.minion_config = servers["minion_config"] 
          salt.minion_key = servers["minion_key"] unless servers["minion_key"].nil?
          salt.minion_pub = servers["minion_pub"] unless servers["minion_pub"].nil?
        end
      end
    end
  end
end

servers.yaml

---
- name: ubuntu
  hostname: ubuntu
  box: ubuntu/trusty64
  ram: 1024
  ip: 10.1.15.11
  bootstrap: ubuntu_bootstrap.sh
  sync-local: ./salt/roots
  sync-remote: /srv/salt
  minion_config: ./salt/configs/ubuntu.conf
  minion_key: ./salt/ssh-key/ubuntu.pem
  minion_pub: ./salt/ssh-key/ubuntu.pub
- name: centos
  hostname: centos
  box: chef/centos-7.0
  ram: 1024
  ip: 10.1.15.12
  bootstrap: centos_bootstrap.sh
  sync-local: ./salt/roots
  sync-remote: /srv/salt 
  minion_config: ./salt/configs/centos.conf
  minion_key: ./salt/ssh-key/centos.pem
  minion_pub: ./salt/ssh-key/centos.pub

ubunt_bootstrap.sh

#!/usr/bin/env bash

echo "Setup saltstack PPA"
sudo apt-get install software-properties-common python-software-properties -y >> /dev/null
sudo add-apt-repository ppa:saltstack/salt -y >> /dev/null
sudo apt-get update >> /dev/null

echo "Installing salt-minion"
sudo apt-get install salt-minion -y >> /dev/null
sudo cp /vagrant/salt/configs/ubuntu.conf /etc/salt/minion

echo "Restart salt-minion"
sudo salt-minion -d
sudo service salt-minion restart
sudo salt-call state.highstate

[amchoukir]

The workaround that worked for me is as follows

In Vagrantfile:

config.vm.provision :shell, path: "provision/base_salt_provision.sh", privileged: false

In provision/base_salt_provision.sh:

start_seconds="$(date +%s)"

if [ ! -d "/etc/salt" ]; then
    echo "Bootstrap salt"
    wget -O - https://bootstrap.saltstack.com | sudo sh
    sudo cp /vagrant/salt/minion /etc/salt/minion

    echo "Restart salt-minion"
    sudo salt-minion -d
    sudo service salt-minion restart
fi

echo "Executing salt highstate (provisionning)"
sudo salt-call state.highstate

end_seconds="$(date +%s)"
echo "-----------------------------"
echo "Provisioning complete in "$(expr $end_seconds - $start_seconds)" seconds"

[CharlesLovely]

Also seeing the same issue.

Also, to add to this current issue, I noticed that when provisioning salt-master, while vagrant won't necessarily error when specifying a configuration file, it won't use the specified file either.

For those looking for yet another workaround, this is what I did:

# SALT MASTER

saltmaster.vm.provision :salt do |salt|
  salt.master_config = "salt/configs/master"  # this line seems ignored, or may be failing silently
  salt.install_master = true
  salt.no_minion = true
  salt.install_type = "git"
  salt.install_args = "v2015.5.1"
end  # salt provisioner

saltmaster.vm.provision "shell",
  inline: "sudo cp /vagrant/salt/configs/master /etc/salt/master && sudo service salt-master restart"

# SALT MINION

machine.vm.provision :salt do |salt|
  # note: no salt.minion_config line here... if you include it here, it will fail
  salt.install_master = false
  salt.no_minion = false
  salt.install_type = "git"
  salt.install_args = "v2015.5.1"
end  # salt provisioner

machine.vm.provision "shell",
  inline: "sudo cp /vagrant/salt/configs/minion_local /etc/salt/minion && sudo service salt-minion restart"

[wwentland]

This is rather unfortunate as, for example, Debian jessie sets PermitRootLogin without-password in its /etc/ssh/sshd_config which prevents @nmadhok's suggestion from working.

Is there a way to utilise sudo in the copy process ?

[nmadhok]

@BABILEN I would recommend using a custom box image

[wwentland]

@nmadhok Well, we are, but I am hesitant to change that globally in our vagrant cloud, simply because the provisioner is buggy. I'd rather recommend that people don't upgrade their vagrant installation from 1.7.2 yet.

[nmadhok]

@BABILEN I agree

[FlorianLoch]

Can confirm this one too, trusty64 seems to restrict root login too...

[davidkarlsen]

Any chance of getting this fixed for 1.7.5?

[nkantar]

Echoing @davidkarlsen's question. Any other info, ETA, etc.?

[jberends]

Worked around this permission problem as well by following @CharlesLovely comment by 'manually' copying over the minion config file and commenting out the salt.minion_config

config.vm.provision :salt do |salt|
    [...]
    #salt.minion_config = "saltstack/etc/minion"
    [...]
end

config.vm.provision "shell",
    inline: "sudo cp /etc/salt/minion{,-dist} && sudo cp /vagrant/saltstack/etc/minion /etc/salt/minion && sudo service salt-minion restart"

[onedeveloper]

I'm on the same boat, tried to use @jberends solution but it comes to a point where the highstate just hangs. My guess salt provisioner is trying to run with default minion config before copying the actual minion config file.

for reference

Project structure:

$ tree
.
├── Vagrantfile
└── salt
    ├── minion
    └── roots
        ├── top.sls
        └── webserver.sls

2 directories, 4 files

Vagrantfile:

# -*- mode: ruby -*-
# vi: set ft=ruby :

VAGRANTFILE_API_VERSION = "2"

# bug with vagrant + salt provisioner
# see => https://github.com/mitchellh/vagrant/issues/5973
$bypass = <<SCRIPT
sudo cp /etc/salt/minion{,-dist} && sudo cp /vagrant/salt/minion /etc/salt/minion && sudo service salt-minion restart
SCRIPT

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.define "web", primary: true do |web|
    web.vm.box = "ubuntu/trusty64"
    web.vm.network "forwarded_port", guest: 80, host: 8080
    web.vm.network "forwarded_port", guest: 443, host: 8443
    web.vm.synced_folder "salt/roots/", "/srv/salt/"
    web.vm.provision :salt do |salt|
      # salt.minion_config = "salt/minion" # bypassing normal minion configuration
      salt.run_highstate = true
      salt.bootstrap_options = "-P"
    end
    web.vm.provision "shell", inline: $bypass
  end
end

minion:

master: localhost
state_top: top.sls
file_client: local

tops.sls:

base:
  '*':
webserver

webserver.sls:

apache2:
  pkg:
    - installed

[pcn]

I've given up on 1.7.3 and 1.7.4. Something regressed after 1.7.2.

[onedeveloper]

@pcn was it broken before 1.7.2 ?

[chrismcmacken]

I ran into the issue on 1.7.2, ended up using @amchoukir s workaround

[FlorianLoch]

I returned to 1.7.2 and can provision again... next timr I check the issues before blindly updating my dev box

[tehmaspc]

this is the 2nd issue w/ salt provisioner today for me (saltstack/salt-bootstrap#648 is the other one); i think @zundra's advice is best for now till vagrant 1.8. or some patch fix to 1.7.x.

substituting the salt provisioner w/ a shell provisioner doing the same thing is not a valid maintainable fix - I could do this and have for other provisioners in the past - but ain't nobody got time for that.

@nmadhok - if you did get this to work w/ your workaround above - can you kindly post your Vagrantfile - I get auth issues upon trying what you suggest; maybe I'm putting it in the wrong spot of my Vagrantfile.

UPDATE (FWIW/ to help anyone else):
downgrading to vagrant 1.7.2 (which also forced me to downgrade from vbox 5.x to latest 4.3.30 - since vbox 5.x support came to vagrant after 1.7.2) worked for me to get a 'ubuntu/trusty64' vagrant box provisioned correctly with my salt formula.

[amchoukir]

@tehmaspc out of curiosity why do you think a shell provisionner is not a maintainable solution. Vagrant caters to type 2 hypervisors, I.e the hypervisors runs as an application on a host OS and allow to run guest operating systems. For type 1 hypervisors, the hypervisor runs bare metal not on top of a host operating system. Examples would be VMWare ESXi, Citrix Xenserver and so on. Type 1 hypervisor are the heavy duty case running hundred if not thousands of VMs and Salt would most likely be installed via a shell script or through a gold image/template.

[ghost]

Excerpt from a working Vagrantfile (with a workaround applied):

  config.vm.provision :salt do |salt|
    salt.run_highstate = false # vagrant issue #5973
  end

  # vagrant issue #5973
  config.vm.provision "shell",
    inline: "sudo cp /etc/salt/minion{,-dist} && sudo cp /vagrant/config/minion /etc/salt/minion && sudo salt-call --local state.highstate"

minion configuration file is kept in config/minion (directory relative to Vagrantfile).

Is this issue still affecting Salt 2015.5.5 or 2015.8.0RCs?

[ev0rtex]

@edvinasme This bug exists regardless of the salt version.

If it helps anyone else, here's what I did. In my case I'm passing pillar data along in my Vagrantfile so I can't just make it provision using a shell command. Instead I make it copy the minion config to /tmp/minion and then pass in the -c /tmp bootstrap option:

    conf.vm.provision :file, source: "provision/minion", destination: "/tmp/minion"
    conf.vm.provision :salt do |salt|
        # Custom pillar
        salt.pillar("user_options" => vmopts['options'].presence || {})
        salt.pillar("user_packages" => vmopts['packages'].presence || {})
        salt.pillar("timezone" => zone)

        # Salt options
        salt.minion_config = "provision/minion"
        salt.run_highstate = true
        salt.install_type = "git"
        salt.install_args = "v2015.8.0rc3"
        salt.bootstrap_options = "-P -c /tmp"
        salt.colorize = true
        salt.verbose = true
        salt.log_level = "info"
        # salt.always_install = true
    end

[joshughes]

@tehmaspc I can confirm vagrant 1.7.4 and Virtualbox 5 works with the above fix... which looks like the following for me..

  config.vm.provision :file, source: "./.vagrant-salt/minion", destination: "/tmp/minion"
  # Provision VM with current formula, in masterless mode
  config.vm.provision :salt do |salt|
    salt.minion_config = "./.vagrant-salt/minion"
    salt.run_highstate = true
    salt.install_type = 'git v2015.5.5'
    salt.bootstrap_options = "-P -c /tmp"
    salt.colorize = true
    salt.verbose = true
  end

[bemeyert]

Hi all, I stumbled over this issue because my setup for minion boxes (pre Vagrant 1.7.2) was no longer working with 1.7.4 too. After some gruesome debugging (eyes still hurting) I found out that vagrant uploads the keys and configuration only to the "tmpdir" (in Linux this is /tmp) as user "vagrant". So I gave the bootstrap script some options via salt.bootstrap_options: -F -c /tmp/. This worked at the end. So my setup looks like this:

[...]
$minion_script = <<EOF
yum remove -y salt\*
echo $(hostname -f) > /etc/salt/minion_id
EOF
[...]
        minionc6.vm.provision "shell",
            inline: $minion_script
        minionc6.vm.provision :salt do |salt|
            salt.bootstrap_options = '-F -c /tmp/'
            salt.minion_config = "provision/minion_config"
            salt.minion_key = "provision/keys/minion.pem"
            salt.minion_pub = "provision/keys/minion.pub"
            salt.colorize = true
            salt.log_level = "info"
            salt.verbose = true
        end
[...]

So I think that either the documentation should state the usage of the switches to bootstrap when provisioning keys and config or vagrant could do it by itself.

[robertwe]

@bemeyert works for me. Many thx for sharing it.

[bemeyert]

@robertw Glad I could help ;)

[bsuh]

TL;DR: Add salt.bootstrap_options = '-F -c /tmp/ -P' to your Vagrantfile

Salt bootstrap options explanations

  Options:
  -c  Temporary configuration directory
  -F  Allow copied files to overwrite existing(config, init.d, etc)
  -P  Allow pip based installations. On some distributions the required salt
      packages or its dependencies are not available as a package for that
      distribution. Using this flag allows the script to use pip as a last
      resort method. NOTE: This only works for functions which actually
      implement pip based installations.

https://github.com/saltstack/salt-bootstrap/blob/b1624cc14a91f21abab1da0fe1ef23d5a95d2ef1/bootstrap-salt.sh#L245

WTF is going on

There's multiple issues messing up provisioning.

#6029 : Bootstrapping Salt Fails Due to Salt-Boostrap Update

A salt-bootstrap script update makes it so that -P needs to be passed to the bootstrap script for PIP based installations to work.

#6011 : Regression: Salt provisioning fails when box is already provisioned

Vagrant no longer passing in -F -c <temporary config dir> to bootstrap options. Committed da67824#diff-c4526b51e4e1b1f3f6bff789181cce2bR107 11 days before v1.7.4
and fixed on ff941ef post v1.7.4.

You can see that the added conditional is written incorrectly as !@machine.config.vm.communicator == :winrm instead of @machine.config.vm.communicator != :winrm or !(@machine.config.vm.communicator == :winrm)

If salt is already installed, then configuration mode fails without temporary configuration directory specified through -c

if [ "$_CONFIG_ONLY" -eq $BS_TRUE ] && [ "$_TEMP_CONFIG_DIR" = "null" ]; then
    echoerror "In order to run the script in configuration only mode you also need to provide the configuration directory."
    exit 1
fi

https://github.com/saltstack/salt-bootstrap/blob/b1624cc14a91f21abab1da0fe1ef23d5a95d2ef1/bootstrap-salt.sh#L381

If salt isn't already installed, then bootstrap-salt.sh uses the temporary configuration directory to copy over configuration files. But since the temporary configuration directory isn't passed in, no configuration files are copied over.

#5973 : Salt provisioner fails to upload minion config

During highstate, vagrant tries to copy the minion config file to /etc/salt/. Committed 74d2206#diff-c4526b51e4e1b1f3f6bff789181cce2bR308 March 11, 2014. The reason was for base boxes with salt already installed, provisioning skipped installing salt and did not copy over minion configuration files (no longer true).

Wait March 11, 2014? Why did it take so long for permission errors to occur?!

vagrant@saltminion1:~$ ls -l /etc/salt/
total 60
-rw-r--r-- 1 vagrant vagrant 23480 Sep  2 19:23 minion
drwxr-xr-x 2 root    root     4096 Sep  2 19:13 minion.d
-rw-r--r-- 1 vagrant vagrant 25854 Jul  7 20:42 minion.dpkg-dist
drwx------ 3 root    root     4096 Sep  2 19:13 pki

So if bootstrap-salt.sh receives a temporary configuration directory, it copies over the files with the same permissions, and it is writeable by the vagrant user. But because of #6011, and there being no existing vagrant user writeable configuration files, #5973 is exposed as a bug.

#6073 : Stop trying to upload minion config to privileged directory
This (unmerged as of now) pull request stops unneeded extra copy to /etc/salt/, since provisioning with salt already installed, does copy over minion config files.

Summary

#6011 causes #5973 to be revealed. #6073 pull request is awaiting merge to fix #5973.
#6011 is fixed in ff941ef
#6029 is not fixed and no pull requests for it.

Bonus

Figuring out how to use git master took me all day to figure out, so I'll document it here. It could be a better option when a bug is fixed on master than waiting for the next release. I found something on Google about using VAGRANT_INSTALLER_EMBEDDED_DIR, but it ended up crashing with plugins and I couldn't figure out how to deal with it. Eventually I just tried installing the master as a gem into the Ruby environment embedded inside the installed vagrant.

Replace /opt/vagrant/ with your vagrant installation directory as necessary.

chown -R <user> /opt/vagrant/
cd
git clone https://github.com/mitchellh/vagrant.git
cd vagrant
GEM_HOME=/opt/vagrant/embedded/gems/ /opt/vagrant/embedded/bin/gem install bundler -v 1.10.5
GEM_HOME=/opt/vagrant/embedded/gems/ /opt/vagrant/embedded/gems/bin/bundle install
GEM_HOME=/opt/vagrant/embedded/gems/ /opt/vagrant/embedded/gems/bin/rake install

[johntron]

causes #5973 to be revealed. #6073 pull request is awaiting merge to fix #5973.
#6011 is fixed in ff941ef
#6029 is not fixed and no pull requests for it.

Exactly what I found as well.

I wasn't sure how #6029 should be fixed. If we include -P for all systems, some may not work due to special cases. The system packages (e.g. Distro packages from APT) may have required special build steps or patches, so installing from pip may not work. Conversely, a system package may not be available in the right version.

Maybe we should include -P by default, since this will be most likely to succeed, and let the user explicitly disable it with an option?

[tehmaspc]

@bsuh - LMAO - great post formatting.

[mitchellh]

Fixed by #6073

[dwickwire]

👍 Thanks!

[kevich]

Hooray! Thaks! Hope we will get new 1.8.0 release soon

[tehmaspc]

Thank You!

[obestwalter]

@mitchellh Wow - lot's of salt provisioner merges today- thanks a lot!

[AndreiPashkin]

@bsuh 👍

[Jetlum]

Got the same error then I just wrote the commands on cmd instead of git console and it worked.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.