Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tmpdir and subprocess fail in CI environment, work locally #2686

Closed
menzenski opened this issue Aug 15, 2017 · 4 comments
Closed

tmpdir and subprocess fail in CI environment, work locally #2686

menzenski opened this issue Aug 15, 2017 · 4 comments
Labels
type: question general question, might be closed after 2 weeks of inactivity

Comments

@menzenski
Copy link

I have some tests that direct output of a directory-creating command to tmpdir, then invoke a command from the Makefile located in the directory which was created.

These tests pass on my local machine but do not pass in my Travis CI environment.

The project uses Pipfile to manage dependencies.

My local machine is running Mac 10.11.6, with Python version 3.5.2 (from a pipenv virtual environment). The Travis CI build runs on Ubuntu Trusty 14.04 with Python 3.5.3.

Minimal Working Example

I have put together a minimal working example repo here: https://github.com/menzenski/pipenv-travis-test with a simplified test which nevertheless exhibits the same behavior: it creates a tempdir and then calls subprocess.check_call with the path to the tempdir as an argument. It passes on my local machine but fails (with the same error) in the Travis CI environment. The pytest version used is 3.0.7 .

Output on my local machine, which passes:

$ make test
pipenv run -- py.test tests -s -v
============================= test session starts ==============================
platform darwin -- Python 3.5.2, pytest-3.0.7, py-1.4.34, pluggy-0.4.0 -- /Users/matt/.local/share/virtualenvs/pipenv-travis-test-LpoC41Ey/bin/python3.5
cachedir: .cache
rootdir: /Users/matt/sandbox/pipenv-travis-test, inifile:
plugins: logging-2015.11.4
collected 1 items

tests/test_main.py::test_using_subprocess_check PASSED

=========================== 1 passed in 0.03 seconds ===========================

Output from the travis build, which fails:

Travis CI
There are no broadcasts

Blog
 
Status
 
Help
Matt Menzenski  Matt Menzenski avatar
menzenski / pipenv-travis-test
 build:failed
More options
Current
 
Branches
 
Build History
 
Pull Requests
 
Build #2
 
Job #2.1
 develop add a test that calls subprocess

 Commit 1862a2c
 Compare 0e18ab5..1862a2c
 Branch develop
Matt Menzenski avatar Matt Menzenski authored and committed

 #2.1 failed

 Ran for 1 min 16 sec
 11 minutes ago
 Restart job
 This job ran on our Trusty environment, which is gradually becoming our default Linux environment. Read all about this in our blog: Trusty as a default Linux is coming and take note that you can add dist: precise in your .travis.yml file to continue using Precise.

Job log
 
View config
 Remove log  Raw log
 worker_info
Worker information
hostname: i-031e0ec-production-2-worker-org-ec2.travisci.net:6325f27e-912c-4f21-bee2-4bf1519a3121
version: v2.9.3 https://github.com/travis-ci/worker/tree/a41c772c638071fbbdbc106f31a664c0532e0c36
instance: 9dd07ea:travis:python (via amqp)
startup: 730.541623ms
system_info
Build system information
Build language: python
Build group: stable
Build dist: trusty
Build id: 264573254
Job id: 264573255
travis-build version: f4d4af482
Build image provisioning date and time
Fri Jul  7 18:52:25 UTC 2017
Operating System Details
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.5 LTS
Release:	14.04
Codename:	trusty
Linux Version
4.4.0-83-generic
Cookbooks Version
15a6f94 https://github.com/travis-ci/travis-cookbooks/tree/15a6f94
git version
git version 2.13.0
bash version
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
gcc version
gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
docker version
Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 16:58:30 2017
 OS/Arch:      linux/amd64
clang version
clang version 3.5.0 (tags/RELEASE_350/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
jq version
jq-1.5
bats version
Bats 0.4.0
shellcheck version
0.4.5
shfmt version
v1.0.0
ccache version
ccache version 3.1.9
Copyright (C) 2002-2007 Andrew Tridgell
Copyright (C) 2009-2011 Joel Rosdahl
This program is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation; either version 3 of the License, or (at your option) any later
version.
cmake version
cmake version 3.2.2
CMake suite maintained and supported by Kitware (kitware.com/cmake).
heroku version
heroku-cli/6.12.5-17216bc (linux-x64) node-v8.1.3
imagemagick version
Version: ImageMagick 6.7.7-10 2017-05-26 Q16 http://www.imagemagick.org
md5deep version
4.2
mercurial version
Mercurial Distributed SCM (version 4.2.2)
(see https://mercurial-scm.org for more information)
Copyright (C) 2005-2017 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
mysql version
mysql  Ver 14.14 Distrib 5.6.33, for debian-linux-gnu (x86_64) using  EditLine wrapper
openssl version
OpenSSL 1.0.2j  26 Sep 2016
packer version
Packer v0.10.1
Your version of Packer is out of date! The latest version
is 1.0.2. You can update by downloading from www.packer.io
postgresql client version
psql (PostgreSQL) 9.6.3
ragel version
Ragel State Machine Compiler version 6.8 Feb 2013
Copyright (c) 2001-2009 by Adrian Thurston
subversion version
svn, version 1.8.8 (r1568071)
   compiled Aug 20 2015, 12:51:30 on x86_64-pc-linux-gnu
Copyright (C) 2013 The Apache Software Foundation.
This software consists of contributions made by many people;
see the NOTICE file for more information.
Subversion is open source software, see http://subversion.apache.org/
The following repository access (RA) modules are available:
* ra_svn : Module for accessing a repository using the svn network protocol.
  - with Cyrus SASL authentication
  - handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
  - handles 'file' scheme
* ra_serf : Module for accessing a repository via WebDAV protocol using serf.
  - using serf 1.3.3
  - handles 'http' scheme
  - handles 'https' scheme
sudo version
Sudo version 1.8.9p5
Configure options: --prefix=/usr -v --with-all-insults --with-pam --with-fqdn --with-logging=syslog --with-logfac=authpriv --with-env-editor --with-editor=/usr/bin/editor --with-timeout=15 --with-password-timeout=0 --with-passprompt=[sudo] password for %p:  --without-lecture --with-tty-tickets --disable-root-mailer --enable-admin-flag --with-sendmail=/usr/sbin/sendmail --with-timedir=/var/lib/sudo --mandir=/usr/share/man --libexecdir=/usr/lib/sudo --with-sssd --with-sssd-lib=/usr/lib/x86_64-linux-gnu --with-selinux
Sudoers policy plugin version 1.8.9p5
Sudoers file grammar version 43
Sudoers path: /etc/sudoers
Authentication methods: 'pam'
Syslog facility if syslog is being used for logging: authpriv
Syslog priority to use when user authenticates successfully: notice
Syslog priority to use when user authenticates unsuccessfully: alert
Send mail if the user is not in sudoers
Use a separate timestamp for each user/tty combo
Lecture user the first time they run sudo
Root may run sudo
Allow some information gathering to give useful error messages
Require fully-qualified hostnames in the sudoers file
Visudo will honor the EDITOR environment variable
Set the LOGNAME and USER environment variables
Length at which to wrap log file lines (0 for no wrap): 80
Authentication timestamp timeout: 15.0 minutes
Password prompt timeout: 0.0 minutes
Number of tries to enter a password: 3
Umask to use or 0777 to use user's: 022
Path to mail program: /usr/sbin/sendmail
Flags for mail program: -t
Address to send mail to: root
Subject line for mail messages: *** SECURITY information for %h ***
Incorrect password message: Sorry, try again.
Path to authentication timestamp dir: /var/lib/sudo
Default password prompt: [sudo] password for %p: 
Default user to run commands as: root
Value to override user's $PATH with: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Path to the editor for use by visudo: /usr/bin/editor
When to require a password for 'list' pseudocommand: any
When to require a password for 'verify' pseudocommand: all
File descriptors >= 3 will be closed before executing a command
Environment variables to check for sanity:
	TZ
	TERM
	LINGUAS
	LC_*
	LANGUAGE
	LANG
	COLORTERM
Environment variables to remove:
	RUBYOPT
	RUBYLIB
	PYTHONUSERBASE
	PYTHONINSPECT
	PYTHONPATH
	PYTHONHOME
	TMPPREFIX
	ZDOTDIR
	READNULLCMD
	NULLCMD
	FPATH
	PERL5DB
	PERL5OPT
	PERL5LIB
	PERLLIB
	PERLIO_DEBUG 
	JAVA_TOOL_OPTIONS
	SHELLOPTS
	GLOBIGNORE
	PS4
	BASH_ENV
	ENV
	TERMCAP
	TERMPATH
	TERMINFO_DIRS
	TERMINFO
	_RLD*
	LD_*
	PATH_LOCALE
	NLSPATH
	HOSTALIASES
	RES_OPTIONS
	LOCALDOMAIN
	CDPATH
	IFS
Environment variables to preserve:
	JAVA_HOME
	TRAVIS
	CI
	DEBIAN_FRONTEND
	XAUTHORIZATION
	XAUTHORITY
	PS2
	PS1
	PATH
	LS_COLORS
	KRB5CCNAME
	HOSTNAME
	HOME
	DISPLAY
	COLORS
Locale to use while parsing sudoers: C
Directory in which to store input/output logs: /var/log/sudo-io
File in which to store the input/output log: %{seq}
Add an entry to the utmp/utmpx file when allocating a pty
PAM service name to use
PAM service name to use for login shells
Create a new PAM session for the command to run in
Maximum I/O log sequence number: 0
Local IP address and netmask pairs:
	172.17.0.2/255.255.0.0
Sudoers I/O plugin version 1.8.9p5
gzip version
gzip 1.6
Copyright (C) 2007, 2010, 2011 Free Software Foundation, Inc.
Copyright (C) 1993 Jean-loup Gailly.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.
Written by Jean-loup Gailly.
zip version
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Currently maintained by E. Gordon.  Please send bug reports to
the authors using the web page at www.info-zip.org; see README for details.
Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip,
as of above date; see http://www.info-zip.org/ for other sites.
Compiled with gcc 4.8.2 for Unix (Linux ELF) on Oct 21 2013.
Zip special compilation options:
	USE_EF_UT_TIME       (store Universal Time)
	BZIP2_SUPPORT        (bzip2 library version 1.0.6, 6-Sept-2010)
	    bzip2 code and library copyright (c) Julian R Seward
	    (See the bzip2 license for terms of use)
	SYMLINK_SUPPORT      (symbolic links supported)
	LARGE_FILE_SUPPORT   (can read and write large files on file system)
	ZIP64_SUPPORT        (use Zip64 to store large files in archives)
	UNICODE_SUPPORT      (store and read UTF-8 Unicode paths)
	STORE_UNIX_UIDs_GIDs (store UID/GID sizes/values using new extra field)
	UIDGID_NOT_16BIT     (old Unix 16-bit UID/GID extra field not used)
	[encryption, version 2.91 of 05 Jan 2007] (modified for Zip 3)
Encryption notice:
	The encryption code of this program is not copyrighted and is
	put in the public domain.  It was originally written in Europe
	and, to the best of our knowledge, can be freely distributed
	in both source and object forms from any country, including
	the USA under License Exception TSU of the U.S. Export
	Administration Regulations (section 740.13(e)) of 6 June 2002.
Zip environment options:
             ZIP:  [none]
          ZIPOPT:  [none]
vim version
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Nov 24 2016 16:43:18)
Included patches: 1-52
Extra patches: 8.0.0056
Modified by pkg-vim-maintainers@lists.alioth.debian.org
Compiled by buildd@
Huge version without GUI.  Features included (+) or not (-):
+acl             +farsi           +mouse_netterm   +syntax
+arabic          +file_in_path    +mouse_sgr       +tag_binary
+autocmd         +find_in_path    -mouse_sysmouse  +tag_old_static
-balloon_eval    +float           +mouse_urxvt     -tag_any_white
-browse          +folding         +mouse_xterm     -tcl
++builtin_terms  -footer          +multi_byte      +terminfo
+byte_offset     +fork()          +multi_lang      +termresponse
+cindent         +gettext         -mzscheme        +textobjects
-clientserver    -hangul_input    +netbeans_intg   +title
-clipboard       +iconv           +path_extra      -toolbar
+cmdline_compl   +insert_expand   -perl            +user_commands
+cmdline_hist    +jumplist        +persistent_undo +vertsplit
+cmdline_info    +keymap          +postscript      +virtualedit
+comments        +langmap         +printer         +visual
+conceal         +libcall         +profile         +visualextra
+cryptv          +linebreak       +python          +viminfo
+cscope          +lispindent      -python3         +vreplace
+cursorbind      +listcmds        +quickfix        +wildignore
+cursorshape     +localmap        +reltime         +wildmenu
+dialog_con      -lua             +rightleft       +windows
+diff            +menu            -ruby            +writebackup
+digraphs        +mksession       +scrollbind      -X11
-dnd             +modify_fname    +signs           -xfontset
-ebcdic          +mouse           +smartindent     -xim
+emacs_tags      -mouseshape      -sniff           -xsmp
+eval            +mouse_dec       +startuptime     -xterm_clipboard
+ex_extra        +mouse_gpm       +statusline      -xterm_save
+extra_search    -mouse_jsbterm   -sun_workshop    -xpm
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H     -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1      
Linking: gcc   -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--as-needed -o vim        -lm -ltinfo -lnsl  -lselinux  -lacl -lattr -lgpm -ldl    -L/usr/lib/python2.7/config-x86_64-linux-gnu -lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions      
iptables version
iptables v1.4.21
curl version
curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3
wget version
GNU Wget 1.15 built on linux-gnu.
rsync version
rsync  version 3.1.0  protocol version 31
gimme version
v1.0.0
nvm version
0.33.0
perlbrew version
/home/travis/perl5/perlbrew/bin/perlbrew  - App::perlbrew/0.80
phpenv version
rbenv 1.1.1-2-g615f844
rvm version
rvm 1.29.2 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io/]
default ruby version
ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
CouchDB version
couchdb 1.6.1
ElasticSearch version
5.4.3
Installed Firefox version
firefox 50.0.2
MongoDB version
MongoDB 3.2.15
Pre-installed PostgreSQL versions
9.2.21
9.3.17
9.4.12
9.5.7
9.6.3
RabbitMQ Version
3.6.10
Redis version
redis-server 3.2.9
riak version
2.2.3
Pre-installed Go versions
1.7.4
ant version
Apache Ant(TM) version 1.9.3 compiled on April 8 2014
mvn version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00)
Maven home: /usr/local/maven-3.3.9
Java version: 1.8.0_131, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-oracle/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.4.0-83-generic", arch: "amd64", family: "unix"
gradle version
------------------------------------------------------------
Gradle 3.2.1
------------------------------------------------------------
Build time:   2016-11-22 15:19:54 UTC
Revision:     83b485b914fd4f335ad0e66af9d14aad458d2cc5
Groovy:       2.4.7
Ant:          Apache Ant(TM) version 1.9.6 compiled on June 29 2015
JVM:          1.8.0_131 (Oracle Corporation 25.131-b11)
OS:           Linux 4.4.0-83-generic amd64
lein version
WARNING: You're currently running as root; probably by accident.
Press control-C to abort or Enter to continue as root.
Set LEIN_ROOT to disable this warning.
Leiningen 2.7.1 on Java 1.8.0_131 Java HotSpot(TM) 64-Bit Server VM
Pre-installed Node.js versions
v4.8.3
v6.11.0
v6.9.4
v7.4
v7.4.0
phpenv versions
  system
  5.6
* 5.6.24 (set by /home/travis/.phpenv/version)
  7.0
  7.0.7
  hhvm
  hhvm-stable
composer --version
Composer version 1.2.0 2016-07-19 01:28:52
Pre-installed Ruby versions
ruby-2.2.7
ruby-2.3.4
ruby-2.4.1
W: http://dl.hhvm.com/ubuntu/dists/trusty/InRelease: Signature by key 36AEF64D0207E7EEE352D4875A16E7281BE7A449 uses weak digest algorithm (SHA1)
W: http://ppa.launchpad.net/couchdb/stable/ubuntu/dists/trusty/Release.gpg: Signature by key 15866BAFD9BCC4F3C1E0DFC7D69548E1C17EAB57 uses weak digest algorithm (SHA1)
Patching redis-server init script
git.checkout
0.46s$ git clone --depth=50 --branch=develop https://github.com/menzenski/pipenv-travis-test.git menzenski/pipenv-travis-test
Cloning into 'menzenski/pipenv-travis-test'...
remote: Counting objects: 16, done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 16 (delta 3), reused 15 (delta 2), pack-reused 0
Unpacking objects: 100% (16/16), done.
$ cd menzenski/pipenv-travis-test
$ git checkout -qf 1862a2c50cf646246e7a88a72b08ae54f7a7115c
Setting environment variables from .travis.yml
$ export PIPENV_VENV_IN_PROJECT=1
$ export PIPENV_IGNORE_VIRTUALENVS=1
0.04s$ source ~/virtualenv/python3.5/bin/activate
$ python --version
Python 3.5.3
$ pip --version
pip 9.0.1 from /home/travis/virtualenv/python3.5.3/lib/python3.5/site-packages (python 3.5)
install
47.47s$ make setup
pip install pipenv
Collecting pipenv
  Downloading pipenv-5.1.2.tar.gz (1.1MB)
    100% |████████████████████████████████| 1.1MB 1.2MB/s 
Collecting virtualenv (from pipenv)
  Downloading virtualenv-15.1.0-py2.py3-none-any.whl (1.8MB)
    100% |████████████████████████████████| 1.8MB 583kB/s 
Collecting pew>=0.1.26 (from pipenv)
  Downloading pew-0.1.26-py2.py3-none-any.whl
Requirement already satisfied: pip in /home/travis/virtualenv/python3.5.3/lib/python3.5/site-packages (from pipenv)
Collecting virtualenv-clone>=0.2.5 (from pew>=0.1.26->pipenv)
  Downloading virtualenv-clone-0.2.6.tar.gz
Collecting pythonz-bd>=1.10.2 (from pew>=0.1.26->pipenv)
  Downloading pythonz_bd-1.11.4-py3-none-any.whl (117kB)
    100% |████████████████████████████████| 122kB 6.1MB/s 
Requirement already satisfied: setuptools>=17.1 in /home/travis/virtualenv/python3.5.3/lib/python3.5/site-packages (from pew>=0.1.26->pipenv)
Collecting resumable-urlretrieve; python_version == "3.5" (from pythonz-bd>=1.10.2->pew>=0.1.26->pipenv)
  Downloading resumable_urlretrieve-0.1.5-py2.py3-none-any.whl
Collecting requests (from resumable-urlretrieve; python_version == "3.5"->pythonz-bd>=1.10.2->pew>=0.1.26->pipenv)
  Downloading requests-2.18.3-py2.py3-none-any.whl (88kB)
    100% |████████████████████████████████| 92kB 6.8MB/s 
Collecting certifi>=2017.4.17 (from requests->resumable-urlretrieve; python_version == "3.5"->pythonz-bd>=1.10.2->pew>=0.1.26->pipenv)
  Downloading certifi-2017.7.27.1-py2.py3-none-any.whl (349kB)
    100% |████████████████████████████████| 358kB 2.4MB/s 
Collecting idna<2.6,>=2.5 (from requests->resumable-urlretrieve; python_version == "3.5"->pythonz-bd>=1.10.2->pew>=0.1.26->pipenv)
  Downloading idna-2.5-py2.py3-none-any.whl (55kB)
    100% |████████████████████████████████| 61kB 6.7MB/s 
Collecting chardet<3.1.0,>=3.0.2 (from requests->resumable-urlretrieve; python_version == "3.5"->pythonz-bd>=1.10.2->pew>=0.1.26->pipenv)
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 3.5MB/s 
Collecting urllib3<1.23,>=1.21.1 (from requests->resumable-urlretrieve; python_version == "3.5"->pythonz-bd>=1.10.2->pew>=0.1.26->pipenv)
  Downloading urllib3-1.22-py2.py3-none-any.whl (132kB)
    100% |████████████████████████████████| 133kB 5.3MB/s 
Building wheels for collected packages: pipenv, virtualenv-clone
  Running setup.py bdist_wheel for pipenv ... done
  Stored in directory: /home/travis/.cache/pip/wheels/48/db/09/235c870e15bc0c59510f62eb69b7cd47ca3205ff80afa7dfc9
  Running setup.py bdist_wheel for virtualenv-clone ... done
  Stored in directory: /home/travis/.cache/pip/wheels/24/51/ef/93120d304d240b4b6c2066454250a1626e04f73d34417b956d
Successfully built pipenv virtualenv-clone
Installing collected packages: virtualenv, virtualenv-clone, certifi, idna, chardet, urllib3, requests, resumable-urlretrieve, pythonz-bd, pew, pipenv
Successfully installed certifi-2017.7.27.1 chardet-3.0.4 idna-2.5 pew-0.1.26 pipenv-5.1.2 pythonz-bd-1.11.4 requests-2.18.3 resumable-urlretrieve-0.1.5 urllib3-1.22 virtualenv-15.1.0 virtualenv-clone-0.2.6
pipenv install --dev --three
Creating a virtualenv for this project...
⠋Running virtualenv with interpreter /home/travis/virtualenv/python3.5.3/bin/python3
Using real prefix '/opt/python/3.5.3'
New python executable in /home/travis/build/menzenski/pipenv-travis-test/.venv/bin/python3
Also creating executable in /home/travis/build/menzenski/pipenv-travis-test/.venv/bin/python
Installing setuptools, pip, wheel...done.
Virtualenv location: /home/travis/build/menzenski/pipenv-travis-test/.venv
No package provided, installing all dependencies.
Pipfile found at /home/travis/build/menzenski/pipenv-travis-test/Pipfile. Considering this to be the project home.
Installing dependencies from Pipfile.lock...
[================================] 17/17 - 00:00:30
To activate this project's virtualenv, run the following:
 $ pipenv shell
1.39s$ make test
pipenv run -- py.test tests -s -v
============================= test session starts ==============================
platform linux -- Python 3.5.3, pytest-3.0.7, py-1.4.34, pluggy-0.4.0 -- /home/travis/build/menzenski/pipenv-travis-test/.venv/bin/python3
cachedir: .cache
rootdir: /home/travis/build/menzenski/pipenv-travis-test, inifile:
plugins: logging-2015.11.4
collected 1 items 
tests/test_main.py::test_using_subprocess_check FAILED
=================================== FAILURES ===================================
_________________________ test_using_subprocess_check __________________________
tmpdir = local('/tmp/pytest-of-travis/pytest-0/test_using_subprocess_check0')
    def test_using_subprocess_check(tmpdir):
        p = tmpdir.mkdir("sub").join("hello.txt")
        p.write("content")
        subprocess.check_call([
            'cd',
>           tmpdir.strpath,
            ])
tests/test_main.py:9: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/python/3.5.3/lib/python3.5/subprocess.py:266: in check_call
    retcode = call(*popenargs, **kwargs)
/opt/python/3.5.3/lib/python3.5/subprocess.py:247: in call
    with Popen(*popenargs, **kwargs) as p:
/opt/python/3.5.3/lib/python3.5/subprocess.py:676: in __init__
    restore_signals, start_new_session)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <subprocess.Popen object at 0x2b429ffbf240>
args = ['cd', '/tmp/pytest-of-travis/pytest-0/test_using_subprocess_check0']
executable = b'cd', preexec_fn = None, close_fds = True, pass_fds = ()
cwd = None, env = None, startupinfo = None, creationflags = 0, shell = False
p2cread = -1, p2cwrite = -1, c2pread = -1, c2pwrite = -1, errread = -1
errwrite = -1, restore_signals = True, start_new_session = False
    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       restore_signals, start_new_session):
        """Execute program (POSIX version)"""
    
        if isinstance(args, (str, bytes)):
            args = [args]
        else:
            args = list(args)
    
        if shell:
            args = ["/bin/sh", "-c"] + args
            if executable:
                args[0] = executable
    
        if executable is None:
            executable = args[0]
        orig_executable = executable
    
        # For transferring possible exec failure from child to parent.
        # Data format: "exception name:hex errno:description"
        # Pickle is not used; it is complex and involves memory allocation.
        errpipe_read, errpipe_write = os.pipe()
        # errpipe_write must not be in the standard io 0, 1, or 2 fd range.
        low_fds_to_close = []
        while errpipe_write < 3:
            low_fds_to_close.append(errpipe_write)
            errpipe_write = os.dup(errpipe_write)
        for low_fd in low_fds_to_close:
            os.close(low_fd)
        try:
            try:
                # We must avoid complex work that could involve
                # malloc or free in the child process to avoid
                # potential deadlocks, thus we do all this here.
                # and pass it to fork_exec()
    
                if env is not None:
                    env_list = [os.fsencode(k) + b'=' + os.fsencode(v)
                                for k, v in env.items()]
                else:
                    env_list = None  # Use execv instead of execve.
                executable = os.fsencode(executable)
                if os.path.dirname(executable):
                    executable_list = (executable,)
                else:
                    # This matches the behavior of os._execvpe().
                    executable_list = tuple(
                        os.path.join(os.fsencode(dir), executable)
                        for dir in os.get_exec_path(env))
                fds_to_keep = set(pass_fds)
                fds_to_keep.add(errpipe_write)
                self.pid = _posixsubprocess.fork_exec(
                        args, executable_list,
                        close_fds, sorted(fds_to_keep), cwd, env_list,
                        p2cread, p2cwrite, c2pread, c2pwrite,
                        errread, errwrite,
                        errpipe_read, errpipe_write,
                        restore_signals, start_new_session, preexec_fn)
                self._child_created = True
            finally:
                # be sure the FD is closed no matter what
                os.close(errpipe_write)
    
            # self._devnull is not always defined.
            devnull_fd = getattr(self, '_devnull', None)
            if p2cread != -1 and p2cwrite != -1 and p2cread != devnull_fd:
                os.close(p2cread)
            if c2pwrite != -1 and c2pread != -1 and c2pwrite != devnull_fd:
                os.close(c2pwrite)
            if errwrite != -1 and errread != -1 and errwrite != devnull_fd:
                os.close(errwrite)
            if devnull_fd is not None:
                os.close(devnull_fd)
            # Prevent a double close of these fds from __init__ on error.
            self._closed_child_pipe_fds = True
    
            # Wait for exec to fail or succeed; possibly raising an
            # exception (limited in size)
            errpipe_data = bytearray()
            while True:
                part = os.read(errpipe_read, 50000)
                errpipe_data += part
                if not part or len(errpipe_data) > 50000:
                    break
        finally:
            # be sure the FD is closed no matter what
            os.close(errpipe_read)
    
        if errpipe_data:
            try:
                os.waitpid(self.pid, 0)
            except ChildProcessError:
                pass
            try:
                exception_name, hex_errno, err_msg = (
                        errpipe_data.split(b':', 2))
            except ValueError:
                exception_name = b'SubprocessError'
                hex_errno = b'0'
                err_msg = (b'Bad exception data from child: ' +
                           repr(errpipe_data))
            child_exception_type = getattr(
                    builtins, exception_name.decode('ascii'),
                    SubprocessError)
            err_msg = err_msg.decode(errors="surrogatepass")
            if issubclass(child_exception_type, OSError) and hex_errno:
                errno_num = int(hex_errno, 16)
                child_exec_never_called = (err_msg == "noexec")
                if child_exec_never_called:
                    err_msg = ""
                if errno_num != 0:
                    err_msg = os.strerror(errno_num)
                    if errno_num == errno.ENOENT:
                        if child_exec_never_called:
                            # The error must be from chdir(cwd).
                            err_msg += ': ' + repr(cwd)
                        else:
                            err_msg += ': ' + repr(orig_executable)
>               raise child_exception_type(errno_num, err_msg)
E               FileNotFoundError: [Errno 2] No such file or directory: 'cd'
/opt/python/3.5.3/lib/python3.5/subprocess.py:1282: FileNotFoundError
=========================== 1 failed in 0.31 seconds ===========================
make: *** [test] Error 1
The command "make test" exited with 2.
Done. Your build exited with 1.
Top

Search all repositories
 
My Repositories
 
 menzenski/pipenv-travis-test  2
 Duration: 2 min 52 sec
 Finished: 11 minutes ago
 menzenski/cookiecutter-python-cli  4
 Duration: 2 min 58 sec
 Finished: 25 minutes ago
 menzenski/conphon  4
 Duration: 2 min 29 sec
 Finished: a day ago
 menzenski/trunc  10
 Duration: 2 min 42 sec
 Finished: 6 days ago
 menzenski/map-generator  8
 Duration: 1 min 2 sec
 Finished: 8 days ago
 menzenski/mtg-sdk-clojure  44
 Duration: 48 sec
 Finished: 3 months ago
@nicoddemus
Copy link
Member

Hi @menzenski,

Thanks for setting up a repository demonstrating the problem, we appreciate it.

Your call to cd feels fishy:

        subprocess.check_call([
            'cd',
           tmpdir.strpath,
           ])

Why do you need this call? There are better ways to change the working directory:

  1. Use the cwd argument of subprocess.check_call if you need to change the working directory just for that subprocess invocation.
  2. Use monkeypatch.chdir to change the working directory for the remaining of the test.

I'm short on time now to research the details, but on Windows cd is not a program but a shell command and to execute subprocess.check_call['cd', ...] one needs to pass shell=True.

@nicoddemus nicoddemus added the type: question general question, might be closed after 2 weeks of inactivity label Aug 15, 2017
@menzenski
Copy link
Author

menzenski commented Aug 16, 2017

Hi @nicoddemus , thank you for the comment.

I did a poor job constructing the MWE repo — cd wasn't the specific command I was actually testing, but the behavior was the same between cd and the actual command under test (which was cookiecutter).

Your tip about using monkeypatch.chdir did actually lead me down a path to solving my issue, though, so thank you! I'm closing this issue because I've got my build passing.

@RonnyPfannschmidt
Copy link
Member

@menzenski thanks for the feedback, if possible could you describe the problem/solution a bit more in detail
so the next person hitting it has a better day

@menzenski
Copy link
Author

@RonnyPfannschmidt sure.

I am working on a fork of https://github.com/nvie/cookiecutter-python-cli that uses pipenv to manage dependencies and virtual environments, and wanted to write a test which would:

  • run the cookiecutter command to generate a python package in a subdirectory
  • run the make setup command inside that subdirectory to install the output package's tests and set up its virtualenv.
  • run the make test command inside that subdirectory to assert that the package's test suite passed.

My first attempt at a test that did these three things looked like this:

import json
import os
import os.path
import subprocess


def test_cookiecuttering(tmpdir):
    # read the package name from the default cookiecutter config file
    root_dir = os.path.join(os.path.dirname(__file__), os.pardir)
    with open(os.path.join(root_dir, 'cookiecutter.json')) as config_file:
        config_data = json.load(config_file)
        repo_name = config_data['repo_name']
        package_name = config_data['package_name']
    assert package_name and isinstance(package_name, str)

    # generate the package with the cookiecutter CLI command
    subprocess.check_call([
        'cookiecutter', 'https://github.com/menzenski/cookiecutter-python-cli',
        '--checkout', 'pipfile', '--no-input', '--overwrite-if-exists',
        '--output-dir', tmpdir.strpath
    ])

    # the directory where the generated package is located
    repo_dir = tmpdir.join(repo_name)

    # run the setup command in the package dir to install dependencies
    subprocess.check_call(['make', '-C', repo_dir.strpath, 'setup'])

    # run the test command in the package dir
    subprocess.check_call(['make', '-C', repo_dir.strpath, 'test'])

That passed locally but failed in Travis CI with FileNotFoundError: [Errno 2] No such file or directory: 'cookiecutter'.

The following test passed both locally and in Travis CI:

import cookiecutter.main as cookiecutter
import json
import os
import os.path
import subprocess


def test_cookiecuttering(monkeypatch, tmpdir):
    # read the package name from the default cookiecutter config file
    root_dir = os.path.join(os.path.dirname(__file__), os.pardir)
    with open(os.path.join(root_dir, 'cookiecutter.json')) as config_file:
        config_data = json.load(config_file)
        repo_name = config_data['repo_name']
        package_name = config_data['package_name']
    assert package_name and isinstance(package_name, str)

    monkeypatch.chdir(root_dir)

    # generate the package with the cookiecutter function
    cookiecutter.cookiecutter(
        'https://github.com/menzenski/cookiecutter-python-cli',
        checkout='pipfile', no_input=True, overwrite_if_exists=True,
        default_config=True)

    # change working directory to the generated project's root directory
    tmpdir.join(repo_name)
    monkeypatch.chdir(repo_name)

    # run the setup command in the package dir to install dependencies
    subprocess.check_call(['make', 'setup'])

    # # run the test command in the package dir
    subprocess.check_call(['make', 'test'])

Part of the solution here was importing and calling cookiecutter in Python in the test instead of calling the CLI command with subprocess, but the other part of the solution was using monkeypatch to change directories rather than trying to use tmpdir or tmpdir.strpath or str(tmpdir) to direct a CLI command at a different directory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: question general question, might be closed after 2 weeks of inactivity
Projects
None yet
Development

No branches or pull requests

3 participants