flakey rust test?

[siggy]

master ci failing with tcp_waits_for_proxies_to_close, from https://travis-ci.org/runconduit/conduit/jobs/367436260#L1:

travis_fold:start:worker_info
�[0K�[33;1mWorker information�[0m
hostname: c7290313-a487-40cb-9417-7775174d2d8a@1.i-04a5ad9-production-2-worker-org-ec2.travisci.net
version: v3.6.0 https://github.com/travis-ci/worker/tree/170b2a0bb43234479fd1911ba9e4dbcc36dadfad
instance: 7f3b4be travisci/ci-garnet:packer-1512502276-986baf0 (via amqp)
startup: 543.840843ms
travis_fold:end:worker_info
�[0Kmode of ‘/usr/local/clang-5.0.0/bin’ changed from 0777 (rwxrwxrwx) to 0775 (rwxrwxr-x)
travis_fold:start:system_info
�[0K�[33;1mBuild system information�[0m
Build language: rust
Build group: stable
Build dist: trusty
Build id: 367436259
Job id: 367436260
Runtime kernel version: 4.14.12-041412-generic
travis-build version: e0f4abce4
�[34m�[1mBuild image provisioning date and time�[0m
Tue Dec  5 20:11:19 UTC 2017
�[34m�[1mOperating System Details�[0m
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.5 LTS
Release:	14.04
Codename:	trusty
�[34m�[1mCookbooks Version�[0m
7c2c6a6 https://github.com/travis-ci/travis-cookbooks/tree/7c2c6a6
�[34m�[1mgit version�[0m
git version 2.15.1
�[34m�[1mbash version�[0m
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
�[34m�[1mgcc version�[0m
gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

�[34m�[1mdocker version�[0m
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:39:28 2017
 OS/Arch:      linux/amd64
�[34m�[1mclang version�[0m
clang version 5.0.0 (tags/RELEASE_500/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/local/clang-5.0.0/bin
�[34m�[1mjq version�[0m
jq-1.5
�[34m�[1mbats version�[0m
Bats 0.4.0
�[34m�[1mshellcheck version�[0m
0.4.6
�[34m�[1mshfmt version�[0m
v2.0.0
�[34m�[1mccache version�[0m
ccache version 3.1.9

Copyright (C) 2002-2007 Andrew Tridgell
Copyright (C) 2009-2011 Joel Rosdahl

This program is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation; either version 3 of the License, or (at your option) any later
version.
�[34m�[1mcmake version�[0m
cmake version 3.9.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).
�[34m�[1mheroku version�[0m
heroku-cli/6.14.39-addc925 (linux-x64) node-v9.2.0
�[34m�[1mimagemagick version�[0m
Version: ImageMagick 6.7.7-10 2017-07-31 Q16 http://www.imagemagick.org
�[34m�[1mmd5deep version�[0m
4.2
�[34m�[1mmercurial version�[0m
Mercurial Distributed SCM (version 4.2.2)
(see https://mercurial-scm.org for more information)

Copyright (C) 2005-2017 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
�[34m�[1mmysql version�[0m
mysql  Ver 14.14 Distrib 5.6.33, for debian-linux-gnu (x86_64) using  EditLine wrapper
�[34m�[1mopenssl version�[0m
OpenSSL 1.0.1f 6 Jan 2014
�[34m�[1mpacker version�[0m
Packer v1.0.2

Your version of Packer is out of date! The latest version
is 1.1.2. You can update by downloading from www.packer.io
�[34m�[1mpostgresql client version�[0m
psql (PostgreSQL) 9.6.6
�[34m�[1mragel version�[0m
Ragel State Machine Compiler version 6.8 Feb 2013
Copyright (c) 2001-2009 by Adrian Thurston
�[34m�[1msubversion version�[0m
svn, version 1.8.8 (r1568071)
   compiled Aug 10 2017, 17:20:39 on x86_64-pc-linux-gnu

Copyright (C) 2013 The Apache Software Foundation.
This software consists of contributions made by many people;
see the NOTICE file for more information.
Subversion is open source software, see http://subversion.apache.org/

The following repository access (RA) modules are available:

* ra_svn : Module for accessing a repository using the svn network protocol.
  - with Cyrus SASL authentication
  - handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
  - handles 'file' scheme
* ra_serf : Module for accessing a repository via WebDAV protocol using serf.
  - using serf 1.3.3
  - handles 'http' scheme
  - handles 'https' scheme

�[34m�[1msudo version�[0m
Sudo version 1.8.9p5
Configure options: --prefix=/usr -v --with-all-insults --with-pam --with-fqdn --with-logging=syslog --with-logfac=authpriv --with-env-editor --with-editor=/usr/bin/editor --with-timeout=15 --with-password-timeout=0 --with-passprompt=[sudo] password for %p:  --without-lecture --with-tty-tickets --disable-root-mailer --enable-admin-flag --with-sendmail=/usr/sbin/sendmail --with-timedir=/var/lib/sudo --mandir=/usr/share/man --libexecdir=/usr/lib/sudo --with-sssd --with-sssd-lib=/usr/lib/x86_64-linux-gnu --with-selinux
Sudoers policy plugin version 1.8.9p5
Sudoers file grammar version 43

Sudoers path: /etc/sudoers
Authentication methods: 'pam'
Syslog facility if syslog is being used for logging: authpriv
Syslog priority to use when user authenticates successfully: notice
Syslog priority to use when user authenticates unsuccessfully: alert
Send mail if the user is not in sudoers
Use a separate timestamp for each user/tty combo
Lecture user the first time they run sudo
Root may run sudo
Allow some information gathering to give useful error messages
Require fully-qualified hostnames in the sudoers file
Visudo will honor the EDITOR environment variable
Set the LOGNAME and USER environment variables
Length at which to wrap log file lines (0 for no wrap): 80
Authentication timestamp timeout: 15.0 minutes
Password prompt timeout: 0.0 minutes
Number of tries to enter a password: 3
Umask to use or 0777 to use user's: 022
Path to mail program: /usr/sbin/sendmail
Flags for mail program: -t
Address to send mail to: root
Subject line for mail messages: *** SECURITY information for %h ***
Incorrect password message: Sorry, try again.
Path to authentication timestamp dir: /var/lib/sudo
Default password prompt: [sudo] password for %p: 
Default user to run commands as: root
Value to override user's $PATH with: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Path to the editor for use by visudo: /usr/bin/editor
When to require a password for 'list' pseudocommand: any
When to require a password for 'verify' pseudocommand: all
File descriptors >= 3 will be closed before executing a command
Environment variables to check for sanity:
	TZ
	TERM
	LINGUAS
	LC_*
	LANGUAGE
	LANG
	COLORTERM
Environment variables to remove:
	RUBYOPT
	RUBYLIB
	PYTHONUSERBASE
	PYTHONINSPECT
	PYTHONPATH
	PYTHONHOME
	TMPPREFIX
	ZDOTDIR
	READNULLCMD
	NULLCMD
	FPATH
	PERL5DB
	PERL5OPT
	PERL5LIB
	PERLLIB
	PERLIO_DEBUG 
	JAVA_TOOL_OPTIONS
	SHELLOPTS
	GLOBIGNORE
	PS4
	BASH_ENV
	ENV
	TERMCAP
	TERMPATH
	TERMINFO_DIRS
	TERMINFO
	_RLD*
	LD_*
	PATH_LOCALE
	NLSPATH
	HOSTALIASES
	RES_OPTIONS
	LOCALDOMAIN
	CDPATH
	IFS
Environment variables to preserve:
	JAVA_HOME
	TRAVIS
	CI
	DEBIAN_FRONTEND
	XAUTHORIZATION
	XAUTHORITY
	PS2
	PS1
	PATH
	LS_COLORS
	KRB5CCNAME
	HOSTNAME
	HOME
	DISPLAY
	COLORS
Locale to use while parsing sudoers: C
Directory in which to store input/output logs: /var/log/sudo-io
File in which to store the input/output log: %{seq}
Add an entry to the utmp/utmpx file when allocating a pty
PAM service name to use
PAM service name to use for login shells
Create a new PAM session for the command to run in
Maximum I/O log sequence number: 0

Local IP address and netmask pairs:
	172.17.0.2/255.255.0.0

Sudoers I/O plugin version 1.8.9p5
�[34m�[1mgzip version�[0m
gzip 1.6
Copyright (C) 2007, 2010, 2011 Free Software Foundation, Inc.
Copyright (C) 1993 Jean-loup Gailly.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.

Written by Jean-loup Gailly.
�[34m�[1mzip version�[0m
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Currently maintained by E. Gordon.  Please send bug reports to
the authors using the web page at www.info-zip.org; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip,
as of above date; see http://www.info-zip.org/ for other sites.

Compiled with gcc 4.8.2 for Unix (Linux ELF) on Oct 21 2013.

Zip special compilation options:
	USE_EF_UT_TIME       (store Universal Time)
	BZIP2_SUPPORT        (bzip2 library version 1.0.6, 6-Sept-2010)
	    bzip2 code and library copyright (c) Julian R Seward
	    (See the bzip2 license for terms of use)
	SYMLINK_SUPPORT      (symbolic links supported)
	LARGE_FILE_SUPPORT   (can read and write large files on file system)
	ZIP64_SUPPORT        (use Zip64 to store large files in archives)
	UNICODE_SUPPORT      (store and read UTF-8 Unicode paths)
	STORE_UNIX_UIDs_GIDs (store UID/GID sizes/values using new extra field)
	UIDGID_NOT_16BIT     (old Unix 16-bit UID/GID extra field not used)
	[encryption, version 2.91 of 05 Jan 2007] (modified for Zip 3)

Encryption notice:
	The encryption code of this program is not copyrighted and is
	put in the public domain.  It was originally written in Europe
	and, to the best of our knowledge, can be freely distributed
	in both source and object forms from any country, including
	the USA under License Exception TSU of the U.S. Export
	Administration Regulations (section 740.13(e)) of 6 June 2002.

Zip environment options:
             ZIP:  [none]
          ZIPOPT:  [none]
�[34m�[1mvim version�[0m
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Nov 24 2016 16:43:18)
Included patches: 1-52
Extra patches: 8.0.0056
Modified by pkg-vim-maintainers@lists.alioth.debian.org
Compiled by buildd@
Huge version without GUI.  Features included (+) or not (-):
+acl             +farsi           +mouse_netterm   +syntax
+arabic          +file_in_path    +mouse_sgr       +tag_binary
+autocmd         +find_in_path    -mouse_sysmouse  +tag_old_static
-balloon_eval    +float           +mouse_urxvt     -tag_any_white
-browse          +folding         +mouse_xterm     -tcl
++builtin_terms  -footer          +multi_byte      +terminfo
+byte_offset     +fork()          +multi_lang      +termresponse
+cindent         +gettext         -mzscheme        +textobjects
-clientserver    -hangul_input    +netbeans_intg   +title
-clipboard       +iconv           +path_extra      -toolbar
+cmdline_compl   +insert_expand   -perl            +user_commands
+cmdline_hist    +jumplist        +persistent_undo +vertsplit
+cmdline_info    +keymap          +postscript      +virtualedit
+comments        +langmap         +printer         +visual
+conceal         +libcall         +profile         +visualextra
+cryptv          +linebreak       +python          +viminfo
+cscope          +lispindent      -python3         +vreplace
+cursorbind      +listcmds        +quickfix        +wildignore
+cursorshape     +localmap        +reltime         +wildmenu
+dialog_con      -lua             +rightleft       +windows
+diff            +menu            -ruby            +writebackup
+digraphs        +mksession       +scrollbind      -X11
-dnd             +modify_fname    +signs           -xfontset
-ebcdic          +mouse           +smartindent     -xim
+emacs_tags      -mouseshape      -sniff           -xsmp
+eval            +mouse_dec       +startuptime     -xterm_clipboard
+ex_extra        +mouse_gpm       +statusline      -xterm_save
+extra_search    -mouse_jsbterm   -sun_workshop    -xpm
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H     -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1      
Linking: gcc   -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--as-needed -o vim        -lm -ltinfo -lnsl  -lselinux  -lacl -lattr -lgpm -ldl    -L/usr/lib/python2.7/config-x86_64-linux-gnu -lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions      
�[34m�[1miptables version�[0m
iptables v1.4.21
�[34m�[1mcurl version�[0m
curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3
�[34m�[1mwget version�[0m
GNU Wget 1.15 built on linux-gnu.
�[34m�[1mrsync version�[0m
rsync  version 3.1.0  protocol version 31
�[34m�[1mgimme version�[0m
v1.2.0
�[34m�[1mnvm version�[0m
0.33.6
�[34m�[1mperlbrew version�[0m
/home/travis/perl5/perlbrew/bin/perlbrew  - App::perlbrew/0.80
�[34m�[1mphpenv version�[0m
rbenv 1.1.1-25-g6aa70b6
�[34m�[1mrvm version�[0m
rvm 1.29.3 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]
�[34m�[1mdefault ruby version�[0m
ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
�[34m�[1mCouchDB version�[0m
couchdb 1.6.1
�[34m�[1mElasticSearch version�[0m
5.5.0
�[34m�[1mInstalled Firefox version�[0m
firefox 56.0.2
�[34m�[1mMongoDB version�[0m
MongoDB 3.4.10
�[34m�[1mPhantomJS version�[0m
2.1.1
�[34m�[1mPre-installed PostgreSQL versions�[0m
9.2.24
9.3.20
9.4.15
9.5.10
9.6.6
�[34m�[1mRabbitMQ Version�[0m
3.6.14
�[34m�[1mRedis version�[0m
redis-server 4.0.6
�[34m�[1mriak version�[0m
2.2.3
�[34m�[1mPre-installed Go versions�[0m
1.7.4
�[34m�[1mant version�[0m
Apache Ant(TM) version 1.9.3 compiled on April 8 2014
�[34m�[1mmvn version�[0m
Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z)
Maven home: /usr/local/maven-3.5.2
Java version: 1.8.0_151, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-oracle/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.4.0-101-generic", arch: "amd64", family: "unix"
�[34m�[1mgradle version�[0m

------------------------------------------------------------
Gradle 4.0.1
------------------------------------------------------------

Build time:   2017-07-07 14:02:41 UTC
Revision:     38e5dc0f772daecca1d2681885d3d85414eb6826

Groovy:       2.4.11
Ant:          Apache Ant(TM) version 1.9.6 compiled on June 29 2015
JVM:          1.8.0_151 (Oracle Corporation 25.151-b12)
OS:           Linux 4.4.0-101-generic amd64

�[34m�[1mlein version�[0m
Leiningen 2.8.1 on Java 1.8.0_151 Java HotSpot(TM) 64-Bit Server VM
�[34m�[1mPre-installed Node.js versions�[0m
v4.8.6
v6.12.0
v6.12.1
v8.9
v8.9.1
�[34m�[1mphpenv versions�[0m
  system
  5.6
* 5.6.32 (set by /home/travis/.phpenv/version)
  7.0
  7.0.25
  7.1
  7.1.11
  hhvm
  hhvm-stable
�[34m�[1mcomposer --version�[0m
Composer version 1.5.2 2017-09-11 16:59:25
�[34m�[1mPre-installed Ruby versions�[0m
ruby-2.2.7
ruby-2.3.4
ruby-2.4.1
travis_fold:end:system_info
�[0K
removed ‘/etc/apt/sources.list.d/basho_riak.list’
�[32;1mNetwork availability confirmed.�[0m
127.0.0.1	localhost
::1	 ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
172.17.0.4	travis-job-runconduit-conduit-367436260.travisci.net travis-job-runconduit-conduit-367436260
W: http://ppa.launchpad.net/couchdb/stable/ubuntu/dists/trusty/Release.gpg: Signature by key 15866BAFD9BCC4F3C1E0DFC7D69548E1C17EAB57 uses weak digest algorithm (SHA1)
travis_fold:start:git.checkout
�[0Ktravis_time:start:10e39da4
�[0K$ git clone --depth=50 --branch=master https://github.com/runconduit/conduit.git runconduit/conduit
Cloning into 'runconduit/conduit'...

travis_time:end:10e39da4:start=1523925457700769720,finish=1523925458393064016,duration=692294296
�[0K$ cd runconduit/conduit
$ git checkout -qf 727521f914b4d04cd02ab57f25f3df1750d34bfa
travis_fold:end:git.checkout
�[0K
�[33;1mSetting environment variables from repository settings�[0m
$ export GCP_PROJECT=runconduit
$ export encrypted_6af64675f81c_key=[secure]
$ export encrypted_6af64675f81c_iv=[secure]
$ export GKE_CLUSTER=[secure]
$ export GCP_ZONE=[secure]

travis_fold:start:cache.1
�[0KSetting up build cache
$ export CASHER_DIR=$HOME/.casher
travis_time:start:2fd142fd
�[0K$ Installing caching utilities

travis_time:end:2fd142fd:start=1523925468090310695,finish=1523925468122787278,duration=32476583
�[0Ktravis_time:start:0bad3a04
�[0K
travis_time:end:0bad3a04:start=1523925468131337354,finish=1523925468137642561,duration=6305207
�[0Ktravis_time:start:0170550c
�[0K�[32;1mattempting to download cache archive�[0m
�[32;1mfetching master/cache-linux-trusty-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855--cargo-stable.tgz�[0m
�[32;1mfound cache�[0m

travis_time:end:0170550c:start=1523925468144579069,finish=1523925492297224534,duration=24152645465
�[0Ktravis_fold:end:cache.1
�[0Ktravis_fold:start:cache.cargo
�[0Ktravis_time:start:30c28e4c
�[0K
travis_time:end:30c28e4c:start=1523925492305373080,finish=1523925492311994385,duration=6621305
�[0Ktravis_time:start:046ef4b2
�[0K�[32;1madding /home/travis/.cargo to cache�[0m
�[32;1mcreating directory /home/travis/.cargo�[0m
�[32;1madding /home/travis/build/runconduit/conduit/target to cache�[0m
�[32;1mcreating directory /home/travis/build/runconduit/conduit/target�[0m

travis_time:end:046ef4b2:start=1523925492319658503,finish=1523925549146473857,duration=56826815354
�[0Ktravis_fold:end:cache.cargo
�[0Ktravis_fold:start:rustup-install
�[0K�[33;1mInstalling Rust�[0m
travis_time:start:0804470d
�[0K$ curl -sSf https://build.travis-ci.org/files/rustup-init.sh | sh -s -- --default-toolchain=$TRAVIS_RUST_VERSION -y
info: downloading installer
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: latest update on 2018-03-29, rust version 1.25.0 (84203cac6 2018-03-25)
info: downloading component 'rustc'
info: downloading component 'rust-std'
info: downloading component 'cargo'
info: downloading component 'rust-docs'
info: installing component 'rustc'
info: installing component 'rust-std'
info: installing component 'cargo'
info: installing component 'rust-docs'
info: default toolchain set to 'stable'

  stable installed - rustc 1.25.0 (84203cac6 2018-03-25)


travis_time:end:0804470d:start=1523925549159082663,finish=1523925572212184792,duration=23053102129
�[0K$ export PATH=$HOME/.cargo/bin:$PATH
travis_fold:end:rustup-install
�[0K$ rustc --version
rustc 1.25.0 (84203cac6 2018-03-25)
$ rustup --version
rustup 1.11.0 (e751ff9f8 2018-02-13)
$ cargo --version
cargo 0.26.0 (41480f5cc 2018-02-26)

travis_time:start:0922c30a
�[0K$ (cd proxy && cargo test --locked --no-default-features)
   Compiling convert v0.3.0 (file:///home/travis/build/runconduit/conduit/proxy/convert)
   Compiling futures-mpsc-lossy v0.3.0 (file:///home/travis/build/runconduit/conduit/proxy/futures-mpsc-lossy)
   Compiling conduit-proxy-router v0.3.0 (file:///home/travis/build/runconduit/conduit/proxy/router)
   Compiling conduit-proxy-controller-grpc v0.3.0 (file:///home/travis/build/runconduit/conduit/proxy/controller-grpc)
   Compiling conduit-proxy v0.3.0 (file:///home/travis/build/runconduit/conduit/proxy)
    Finished dev [unoptimized + debuginfo] target(s) in 55.19 secs
     Running /home/travis/build/runconduit/conduit/target/debug/deps/conduit_proxy-7dbcf7583401ab30

running 24 tests
test ctx::transport::tests::same_addr_ip6_compat_ipv4 ... ok
test ctx::transport::tests::same_addr_ip6_mapped_ipv4 ... ok
test ctx::transport::tests::same_addr_ipv4 ... ok
test telemetry::tap::match_::tests::tcp_matches ... ok
test ctx::transport::tests::same_addr_ipv6 ... ok
test dns::tests::test_dns_name_parsing ... ok
test drain::tests::watch ... ok
test drain::tests::watch_clones ... ok
test transparency::tcp::tests::duplex_doesnt_hang_when_one_half_finishes ... ok
test telemetry::metrics::prometheus::labels::test::label_updates ... ok
test inbound::tests::recognize_default_no_ctx ... ok
test inbound::tests::recognize_default_no_loop ... ok
test telemetry::metrics::prometheus::labels::test::no_labels ... ok
test telemetry::metrics::prometheus::labels::test::one_label ... ok
test control::fully_qualified_authority::tests::test_normalized_authority ... ok
test telemetry::tap::match_::tests::tcp_from_proto ... ok
test control::cache::tests::update_union_reset_on_next_modification ... ok
test control::cache::tests::update_union_fires_events ... ok
test control::cache::tests::clear_reset_on_next_modification ... ok
test control::cache::tests::remove_reset_on_next_modification ... ok
test control::cache::tests::clear_fires_removal_event ... ok
test inbound::tests::recognize_default_no_orig_dst ... ok
test inbound::tests::recognize_orig_dst ... ok
test telemetry::tap::match_::tests::http_from_proto ... ok

test result: ok. 24 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running /home/travis/build/runconduit/conduit/target/debug/deps/conduit_proxy-7f27b9d5c9413277

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running /home/travis/build/runconduit/conduit/target/debug/deps/discovery-c79f040c87490fe7

running 23 tests
test http1::absolute_uris::outbound_destinations_reset_on_reconnect_followed_by_add_none ... ignored
test http1::absolute_uris::outbound_destinations_reset_on_reconnect_followed_by_no_endpoints_exists ... ignored
test http1::absolute_uris::outbound_destinations_reset_on_reconnect_followed_by_remove_none ... ignored
test http1::absolute_uris::outbound_times_out ... ignored
test http1::outbound_destinations_reset_on_reconnect_followed_by_add_none ... ignored
test http1::outbound_destinations_reset_on_reconnect_followed_by_no_endpoints_exists ... ignored
test http1::outbound_destinations_reset_on_reconnect_followed_by_remove_none ... ignored
test http1::outbound_times_out ... ignored
test http2::outbound_destinations_reset_on_reconnect_followed_by_add_none ... ignored
test http2::outbound_destinations_reset_on_reconnect_followed_by_no_endpoints_exists ... ignored
test http2::outbound_destinations_reset_on_reconnect_followed_by_remove_none ... ignored
test http2::outbound_times_out ... ignored
test support::assert_eventually ... ok
server h1 error: invalid HTTP version specified
ERROR 2018-04-17T00:40:28Z: conduit_proxy: turning Error caused by underlying HTTP/2 error: protocol error: frame with invalid size into 500
test http1::outbound_asks_controller_api ... ok
test http2::outbound_asks_controller_api ... ok
test http1::absolute_uris::outbound_asks_controller_without_orig_dst ... ok
test http1::absolute_uris::outbound_reconnects_if_controller_stream_ends ... ok
test http1::outbound_reconnects_if_controller_stream_ends ... ok
test outbound_updates_newer_services ... ok
test http2::outbound_reconnects_if_controller_stream_ends ... ok
test http2::outbound_asks_controller_without_orig_dst ... ok
test http1::absolute_uris::outbound_asks_controller_api ... ok
test http1::outbound_asks_controller_without_orig_dst ... ok

test result: ok. 11 passed; 0 failed; 12 ignored; 0 measured; 0 filtered out

     Running /home/travis/build/runconduit/conduit/target/debug/deps/shutdown-c1b927fbd5f78a47

running 5 tests
thread 'support server' panicked at 'assertion failed: `(left == right)`
  left: `[]`,
 right: `[99, 117, 115, 116, 111, 109, 32, 116, 99, 112, 32, 104, 101, 108, 108, 111]`', proxy/tests/shutdown.rs:139:21
note: Run with `RUST_BACKTRACE=1` for a backtrace.
test tcp_waits_for_proxies_to_close ... FAILED
test support::assert_eventually ... ok
test h2_goaways_connections ... ok
test http1_closes_idle_connections ... ok
test h2_exercise_goaways_connections ... ok

failures:

---- tcp_waits_for_proxies_to_close stdout ----
	thread 'tcp_waits_for_proxies_to_close' panicked at 'read: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }', libcore/result.rs:945:5


failures:
    tcp_waits_for_proxies_to_close

test result: FAILED. 4 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--test shutdown'

travis_time:end:0922c30a:start=1523925572285180949,finish=1523925628997584875,duration=56712403926
�[0K
�[31;1mThe command "(cd proxy && cargo test --locked --no-default-features)" exited with 101.�[0m
travis_fold:start:cache.2
�[0Kstore build cache
travis_time:start:03fb1ebf
�[0K
travis_time:end:03fb1ebf:start=1523925629006748191,finish=1523925629013762879,duration=7014688
�[0Ktravis_time:start:1994de46
�[0K�[32;1mchange detected (content changed, file is created, or file is deleted):
/home/travis/build/runconduit/conduit/target/debug/conduit_proxy-7dbcf7583401ab30.d
/home/travis/build/runconduit/conduit/target/debug/conduit_proxy-7f27b9d5c9413277.d
/home/travis/build/runconduit/conduit/target/debug/discovery-c79f040c87490fe7.d
/home/travis/build/runconduit/conduit/target/debug/.fingerprint/conduit-proxy-273250fc6d2eaea0/test-integration-test-transparency-273250fc6d2eaea0
/home/travis/build/runconduit/conduit/target/debug/.fingerprint/conduit-proxy-273250fc6d2eaea0/test-integration-test-transparency-273250fc6d2eaea0.json
/home/travis/build/runconduit/conduit/target/debug/.fingerprint/conduit-proxy-470063197a7ade50/lib-conduit_proxy-470063197a7ade50
/home/travis/build/runconduit/conduit/target/debug/.fingerprint/conduit-proxy-470063197a7ade50/lib-conduit_proxy-470063197a7ade50.json
/home/travis/build/runconduit/conduit/target/debug/.fingerprint/conduit-proxy-7dbcf7583401ab30/test-lib-conduit_proxy-7dbcf7583401ab30
/home/travis/build/runconduit/conduit/target/debug/.fi
�[0m
�[32;1m...
�[0m
�[32;1mchanges detected, packing new archive�[0m
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
�[32;1muploading archive�[0m
running `casher push` took longer than 180 seconds and has been aborted.
You can extend the timeout with `cache.timeout`. See https://docs.travis-ci.com/user/caching/#Setting-the-timeout for details

travis_time:end:1994de46:start=1523925629021599611,finish=1523925811156426371,duration=182134826760
�[0Ktravis_fold:end:cache.2
�[0K
Done. Your build exited with 1.

[olix0r]

probably. introduced in 71370ac. @seanmonstar, do you mind taking a look into this?

[hawkw]

IIRC this test was "fixed" once already in 7f54b52...evidentially there's more going on here.

[seanmonstar]

I've been looking at this, and so far cannot determine why this error is happening. I don't know if it's flakiness in the test, but I'm a little worried that it's actually flakiness in our TCP proxy... 😢

The logs here, combined with my commentary on how the code works, shows this:

1. The unit test TCP client connects to the proxy.
2. The proxy connects the unit test TCP server.
3. The test server waits for a read on the connected socket, and when it returns, gets a read of 0 bytes. It then panics as assert_eq!(&read[..n], expected_msg) is clearly not true ("" != "blah blah").
4. Since it panics, the connection closes abruptly. The proxy sees the closure and drops the TCP pipe.
5. The unit test TCP client was waiting on a read to get back expected_msg2, and then it sees the connection aborted, and the unit test just completely fails.

The read of 0 bytes in (3) worries me. That means the proxy closed (or SHUT_WR at least) the connection to the TCP server. I don't know why it would have done so if both sides hadn't closed yet.

Reproducing this is not easy, so trying to capture some logs is 💀 .

[olix0r]

Just hit this again in CI...

---- inbound_tcp stdout ----
	thread 'inbound_tcp' panicked at 'read: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }', libcore/result.rs:945:5
failures:
    inbound_tcp
test result: FAILED. 20 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
error: test failed, to rerun pass '--test transparency'

perhaps the CI networking stack is... very unreliable?

[olix0r]

similar but different

---- tcp_waits_for_proxies_to_close stdout ----
	thread 'tcp_waits_for_proxies_to_close' panicked at 'read: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }', libcore/result.rs:945:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.

[olix0r]

https://gist.github.com/olix0r/7fb373db7d7825ad4545daaf1426e4ec

[olix0r]

---- inbound_tcp stdout ----
	tcp server (addr=127.0.0.1:40015): running
proxy running; control=127.0.0.1:45837, inbound=127.0.0.1:45659 (SO_ORIGINAL_DST=127.0.0.1:40015), outbound=127.0.0.1:40987, metrics=127.0.0.1:40681
tcp client (addr=127.0.0.1:45659) thread running
tcp client (addr=127.0.0.1:45659): connected
tcp client (addr=127.0.0.1:45659): write
tcp client (addr=127.0.0.1:45659): read
thread 'inbound_tcp' panicked at 'TcpConn(addr=127.0.0.1:45659) read() error: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }', proxy/tests/support/tcp.rs:109:17
server Listening dropped; addr=127.0.0.1:40015
failures:
    inbound_tcp

[hawkw]

Just saw inbound_tcp flake out on Travis again. Test output looks like it's pretty much the same as what @olix0r posted above but I figured I'd post it here just in case.

test output:

     Running /home/travis/build/runconduit/conduit/target/debug/deps/transparency-d7634b2c8bfe1ba2
running 21 tests
test support::assert_eventually ... ok
thread 'support tcp server (addr=127.0.0.1:41247)' panicked at 'assertion failed: `(left == right)`
  left: `[]`,
 right: `[99, 117, 115, 116, 111, 109, 32, 116, 99, 112, 32, 104, 101, 108, 108, 111]`', proxy/tests/transparency.rs:199:13
test inbound_tcp ... FAILED
test http1_removes_connection_headers ... ok
test http1_connect_not_supported ... ok
test outbound_tcp ... ok
test http1_head_responses ... ok
test inbound_http1 ... ok
test tcp_connections_close_if_client_closes ... ok
test http10_without_host ... ok
test http10_with_host ... ok
controller error: Protocol(Error { kind: Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }) })
test tcp_with_no_orig_dst ... ok
ERROR 2018-05-02T18:20:47Z: conduit_proxy: turning Error caused by underlying HTTP/2 error: protocol error: unexpected internal error encountered into 500
test http11_upgrade_not_supported ... ok
test http11_absolute_uri_differs_from_host ... ok
test outbound_http1 ... ok
test http1_response_end_of_file ... ok
test http1_requests_without_body_doesnt_add_transfer_encoding ... ok
test http1_bodyless_responses ... ok
test tcp_server_first ... ok
test http1_one_connection_per_host ... ok
test http1_content_length_zero_is_preserved ... ok
test http1_requests_without_host_have_unique_connections ... ok
failures:
---- inbound_tcp stdout ----
	tcp server (addr=127.0.0.1:41247): running
proxy running; control=127.0.0.1:41409, inbound=127.0.0.1:33227 (SO_ORIGINAL_DST=127.0.0.1:41247), outbound=127.0.0.1:37519, metrics=127.0.0.1:39855
tcp client (addr=127.0.0.1:33227) thread running
tcp client (addr=127.0.0.1:33227): connected
tcp client (addr=127.0.0.1:33227): write
tcp client (addr=127.0.0.1:33227): read
thread 'inbound_tcp' panicked at 'TcpConn(addr=127.0.0.1:33227) read() error: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }', proxy/tests/support/tcp.rs:109:17
server Listening dropped; addr=127.0.0.1:41247
failures:
    inbound_tcp
test result: FAILED. 20 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

[hawkw]

Just saw a failure of tcp_connections_close_if_client_closes which also seems spurious (on a build of PR #865):

test output:

failures:
---- tcp_connections_close_if_client_closes stdout ----
	tcp server (addr=127.0.0.1:41991): running
proxy running; control=127.0.0.1:35368, inbound=127.0.0.1:33577 (SO_ORIGINAL_DST=127.0.0.1:41991), outbound=127.0.0.1:37026, metrics=127.0.0.1:42459
tcp client (addr=127.0.0.1:33577) thread running
tcp client (addr=127.0.0.1:33577): connected
tcp client (addr=127.0.0.1:33577): write
tcp client (addr=127.0.0.1:33577): read
thread 'tcp_connections_close_if_client_closes' panicked at 'assertion failed: `(left == right)`
  left: `[]`,
 right: `[99, 117, 115, 116, 111, 109, 32, 116, 99, 112, 32, 98, 121, 101]`', proxy/tests/transparency.rs:314:5
server Listening dropped; addr=127.0.0.1:41991
failures:
    tcp_connections_close_if_client_closes
test result: FAILED. 20 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

[seanmonstar]

I have been pondering setting our Rust test concurrency to 1... The suite would take slightly longer to complete, but we don't have that many tests...

[siggy]

Closing due to inactivity.