Bisect tests failing

[mikebentley15]

Bug Report

Describe the problem
All of a sudden, the tst_bisect.py test is failing on one of my feature branches.

**********************************************************************
File "tst_bisect.py", line 187, in __main__
Failed example:
    print('\n'.join(
        sorted([' '.join(x.split()[4:]) for x in bisect_out
                if x.startswith('    Found differing symbol on line')])))
Expected:
    line 100 -- file1_func3_PROBLEM() (score 2.0)
    line 103 -- file3_func5_PROBLEM() (score 3.0)
    line 106 -- file4_all() (score 30.0)
    line 108 -- file1_func4_PROBLEM() (score 3.0)
    line 90 -- file2_func1_PROBLEM() (score 7.0)
    line 92 -- file1_func2_PROBLEM() (score 5.0)
    line 92 -- file3_func2_PROBLEM() (score 1.0)
    line 95 -- A::fileA_method1_PROBLEM() (score 2.0)
    line 96 -- real_problem_test(int, char**) (score 50.0)
Got:
    line 100 -- file1_func3_PROBLEM() (score 2.0)
    line 103 -- file3_func5_PROBLEM() (score 3.0)
    line 108 -- file1_func4_PROBLEM() (score 3.0)
    line 90 -- file2_func1_PROBLEM() (score 7.0)
    line 92 -- file1_func2_PROBLEM() (score 5.0)
    line 92 -- file3_func2_PROBLEM() (score 1.0)
    line 95 -- A::fileA_method1_PROBLEM() (score 2.0)
    line 96 -- real_problem_test(int, char**) (score 50.0)
**********************************************************************
File "tst_bisect.py", line 225, in __main__
Failed example:
    idx = bisect_out.index('  All differing symbols in tests/file4.cxx:')
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.5/doctest.py", line 1321, in __run
        compileflags, 1), test.globs)
      File "<doctest __main__[25]>", line 1, in <module>
        idx = bisect_out.index('  All differing symbols in tests/file4.cxx:')
    ValueError: '  All differing symbols in tests/file4.cxx:' is not in list
**********************************************************************
File "tst_bisect.py", line 226, in __main__
Failed example:
    print('\n'.join(sorted(bisect_out[idx+1:idx+2])))
Expected:
        line 106 -- file4_all() (score 30.0)
Got:
        line 103 -- file3_func5_PROBLEM() (score 3.0)
**********************************************************************
File "tst_bisect.py", line 228, in __main__
Failed example:
    bisect_out[idx+2].startswith(' ')
Expected:
    False
Got:
    True
**********************************************************************
File "tst_bisect.py", line 247, in __main__
Failed example:
    print('\n'.join(bisect_out[idx+1:])) # doctest:+ELLIPSIS
Expected:
      tests/BisectTest.cpp:96 ... -- real_problem_test(int, char**) (score 50.0)
      tests/file4.cxx:106 ... -- file4_all() (score 30.0)
      tests/file2.cpp:90 ... -- file2_func1_PROBLEM() (score 7.0)
      tests/file1.cpp:92 ... -- file1_func2_PROBLEM() (score 5.0)
      tests/file1.cpp:108 ... -- file1_func4_PROBLEM() (score 3.0)
      tests/file3.cpp:103 ... -- file3_func5_PROBLEM() (score 3.0)
      tests/A.cpp:95 ... -- A::fileA_method1_PROBLEM() (score 2.0)
      tests/file1.cpp:100 ... -- file1_func3_PROBLEM() (score 2.0)
      tests/file3.cpp:92 ... -- file3_func2_PROBLEM() (score 1.0)
Got:
      tests/BisectTest.cpp:96 _Z17real_problem_testiPPc -- real_problem_test(int, char**) (score 50.0)
      tests/file2.cpp:90 _Z19file2_func1_PROBLEMv -- file2_func1_PROBLEM() (score 7.0)
      tests/file1.cpp:92 _Z19file1_func2_PROBLEMv -- file1_func2_PROBLEM() (score 5.0)
      tests/file1.cpp:108 _Z19file1_func4_PROBLEMv -- file1_func4_PROBLEM() (score 3.0)
      tests/file3.cpp:103 _Z19file3_func5_PROBLEMv -- file3_func5_PROBLEM() (score 3.0)
      tests/A.cpp:95 _ZN1A21fileA_method1_PROBLEMEv -- A::fileA_method1_PROBLEM() (score 2.0)
      tests/file1.cpp:100 _Z19file1_func3_PROBLEMv -- file1_func3_PROBLEM() (score 2.0)
      tests/file3.cpp:92 _Z19file3_func2_PROBLEMv -- file3_func2_PROBLEM() (score 1.0)
**********************************************************************
1 items had failures:
   5 of  38 in __main__
***Test Failed*** 5 failures.

This test is missing file4_all() as a contributing symbol. The only reason that I can think of that this is failing now (consistently) and was not failing before is an update to a dependency. My guess is an update to pyelftools. This needs to be investigated.

I have recreated an environment on my developer system using Docker with the following Dockerfile:

FROM ubuntu:xenial
MAINTAINER mikebentley15@gmail.com

# Setup development environment
RUN apt-get update && \
    apt-get install -y \
      cmake \
      curl \
      g++ \
      gcc \
      gdb \
      git \
      htop \
      make \
      python3 \
      python3-pip \
      screen \
      sudo \
      tmux \
      tree \
      vim \
      vim-gtk \
      wget \
      && \
    rm -rf /var/lib/apt/lists/*

RUN useradd --create-home --shell /bin/bash --gid sudo dockuser && \
    echo 'dockuser:dockpass' | chpasswd

USER dockuser
WORKDIR /home/dockuser

# Setup my dotfiles
RUN mkdir git && \
    cd git && \
    git clone https://bitbucket.org/mikebentley15/configurations.git && \
    cd configurations && \
    make -f link-config.mk backup && \
    make -f link-config.mk all

RUN rmdir ~/.vim/bundle/Vundle.vim && \
    git clone https://github.com/VundleVim/Vundle.vim.git ~/.vim/bundle/Vundle.vim && \
    vim +PluginInstall +qall

COPY Dockerfile Dockerfile-ubuntu-dev

CMD ["/bin/bash"]

Suggested Fix
Probably to get it working with the updated version of pyelftools (if that is the problem)

Alternative approaches:
Move to another way of parsing the elf binaries.