Debugger Heisenbug where integer comparison appears to fail within a specific context

[dddvision]

Environment data

• VS Code version:
  Version: 1.47.2
  Commit: 17299e413d5590b14ab0340ea477cdd86ff13daf
  Date: 2020-07-15T18:22:15.161Z
  Electron: 7.3.2
  Chrome: 78.0.3904.130
  Node.js: 12.8.1
  V8: 7.8.279.23-electron.0
  OS: Linux x64 5.4.0-42-generic snap

• Extension version (available under the Extensions sidebar):
  ms-python.python v2020.7.96456

• OS and version:
  LSB Version: core-9.20170808ubuntu1-noarch:security-9.20170808ubuntu1-noarch
  Distributor ID: Ubuntu
  Description: Ubuntu 18.04.4 LTS
  Release: 18.04
  Codename: bionic

• Python version (& distribution if applicable, e.g. Anaconda):
  Anaconda Python 3.7.7

• Type of virtual environment used (N/A | venv | virtualenv | conda | ...):
  Python 3.7.7 64-bit ('base':conda)

• Relevant/affected Python packages and their versions:
  jsonpickle 1.4.1
  numpy 1.18.1
  SQLAlchemy 1.3.17

• Relevant/affected Python-related VS Code extensions and their versions:
  ms-python.python v2020.7.96456

• Value of the python.languageServer setting:
  Microsoft

Expected behaviour

When debugging the attached file and stepping into main on line 33 with no arguments, the value of len(sys.argv) is 1, so the comparison N>=2 on line 35 should be False and the debugger should step to the else statement on line 39.

Actual behaviour

The debugger steps to line 36 as if N>=2 were True. It also incorrectly steps to line 42 after comparing N>=3.

Steps to reproduce:

Debug the attached file and step into main on line 33 with no arguments, then see if the debugger steps to line 36 or 39. All settings are default with a fresh install of vscode and the python extension.

I have spent many hours trying to isolate the minimal code required to demonstrate this bug. As far as I can tell, every line of uncommented code in this file is required to demonstrate the problem. Furthermore, adding print statements before or after the bug makes it disappear.

If further evidence of this bug is needed, then please ask and I will provide a .gif.

Logs

Nothing appears in the Output panel. In the terminal, the following appears:
cd /home/username ; env /home/username/anaconda3/bin/python /home/username/.vscode/extensions/ms-python.python-2020.7.96456/pythonFiles/lib/python/debugpy/launcher 39939 -- /home/username/vscode_debuger_bug.py
1

Code

import os
import sys
import time
import random
import jsonpickle
import numpy as np
import sqlalchemy as sa
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker

Base = declarative_base()

def MyFunction():
	pass

class MyClassA(Base):
	__tablename__ = 'tableA'
	n = sa.Column(sa.BigInteger, primary_key=True)

class MyClassB(Base):
	__tablename__ = 'tableB'
	n = sa.Column(sa.BigInteger, primary_key=True)

class MyClassC(object):
	def __init__(self):
		pass

class MyClassD(object):
	def __init__(self):
		pass

if __name__ == "__main__":
	N = len(sys.argv)  # set breakpoint here and pass no input arguments

	if N >= 2:
		arg1 = sys.argv[1]  # the debugger gets here even though N=1
		print(arg1)
	else:
		arg1 = 'MyString'

	if N >= 3:
		arg2 = int(sys.argv[2])  # the debugger gets here even though N=1
	else:
		arg2 = int(0)

	print(N)  # still N=1
	# print(arg1)  # if you print this then it changes the debugger behavior above

[int19h]

I couldn't readily repro it with these steps, unfortunately. It might be that there's some hidden variable at play here....

Can you check something on your repro? When you step through the program, if you have the Run view open, you should be seeing line numbers in the Call Stack section. Do they reflect the highlighted line in the editor? Or are they correct, but the editor is highlighting the wrong line?

[dddvision]

I checked the line numbers in the Call Stack, and they are correct.

I also recorded a gif animation below.

I also updated my sqlalchemy library to the following and confirmed the same result.
SQLAlchemy 1.3.18

Other libraries related to sqlalchemy that possibly could modify it are:
pyrqlite HEAD
sqlalchemy-rqlite HEAD

And, just in case it is helpful, here is the full output of 'pip list':

Click to expand!

absl-py                       0.9.0
actionlib                     1.12.0
anaconda-client               1.7.2
anaconda-navigator            1.9.12
angles                        1.9.12
appdirs                       1.4.3
arviz                         0.7.0
astor                         0.8.1
astroid                       2.3.3
astropy                       4.0
attrs                         19.3.0
Automat                       20.2.0
autopep8                      1.5
autosub                       0.3.12
Babel                         2.8.0
backcall                      0.2.0
backports.functools-lru-cache 1.6.1
backports.tempfile            1.0
backports.weakref             1.0.post1
beautifulsoup4                4.9.1
bleach                        3.1.5
bokeh                         1.4.0
bondpy                        1.8.5
brotlipy                      0.7.0
cachetools                    4.0.0
camera-calibration            1.15.0
camera-calibration-parsers    1.11.13
catkin                        0.7.26
catkin-pkg                    0.4.16
certifi                       2020.6.20
cffi                          1.14.0
cftime                        1.1.0
chardet                       3.0.4
click                         7.1.2
cloudpickle                   1.3.0
clyent                        1.2.2
colorama                      0.4.3
conda                         4.8.3
conda-build                   3.18.11
conda-package-handling        1.7.0
conda-verify                  3.4.2
constantly                    15.1.0
contextlib2                   0.6.0.post1
controller-manager            0.18.1
controller-manager-msgs       0.18.1
coverage                      5.0.3
cryptography                  2.9.2
cv-bridge                     1.13.0
cycler                        0.10.0
Cython                        0.29.21
dask                          2.12.0
decorator                     4.4.2
deepdiff                      4.2.0
defusedxml                    0.6.0
diagnostic-analysis           1.9.3
diagnostic-common-diagnostics 1.9.3
diagnostic-updater            1.9.3
diff-match-patch              20181111
distributed                   2.12.0
distro                        1.4.0
docutils                      0.16
dynamic-reconfigure           1.6.3
entrypoints                   0.3
enum34                        1.1.9
fastcache                     1.1.0
filelock                      3.0.12
flake8                        3.7.9
Flask                         1.1.1
Flask-Cors                    3.0.8
future                        0.18.2
gast                          0.2.2
gazebo-plugins                2.8.7
gazebo-ros                    2.8.7
gencpp                        0.6.5
geneus                        2.2.6
genlisp                       0.4.16
genmsg                        0.5.16
gennodejs                     2.0.1
genpy                         0.6.12
gevent                        1.4.0
glob2                         0.7
google-api-core               1.16.0
google-api-python-client      1.7.11
google-auth                   1.11.2
google-auth-httplib2          0.0.3
google-auth-oauthlib          0.4.1
google-cloud-texttospeech     1.0.1
google-pasta                  0.1.8
googleapis-common-protos      1.51.0
graphviz                      0.13.2
grpcio                        1.27.2
h5py                          2.10.0
helpdev                       0.6.10
httplib2                      0.17.0
hyperlink                     19.0.0
hypothesis                    5.6.0
idna                          2.10
image-geometry                1.13.0
imageio                       2.8.0
imagesize                     1.2.0
importlib-metadata            1.7.0
incremental                   17.5.0
interactive-markers           1.11.5
intervaltree                  3.0.2
ipykernel                     5.3.3
ipython                       7.16.1
ipython-genutils              0.2.0
ipywidgets                    7.5.1
isort                         4.3.21
jdcal                         1.4.1
jedi                          0.17.1
jeepney                       0.4.3
Jinja2                        2.11.2
joblib                        0.14.1
joint-state-publisher         1.12.15
json5                         0.9.5
jsonpickle                    1.4.1
jsonschema                    3.2.0
jupyter-client                6.1.6
jupyter-core                  4.6.3
jupyterlab                    2.1.5
jupyterlab-server             1.2.0
kdl-parser-py                 1.13.1
Keras                         2.3.1
Keras-Applications            1.0.8
Keras-Preprocessing           1.1.0
keras-segmentation            0.2.0
keyring                       21.1.1
kiwisolver                    1.2.0
laser-geometry                1.6.5
lazy-object-proxy             1.4.3
libarchive-c                  2.9
lief                          0.10.1
lxml                          4.5.0
Markdown                      3.2.1
MarkupSafe                    1.1.1
matplotlib                    3.2.0
message-filters               1.14.6
mistune                       0.8.4
mkl-fft                       1.0.15
mkl-random                    1.1.0
mkl-service                   2.3.0
mock                          4.0.1
more-itertools                8.2.0
mpyq                          0.2.5
msgpack                       1.0.0
msgpack-python                0.5.6
navigator-updater             0.2.1
nbconvert                     5.6.1
nbformat                      5.0.7
netCDF4                       1.5.3
nltk                          3.4.5
notebook                      6.0.3
numexpr                       2.7.1
numpy                         1.18.1
numpydoc                      0.9.2
oauthlib                      3.1.0
olefile                       0.46
openpyxl                      3.0.3
opt-einsum                    3.2.0
ordered-set                   3.1.1
packaging                     20.4
pandas                        1.0.1
pandocfilters                 1.4.2
parso                         0.7.0
partd                         1.1.0
path                          13.1.0
path.py                       12.4.0
pathlib2                      2.3.5
pathtools                     0.1.2
pbr                           5.4.4
peewee                        3.13.1
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        7.2.0
pip                           20.1.1
pkginfo                       1.5.0.1
pluggy                        0.13.1
portpicker                    1.3.1
progressbar                   2.5
prometheus-client             0.8.0
prompt-toolkit                3.0.5
protobuf                      3.8.0
protobuf3                     0.2.1
psutil                        5.7.0
ptyprocess                    0.6.0
py                            1.8.1
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycodestyle                   2.5.0
pycosat                       0.6.3
pycparser                     2.20
pydocstyle                    5.0.2
pydot                         1.4.1
pyflakes                      2.1.1
pygame                        1.9.6
Pygments                      2.6.1
PyHamcrest                    2.0.2
pylint                        2.4.4
pymc3                         3.8
pyodbc                        4.0.30
pyOpenSSL                     19.1.0
pyparsing                     2.4.7
PyQt5                         5.12.3
PyQt5-sip                     12.7.1
PyQtWebEngine                 5.12.1
pyreadline                    2.1
pyrqlite                      HEAD
pyrsistent                    0.16.0
PySC2                         3.0.0
PySocks                       1.7.1
pysrt                         1.1.2
pytest                        5.3.5
pytest-astropy                0.8.0
pytest-astropy-header         0.1.2
pytest-cov                    2.8.1
pytest-doctestplus            0.5.0
pytest-filter-subpackage      0.1.1
pytest-openfiles              0.4.0
python-dateutil               2.8.1
python-jsonrpc-server         0.3.4
python-language-server        0.31.8
python-qt-binding             0.4.3
pytools                       2020.1
pytz                          2020.1
pyxdg                         0.26
PyYAML                        5.3.1
pyzmq                         19.0.1
QDarkStyle                    2.8
qt-dotgraph                   0.4.1
qt-gui                        0.4.1
qt-gui-cpp                    0.4.1
qt-gui-py-common              0.4.1
QtAwesome                     0.7.0
qtconsole                     4.7.1
QtPy                          1.9.0
requests                      2.24.0
requests-oauthlib             1.3.0
resource-retriever            1.12.6
rope                          0.16.0
rosbag                        1.14.3
rosboost-cfg                  1.14.9
rosclean                      1.14.9
roscreate                     1.14.9
rosdep                        0.18.0
rosdistro                     0.8.0
rosgraph                      1.14.6
roslaunch                     1.14.6
roslib                        1.14.9
roslint                       0.11.2
roslz4                        1.14.6
rosmake                       1.14.9
rosmaster                     1.14.6
rosmsg                        1.14.6
rosnode                       1.14.6
rosparam                      1.14.6
rospkg                        1.2.3
rospy                         1.14.6
rosservice                    1.14.6
rostest                       1.14.6
rostopic                      1.14.6
rosunit                       1.14.9
roswtf                        1.14.6
rqt-action                    0.4.9
rqt-bag                       0.4.12
rqt-bag-plugins               0.4.12
rqt-console                   0.4.9
rqt-dep                       0.4.9
rqt-graph                     0.4.11
rqt-gui                       0.5.2
rqt-gui-py                    0.5.2
rqt-image-view                0.4.14
rqt-launch                    0.4.8
rqt-logger-level              0.4.8
rqt-moveit                    0.5.7
rqt-msg                       0.4.8
rqt-nav-view                  0.5.7
rqt-plot                      0.4.9
rqt-pose-view                 0.5.8
rqt-publisher                 0.4.8
rqt-py-common                 0.5.2
rqt-py-console                0.4.8
rqt-reconfigure               0.5.3
rqt-robot-dashboard           0.5.7
rqt-robot-monitor             0.5.9
rqt-robot-steering            0.5.10
rqt-runtime-monitor           0.5.7
rqt-rviz                      0.6.0
rqt-service-caller            0.4.8
rqt-shell                     0.4.9
rqt-srv                       0.4.8
rqt-tf-tree                   0.6.0
rqt-top                       0.4.8
rqt-topic                     0.4.11
rqt-web                       0.4.8
rsa                           4.0
ruamel-yaml                   0.15.87
ruamel.yaml.clib              0.2.0
rviz                          1.13.13
s2clientprotocol              4.11.4.78285.0
s2protocol                    4.11.4.78285.0
scikit-image                  0.16.2
scikit-learn                  0.22.2.post1
scipy                         1.4.1
seaborn                       0.10.0
SecretStorage                 3.1.2
Send2Trash                    1.5.0
sensor-msgs                   1.12.7
service-identity              18.1.0
setuptools                    49.2.0.post20200714
six                           1.15.0
sk-video                      1.1.10
smach                         2.0.1
smach-ros                     2.0.1
smclib                        1.8.5
sortedcollections             1.1.2
soupsieve                     2.0.1
Sphinx                        2.4.4
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        1.0.3
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.4
sphinxcontrib-websupport      1.2.0
spyder                        4.0.1
spyder-kernels                1.8.1
SQLAlchemy                    1.3.18
sqlalchemy-rqlite             HEAD
sqlite-web                    0.3.6
srt                           3.0.0
statsmodels                   0.11.1
stompest                      2.3.0
tblib                         1.6.0
tensorboard                   2.1.1
tensorflow-cpu                2.1.0
tensorflow-estimator          2.1.0
termcolor                     1.1.0
terminado                     0.8.3
testpath                      0.4.4
tf                            1.12.1
tf-conversions                1.12.1
tf2-geometry-msgs             0.6.5
tf2-kdl                       0.6.5
tf2-py                        0.6.5
tf2-ros                       0.6.5
Theano                        1.0.4
topic-tools                   1.14.6
tornado                       6.0.4
tqdm                          4.47.0
traitlets                     4.3.3
Twisted                       19.10.0
typed-ast                     1.4.1
ujson                         1.35
urdfdom-py                    0.4.3
uritemplate                   3.0.1
urllib3                       1.25.9
watchdog                      0.10.2
wcwidth                       0.2.5
webencodings                  0.5.1
websocket-client              0.57.0
Werkzeug                      1.0.0
wheel                         0.34.2
whichcraft                    0.6.1
widgetsnbextension            3.5.1
wrapt                         1.12.1
wurlitzer                     2.0.0
wxPython                      4.0.4
xacro                         1.13.5
xarray                        0.15.0
XlsxWriter                    1.2.8
xmltodict                     0.12.0
yapf                          0.29.0
zict                          2.0.0
zipp                          3.1.0
zope.interface                4.7.1

[int19h]

Can you try setting PYDEVD_USE_FRAME_EVAL=NO in environment variables, and see if that helps? Since you're launching the program using the debugger, you should be able to set it via "env" in launch.json.

[dddvision]

Setting "env":{"PYDEVD_USE_FRAME_EVAL": "NO"} in launch.json solves the problem. Thank you!

Should I add this to all of my launch.json files until a bug fix is pushed out?

[int19h]

It would probably be easiest to just set it as a global environment variable - that way it'll also apply to "attach" sessions.

[dddvision]

I'm looking forward to a more permanent bug fix.

[int19h]

@fabioz, can you take a look? It might be a rare corner case, but changing behavior of running code like that is very concerning.

[fabioz]

Sure, I'll give priority to this issue.

[int19h]

The crucial requirement for the repro is Python 3.7, although I suspect this has more to do with us only shipping Cython binaries for that.

[fabioz]

Note: I'm able to reproduce this and I'm investigating it (took me a bit more than I expected today because I ended up having to reinstall my Linux VirtualBox image which became corrupted).

[fabioz]

The issue is related to the bytecode modification in the frame eval mode -- I'm still working on it.

[fabioz]

Note: just to give more feedback, I'm still working on this.

I've actually found more issues related to that on the frame eval mode as I started doing some more tests on this, so, I'm actually scrapping the code related to the bytecode rewrite and doing the bytecode rewrite using https://github.com/vstinner/bytecode, which should allow for a better API for bytecode manipulation.

I think I'll be able to finish it next week.

[marknelson]

I'm seeing the same problem, python 3.7, the environment variable fix resolves the problem.

[BraveDistribution]

This is still valid on build: 1.74.3

"PYDEVD_USE_FRAME_EVAL": "NO" doesn't work as a fix anymore.

[int19h]

Are you doing an attach, by chance? Putting stuff in "env" only works for launch, otherwise environment variable has to be set specifically for the process you're attaching to when it's started.

If that doesn't solve the problem, can you please file a new issue?