BUG: Calling pop() on a row inside apply() affects all subsequent applied rows

[flogbert]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import pandas as pd
def f(s):
  s.pop("A")
df = pd.DataFrame({"A": [1, 2, 3]})
df.apply(f, axis="columns")

Traceback (most recent call last):
  File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3080, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 70, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 101, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 4554, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 4562, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'A'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/frame.py", line 7765, in apply
    return op.get_result()
  File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/apply.py", line 185, in get_result
    return self.apply_standard()
  File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/apply.py", line 276, in apply_standard
    results, res_index = self.apply_series_generator()
  File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/apply.py", line 290, in apply_series_generator
    results[i] = self.f(v)
  File "<stdin>", line 1, in f
  File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/series.py", line 4467, in pop
    return super().pop(item=item)
  File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 768, in pop
    result = self[item]
  File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/series.py", line 824, in __getitem__
    return self._get_value(key)
  File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/series.py", line 932, in _get_value
    loc = self.index.get_loc(label)
  File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3082, in get_loc
    raise KeyError(key) from err
KeyError: 'A'

Problem description

When the function being applied by apply pops a column from the row, the popped column is missing on subsequent calls of the function. This seems to stem from apply reusing the same Series instance across multiple invocations but failing to fully reset its state (i.e., the popped column stays popped).

This is an apparent regression from the behavior in v1.0.5 which returned a DataFrame with three rows. The timing of the regression suggests that it is possibly an unintended consequence of #34909.

The problem is relatively straightforward to work around. Performing a shallow copy on the Series fixes it:

>>> def f(s):
...   s.copy(deep=False).pop("A")
... 
>>> df = pd.DataFrame({"A": [1, 2, 3]})
>>> df.apply(f, axis="columns")
0    None
1    None
2    None
dtype: object

Expected Output

The expected output is what is shown above with the workaround which is what v1.0.5 output.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 9d598a5
python : 3.8.7.final.0
python-bits : 64
OS : Linux
OS-release : 5.7.17-1rodete4-amd64
Version : #1 SMP Debian 5.7.17-1rodete4 (2020-10-01)
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.2.1
numpy : 1.19.4
pytz : 2020.5
dateutil : 2.8.1
pip : 20.1.1
setuptools : 51.1.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : 4.9.3
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[rhshadrach]

Thanks for the report! In general, one shouldn't mutate a container when iterating over it. I think your shallow copy is the appropriate solution here. Related to #12653.

[flogbert]

Thanks for the prompt response and the helpful explanation! Is this explained in the documentation somewhere? I couldn't find it.

[rhshadrach]

No, but it should be. See #12653.

[MarcoGorelli]

@rhshadrach since #39762 was merged, can this be closed?

I'll go ahead with closing, but please do let me know if I've misunderstood / not read this properly

[rhshadrach]

Yep, agreed. Thanks @MarcoGorelli