Acquire spill lock in to/from_arrow

[shwina]

Description

Currently, calling to_arrow will render a cuDF Series/DataFrame unspillable. I believe it's just an oversight that we don't call to/from_arrow in an acquire_spill_lock context. Neither of those methods should expose the device pointer to a cuDF object externally.

cc: @madsbk @vyasr -- could you take a look please?

In [1]: import cudf
In [2]: s = cudf.Series([1, 2, 3])
In [3]: cudf.core.buffer.spill_manager.get_global_manager()
Out[3]: <SpillManager spill_on_demand=True device_memory_limit=N/A | 0B spilled | 24B (0%) unspilled (unspillable)>
In [4]: s.to_arrow()
Out[4]:
<pyarrow.lib.Int64Array object at 0x7f31e8f7ba60>
[
  1,
  2,
  3
]
In [5]: cudf.core.buffer.spill_manager.get_global_manager()
Out[5]: <SpillManager spill_on_demand=True device_memory_limit=N/A | 0B spilled | 24B (100%) unspilled (unspillable)>

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[vyasr]

Yeah I think this is just an oversight. Adding more acquire_spill_lock calls was always intended to be an ongoing task IIRC, although @madsbk can correct me if I've forgotten something.

[madsbk]

Right, since cudf::to_arrow returns a arrow::Table that owns the data, we can spill.
Good catch @shwina

[pentschev]

Although this is resolving a real problem that needs to be fixed, just wanted to report here that it doesn't resolve the P2P shuffle issue we've seen and we still end up with errors such as below:

MemoryError: std::bad_alloc: CUDA error at: /datasets/pentschev/miniconda3/envs/cudf-invalid-address-src/include/rmm/mr/device/cuda_async_view_memory_resource.hpp:121: cudaErrorIllegalAddress an illegal memory access was encountered

With that said, the illegal address errors are probably due to something else.

[pentschev]

cc @quasiben @rjzamora @VibhuJawa for awareness, it doesn't resolve the illegal address errors as I mentioned in #13646 (review) .

[shwina]

/merge