Some builtin single phase modules do get loaded into subinterpreters which do not support them

[sebsura]

Bug description:

If a subinterpreter is started with

PyInterpreterConfig cfg = {
...
    .check_multi_interp_extensions = 1,
};
Py_NewInterpreterFromConfig(&ts, &cfg);

I expect that no single phase module is allowed to be loaded. This works for all self build modules and with most builtin ones, but not all. Some modules i found to be misbehaving are e.g. datetime, tracemalloc.

This can easily cause a crash:

#include <Python.h>
#include <assert.h>

// make subinterpreter from config
PyThreadState* make_sub(PyInterpreterConfig* config)
{
  PyThreadState* sub = NULL;
  PyStatus status = Py_NewInterpreterFromConfig(&sub, config);

  if (PyStatus_Exception(status)) {
    return NULL;
  }

  return sub;
}

// create an isolated subinterpreter, then load a module with name 'name'
void sub_import(PyThreadState* main, const char* name)
{
  PyEval_AcquireThread(main);

  PyInterpreterConfig config = {
      .use_main_obmalloc = 0,
      .allow_fork = 0,
      .allow_exec = 0,
      .allow_threads = 1,
      .allow_daemon_threads = 0,
      .check_multi_interp_extensions = 1,
      .gil = PyInterpreterConfig_OWN_GIL,
  };

  PyThreadState* sub = make_sub(&config);
  assert(sub);

  PyObject* module = PyImport_ImportModule(name);
  
  printf("Loaded: %s\n", module ? "yes" : "no");

  Py_XDECREF(module);

  Py_EndInterpreter(sub);
}

int main()
{
  Py_InitializeEx(0);

  PyThreadState* interp = PyEval_SaveThread();

  const char* module_name = "datetime";

  sub_import(interp, module_name);
  sub_import(interp, module_name); // <- crash

  PyEval_AcquireThread(interp);
  Py_FinalizeEx();
}

This is because the module will get loaded a second time, which will cause python to try and cleanup the old state from the old interpreter inside the new interpreter.

// obmalloc.c
void
_PyObject_Free(void *ctx, void *p) // p will point to an object allocated by pymalloc from the old interpreter here
{
    /* PyObject_Free(NULL) has no effect */
    if (p == NULL) {
        return;
    }

    OMState *state = get_state();
    if (UNLIKELY(!pymalloc_free(state, ctx, p))) { // <- this returns false, since the object does not belong to the new interpreter
        /* pymalloc didn't allocate this address */
        PyMem_RawFree(p); // <- this will crash, since the pointer was not allocated directly through malloc
        raw_allocated_blocks--;
    }
}

This may be related to #117953 and #104621 but its important to note that this does not happen with the readline module as it correctly refuses to load.

CPython versions tested on:

3.12

Operating systems tested on:

Linux

[sebsura]

The deallocation is caused by this:

// _tetimemodule.c:6755
static int
_datetime_exec(PyObject* module)
{
    ...
    PyObject *d = PyDateTime_DeltaType.tp_dict; // <- this is a global variable
    DATETIME_ADD_MACRO(d, "resolution", new_delta(0, 0, 1, 0));

[ericsnowcurrently]

This should be resolved now for 3.13+, as of gh-121060. Sadly, the fix is a bit too complex to backport to 3.12.

[sebsura]

Thanks for the heads up. I can now run the sample code i gave above without crashing!