[3.13] gh-113993: Allow interned strings to be mortal, and fix related issues (GH-120520)

[encukou]

(cherry-picked from 6f1d448)

• Add an InternalDocs file describing how interning should work and how to use it.

• Add internal functions to explicitly request what kind of interning is done:

  • _PyUnicode_InternMortal
  • _PyUnicode_InternImmortal
  • _PyUnicode_InternStatic

• Switch uses of PyUnicode_InternInPlace to those.

• Disallow using _Py_SetImmortal on strings directly. You should use _PyUnicode_InternImmortal instead:

  • Strings should be interned before immortalization, otherwise you're possibly interning a immortalizing copy.
  • _Py_SetImmortal doesn't handle the SSTATE_INTERNED_MORTAL to SSTATE_INTERNED_IMMORTAL update, and those flags can't be changed in backports, as they are now part of public API and version-specific ABI.

• Add private _only_immortal argument for sys.getunicodeinternedsize, used in refleak test machinery.

• Make sure the statically allocated string singletons are unique. This means these sets are now disjoint:

  • _Py_ID
  • _Py_STR (including the empty string)
  • one-character latin-1 singletons

  Now, when you intern a singleton, that exact singleton will be interned.

• Add a _Py_LATIN1_CHR macro, use it instead of _Py_ID/_Py_STR for one-character latin-1 singletons everywhere (including Clinic).

• Intern _Py_STR singletons at startup.

• For free-threaded builds, intern _Py_LATIN1_CHR singletons at startup.

• Beef up the tests. Cover internal details (marked with @cpython_only).

• Add lots of assertions

• Issue: Interned strings are immortal, despite what the documentation says #113993

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @encukou for commit 70bfeae 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.