Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-114099 - Add iOS framework loading machinery. #116454

Merged
merged 11 commits into from
Mar 19, 2024
Merged
77 changes: 76 additions & 1 deletion Lib/importlib/_bootstrap_external.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@

# Bootstrap-related code ######################################################
_CASE_INSENSITIVE_PLATFORMS_STR_KEY = 'win',
_CASE_INSENSITIVE_PLATFORMS_BYTES_KEY = 'cygwin', 'darwin'
_CASE_INSENSITIVE_PLATFORMS_BYTES_KEY = 'cygwin', 'darwin', 'ios', 'tvos', 'watchos'
_CASE_INSENSITIVE_PLATFORMS = (_CASE_INSENSITIVE_PLATFORMS_BYTES_KEY
+ _CASE_INSENSITIVE_PLATFORMS_STR_KEY)

Expand Down Expand Up @@ -1711,6 +1711,77 @@ def __repr__(self):
return f'FileFinder({self.path!r})'


class AppleFrameworkLoader(ExtensionFileLoader):
"""A loader for modules that have been packaged as frameworks for
compatibility with Apple's App Store policies.
freakboy3742 marked this conversation as resolved.
Show resolved Hide resolved

For compatibility with the App Store, *all* binary modules must be .dylib
objects, contained in a framework, stored in the ``Frameworks`` folder of
the packaged app. There can be only a single binary per framework, and
there can be no executable binary material outside the Frameworks folder.

If you're trying to run ``from foo.bar import _whiz``, and ``_whiz`` is
implemented with the binary module ``foo/bar/_whiz.abi3.dylib`` (or any
other ABI .dylib extension), this loader will look for
``{sys.executable}/Frameworks/foo.bar._whiz.framework/_whiz.abi3.dylib``
(forming the package name by taking the full import path of the library,
and replacing ``/`` with ``.``).

However, the ``__file__`` attribute of the ``_whiz`` module will report as
the original location inside the ``foo/bar`` subdirectory. This so that
code that depends on walking directory trees will continue to work as
expected based on the *original* file location.

The Xcode project building the app is responsible for converting any
``.dylib`` files from wherever they exist in the ``PYTHONPATH`` into
frameworks in the ``Frameworks`` folder (including the addition of
framework metadata, and signing the resulting framework). This will usually
be done with a build step in the Xcode project; see the iOS documentation
for details on how to construct this build step.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see this in the importlib docs. It would probably make sense to have it there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense; I'll add it in.

"""
def __init__(self, fullname, dylib_file, path=None):
super().__init__(fullname, dylib_file)
self.parent_paths = path

def create_module(self, spec):
mod = super().create_module(spec)
if self.parent_paths:
for parent_path in self.parent_paths:
if _path_isdir(parent_path):
mod.__file__ = _path_join(
parent_path,
_path_split(self.path)[-1],
)
continue
freakboy3742 marked this conversation as resolved.
Show resolved Hide resolved
return mod


class AppleFrameworkFinder:
"""A finder for modules that have been packaged as Apple Frameworks
for compatibility with Apple's App Store policies.

See AppleFrameworkLoader for details.
"""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this a meta-path finder and not a path-entry finder? In other words, why don't you subclass FileFinder and then plug this in to sys.path_hooks? This is certainly a file-focused finder.

FWIW, I think I know the answer already. 😄 Correct me if I'm wrong, but that would require FRAMEWORKS/FULLNAME.framework to be on sys.path. It would also mean that it would be checked (unnecessarily) for any other sys.path entries and that any other path-entry finders would check the framework-specific sys.path entries unnecessarily.

That said, would it be feasible to write a custom FileFinder subclass that operated relative to the app's frameworks directory. Then you'd put that on sys.path and wouldn't need a custom __init__(). I'm not saying you need to do this (or even that it is doable, though my gut says it is). Mostly I'm asking if you had considered this and, if not, recommend that you take a look.

One reason I bring it up is because, any time we step outside the normal flow of things, we run the risk of disrupting users. For example, in this case a user might expect extension modules to be found via a path-entry finder rather than a meta-path finder. They might mess with things under that assumption and then get grumpy when things break (most likely mysteriously). Of course the risk here is supremely small, but users have a habit of surprising us. 😄

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would also mean that it would be checked (unnecessarily) for any other sys.path entries

Out of curiosity, what should one expect sys.path to look like on iOS?

Also, if I've understood correctly then extension modules can only be found under the app's frameworks folder and never on any other sys.path entry. If that's the case then should we remove the ExtensionFileLoader entry from _get_supported_file_loaders() in Lib/importlib/_bootstrap_external.py? Also, are the restrictions on extension modules (location, suffix) technical restrictions or strictly a consequence of the fact that apps cannot ship with non-conforming binary files, as enforced by the app store police? Is there a possibility that one could find an extension file out of place somehow, like in some system directory?

IIUC, .py files may still be imported using the normal path-entry finder (FileFinder), right? I'm not familiar with iOS apps to even imagine when the finder might be looking (i.e. what's in sys.path).

Relatedly, what happens if an app maintainer (or, somehow, a user) registers additional importers or messes with sys.path? Could that cause problems for this finder? Could it circumvent the app store rules?

and that any other path-entry finders would check the framework-specific sys.path entries unnecessarily.

FYI, the import machinery already makes this (almost) a non-issue with path hooks and sys.path_importer_cache.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're correct that the reason for the metapath finder is the structure of the Frameworks folder. I hadn't considered using a custom FileFinder; I agree we should do anything we can to avoid breaking user expectations (however eccentric or unexpected those might be), so I'll take a look and see if I can make this work.

As for ExtensionFileLoader - that's an interesting case. iOS apps are entirely capable of loading binary modules from the PYTHONPATH; it's entirely an issue of the resulting app being acceptable to the App Store. If you've got a project that doesn't have the "move dylibs to the Frameworks folder" build step, the app will still run - it will just be rejected when you try to submit it to the App store (and it will be rejected by an automated check at time of submission, not hours/days/weeks later as a result of the review process). I've also had a couple of bugs (including one I'm still chasing with cryptography) where it's useful to be able to confirm if the problem you're seeing is because the .dylib has been moved, or because the .dylib isn't working. I therefore opted to retain the ExtensionFileLoader, just in case it was useful.

The SourceFileLoader and SourcelessFileLoader both work exactly the same, however. Loading .py files and .pyc files from anywhere in the app bundle isn't an issue; it's only binary modules that the App Store restricts. sys.path for a running app will contain {dirname(sys.executable)}/python/lib/python3.X at a minimum (i.e., prefix=={dirname(sys.executable)}/python, but the bin/include/man/share folders aren't installed). BeeWare apps usually end up with {dirname(sys.executable)}/app and {dirname(sys.executable)}/app_packages as well as the location of app code and app dependencies, but that's entirely up to the app itself.

If a user installs an additional importer... I guess that depends on what the importer is doing. The entire contents of the app bundle is fair game for reading; so a novel mechanism for finding .py files shouldn't be an issue. The only place I could force a problem is if the importer is expecting to find an executable binary in a specific location outside the Frameworks folder - but the app won't be accepted into the App Store in the first place if they try to do this. As for circumventing the App Store rules - it might be a lack of imagination on my part, but I'm having difficulty imaging what you'd be able to do here. Normal filesystem write protections should prevent even a malicious user from doing anything damaging, and you won't be able to use anything binary that isn't in the app bundle and signed on that basis.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. Thanks for the detailed explanation!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hadn't considered using a custom FileFinder; I agree we should do anything we can to avoid breaking user expectations (however eccentric or unexpected those might be), so I'll take a look and see if I can make this work.

Let me know if you have any questions on this.

def __init__(self, path):
self.frameworks_path = path

def find_spec(self, fullname, path, target=None):
name = fullname.split(".")[-1]
freakboy3742 marked this conversation as resolved.
Show resolved Hide resolved

for extension in EXTENSION_SUFFIXES:
ericsnowcurrently marked this conversation as resolved.
Show resolved Hide resolved
dylib_file = _path_join(self.frameworks_path, f"{fullname}.framework", f"{name}{extension}")
_bootstrap._verbose_message("Looking for Apple Framework dylib {}", dylib_file)
try:
dylib_exists = _path_isfile(dylib_file)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mentioned yesterday that this logic is difficult to test completely when the stdlib doesn't have any extension modules inside packages. But couldn't it be tested with any directory containing files with the correct suffixes? They don't actually have to be dylibs, and the test doesn't actually have to load them, it just needs to verify that the correct spec is returned.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually found a way to test this by faking the request; see test_file_origin in the loader tests.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That tests the loader, and I see extension/test_finder.py gives some coverage of the finder for top-level modules, but the distinction between fullname and name for modules inside a package should be tested somehow as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code no longer exists, so I guess extra tests aren't needed either.

except ValueError:
pass
freakboy3742 marked this conversation as resolved.
Show resolved Hide resolved
else:
if dylib_exists:
loader = AppleFrameworkLoader(fullname, dylib_file, path)
return _bootstrap.spec_from_loader(fullname, loader)

return None

# Import setup ###############################################################

def _fix_up_module(ns, name, pathname, cpathname=None):
Expand Down Expand Up @@ -1760,3 +1831,7 @@ def _install(_bootstrap_module):
supported_loaders = _get_supported_file_loaders()
sys.path_hooks.extend([FileFinder.path_hook(*supported_loaders)])
sys.meta_path.append(PathFinder)
if sys.platform in {"ios", "tvos", "watchos"}:
frameworks_folder = _path_join(_path_split(sys.executable)[0], "Frameworks")
ericsnowcurrently marked this conversation as resolved.
Show resolved Hide resolved
_bootstrap._verbose_message("Adding Apple Framework dylib finder at {}", frameworks_folder)
sys.meta_path.append(AppleFrameworkFinder(frameworks_folder))
2 changes: 2 additions & 0 deletions Lib/importlib/machinery.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@
from ._bootstrap_external import SourceFileLoader
from ._bootstrap_external import SourcelessFileLoader
from ._bootstrap_external import ExtensionFileLoader
from ._bootstrap_external import AppleFrameworkLoader
from ._bootstrap_external import AppleFrameworkFinder
freakboy3742 marked this conversation as resolved.
Show resolved Hide resolved
from ._bootstrap_external import NamespaceLoader


Expand Down
34 changes: 25 additions & 9 deletions Lib/test/test_import/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
import importlib.util
from importlib._bootstrap_external import _get_sourcefile
from importlib.machinery import (
BuiltinImporter, ExtensionFileLoader, FrozenImporter, SourceFileLoader,
AppleFrameworkLoader, BuiltinImporter, ExtensionFileLoader, FrozenImporter, SourceFileLoader,
)
import marshal
import os
Expand All @@ -25,7 +25,7 @@

from test.support import os_helper
from test.support import (
STDLIB_DIR, swap_attr, swap_item, cpython_only, is_emscripten,
STDLIB_DIR, swap_attr, swap_item, cpython_only, is_apple_mobile, is_emscripten,
is_wasi, run_in_subinterp, run_in_subinterp_with_config, Py_TRACE_REFS)
from test.support.import_helper import (
forget, make_legacy_pyc, unlink, unload, ready_to_import,
Expand Down Expand Up @@ -66,6 +66,7 @@ def _require_loader(module, loader, skip):
MODULE_KINDS = {
BuiltinImporter: 'built-in',
ExtensionFileLoader: 'extension',
AppleFrameworkLoader: 'framework extension',
FrozenImporter: 'frozen',
SourceFileLoader: 'pure Python',
}
Expand All @@ -91,7 +92,10 @@ def require_builtin(module, *, skip=False):
assert module.__spec__.origin == 'built-in', module.__spec__

def require_extension(module, *, skip=False):
_require_loader(module, ExtensionFileLoader, skip)
if is_apple_mobile:
_require_loader(module, AppleFrameworkLoader, skip)
else:
_require_loader(module, ExtensionFileLoader, skip)

def require_frozen(module, *, skip=True):
module = _require_loader(module, FrozenImporter, skip)
Expand Down Expand Up @@ -360,7 +364,7 @@ def test_from_import_missing_attr_has_name_and_so_path(self):
self.assertEqual(cm.exception.path, _testcapi.__file__)
self.assertRegex(
str(cm.exception),
r"cannot import name 'i_dont_exist' from '_testcapi' \(.*\.(so|pyd)\)"
r"cannot import name 'i_dont_exist' from '_testcapi' \(.*\.(so|dylib|pyd)\)"
)
else:
self.assertEqual(
Expand Down Expand Up @@ -1689,6 +1693,12 @@ def pipe(self):
os.set_blocking(r, False)
return (r, w)

def create_extension_loader(self, modname, filename):
if is_apple_mobile:
return AppleFrameworkLoader(modname, filename, None)
else:
return ExtensionFileLoader(modname, filename)

def import_script(self, name, fd, filename=None, check_override=None):
override_text = ''
if check_override is not None:
Expand Down Expand Up @@ -1883,7 +1893,7 @@ def test_multi_init_extension_compat(self):
def test_multi_init_extension_non_isolated_compat(self):
modname = '_test_non_isolated'
filename = _testmultiphase.__file__
loader = ExtensionFileLoader(modname, filename)
loader = self.create_extension_loader(modname, filename)
spec = importlib.util.spec_from_loader(modname, loader)
module = importlib.util.module_from_spec(spec)
loader.exec_module(module)
Expand All @@ -1901,7 +1911,7 @@ def test_multi_init_extension_non_isolated_compat(self):
def test_multi_init_extension_per_interpreter_gil_compat(self):
modname = '_test_shared_gil_only'
filename = _testmultiphase.__file__
loader = ExtensionFileLoader(modname, filename)
loader = self.create_extension_loader(modname, filename)
spec = importlib.util.spec_from_loader(modname, loader)
module = importlib.util.module_from_spec(spec)
loader.exec_module(module)
Expand Down Expand Up @@ -2034,10 +2044,13 @@ class SinglephaseInitTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
spec = importlib.util.find_spec(cls.NAME)
from importlib.machinery import ExtensionFileLoader
from importlib.machinery import AppleFrameworkLoader, ExtensionFileLoader
cls.FILE = spec.origin
cls.LOADER = type(spec.loader)
assert cls.LOADER is ExtensionFileLoader
if is_apple_mobile:
assert cls.LOADER is AppleFrameworkLoader
else:
assert cls.LOADER is ExtensionFileLoader

# Start fresh.
cls.clean_up()
Expand Down Expand Up @@ -2077,7 +2090,10 @@ def _load_dynamic(self, name, path):
"""
# This is essentially copied from the old imp module.
from importlib._bootstrap import _load
loader = self.LOADER(name, path)
if is_apple_mobile:
loader = self.LOADER(name, path, None)
else:
loader = self.LOADER(name, path)

# Issue bpo-24748: Skip the sys.modules check in _load_module_shim;
# always load new extension.
Expand Down
24 changes: 20 additions & 4 deletions Lib/test/test_importlib/extension/test_finder.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
from test.support import is_apple_mobile
from test.test_importlib import abc, util

machinery = util.import_importlib('importlib.machinery')

import unittest
import os
import sys


Expand All @@ -19,11 +21,25 @@ def setUp(self):
)

def find_spec(self, fullname):
importer = self.machinery.FileFinder(util.EXTENSIONS.path,
(self.machinery.ExtensionFileLoader,
self.machinery.EXTENSION_SUFFIXES))
if is_apple_mobile:
# Apple extensions must be distributed as frameworks. This requires
# a specialist finder.
frameworks_folder = os.path.join(
os.path.split(sys.executable)[0], "Frameworks"
)
importer = self.machinery.AppleFrameworkFinder(frameworks_folder)

return importer.find_spec(fullname, None)
else:
importer = self.machinery.FileFinder(
util.EXTENSIONS.path,
(
self.machinery.ExtensionFileLoader,
self.machinery.EXTENSION_SUFFIXES
)
)

return importer.find_spec(fullname)
return importer.find_spec(fullname)

def test_module(self):
self.assertTrue(self.find_spec(util.EXTENSIONS.name))
Expand Down
Loading
Loading