Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Partially load modules from compiler cache, handling possible race condition #440

Open
wants to merge 5 commits into
base: dev
Choose a base branch
from

Commits on Feb 24, 2023

  1. Limit compiler cache interface to getTarget and addModule

    Current implementation of the interface, used via
    _loadFromCompilerCache, runs loadForSymbol for a given link name and
    then returns two dicts representing everything the cache touched during
    this load process. Maintaining this interface makes the partial-load
    refactor difficult, and muddies the converter layers/cache relationship.
    Here we alter the API so that we can ask the cache for
    a TypedCallTarget, and add modules, and that's it. This means getting
    rid of _loadFromCompilerCache, and associated registers for tracking
    what's being converted. Also means passing the cache down to the
    native_ast_to_llvm layer.
    Augustus Lonergan authored and William Grant committed Feb 24, 2023
    Configuration menu
    Copy the full SHA
    65b622c View commit details
    Browse the repository at this point in the history
  2. Allow for partial module loads in compiler cache.

    Previously we would always attempt to link and validate all global
    variables when loading a module from the cache. This caused linking
    errors, or validation errors, or deserialization errors, and meant we
    needed the mark_invalid mechanism for handling modules with outdated
    global variables. Here we add double-serialised global variables, and
    only deserialize,link&validate the subset required for the function
    required (and its dependencies). This requires the cache to store
    a function and global_var dependency graph. Also add utility methods
    for GlobalVariableDefinition.
    William Grant committed Feb 24, 2023
    Configuration menu
    Copy the full SHA
    f0747ba View commit details
    Browse the repository at this point in the history
  3. Ensure that serializer can use the 'name' pickle protocol.

    Pickle supports a protocol where __reduce__returns a string giving
    the global name. Implementing this behaviour lets us serialize numpy
    ufuncs. Also adjust installInflightFunctions to handle new load
    behaviour, fix an instability caused by not leaving LoadedModule
    objects in memory, and adjust alternative test. Also ensure that the new
    pickle protocol support works with 'local names' (e.g. dotted method names).
    William Grant committed Feb 24, 2023
    Configuration menu
    Copy the full SHA
    12e35a8 View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2023

  1. Allow for multiple copies of the same function in different modules.

    Previous iterations of the cache assumed a one-one-one function name to
    module mapping, however it's possible to end up with many modules which
    contain the same function (e.g due to a race condition when using multiple
    processes, but other scenarios could exist in the future). This commit
    separates the func_name (the id for the function) with the link_name
    (the unique id for a given function in a given module). This distinction
    is not exposed outside the cache - when asked for a target the cache
    chooses which version to return (currently just the first one it sees).
    William Grant committed Mar 2, 2023
    Configuration menu
    Copy the full SHA
    ad77e7d View commit details
    Browse the repository at this point in the history
  2. Disallow compilation during module import

    During the compilation of one Entrypointed function it's possible to import a module which
    calls a second Entrypointed function. This breaks our model of the
    compilation process and could cause deadlocks, so instead throw an
    ImportError.
    William Grant committed Mar 2, 2023
    Configuration menu
    Copy the full SHA
    e2d754a View commit details
    Browse the repository at this point in the history