Dynamically import configuration module from specific path

[andcaspe]

Describe the problem

First of all, thank you for your job and for making this repo available for everyone!

I was trying to understand how your codec generator works. While using the repository as a submodule of another repo, I noticed that the function get_config_module from tools_config.py allways attempts to dynamically import a module based on the file name in CONFIG_ARGS['config_file']. However, it exclusively uses importlib.import_module, which relies on the module being discoverable in Python's sys.path. This introduces a minor limitation: if the specified file is not located in a directory already included in sys.path, the function will fail with a ModuleNotFoundError.

Here are two specific cases I encountered:

• If I generate a configuration file called config.py in a directory different from the current working directory (as shown in the attached image), cbexigen imports the config.py file located in the current working directory instead of the one in config_dir directory
  
• If I generate a configuration file named config_default located in a directory other than the current working directory, I get a ModuleNotFoundError:
  

I think that it is a path dependency issue: The code does not use the full path of the configuration file; it extracts only the file name. This assumes the file resides in a directory that is already accessible via sys.path, which may not always be the case.

The function we are talking about is:

def get_config_module():
    global __CONFIG_MODULE

    if __CONFIG_MODULE is None:
        config_module_name = Path(CONFIG_ARGS['config_file']).name
        if config_module_name.endswith('.py'):
            config_module_name = config_module_name[:-3]
        __CONFIG_MODULE = importlib.import_module(config_module_name)

    return __CONFIG_MODULE

Describe your solution

I tried implementing a solution using importlib.util, which provides a more robust approach by loading the module directly from its specified file path. This method does not rely on the file being located in sys.path or having a unique name.

Here’s how the proposed solution addresses the issues:

• Direct Path Import: By using importlib.util.spec_from_file_location and module_from_spec, the module is loaded explicitly from the provided path. This bypasses the sys.path mechanism, ensuring that the correct file is always imported.
• Name Handling: The module name is derived programmatically from the file's base name using stem property (e.g., config.py becomes config), without manually checking if the string .py is contained in the name.

Here’s the updated implementation, which also ensures backward compatibility:

def get_config_module():
    global __CONFIG_MODULE

    if __CONFIG_MODULE is None:

        config_module = Path(CONFIG_ARGS['program_dir'], CONFIG_ARGS['config_file']).resolve()
        config_module_name = config_module.stem

        spec = importlib.util.spec_from_file_location(config_module_name, config_module)
        __CONFIG_MODULE = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(__CONFIG_MODULE)

    return __CONFIG_MODULE

Additional context

My project is formed with the following structure:

• src: source code, where I have configuration files and some utilities to craft V2G messages via Python
• cbexigen: repo as submodule
• README
• etc

My working environment was formed by:

• Pycharm IDE
• Python virtual environment based on Python3.11
• I have also created a pyproject.toml in order to make your library installable and allow Pycharm to automatically resolve all dependencies without the need of manually adding the cbexigen/src to sys.path. It also allow autocompletion when the instruction import cbexigen is used

The pyproject.toml I am using is quite similar to:

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "my_project"
version = "0.0.1rc0"
authors = [
]
readme = "README.md"
requires-python = ">=3.9"
dependencies = [
  "Jinja2",
  "xmlschema",
]

[project.optional-dependencies]
dev = [
]

[tool.setuptools.packages.find]
where = ["src", "cbexigen/src"]

I hope something of this information could be useful, thank you in advance!