Multiprocessing and running frozen on Windows

[1marc1]

I recently upgraded my environment, along with it I upgraded from logger_tt 1.7.3 to 1.7.4. I found that my application crashes when I run it frozen on Windows. The crash happens as soon as my code calls pool.map(). I then get numerous messages on the screen, saying that the file C:\Program Files\My Program\logs\log.txt does not exist.

I did a lot of digging around and found the reference to logs/log.txt in my configuration file:

{
  "version": 1,
  "disable_existing_loggers": false,
  "formatters": {
    "simple": {
      "format": "%(asctime)s %(name)-8s %(levelname)-8s %(processName)-10s %(message)s"
    },
    "brief": {
      "format": "[%(asctime)s] %(levelname)s: %(message)s",
      "datefmt": "%Y-%m-%d %H:%M:%S"
    }
  },

  "handlers": {
    "console": {
      "class": "logging.StreamHandler",
      "level": "INFO",
      "formatter": "brief",
      "stream": "ext://sys.stdout"
    },

    "error_file_handler": {
      "class": "logging.handlers.TimedRotatingFileHandler",
      "level": "DEBUG",
      "formatter": "simple",
      "filename": "logs/log.txt",
      "backupCount": 3,
      "encoding": "utf8",
      "when": "midnight"
    }
  },

  "loggers": {
    "urllib3": {
      "level": "ERROR",
      "handlers": ["error_file_handler"],
      "propagate": false
    }
  },

  "root": {
    "level": "DEBUG",
    "handlers": ["error_file_handler"]
  },

  "logger_tt": {
    "suppress": ["exchangelib"]
  }
}

My program sets up the logging as follows: setup_logging(config_path=_jsonfile, log_path=_logfile, use_multiprocessing=True), whereby _jsonfile references the configuration file listed above and _logfile references a user-writable location to keep the log file.

When I manually create the C:\Program Files\My Program\logs directory and run my program as Administrator, I don't run into the problem. I then notice that a "log.txt" file is created in that directory with a size of 0 bytes. I attribute this to the fact that an ordinary user does not have write access inside the C:\Program Files directory structure, but an administrator does.

When I revert back to logger_tt 1.7.3, using the same config file, I don't experience this issue. Can someone look into this to work out what has changed between versions that causes this behaviour and perhaps update the code so this doesn't happen?

Thank you!

[Dragon2fly]

Hi @1marc1

Can you provide a minimal code with pool.map() that reproduces the issue?
Also, which Python version are you using?

[1marc1]

I am using Python 3.13.9. I have been trying for several hours to create a minimal code example, but haven't been successful. I will do some further testing.

[1marc1]

OK, I may have something, but I don't think it perfectly highlights my issue as when I use logger_tt 1.7.3 I don't see anything in the log created by process_item(). In my actual program that is working. What I do notice with the minimal code below is that when I use logger_tt 1.7.4 and create a frozen Windows executable, I am experiencing the issue I am reporting about. Sorry if this is confusing.

Here is my main program example.py

import multiprocessing
from logging import getLogger
from logger_tt import setup_logging
import external as ext

_jsonfile = 'log_config.json'
_logfile = 'example.log'
logger_port = ext.get_logger_tt_port_number()
if logger_port == 0:
    print ('No appropriate port for the logger function could be found.')
    sys.exit(1)
        
setup_logging(config_path=_jsonfile, log_path=_logfile, use_multiprocessing=True, host='127.0.0.1', port=logger_port)

logger = getLogger('Example')
logger.info("Main process initialized and logging configured.")


# --- Main Program Execution ---
if __name__ == '__main__':
    multiprocessing.freeze_support()
    
    # Data to be processed
    data_list = [10, 25, 30, 45, 50, 65, 70, 85]
    
    logger.info(f"Starting `pool.map()` for data: {data_list}")

    pool = multiprocessing.Pool()
    results = pool.map(ext.process_item, data_list)
    pool.terminate()

Here is the contents of external.py

import sys
import socket
import multiprocessing
import time
from logging import getLogger

def get_logger_tt_port_number():
    try:
        # Create a socket (IPv4, TCP)
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        # Bind to '127.0.0.1' and port 0 (OS chooses a free port)
        s.bind(('127.0.0.1', 0))
        # Get the port number the OS assigned
        _, port = s.getsockname()
        s.close()
        return port
    except Exception:
        # Return 0 on any error (e.g., no available ports)
        return 0

# --- Worker Function ---
def process_item(item):
    logger = getLogger('Process')

    # 2. Simulate work
    process_id = multiprocessing.current_process().pid
    logger.info(f"Process {process_id} started processing item: {item}")
    time.sleep(0.5) # Simulate some time-consuming operation
    result = item * 2

    # 3. Log the result
    logger.info(f"Process {process_id} finished item {item}, result: {result}")
    return result

When I run python example.py in Python 3.13.9 venv with logger_tt 1.7.4 on Windows 11 it works without any problems. When I freeze this program and run the resulting executable I am getting lots of errors that seem to be in an endless loop. Part of the output that caught my eye in my actual (GUI) program was the reference to the logs/log.txt file. This reference also shows up in this CLI example program.

[Dragon2fly]

Hi @1marc1

Can you provide the pyinstaller command that was used to freeze the sample program above?
Also, will the error appear if you put your freezed program in different location other than C:\Program Files\My Program\ ?

[1marc1]

I am using cx_Freeze version 8.4.1 to freeze the example program. Here is the content of the setup.py file:

import sys
from cx_Freeze import setup, Executable

base = None

build_exe_options = {
    "packages": [], 
    "excludes": ["tkinter", "unittest", "pydoc"],
}

setup(
    name="ExampleApp",
    version="0.1",
    description="A frozen example application",
    options={"build_exe": build_exe_options},
    executables=[Executable("example.py", base=base)]
)

I then run python setup.py build to create the executable.

[1marc1]

And to answer your second question: yes. I have been running the frozen example program from a sub-directory of my "Downloads" directory and the endless loop with the references to logs/log.txt show up there.