Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Misleading RAPTOR message pyfuncs and suppressed error message. #3011

Open
AymenFJA opened this issue Aug 14, 2023 · 0 comments
Open

Misleading RAPTOR message pyfuncs and suppressed error message. #3011

AymenFJA opened this issue Aug 14, 2023 · 0 comments

Comments

@AymenFJA
Copy link
Contributor

The RAPTOR code, specifically in

to_call, _args, _kwargs = PythonTask.get_func_attr(func)

If this line fails, our RAPTOR logger will print:

self._log.warn('function is not a PythonTask [%s] ', uid)

Two problems here:

  1. The message of the get_func_attr is suppressed, and we just report a hard-coded message (we need to at least report the correct message, not the entire exception)
  2. I tested it with a Python function with some module dependencies that were unavailable in the namespace where the function was deserialized, and I printed the exception. Something like this:
except Exception as e:
          self._log.warn('function is not a PythonTask [%s] ', uid)
          self._log.warn(e)

This is the modified output message from our raptor.worker:

1691803277.803 : master.000000.worker.0000 : 28773 : 140004894422784 : WARNING  : function is not a PythonTask [task.000000]
1691803277.803 : master.000000.worker.0000 : 28773 : 140004894422784 : WARNING  : No module named 'mpi_funcs'
1691803277.804 : master.000000.worker.0000 : 28777 : 139872066201344 : DEBUG    : orig args: [] : {}

As you can see, the first line is what RAPTOR reported and marked this function as not PythonTask which is not correct as this task was serialized by our function serializer.

The 2nd line shows the actual error of why our serialization failed, and that is because No module named 'mpi_funcs' which my function requires during the execution.

@AymenFJA AymenFJA self-assigned this Aug 14, 2023
@AymenFJA AymenFJA changed the title Misleading RAPTOR pyfuncs and suppressed error message. Misleading RAPTOR message pyfuncs and suppressed error message. Aug 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant