Path to logs on email notification

[kinow]

In GitLab by @cguzman95 on Oct 9, 2024, 10:58

Hello @dbeltrankyl and @kinow ,

I would like to have the path to the err and out logs on receiving email notifications.

Currently, once I receive the email, I go to the GUI, graph, and copy the error path.

This suggestion would simplify the process by just copying the path from the email.

Thank you

[kinow]

In GitLab by @isimo00 on Oct 10, 2024, 13:43

@LuiggiTenorioK can you point me to where in the GUI scripts these paths are handled, please?

[kinow]

In GitLab by @LuiggiTenorioK on Oct 10, 2024, 14:05

Hi Irene, this data should be available inside the experiment .pkl file. In the API, I have a PklReader class to read it but I'm sure that there should be an equivalent in Autosubmit's codebase that you should use.

[kinow]

In GitLab by @isimo00 on Oct 22, 2024, 12:41

@kinow I think it would be nice to add the <EXPID>/tmp/LOG_<EXPID>/ path into autosubmitconfigparser for this, so that we could call something along the lines of BasicConfig.EXPID_LOG_PATH(expid). We create and call it constantly in autosubmit.py, and it would also make refactoring easier.

[kinow]

No objections from me, @isimo00 . I think at the moment we have to get the root path, and append the rest of the directories, expid, etc. If we are able to abstract that into a single function, that'd indeed a lot more efficient :nerd: Are you volunteering to do that too in the config parser? 😬

[kinow]

In GitLab by @isimo00 on Oct 22, 2024, 14:49

Sure, I can do that! I'm unsure when the migration to GitHub is planned for autosubmitconfigparser, but I'll be on it 😄

[kinow]

The migration has finished! @LuiggiTenorioK started early this morning, and by lunch time we were already automating some tasks with GitHub Actions :nerd: :nerd: :nerd: 💯

Feel free to switch to GitHub for the autosubmit-config-parser project. I'm rebasing one of my branches right at this moment.

[kinow]

In GitLab by @LuiggiTenorioK on Oct 22, 2024, 14:50

I try to manage paths in an additional class in the API to avoid having to complicate with the APIBasicConfig. You can check it if you want: https://earth.bsc.es/gitlab/es/autosubmit-api/-/blob/main/autosubmit_api/persistance/experiment.py

Note: In the long term, this facilitates mocking in future tests since the APIBasicConfig.read(...) might do undesired changes even if it is mocked.

[kinow]

In GitLab by @isimo00 on Oct 22, 2024, 14:58

I would consider a solution similar to this in general, but we do re-declare the expid_log_path - or similar - quite a bit, and since we are planning on refactoring I think it makes the most sense to modify the config parser. But if there's any reason why you would recommend working around the config parser instead, let me know!

[kinow]

In GitLab by @isimo00 on Oct 29, 2024, 16:41

Updated with commit 3ade74b5

[kinow]

In GitLab by @isimo00 on Nov 5, 2024, 11:29

mentioned in commit 1dbf37a0ee36932fda2ccbe39342c740290b63eb

[kinow]

In GitLab by @isimo00 on Nov 5, 2024, 12:49

mentioned in commit daeeaf8f476bb817046faefab6f9f95d11ff6b65

[kinow]

In GitLab by @isimo00 on Nov 5, 2024, 15:33

mentioned in commit 867e93ff52b3583772ea6e97371c5dabf5619ea0

[kinow]

In GitLab by @isimo00 on Dec 5, 2024, 10:46

mentioned in commit df8c557e5713428f0c8e1afca942081ed958e55e

[cguzman95]

Hi @kinow , is this implemented?

[kinow]

Hi @cguzman95 , not yet, the PR for this is this one: #1997 (cc @isimo00 -- maybe worth to have a read of the previous comments here to confirm the PR is addressing what was requested 👍 )

[isimo00]

@cguzman95 A summary: it should be merged pretty soon. Besides adding the paths to the log files in the email body, the _run.log file(s) will also be attached to the email

[cguzman95]

Why attach the _run.log file(s) ? @isimo00

[cguzman95]

Could attach the log files saturate the email? (Because they may have big size) I don't want to have to clean up my email from time to time

[dbeltrankyl]

Hello everyone,

Could attach the log files saturate the email? (Because they may have big size) I don't want to have to clean up my email from time to time

If that is a concern, we could add an optional flag at the job level to enable/disable it. Additionally, we could add compression as ( not sure if it is already added) well to the logs so their space is reduced

[isimo00]

@cguzman95 the user will have the option to enable or disable this in the configuration, but even if they choose to receive the files as attachments, the files will be compressed. The *_run.log files contain all logged information of Autosubmit's run command execution, which is relevant for analysis upon experiment malfunctioning.

[dbeltrankyl]

Thanks @isimo00 , I was not sure about the current status

[kinow]

@cguzman95 the user will have the option to enable or disable this in the configuration, but even if they choose to receive the files as attachments, the files will be compressed. The *_run.log files contain all logged information of Autosubmit's run command execution, which is relevant for analysis upon experiment malfunctioning.

Maybe that should be disabled by default, to avoid flooding user inboxes.