[v7r3] added a flag for disabling the use of SecurityLogging service

[fstagni]

See https://gitlab.cern.ch/ai/it-puppet-module-dirac/-/merge_requests/75

BEGINRELEASENOTES

*Framework
CHANGE: added a flag for disabling the use of SecurityLogging service

ENDRELEASENOTES

[fstagni]

Seems like there's a psutil bug: giampaolo/psutil#2048

[fstagni]

54b6db3 is very much subject to review from @chaen @TaykYoku

[TaykYoku]

SecurityLogClient uses DIRAC.Core.Utilities.ThreadScheduler for reporting.
Since we are moving from DISET to Tornado, it is worth using tornado.web.IOLoop to run periodic tasks. I think the following PR will be useful for this implementation.

[chaen]

Given that we want to ditch it, I really do not think it is worth the pain of adding the securityLog to TornadoService

[fstagni]

Given that we want to ditch it, I really do not think it is worth the pain of adding the securityLog to TornadoService

Yes, we want to ditch the securityLog, but ATM it's still needed. In order to ditch it, we need in fact to store on ES 6 months worth of log data, for each of the running DIRAC services. This is not impossible but rather expensive, considering that ATM we are correctly sending to ES everything that we put in verbose level. What is stored on the current securityLog is a rather minimal subset of that. So, ditching "tomorrow" the current securityLog service is not possible.

What we can do is to use again ES, in combination with logstash: meaning that we create ES indices with the content of what would normally go in the CSV files created by SecurityLog services, driven by a logstash rule. Would this work?

[chaen]

yes it would work. But I still would not add it to Tornado now

[fstagni]

I have removed the changes to tornado, and added changes for logstash configuration. Please review this PR, and the one for puppet/logstash. If you are happy we can try this in the coming hackathon.

[chaen]

same here, the ConfReference is going away

[chaen]

The code and puppet code seems fine. Just you need to add the option doc to the dirac.cfg instead of the ConfReference

[DIRACGridBot]

Sweep summary

Sweep ran in https://github.com/DIRACGrid/DIRAC/actions/runs/1755279881

Failed:

• integration
  cherry-pick a0651b3 into integration failed
  check merge conflicts on a local copy of this repository

  git fetch upstream
  git checkout upstream/integration -b cherry-pick-2-a0651b3e9-integration
  git cherry-pick -x -m 1 a0651b3e9
  # Fix the conflicts
  git cherry-pick --continue
  git commit --amend -m 'sweep: #5760 added a flag for disabling the use of SecurityLogging service' --author='fstagni <federico.stagni@cern.ch>'
  git push -u origin cherry-pick-2-a0651b3e9-integration
  
  # If you have the GitHub CLI installed the PR can be made with
  gh pr create \
       --label 'sweep:from rel-v7r3' \
       --base integration \
       --repo DIRACGrid/DIRAC \
       --title '[sweep:integration] added a flag for disabling the use of SecurityLogging service' \
       --body 'Sweep #5760 `added a flag for disabling the use of SecurityLogging service` to `integration`.
  
  Adding original author @fstagni as watcher.
  
  BEGINRELEASENOTES
  
  *Framework
  CHANGE: added a flag for disabling the use of SecurityLogging service
  
  ENDRELEASENOTES
  Closes #1115939097'