tests drop privileges too late, e.g. causing TestMISPFeedOutputBot failure

[bernhardreiter]

Trying to run the test for the bots/outputs/misp/output_feed.py which fails on me.

/opt/dev_intelmq# INTELMQ_TEST_EXOTIC=1 python3 -m unittest intelmq.tests.bots.outputs.misp.test_output_feed -v
test_bot_name (intelmq.tests.bots.outputs.misp.test_output_feed.TestMISPFeedOutputBot) ... ok
test_event (intelmq.tests.bots.outputs.misp.test_output_feed.TestMISPFeedOutputBot) ... ERROR
ERROR

======================================================================
ERROR: test_event (intelmq.tests.bots.outputs.misp.test_output_feed.TestMISPFeedOutputBot)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/dev_intelmq/intelmq/tests/bots/outputs/misp/test_output_feed.py", line 50, in test_event
    self.run_bot()
  File "/opt/dev_intelmq/intelmq/lib/test.py", line 267, in run_bot
    self.prepare_bot(parameters=parameters)
  File "/opt/dev_intelmq/intelmq/lib/test.py", line 234, in prepare_bot
    self.bot = self.bot_reference(self.bot_id)
  File "/opt/dev_intelmq/intelmq/lib/bot.py", line 1179, in __init__
    super().__init__(bot_id=bot_id)
  File "/opt/dev_intelmq/intelmq/lib/bot.py", line 158, in __init__
    self.init()
  File "/opt/dev_intelmq/intelmq/bots/outputs/misp/output_feed.py", line 51, in init
    if (self.output_dir / '.current').exists():
  File "/usr/lib/python3.7/pathlib.py", line 1339, in exists
    self.stat()
  File "/usr/lib/python3.7/pathlib.py", line 1161, in stat
    return self._accessor.stat(self)
PermissionError: [Errno 13] Permission denied: '/tmp/tmplgfpsol_/.current'

======================================================================
ERROR: tearDownClass (intelmq.tests.bots.outputs.misp.test_output_feed.TestMISPFeedOutputBot)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/dev_intelmq/intelmq/tests/bots/outputs/misp/test_output_feed.py", line 54, in tearDownClass
    cls.directory.cleanup()
  File "/usr/lib/python3.7/tempfile.py", line 948, in cleanup
    _rmtree(self.name)
  File "/usr/lib/python3.7/shutil.py", line 487, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/usr/lib/python3.7/shutil.py", line 485, in rmtree
    fd = os.open(path, os.O_RDONLY)
PermissionError: [Errno 13] Permission denied: '/tmp/tmplgfpsol_'

----------------------------------------------------------------------
Ran 2 tests in 0.007s

FAILED (errors=2)
root@vm1:/opt/dev_intelmq# ls -al /tmp/tmplgfpsol_
total 8
drwx------  2 root root 4096 Jan 15 10:02 .
drwxrwxrwt 12 root root 4096 Jan 15 10:02 ..

Tested with
bb205e9 (HEAD -> develop, origin/develop, origin/HEAD)
On Debian Buster with PyMISP 2e7215bbec6c2fa1d527e09be99d4280fdda3fd1

@Rafiot

[Rafiot]

Well, the intelmq user isn't allowed to write in that directory, not sure what else I can tell you there ;)

[bernhardreiter]

@Rafiot Thanks for responding and for the hint!

Note that I had called it as root (as suggested by https://github.com/certtools/intelmq/blob/develop/docs/Developers-Guide.md#run-the-tests ) and and the directory is read and writable by root. So there is no obvious reason that I could see.

I did not analyse further though.

I also don't know if the test has problems in other settings, because the current continuous build does not run the tests that need "exotic" libraries if I remember correctly. I'll take it that the test runs fine for you, that's good.

Detail, its Python 3.7.3

[bernhardreiter]

Okay, the test runner switches to user intelmq, but only after creating the tmpdirectory.
That is a new feature to be introduced by 2.1.2.

@wagner-certat : What's the rationale behind dropping priviledges, wouldn't it be better to fail loudly and stop hard in the case of anything of intelmq gettings started by root?
(Could not find it in the relevant commit 853ba5b )

At first thought I'd say that failing hard is better in this situation.
(And administrators can always create an intelmq user.)

[bernhardreiter]

When dropping priviledges earlier, the test start but one fails see below

--- a/intelmq/lib/test.py
+++ b/intelmq/lib/test.py
@@ -129,6 +129,11 @@ def setUpClass(cls):
         """
         Set default values and save original functions.
         """
+        if not utils.drop_privileges():
+            logger.error('Running intelmqctl as root is highly discouraged!')
+            sys.exit(1)
+            
+
         cls.bot_id = 'test-bot'
         cls.bot_name = None
         cls.bot = None

INTELMQ_TEST_EXOTIC=1 python3 -m unittest intelmq.tests.bots.outputs.misp.test_output_feed 
.F
======================================================================
FAIL: test_event (intelmq.tests.bots.outputs.misp.test_output_feed.TestMISPFeedOutputBot)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/dev_intelmq/intelmq/tests/bots/outputs/misp/test_output_feed.py", line 50, in test_event
    self.run_bot()
  File "/opt/dev_intelmq/intelmq/lib/test.py", line 316, in run_bot
    self.assertNotRegexpMatchesLog("(ERROR.*?){%d}" % (self.allowed_error_count + 1))
  File "/opt/dev_intelmq/intelmq/lib/test.py", line 480, in assertNotRegexpMatchesLog
    self.assertNotRegex(self.loglines_buffer, pattern)
AssertionError: Regex matched: 'ERROR' matches '(ERROR.*?){1}' in '2020-01-15 12:35:58,864 - test-bot - INFO - MISPFeedOutputBot initialized with id test-bot and intelmq 2.2.0.alpha.1 and python 3.7.3 (default, Apr  3 2019, 05:39:12) as process 1575.\n2020-01-15 12:35:58,864 - test-bot - INFO - Bot is starting.\n2020-01-15 12:35:58,865 - test-bot - INFO - Bot initialization completed.\n2020-01-15 12:35:58,872 - test-bot - ERROR - Bot has found a problem.\nTraceback (most recent call last):\n  File "/opt/dev_intelmq/intelmq/lib/bot.py", line 267, in start\n    self.process()\n  File "/opt/dev_intelmq/intelmq/bots/outputs/misp/output_feed.py", line 102, in process\n    feed_output = self.current_event.to_feed(with_meta=False)\n  File "/usr/local/lib/python3.7/dist-packages/pymisp/mispevent.py", line 833, in to_feed\n    to_return = super()._to_feed()\n  File "/usr/local/lib/python3.7/dist-packages/pymisp/abstract.py", line 209, in _to_feed\n    raise PyMISPError(\'Unable to export in the feed format, _fields_for_feed is missing.\')\npymisp.exceptions.PyMISPError: Unable to export in the feed format, _fields_for_feed is missing.\n2020-01-15 12:35:58,873 - test-bot - INFO - Bot stopped.\n'

----------------------------------------------------------------------
Ran 2 tests in 0.013s

FAILED (failures=1)

[bernhardreiter]

@Rafiot there seems to be something else wrong with the test (see last message)
pymisp.exceptions.PyMISPError: Unable to export in the feed format, _fields_for_feed is missing.

[ghost]

Why you want to run the tests with root privileges? Same applies for normal operation too.

And we drop root privileges mostly because of security reasons in case the user started intelmq as root. Further, the users' will get problems with file permissions for files (pid and log files) created as root when starting it properly.

[bernhardreiter]

@wagner-certat I don't want to run tests with root priviledges, I just missunderstood https://github.com/certtools/intelmq/blob/develop/docs/Developers-Guide.md#run-the-tests I was thinking it was suggesting it (because there is only a 'It may be necessary to switch the user'), then I've looked in the source and found the new drop_privileges feature. Next I found out that
it allows to continue to run when started as root and tests are not ready for this.

My suggestion is to

• make any intelmq starting fail hard with root priviledges (rrename, rework drop_priviledges)
• use the check also for tests
• improve phrasing in the Dev Guide

In addition it there seems to be an unrelated problem in TestMISPFeedOutputBot
which was discovered. (Only for the tested combination, which is both current master for intelmq and pymisp).

[ghost]

• intelmqctl will now also fail if dropping privs did not work
• The bots test cases will now automatically change to the intelmq user if root is detected. This prevents the situation which caused this discovery.
• Further, I rephrased the testing section in the dev guide.