Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MongoDB authentication: compatibility on different MongoDB and pymongo versions #1439

Closed
kalyparker opened this issue Aug 28, 2019 · 6 comments
Labels
bug Indicates an unexpected problem or unintended behavior component: bots
Milestone

Comments

@kalyparker
Copy link
Contributor

kalyparker commented Aug 28, 2019

Hi,

In the new version of intelmq available in the repositories, I faced an issue regarding the bot MongoDB.

        self.pymongo_3 = pymongo.version_tuple >= (3, )
        self.pymongo_35 = pymongo.version_tuple >= (3, 5)`

pymongo 3.5.1 is the last version supporting the old way to connect and the new.
You should change by.

        self.pymongo_3 = pymongo.version_tuple >= (3, )
        self.pymongo_35 = pymongo.version_tuple > (3, 5, 1)`
@ghost ghost self-assigned this Aug 28, 2019
@ghost ghost added bug Indicates an unexpected problem or unintended behavior component: bots labels Aug 28, 2019
@ghost ghost added this to the 2.0.2 milestone Aug 28, 2019
@ghost
Copy link

ghost commented Aug 28, 2019

Thanks for the report, I will adapt the version check.

In the new version of intelmq available in the repositories

Do you mean pip or deb/rpm?

@ghost
Copy link

ghost commented Aug 28, 2019

Looking at the source code of pymongo, the handling of username/password as kwargs looks the same. In the changelog, the change is documented for version 3.5

With any version bigger than or equal to 3.5, authentication with kwargs should be possible.

@kalyparker
Copy link
Contributor Author

kalyparker commented Aug 28, 2019

For the version, sorry, I means deb/rpm.

With any version bigger than or equal to 3.5, authentication with kwargs should be possible.

The problem is related to the version of the mongodb and not directly to pymongo.
In 3.5.1 this is the last version which support "old" mongodb < 3.4, and unfortunately these mongodb does not support this authentification mode.
I don't know if I'm the only one to notice that.

Anyway, force usage of this authentification mode based on the version of pymongo seems dangerous, maybe a parameter do the trick.

@ghost
Copy link

ghost commented Aug 29, 2019

With any version bigger than or equal to 3.5, authentication with kwargs should be possible.

The problem is related to the version of the mongodb and not directly to pymongo.
In 3.5.1 this is the last version which support "old" mongodb < 3.4, and unfortunately these mongodb does not support this authentification mode.
I don't know if I'm the only one to notice that.

Anyway, force usage of this authentification mode based on the version of pymongo seems dangerous, maybe a parameter do the trick.

Okay, so we do not only need to check for the pymongo version, but also for the server version with self.client.server_info()['version']? So something like (pseudo-code)

if self.pymongo_35 and self.username and self.server_version >= 3.4:
    self.client = (..., username=self.username, password=self.password)
else:
    self.client = (...)
db = self.client...
if self.usernameand (not self.pymongo_35 or self.server_version < 3.4):
    db.authenticate(...)

But Database.authenticate is deprecated too: https://api.mongodb.com/python/current/api/pymongo/database.html?highlight=authenticate#pymongo.database.Database.authenticate

@kalyparker
Copy link
Contributor Author

yes, could do the job.

@ghost ghost changed the title MongoDB and pymongo MongoDB authentication: compatibility on different MongoDB and pymongo versions Oct 14, 2019
@ghost ghost modified the milestones: 2.1.1, 2.1.2 Nov 13, 2019
@ghost ghost modified the milestones: 2.1.2, 2.1.3 Jan 28, 2020
ghost pushed a commit that referenced this issue Jan 28, 2020
2.1.2

 ### Core
- `__init__`: Resolve absolute path for `STATE_FILE_PATH` variable (resolves `..`).
- `intelmq.lib.utils`:
  - log: Do not raise an exception if logging to neither file nor syslog is requested.
  - logging StreamHandler: Colorize all warning and error messages red.
  - logging FileHandler: Strip all shell colorizations from the messages (#1436).
- `intelmq.lib.message`:
  - `Message.to_json`: Set `sort_keys=True` to get reproducible results.
  - `drop_privileges`: Handle situations where the user or group `intelmq` does not exist.
- `intelmq.lib.pipeline`:
  - `Amqp._send` and `Amqp._acknowledge`: Log traceback in debug mode in case of errors and necessary re-connections.
  - `Amqp._acknowledge`: Reset delivery tag if acknowledge was successful.

 ### Bots
 #### Collectors
- `intelmq.bots.collectors.misp.collector`:
  - Add compatibility with current pymisp versions and versions released after January 2020 (PR #1468).

 #### Parsers
- `intelmq.bots.parsers.shadowserver.config`: Add some missing fields for the feed `accessible-rdp` (#1463).
- `intelmq.bots.parsers.shadowserver.parser`:
  - Feed-detection based on file names: The prefixed date is optional now.
  - Feed-detection based on file names: Re-detect feed for every report received (#1493).

 #### Experts
- `intelmq.bots.experts.national_cert_contact_certat`: Handle empty responses by server (#1467).
- `intelmq.bots.experts.maxmind_geoip`: The script `update-geoip-data` now requires a license key as second parameter because of upstream changes (#1484)).

 #### Outputs
- `intelmq.bots.outputs.restapi.output`: Fix logging of response body if response status code was not ok.

 ### Documentation
- Remove some hardcoded `/opt/intelmq/` paths from code comments and program outputs.

 ### Packaging
- debian/rules: Only replace `/opt/intelmq/` with LSB-paths in some certain files, not the whole tree, avoiding wrong replacements.
- debian/rules and debian/intelmq.install: Do install the examples configuration directly instead of working around the abandoned examples directory.

 ### Tests
- `lib/test_utils`: Skip some tests on Python 3.4 because `contextlib.redirect_stdout` and `contextlib.redirect_sterr` are not supported on this version.
- Travis: Stop running tests with all optional dependencies on Python 3.4, as more and more libraries are dropping support for it. Tests on the core and code without non-optional requirements are not affected.
- `tests.bots.parsers.html_table`: Make tests independent of current year.

 ### Tools
- `intelmqctl upgrade-config`: Fix missing substitution in error message "State file %r is not writable.".

 ### Known issues
- bots trapped in endless loop if decoding of raw message fails (#1494)
- intelmqctl status of processes: need to check bot id too (#1492)
- MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
2.1.2

 ### Core
- `__init__`: Resolve absolute path for `STATE_FILE_PATH` variable
(resolves `..`).
- `intelmq.lib.utils`:
  - log: Do not raise an exception if logging to neither file nor syslog
is requested.
  - logging StreamHandler: Colorize all warning and error messages red.
  - logging FileHandler: Strip all shell colorizations from the messages
(#1436).
- `intelmq.lib.message`:
  - `Message.to_json`: Set `sort_keys=True` to get reproducible results.
  - `drop_privileges`: Handle situations where the user or group
`intelmq` does not exist.
- `intelmq.lib.pipeline`:
  - `Amqp._send` and `Amqp._acknowledge`: Log traceback in debug mode in
case of errors and necessary re-connections.
  - `Amqp._acknowledge`: Reset delivery tag if acknowledge was
successful.

 ### Bots
 #### Collectors
- `intelmq.bots.collectors.misp.collector`:
  - Add compatibility with current pymisp versions and versions released
after January 2020 (PR #1468).

 #### Parsers
- `intelmq.bots.parsers.shadowserver.config`: Add some missing fields
for the feed `accessible-rdp` (#1463).
- `intelmq.bots.parsers.shadowserver.parser`:
  - Feed-detection based on file names: The prefixed date is optional
now.
  - Feed-detection based on file names: Re-detect feed for every report
received (#1493).

 #### Experts
- `intelmq.bots.experts.national_cert_contact_certat`: Handle empty
responses by server (#1467).
- `intelmq.bots.experts.maxmind_geoip`: The script `update-geoip-data`
now requires a license key as second parameter because of upstream
changes (#1484)).

 #### Outputs
- `intelmq.bots.outputs.restapi.output`: Fix logging of response body if
response status code was not ok.

 ### Documentation
- Remove some hardcoded `/opt/intelmq/` paths from code comments and
program outputs.

 ### Packaging
- debian/rules: Only replace `/opt/intelmq/` with LSB-paths in some
certain files, not the whole tree, avoiding wrong replacements.
- debian/rules and debian/intelmq.install: Do install the examples
configuration directly instead of working around the abandoned examples
directory.

 ### Tests
- `lib/test_utils`: Skip some tests on Python 3.4 because
`contextlib.redirect_stdout` and `contextlib.redirect_sterr` are not
supported on this version.
- Travis: Stop running tests with all optional dependencies on Python
3.4, as more and more libraries are dropping support for it. Tests on
the core and code without non-optional requirements are not affected.
- `tests.bots.parsers.html_table`: Make tests independent of current
year.

 ### Tools
- `intelmqctl upgrade-config`: Fix missing substitution in error message
"State file %r is not writable.".

 ### Known issues
- bots trapped in endless loop if decoding of raw message fails (#1494)
- intelmqctl status of processes: need to check bot id too (#1492)
- MongoDB authentication: compatibility on different MongoDB and pymongo
versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
@ghost ghost closed this as completed in f0afacf Mar 27, 2020
@ghost
Copy link

ghost commented Mar 27, 2020

I applied the proposed fix. I only tested with two different server version, so if something else needs to be adapted, please speak up.

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Indicates an unexpected problem or unintended behavior component: bots
Projects
None yet
Development

No branches or pull requests

1 participant