Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

http stream collector: retry on regular connection problems? #1435

Closed
ghost opened this issue Aug 14, 2019 · 3 comments · Fixed by #1747
Closed

http stream collector: retry on regular connection problems? #1435

ghost opened this issue Aug 14, 2019 · 3 comments · Fixed by #1747
Assignees
@waldbauer-certat
Labels
component: bots feature Indicates new feature requests or new features
Milestone
2.3.0

Comments

@ghost
Copy link

ghost commented Aug 14, 2019

http streams have a different error behavior than typical http connections. They can't timeout easily, but get interrupted which requires a restart. That happens often, but in case the error is mitigated easily, we do not need administrator's attention - an error.

This is a list of some errors happening in the last days for one of our bots

      2 requests.exceptions.ConnectionError: HTTPSConnectionPool(host='[...]', port=443): Read timed out.
      2 requests.exceptions.ProxyError: HTTPSConnectionPool(host='[...]', port=443): Max retries exceeded with url: [...] (Caused by ProxyError('Cannot connect to proxy.', timeout('timed out',)))
      5 requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(505 bytes read, 7 more expected)', IncompleteRead(505 bytes read, 7 more expected))
      6 requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(503 bytes read, 9 more expected)', IncompleteRead(503 bytes read, 9 more expected))
      9 requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='[...]', port=443): Read timed out. (read timeout=30)
     19 requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(504 bytes read, 8 more expected)', IncompleteRead(504 bytes read, 8 more expected))
     68 requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

Maybe some of them can be treated like timeouts in urllib3's Retry class to be used in the HTTPAdapter
@ghost ghost added the component: bots label Aug 14, 2019
@ghost ghost added this to the 2.0.1 milestone Aug 14, 2019
@ghost
Copy link
Author

ghost commented Aug 22, 2019

With 4262916 the retries are handled by requests itself, maybe that fixes the requests.exceptions.ReadTimeout exceptions.

@ghost ghost modified the milestones: 2.0.1, 2.0.2 Aug 28, 2019
@ghost ghost modified the milestones: 2.1.1, 2.1.2 Nov 13, 2019
@ghost ghost modified the milestones: 2.1.2, 2.1.3 Jan 28, 2020
ghost pushed a commit that referenced this issue Jan 28, 2020
Merge tag '2.1.2'
f9c782d 
2.1.2

 ### Core
- `__init__`: Resolve absolute path for `STATE_FILE_PATH` variable (resolves `..`).
- `intelmq.lib.utils`:
  - log: Do not raise an exception if logging to neither file nor syslog is requested.
  - logging StreamHandler: Colorize all warning and error messages red.
  - logging FileHandler: Strip all shell colorizations from the messages (#1436).
- `intelmq.lib.message`:
  - `Message.to_json`: Set `sort_keys=True` to get reproducible results.
  - `drop_privileges`: Handle situations where the user or group `intelmq` does not exist.
- `intelmq.lib.pipeline`:
  - `Amqp._send` and `Amqp._acknowledge`: Log traceback in debug mode in case of errors and necessary re-connections.
  - `Amqp._acknowledge`: Reset delivery tag if acknowledge was successful.

 ### Bots
 #### Collectors
- `intelmq.bots.collectors.misp.collector`:
  - Add compatibility with current pymisp versions and versions released after January 2020 (PR #1468).

 #### Parsers
- `intelmq.bots.parsers.shadowserver.config`: Add some missing fields for the feed `accessible-rdp` (#1463).
- `intelmq.bots.parsers.shadowserver.parser`:
  - Feed-detection based on file names: The prefixed date is optional now.
  - Feed-detection based on file names: Re-detect feed for every report received (#1493).

 #### Experts
- `intelmq.bots.experts.national_cert_contact_certat`: Handle empty responses by server (#1467).
- `intelmq.bots.experts.maxmind_geoip`: The script `update-geoip-data` now requires a license key as second parameter because of upstream changes (#1484)).

 #### Outputs
- `intelmq.bots.outputs.restapi.output`: Fix logging of response body if response status code was not ok.

 ### Documentation
- Remove some hardcoded `/opt/intelmq/` paths from code comments and program outputs.

 ### Packaging
- debian/rules: Only replace `/opt/intelmq/` with LSB-paths in some certain files, not the whole tree, avoiding wrong replacements.
- debian/rules and debian/intelmq.install: Do install the examples configuration directly instead of working around the abandoned examples directory.

 ### Tests
- `lib/test_utils`: Skip some tests on Python 3.4 because `contextlib.redirect_stdout` and `contextlib.redirect_sterr` are not supported on this version.
- Travis: Stop running tests with all optional dependencies on Python 3.4, as more and more libraries are dropping support for it. Tests on the core and code without non-optional requirements are not affected.
- `tests.bots.parsers.html_table`: Make tests independent of current year.

 ### Tools
- `intelmqctl upgrade-config`: Fix missing substitution in error message "State file %r is not writable.".

 ### Known issues
- bots trapped in endless loop if decoding of raw message fails (#1494)
- intelmqctl status of processes: need to check bot id too (#1492)
- MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
2.1.2

 ### Core
- `__init__`: Resolve absolute path for `STATE_FILE_PATH` variable
(resolves `..`).
- `intelmq.lib.utils`:
  - log: Do not raise an exception if logging to neither file nor syslog
is requested.
  - logging StreamHandler: Colorize all warning and error messages red.
  - logging FileHandler: Strip all shell colorizations from the messages
(#1436).
- `intelmq.lib.message`:
  - `Message.to_json`: Set `sort_keys=True` to get reproducible results.
  - `drop_privileges`: Handle situations where the user or group
`intelmq` does not exist.
- `intelmq.lib.pipeline`:
  - `Amqp._send` and `Amqp._acknowledge`: Log traceback in debug mode in
case of errors and necessary re-connections.
  - `Amqp._acknowledge`: Reset delivery tag if acknowledge was
successful.

 ### Bots
 #### Collectors
- `intelmq.bots.collectors.misp.collector`:
  - Add compatibility with current pymisp versions and versions released
after January 2020 (PR #1468).

 #### Parsers
- `intelmq.bots.parsers.shadowserver.config`: Add some missing fields
for the feed `accessible-rdp` (#1463).
- `intelmq.bots.parsers.shadowserver.parser`:
  - Feed-detection based on file names: The prefixed date is optional
now.
  - Feed-detection based on file names: Re-detect feed for every report
received (#1493).

 #### Experts
- `intelmq.bots.experts.national_cert_contact_certat`: Handle empty
responses by server (#1467).
- `intelmq.bots.experts.maxmind_geoip`: The script `update-geoip-data`
now requires a license key as second parameter because of upstream
changes (#1484)).

 #### Outputs
- `intelmq.bots.outputs.restapi.output`: Fix logging of response body if
response status code was not ok.

 ### Documentation
- Remove some hardcoded `/opt/intelmq/` paths from code comments and
program outputs.

 ### Packaging
- debian/rules: Only replace `/opt/intelmq/` with LSB-paths in some
certain files, not the whole tree, avoiding wrong replacements.
- debian/rules and debian/intelmq.install: Do install the examples
configuration directly instead of working around the abandoned examples
directory.

 ### Tests
- `lib/test_utils`: Skip some tests on Python 3.4 because
`contextlib.redirect_stdout` and `contextlib.redirect_sterr` are not
supported on this version.
- Travis: Stop running tests with all optional dependencies on Python
3.4, as more and more libraries are dropping support for it. Tests on
the core and code without non-optional requirements are not affected.
- `tests.bots.parsers.html_table`: Make tests independent of current
year.

 ### Tools
- `intelmqctl upgrade-config`: Fix missing substitution in error message
"State file %r is not writable.".

 ### Known issues
- bots trapped in endless loop if decoding of raw message fails (#1494)
- intelmqctl status of processes: need to check bot id too (#1492)
- MongoDB authentication: compatibility on different MongoDB and pymongo
versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
@ghost ghost modified the milestones: 2.1.3, 2.2.0 May 26, 2020
@ghost ghost modified the milestones: 2.2.0, 2.3.0 Jun 17, 2020
@ghost
Copy link
Author

ghost commented Feb 4, 2021

I regularly see incomplete reads. Some unordered excerpts from the logs:

http.client.IncompleteRead: IncompleteRead(0 bytes read)
http.client.IncompleteRead: IncompleteRead(504 bytes read, 8 more expected)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(504 bytes read, 8 more expected)', IncompleteRead(504 bytes read, 8 more expected))
requests.packages.urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(504 bytes read, 8 more expected)', IncompleteRead(504 bytes read, 8 more expected))
requests.packages.urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

@ghost ghost assigned waldbauer-certat Feb 5, 2021
@ghost
Copy link
Author

ghost commented Feb 5, 2021

Idea: catch IncompleteRead exceptions up to error_max_retries times in a row, otherwise really raise it. Update documentation on this behavior. Errors can (should?) still be shown in lower logging levels (INFO?)

@waldbauer-certat waldbauer-certat mentioned this issue Feb 17, 2021
Merged
@ghost ghost self-assigned this Feb 18, 2021
@ghost ghost added the feature Indicates new feature requests or new features label Feb 18, 2021
@ghost ghost closed this as completed in 9de39ee Feb 22, 2021
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@waldbauer-certat waldbauer-certat

Labels
component: bots feature Indicates new feature requests or new features
Projects
None yet
Milestone
2.3.0
Development

Successfully merging a pull request may close this issue.

[FIX] HTTP Stream Collector better error handling certtools/intelmq
1 participant
@waldbauer-certat