Add logic to send a ping to connector's backend every loop #2105
Conversation
| data_source.validate_config_fields() | ||
| await data_source.validate_config() | ||
|
|
||
| connector.log_debug("Pinging the backend") | ||
| await data_source.ping() |
There was a problem hiding this comment.
Should we try/catch for a NotImplementedError? And just skip past it if it's not implemented? https://github.com/elastic/connectors/blob/main/connectors/source.py#L557-L562
There was a problem hiding this comment.
Do we actually have data sources, which do not implement ping? Otherwise I would either keep it like that or log a clear error that ping is missing, if we expect it from every data source.
There was a problem hiding this comment.
We currently have no sources that do not implement ping.
I copied the logic from https://github.com/elastic/connectors/blob/main/connectors/sync_job_runner.py#L129-L134:
self.sync_job.log_debug("Validating configuration")
self.data_provider.validate_config_fields()
await self.data_provider.validate_config()
self.sync_job.log_debug("Pinging the backend")
await self.data_provider.ping()I'd like to keep logic similar between both places, then if needed refactor and introduce NotImplementedError handling.
Co-authored-by: Sean Story <sean.j.story@gmail.com>
artem-shelkovnikov
deleted the
artem/add-job-scheduling-ping-every-loop
branch
💔 Failed to create backport PR(s)The backport operation could not be completed due to the following error: The backport PRs will be merged automatically after passing CI. To backport manually run: |
Part of https://github.com/elastic/enterprise-search-team/issues/5678
Additionally to running RCF validation (see #2093) every time we check connector for whether the job should be scheduled, now we send a ping to the backend of this connector as well to verify, that it's available.
Before this logic would happen only before sync, which is too late.
Checklists
Pre-Review Checklist
v7.13.2,v7.14.0,v8.0.0)Related Pull Requests
Release Note
Now configurations checks health of 3rd-party system periodically and reports errors to Kibana. Previously it was only done before the sync is started.