Improve retry when influxdb backend is down

[anwar6953]

It seems like InfluxDB Cloud on GCP[1] was down sometime between 14:16 to 15:12 PDT (potentially even broader than this). I wasn't able to use the InfluxDB dashboard to view the metrics (from my home network) and on several of my applications where I am trying out InfluxDB from (running on multiple different GCP VMs), I also got Connection timed out [1]. When the backend comes back up, ideally the client should retry. Instead, I see a dip in the dashboard for a certain period[3], although I am confident that the metrics client was being called to write the metrics.
I did not restart my system, yet writes after this influxDB downtime are visible in the dashboard.
If retry is already implemented, how can I configure the retry strategy to have more retries over longer period of time?

[1] My host is https://us-central1-1.gcp.cloud2.influxdata.com/.
[2] The batch item wasn't processed successfully because: HTTPSConnectionPool(host='us-central1-1.gcp.cloud2.influxdata.com', port=443): Max retries exceeded with url: /api/v2/write?org=<REDACTED>&bucket=<REDACTED>+Bucket&precision=ns (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa44a34b6d8>: Failed to establish a new connection: [Errno 110] Connection timed out'))
[3]

[bednar]

Hi @anwar6953,

thanks for using our client.

The PR #140 brings fully configurable retry strategy. The retry will be configurable by these options: retry_interval, max_retries and max_retry_delay.

For more detail see - https://github.com/influxdata/influxdb-client-python/blob/60676599d6e5b15d1581170575542b1f53783a14/README.rst#batching.

The #140 comes in version 1.10.0 which will be released probably this Friday.

If you would like to use a dev version then install client via:

pip install git+https://github.com/influxdata/influxdb-client-python.git@feat/retry

Regards

[anwar6953]

Thanks, @bednar. I'm curious what the behavior is now, for when there is a write failure, using async. Does the client drop the metrics and move on? Or just keep them in memory still?

[bednar]

• If a response from InfluxDB has status code 429 or 503 the client try it again after retry_interval (https://github.com/influxdata/influxdb-client-python#batching).
• If there is a connection error the client try it again immediately (total 3 attempts)
• After that the metrics are dropped and client move on

[anwar6953]

If there is a connection error, shouldn't it also use the same retry strategy (including retry_interval), instead of limiting to 3 attempts?

[bednar]

Yeah, definitely. It is fixed in #140.

[bednar]

Hi @anwar6953,

The PR #140 is fixed in 1.10.0 milestone.

If you would like to use a dev version then install client via:

pip install git+https://github.com/influxdata/influxdb-client-python.git@master

Regards