Carbonlink cache issues

[ajm-dev]

Hi There

I get the following msg when using graphite-web with go-carbon. It seems like sometimes it does not read the cached metrics in memory correctly but when we refresh the metrics returns.

See graphite-web cache.log below.

CarbonLink sending request for go-carbon.persister.committedPoints to ('127.0.0.1', 'a')
2019-12-02,12:32:51.666 :: Exception getting data from cache ('127.0.0.1', 'a'): [Errno 104] Connection reset by peer
2019-12-02,12:32:51.666 :: CarbonLink cache-query request for go-carbon.persister.committedPoints returned 0 datapoints

Do I need to make any changes on graphite-web? or is this issue related to go-carbon settings.

Below is my go-carbon.conf settings:
[common]
user = "carbon"
graph-prefix = "go-carbon"
metric-endpoint = "tcp://127.0.0.1:2003"
metric-interval = "0m10s"
max-cpu = 12

[whisper]
data-dir = "/data01/whisper/"
schemas-file = "/etc/go-carbon/storage-schemas.conf"
aggregation-file = "/etc/go-carbon/storage-aggregation.conf"
workers = 4
max-updates-per-second = 2500
max-creates-per-second = 100
hard-max-creates-per-second = false
sparse-create = false
flock = true
enabled = true
hash-filenames = true

[cache]
max-size = 900000000
write-strategy = "noop"

[udp]
listen = ":2003"
enabled = true
buffer-size = 0

[tcp]
listen = ":2003"
enabled = true
buffer-size = 0

[pickle]
listen = ":2004"
max-message-size = 67108864
enabled = true
buffer-size = 0

[carbonlink]
listen = "127.0.0.1:7002"
enabled = true
read-timeout = "30s"

[grpc]
listen = ":7003"
enabled = true

[tags]
enabled = false
tagdb-url = "http://127.0.0.1:8000"
tagdb-chunk-size = 32
tagdb-update-interval = 100
local-dir = "/var/lib/graphite/tagging/"
tagdb-timeout = "1s"

[carbonserver]
listen = ":8080"
enabled = false
buckets = 10
metrics-as-counters = false
read-timeout = "60s"
write-timeout = "60s"
query-cache-enabled = true
query-cache-size-mb = 0
find-cache-enabled = true
trigram-index = true
scan-frequency = "5m0s"
max-globs = 100
fail-on-max-globs = false
graphite-web-10-strict-mode = true
internal-stats-dir = ""
stats-percentiles = [99, 98, 95, 75, 50]

[dump]
enabled = false
path = "/var/lib/graphite/dump/"
restore-per-second = 0

[pprof]
listen = "localhost:7007"
enabled = false

[[logging]]
logger = ""
file = "/var/log/go-carbon/go-carbon.log"
level = "error"
encoding = "mixed"
encoding-time = "iso8601"
encoding-duration = "seconds"

Below is grafana graphs of the metrics that is missing and when we refresh it returns.

[ajm-dev]

Hi Alikhan Currently we are running Python 2.7.5 Should we try running it on version 3.0+ ? Regards

…

On Tue, Apr 21, 2020 at 12:03 AM Alikhan ***@***.***> wrote: I hope my reply is not too late. Is graphite-web running in python 3.0+? Because I am seeing similar issue with cache not being read using carbonlink. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#326 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AN5YTCSWODGP6ZKRPQFKUJTRNTBEJANCNFSM4JTVFGLQ> .

-- Andries Malan 0824513047

[alikhtag]

Ah there is an issue with Python 3.0+ Carbonlink (EDIT: PR #340 )
So using Python 2.7 should be fine.

[azhiltsov]

Can you confirm that go-carbon 0fdd9e5 fixed it?

[alikhtag]

Can you confirm that go-carbon 0fdd9e5 fixed it?

The PR #340 should have fixed python 3 issue only, if py2.7 was used I doubt the issue was fixed :/

[ajm-dev]

Hi There We are using py2.7 like Alik mentioned still waiting for a fix. Regards

…

On Sun, Jun 7, 2020 at 5:57 PM Alik ***@***.***> wrote: Can you confirm that go-carbon 0fdd9e5 <0fdd9e5> fixed it? The PR should have fixed python 3 issue only, if py2.7 was used I doubt the issue was fixed :/ — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#326 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AN5YTCW2HPKQ2WA47KY4K33RVO2GVANCNFSM4JTVFGLQ> .

-- Andries Malan 0824513047

[deniszh]

@ajm-dev :
I can't reproduce that on python 3 btw (with PR #340). But I didn't use carbonlink massively. You can try to disable carbonlink and enable carbonserver (then use CLUSTER_SERVERS=["127.0.0.1:8080"])

[piotr1212]

Not sure if this is related at all with the pickle version.

I've run into a similar issues a long time ago which was caused by go-carbon closing inactive connections. Fixed it by enabling tcp_keepalive in the os. Never bothered to create a PR to enable keepalive in go-carbon. This was a long time ago, maybe it is improved by now.