missing logs #2272
Comments
|
Hello, we need more information to help you. Which client are you using to push log ? what configuration are you using ? Can you activate debug log for Loki and verify if anything stands out ? Which version are you using ? Thanks ! |
|
I have 3 hosts: A, B, C ;the deploy architecture is :
I built loki and promtail from master branch(version master-e400b4b (branch: master, revision: e400b4b1): config: positions: clients:
scrape_configs:
Besides, it only happens when the log is big. If I remove the ./text file context from the shell script, everything is fine. |
|
This architecture is neither the one we provide with helm nor tanka. You should start with those first. See if the issue persist. |
|
By redirecting all entries of promtail/distributor/ingester to local files, I gradually narrowed the bug down to
` ... So, I test with ` var line = " type Case struct { func TestLargeBlock(t *testing.T) { } ` I don't know what to do next, I was wondering if you have any suggestion. |
|
Okay, I found where the bug is, in `// moveNext moves the buffer to the next entry } In the for loop, should change the if clause to |
|
Nice found ! I think you’re right, you’re unlucky cause this code triggers with only very long line. I’ll try to fix it today. |
cyriltovena
added
keepalive
Funny enough this was triggered only with gzip and only with a certain size. But thanks for investigation here grafana#2272 I was able to reproduce this with a test and indeed this was incorrect. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
|
Interesting. I digged a little into it, and found that gzip's |
Funny enough this was triggered only with gzip and only with a certain size. But thanks for investigation here #2272 I was able to reproduce this with a test and indeed this was incorrect. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
|
Yes I was able to reproduce only with GZIP and not always. Anyway that's seems to be fixed now. Btw @quchenyuan if you have a lot of logs and Loki is not performing as fast as you want you can switch to snappy. You'll trade a bit of compression ratio but at least it will be way faster. Depends on your use case though gzip is 10x compression ratio and snappy is 5x. |
|
@cyriltovena Thanks! This looks great to me. |
I test with a pod runs the following shell script:
`
#! /bin/bash
i=0
while true
do
d=$(date "+%Y-%m-%d %H:%M:%S.%N")
printf "%-10d %20s %-10s %s\n" $i "$d" "---" "$(cat ./text)"
sleep 0.1;
((i++));
done
`
./text is a file with a single line, which has a length of 25214
I query with logcli and found that some lines are missing:
`
2020-06-29T17:37:09+08:00 2020-06-29T17:37:09.788688333+08:00 stdout F 9 2020-06-29 09:37:09. ---
2020-06-29T17:37:09+08:00 2020-06-29T17:37:09.99691343+08:00 stdout F 11 2020-06-29 09:37:09. ---
2020-06-29T17:37:10+08:00 2020-06-29T17:37:10.100728939+08:00 stdout F 12 2020-06-29 09:37:10. ---
2020-06-29T17:37:10+08:00 2020-06-29T17:37:10.829034129+08:00 stdout F 19 2020-06-29 09:37:10. ---
2020-06-29T17:37:10+08:00 2020-06-29T17:37:10.932854453+08:00 stdout F 20 2020-06-29 09:37:10. ---
2020-06-29T17:37:11+08:00 2020-06-29T17:37:11.140982502+08:00 stdout F 22 2020-06-29 09:37:11. ---
`
and I checked the original log files, the number sequences are continuous.
Besides, no errors or exceptions are found in promtail logs.
Does anyone have a clue with this?
Originally posted by @quchenyuan in #246 (comment)
The text was updated successfully, but these errors were encountered: