-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
missing logs #2272
Comments
Hello, we need more information to help you. Which client are you using to push log ? what configuration are you using ? Can you activate debug log for Loki and verify if anything stands out ? Which version are you using ? Thanks ! |
I have 3 hosts: A, B, C ;the deploy architecture is : I built loki and promtail from master branch(version master-e400b4b (branch: master, revision: e400b4b1): config: positions: clients:
scrape_configs:
Besides, it only happens when the log is big. If I remove the ./text file context from the shell script, everything is fine. |
This architecture is neither the one we provide with helm nor tanka. You should start with those first. See if the issue persist. |
By redirecting all entries of promtail/distributor/ingester to local files, I gradually narrowed the bug down to
`
... So, I test with ` var line = " type Case struct { func TestLargeBlock(t *testing.T) {
} ` I don't know what to do next, I was wondering if you have any suggestion. |
Okay, I found where the bug is, in `// moveNext moves the buffer to the next entry
} In the for loop, should change the if clause to |
Nice found ! I think you’re right, you’re unlucky cause this code triggers with only very long line. I’ll try to fix it today. |
Funny enough this was triggered only with gzip and only with a certain size. But thanks for investigation here grafana#2272 I was able to reproduce this with a test and indeed this was incorrect. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Interesting. I digged a little into it, and found that gzip's |
Funny enough this was triggered only with gzip and only with a certain size. But thanks for investigation here #2272 I was able to reproduce this with a test and indeed this was incorrect. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Yes I was able to reproduce only with GZIP and not always. Anyway that's seems to be fixed now. Btw @quchenyuan if you have a lot of logs and Loki is not performing as fast as you want you can switch to snappy. You'll trade a bit of compression ratio but at least it will be way faster. Depends on your use case though gzip is 10x compression ratio and snappy is 5x. |
@cyriltovena Thanks! This looks great to me. |
I test with a pod runs the following shell script:
`
#! /bin/bash
i=0
while true
do
d=$(date "+%Y-%m-%d %H:%M:%S.%N")
printf "%-10d %20s %-10s %s\n" $i "$d" "---" "$(cat ./text)"
sleep 0.1;
((i++));
done
`
./text is a file with a single line, which has a length of 25214
I query with logcli and found that some lines are missing:
`
2020-06-29T17:37:09+08:00 2020-06-29T17:37:09.788688333+08:00 stdout F 9 2020-06-29 09:37:09. ---
2020-06-29T17:37:09+08:00 2020-06-29T17:37:09.99691343+08:00 stdout F 11 2020-06-29 09:37:09. ---
2020-06-29T17:37:10+08:00 2020-06-29T17:37:10.100728939+08:00 stdout F 12 2020-06-29 09:37:10. ---
2020-06-29T17:37:10+08:00 2020-06-29T17:37:10.829034129+08:00 stdout F 19 2020-06-29 09:37:10. ---
2020-06-29T17:37:10+08:00 2020-06-29T17:37:10.932854453+08:00 stdout F 20 2020-06-29 09:37:10. ---
2020-06-29T17:37:11+08:00 2020-06-29T17:37:11.140982502+08:00 stdout F 22 2020-06-29 09:37:11. ---
`
and I checked the original log files, the number sequences are continuous.
Besides, no errors or exceptions are found in promtail logs.
Does anyone have a clue with this?
Originally posted by @quchenyuan in #246 (comment)
The text was updated successfully, but these errors were encountered: