A script for reproducing some out-or-order peculiarities in loki, and a tiny flask app for making sure that the script works correctly.
The guys at Loki eventually figured out that the out of order stuff was caused by a hash collision, and this should be fixed as of the latest versions of Loki. More info here.
-
Clone this repo
-
Install requirements
your-favourite-package-manager install snappy
pip3 install -r requirements.txt
On macOS, you might encounter some errors with installing python-snappy - this stackoverflow answer might help you in that case!
- Run loki locally
git clone git@github.com:grafana/loki.git
cd loki/procution
docker-compose pull
docker-compose up
- Run lokitoy
python3 lokitoy.py
- Watch the world burn
CHUNK: {(...)}, RESULT: <Response [400]>, RESULT TEXT: entry out of order for stream: {app="m", uniq0="0", uniq1="0"}
Lokitoy now has some command line options! Try python3 lokitoy.py --help
to see them.
-
Make sure that you've installed the requirements
-
Set the
FLASK_APP
environment variable to point tolokimock.py
export FLASK_APP=lokimock.py
- Run lokimock with flask
flask run
- Run lokitoy with the
--mock
flag
python3 lokitoy.py --mock
- Watch the world not burn
You can see that lokimock catches out-of-orders as it should by telling lokitoy to deliberately send stuff out-of-order:
python3 lokitoy.py --mock --deliberate-ooo
Check out the documentation comments in lokitoy.py - they are quite extensive!
- Using multiple shared labels of the form
"sharedN": ["random", "words", "here"]
doesn't seem to result in errors, but including"app": ['h', 'i', 'j', 'k', 'l', 'm']
in addition to those results in errors. - If you use
['a', 'b', 'c', 'd', 'e', 'f']
,["foo", "bar", "baz", "beep", "boop", "boom"]
or["lokitest"]
as the selection for the"app"
shared label, the errors don't seem to happen, but when you use['h', 'i', 'j', 'k', 'l', 'm']
, the errors do happen - If you rename the
"app"
shared label to"test"
, there are no errors