Received failure on HTTP shipping (400): error at least one label pair is required per stream. n log events will be dropped

[poostwoud]

Hi,

I'm running the following configuration:

{
	"Serilog": {
		"Using": [
			"Serilog.Sinks.Grafana.Loki"
		],
		"WriteTo": [
			{
				"Name": "GrafanaLoki",
				"Args": {
					"uri": "http://localhost:3100",
					"outputTemplate": "{Timestamp:dd-MM-yyyy HH:mm:ss} [{Level:u3}] [{ThreadId}] {Message}{NewLine}{Exception}",
					"restrictedToMinimumLevel": "Debug",
					"filtrationMode": "Include",
					"filtrationLabels": [
						"Application"
					],
					"textFormatter": "Serilog.Sinks.Grafana.Loki.LokiJsonTextFormatter, Serilog.Sinks.Grafana.Loki"
				}
			}
		],
		"Enrich": [
			"FromLogContext",
			"WithEnvironmentName",
			"WithExceptionDetails"
		],
		"Destructure": [
			{
				"Name": "ToMaximumDepth",
				"Args": {
					"maximumDestructuringDepth": 5
				}
			},
			{
				"Name": "ToMaximumStringLength",
				"Args": {
					"maximumStringLength": 8192
				}
			},
			{
				"Name": "ToMaximumCollectionCount",
				"Args": {
					"maximumCollectionCount": 16
				}
			}
		],
		"MinimumLevel": {
			"Default": "Debug"
		}
	},
	"Properties": {
		"Application": "MyService"
	}
}

This results in Received failure on HTTP shipping (400): error at least one label pair is required per stream. n log events will be dropped.

Then when I'm adding the following args to the Loki section:

      {
        "Name": "GrafanaLoki",
        "Args": {
          // ...
          "labels": [
            {
              "key": "Application",
              "value": "MyService" 
            }
          ],
          // ...
      }

it does work. However according to https://github.com/serilog/serilog-settings-configuration the Properties section is a supported section.

The same issue occurs when using the enviroment enricher https://github.com/serilog/serilog-enrichers-environment.

	{
		"Name": "GrafanaLoki",
		"Args": {
			// ...
			"filtrationLabels": [
				"EnvironmentName",
				"Application"
			],
			// ...
		}
	}

The EnvironmentName label doesn't get added.

I've been pushing them to an internal endpoint to see what happens:

public class LokiController : ControllerBase
{
    [HttpPost("loki/api/v1/push")]
    [AllowAnonymous]
    public IActionResult Do([FromBody] JObject body)
    {
        var json = JsonConvert.SerializeObject(body, new JsonSerializerSettings
        {
            ContractResolver = new CamelCasePropertyNamesContractResolver()
        });
        return BadRequest();
    }
}

Example output:

{
	"streams": [
		{
			"stream": {}, // Nothing there ???
			"values": [
				[
					"1673988513748000000",
					"{ ... }"
				],
				[
					"1673988513819000000",
					"{ ... }"
				]			
			]
		}
	]
}

My service:

• .NET Core 3.1
• Serilog 2.12.0
• Serilog.AspNetCore 6.1.0
• Serilog.Enrichers.Environment 2.2.0
• Serilog.Exceptions 8.4.0
• Serilog.Sinks.Grafana.Loki 8.1.0

Any thoughts?

Cheers,

Peter

[mishamyte]

Hi @poostwoud,

At the first sight, seems like you have a misconfiguration. Because filtrationMode and filtrationLabels are deprecated in v8.
I recommend to read #99 and recheck the configuration doc and samples

[mishamyte]

I will quote the important part of the reference for your situation:

💡 Mapping log event properties to labels
Label is one of most important thinks in Loki. It is a key-value pair, used by Loki to organize events. The main use case of label is querying and filtering the data. By according to internals of Loki, incorrect use of labels led to high cardinality problem. And there are some best practices for labels.

First public version (v3) of this sink has only possibility to exclude some properties from mapping to the labels. This wasn't a good solution, cause you needed to add all exclusion rules explicitly, what becomes a hell with a huge amount of labels (especially in cases with external libs.

We have changed this behavior in v4. It brought us a filtrationMode and filtrationLabels, where you could select one of modes: Include/Exclude and define properties' names to map 'em to the labels. Or you could leave it blank and in this case you will face the high cardinality problem. We always recommended to use Include mode with some always present labels (such a app name). But in last #81 (comment) we have found that this behavior is straightforward, but has no sense.

From v8 we propose you a clear way - you define only properties that should be mapped into a labels (or none if you don't need it). For this case use a new propertiesAsLabels configuration parameters. The rest of properties would be passed to a formatted and should be included as a part of log entry.

[mishamyte]

I think this config file from the sample could be useful for you

[poostwoud]

@mishamyte 🤦‍♂️ thanks I missed the fact filtrationMode and filtrationLabels are deprecated in v8. Using propertiesAsLabels now, works great. Thanks 👍 !

[mishamyte]

I was happy to help you!)

You're welcome)