4.9.0

AxoSyslog is binary-compatible with syslog-ng [1] and serves as a drop-in replacement.

Explore and learn more about the new features in our release announcement blog post.

We provide cloud-ready container images and Helm charts.

Packages are available for Debian and Ubuntu from our APT repository.
RPM packages are available in the Assets section (we’re working on an RPM repository as well, and hope to have it up and running for the next release).

FilterX (AxoSyslog's advanced parsing and filtering language) became a publicly available feature in AxoSyslog after the 4.8 release.
As it is currently under heavy development, FilterX related news entries can be found in separate sections.
Please note that although its syntax approaches its final form, it may break in subsequent releases.

Check out the AxoSyslog documentation for all the details.

Highlights

Sending data to ClickHouse

The new clickhouse() destination uses ClickHouse's gRPC
interface to insert logs.

Please note, that as of today, ClickHouse Cloud does not support
the gRPC interface. The clickhouse() destination is currently
only useful for self hosted ClickHouse servers.

If you would like to send logs to ClickHouse Cloud, gRPC support
can be requested from the ClickHouse Cloud team or a HTTP based
driver can be implemented in AxoSyslog.

Example config:

clickhouse(
  database("default")
  table("my_first_table")
  user("default")
  password("pw")
  schema(
    "user_id" UInt32 => $R_MSEC,
    "message" String => "$MSG",
    "timestamp" DateTime => "$R_UNIXTIME",
    "metric" Float32 => 3.14
  )
  workers(4)
  batch-lines(1000)
  batch-timeout(1000)
);

(#354)

Features

opentelemetry(), loki() destination: Added support for templated header() values.
(#334)
opentelemetry(), axosyslog-otlp(): Added keep-alive() options.

Keepalive can be configured with the time(), timeout()
and max-pings-without-data() options of the keep-alive() block.
```
opentelemetry(
    ...
    keep-alive(time(20000) timeout(10000) max-pings-without-data(0))
);
```
(#276)
bigquery(): Added auth() options.

Similarly to other gRPC based destination drivers, the bigquery()
destination now accepts different authentication methods, like
adc(), alts(), insecure() and tls().
```
bigquery (
    ...
    auth(
        tls(
            ca-file("/path/to/ca.pem")
            key-file("/path/to/key.pem")
            cert-file("/path/to/cert.pem")
        )
    )
);
```
(#276)
loki(): Added batch-bytes() and compression() options.
(#276)
socket based sources: Added a new option called idle-timeout().

Setting this option makes AxoSyslog close the client connection
if no data is received for the set amount of seconds.
(#355)
socket based sources: Added new flag, called exit-on-eof.

Setting this flag to a source makes AxoSyslog stop,
when EOF is received.
(#351)

syslog-ng-ctl: Added attach subcommand.

With attach, it is possible to attach to the
standard IO of the syslog-ng proccess.

Example usage:

# takes the stdio fds for 10 seconds and displays syslog-ng output in that time period
$ syslog-ng-ctl attach stdio --seconds 10

# steal trace level log messages for 10 seconds
$ syslog-ng-ctl attach logs --seconds 10 --log-level trace

(#326)

Bugfixes

Config @version: Fixed compat-mode inconsistencies when @version
was not specified at the top of the configuration file or was not specified at all.
(#312)
s3(): Eliminated indefinite memory usage increase for each reload.

The increased memory usage is caused by the botocore library, which
caches the session information. We only need the Session object, if
role() is set. The increased memory usage still happens with that set,
currently we only fixed the unset case.
(#318)
opentelemetry(), axosyslog-otlp() sources: Fixed source hang-up on flow-controlled paths.
(#314)
opentelemetry(), axosyslog-otlp() sources: Fixed a crash when workers() is set to > 1.
(#310)
file(), wildcard-file(): Fixed a crash and persist name collision issues.

If multiple wildcard-file() sources or a wildcard-file() and a file() source were
reading the same input file, it could result in log loss, log duplication, and various crashes.
(#291)
wildcard-file(): Fixed a crash that occurs after config reload when the source is flow-controlled.
(#293)
file(), stdout(): Fixed log sources getting stuck.

Due to an acknowledgment bug in the file() and stdout() destinations,
sources routed to those destinations may have gotten stuck as they were
flow-controlled incorrectly.

This issue occured only in extremely rare cases with regular files, but it
occured frequently with /dev/stderr and other slow pseudo-devices.
(#303)
metrics: syslog-ng-ctl --reset will no longer reset Prometheus metrics
(#370)
stats: Fixed free_window counters.
(#296)

FilterX features

Added new filterx code flow controls.
- drop: Drops the currently processed message and returns success.
- done: Stops the processing and returns success.
  (#269)

update_metric(): Added a new function similar to metrics-probe parser.

Example usage:

update_metric("filterx_metric", labels={"msg": $MSG, "foo": "foovalue"}, level=1, increment=$INCREMENT);

(#220)

startswith(), endswith(), includes(): Added string matching functions.

First argument is the string that is being matched.
Second argument is either a single substring or a list of substrings.
Optionally the ignorecase argument can be set to configure case sensitivity
- default: false

Example usage:

startswith(string, prefix, ignorecase=false);
startswith(string, [prefix_1, prefix_2], ignorecase=true);

endswith(string, suffix, ignorecase=false);
endswith(string, [suffix_1, suffix_2], ignorecase=true);

includes(string, substring, ignorecase=false);
includes(string, [substring_1, substring_2], ignorecase=true);

(#297)

parse_xml(): Added new function to parse XMLs.

Example usage:
```
my_structured_data = parse_xml(raw_xml);
```
Converting XML to a dict is not standardized.

Our intention is to create the most compact dict as possible,
which means certain nodes will have different types and
structures based on a number of different qualities of the
input XML element.

The following points will demonstrate the choices we made in our parser.
In the examples we will use the JSON dict implementation.
1. Empty XML elements become empty strings.
```
  XML:  <foo></foo>
  JSON: {"foo": ""}
```
1. Attributions are stored in @attr key-value pairs,
  similarly to some other converters (e.g.: python xmltodict).
```
  XML:  <foo bar="123" baz="bad"/>
  JSON: {"foo": {"@bar": "123", "@baz": "bad"}}
```
1. If an XML element has both attributes and a value,
  we need to store them in a dict, and the value needs a key.
  We store the text value under the #text key.
```
  XML:  <foo bar="123">baz</foo>
  JSON: {"foo": {"@bar": "123", "#text": "baz"}}
```
1. An XML element can have both a value and inner elements.
  We use the #text key here, too.
```
  XML:  <foo>bar<baz>123</baz></foo>
  JSON: {"foo": {"#text": "bar", "baz": "123"}}
```
1. An XML element can have multiple values separated by inner elements.
  In that case we concatenate the values.
```
  XML:  <foo>bar<a></a>baz</foo>
  JSON: {"foo": {"#text": "barbaz", "a": ""}}
```
(#251)
parse_windows_eventlog_xml(): Added a new function to parse Windows EventLog XMLs.

This parser is really similar to parse_xml() with
a couple of small differences:
1. There is a quick schema validation.
2. The Event->EventData field automatically handles named Data elements.
  (#282)
parse_cef(), `parse_le...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

4.9.0

Highlights

Sending data to ClickHouse

Features

Bugfixes

FilterX features

Releases: mitzkia/axosyslog

axosyslog-4.9.0

4.9.0

Highlights

Sending data to ClickHouse

Features

Bugfixes

FilterX features

axosyslog-charts-0.12.0

axosyslog-charts-0.11.0

axosyslog-charts-0.10.1