syslog-ng-3.15.1

3.15.1

Features

• Support added for if/elif/else blocks to the configuration file syntax.
  (#1856)
• Dramatically improved debug messages during filter/parser evaluation. (#1898)
• Similarly improved the error messages shown on syntax errors, they now show a
  full backtrace of inclusions, among other things. (#1932)
• The hook-commands module was added, allowing one to run custom commands on
  source or destination setup and teardown. (#1951)
• Implemented a way to skip processing included config file snippets in case a
  dependency is missing: The @requires json pragma. (#827, #1956)
• Basic client-side failover support was implemented. (#1905)
• Errors from python destinations are now reported together with any exception
  text (if any). (#1931)
• add-contextual-data gained a new ignore-case() option. (#1911)

Bugfixes

• Fix a crash that happened on disk queue restart. (#1886)
• Fixed another crash when a corrupted disk queue file was being moved away.
  (#1924)
• Fixed a crash that could happen during nvtable deserialization. (#1967)
• Fixed a crash that occurred when NVTables were stored on low memory
  addresses. (#1970)
• Fixed an issue with TLS session resumption, the session id context value is
  now properly set. (#1936, #2000)
• We now link directly to the evtlog shipped with syslog-ng, and are not
  using the system library, not even when present. (#1915)
• TLS destinations now work again without key-file or cert-file specified.
  (#1916, #1917)
• SDATA block names are now sanitized, in order to not break the spec when we
  get our SDATA from sources that are more lax (such as JSON). (#1948)
• Some internal messages contained key-value pairs where the key had spaces in
  it, this has been addressed, they do not contain spaces anymore.
• The STOMP destination will now correctly use template options when formatting
  its body part. (#1957)
• Fix compilation with OpenSSL 1.1.0 (#1921, #1997)
• Fix compilation on FreeBSD. (#1901)
• Fix compilation on SLES 11. (#1897)
• Fix compilation on Hurd. (#1912, #1914)
• Fix compiltaion on Solaris 10. (#1982, #1983)
• Fix compilation on MacOS.
• Fixed a value conflict in the afstreams module's grammar file.
• Various compiler warning-related fixes all over the codebase.

Other changes

• POSIX RegExp support was dropped from the filters, PCRE remains available. (#1899)
• Miscellaneous build-system related fixes and improvements (both autotools and
  CMake).
• Update lib/json-c to json-c-0.13-20171207. (#1900)

Notes to the developers

• The init() function is now optional for Python destinations. (#1756)
• The Docker environment (dbld/) has seen significant changes, among them an
  upgrade to Ubuntu Xenial. (#1876)
• dbld/rules gained two new targets: login and build, that do what their
  names suggest. (#1927)
• The LogPipe object gained a pre_init() and a post_deinit() method, used
  by the hook-commands module.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:
Andras Mitzki, Antal Nemes, Balazs Scheidler, Budai Laszlo, Gabor Nagy, Gábor
Nagy, Gergely Nagy, Juhasz Viktor, Kókai Péter, Laszlo Budai, László Szemere,
László Várady, Mehul Prajapati, Norbert Takacs, Robert Fekete, SZALAY Attila,
Tamas Nagy, Terez Nemes, Utsav Krishnan, Videet Singhai, Vivek Raj

syslog-ng-3.14

3.14.1

Features

• Password protected ssl keys (#1888)
• Add OpenBSD module to system() source (#1875)
• Add Ubuntu Trusty support to Docker build (#1849)

Bugfixes

• Fix increased memory usage during saving disk-buffer (#1867)
• Fix maximum record length limitations of disk-buffer (#1874)
• Fix a memory leak in cfg-lexer (#1843)
• Fix some issues found by pylint in python module (#1881, #1830)
• Fix a crash due to a race condition in kv-parser() (#1871)
• Fix a crash due to a race condition in file() destination (#1858)
• Fix deprecated API usage in python module tests (#1829)
• Fix a race condition in internal() source (#1815)
• Fix a locale issue in merge-grammar python tool (#1868)
• Fix compile problems with autotools when '--disable-all-modules' used (#1853)
• Fix a file descriptor leak in persist-state (#1847)
• Fix a file descriptor leak in pseudofile() (#1846)
• Fix memory/fd leaks in loggen tool (#1844, #1845)
• Fix compile problems on Fedora, RHEL6, CentOS6 and SUSE based platforms (#1837)
• Fix a crash when large variety of keys added to messages (#1836)
• Fix compile problems when PATH_MAX not defined (#1828)
• Fix integer overflow problems in grammar (#1823)
• Fix a memory leak in filter() (#1812)
• Fix memory leak of persist-name() option (#1816)
• Fix message corruption caused by a bug in the subst() rewrite rule (#1801)
• Fix silently dropped messages in elasticsearch2() when sending in bulk mode (#1800)
• Fix broken disk-buffer() support in elasticsearch2() (#1807)
• Fix Hy support in python module (#1754)
• Fix an event scheduler related crash during reloading syslog-ng (#1711)
• Fix a crash with SIGBUS when persist file cannot grow (#1785)

Other changes

• Improve error reporting in "block" definitions in config (#1809)
• Add warning message when disk-buffer() directory is changed in configuration (#1861)
• Syslog-ng debun improvements (#1840)
• Refactor in rewrite() module init (#1818)
• Missing child program (exit status 127) handling is changed in program() destination:
  stopping destination instead of polling for the child program (#1817)
• Refactor in filter() module (#1814)
• Improve thread synchronization in mainloop and refactor (#1813)
• Adapted json-c v0.13 API changes to json-parser (#1810)
• Add filters as selectors in contextual data (#1838)

Notes to the developers

• Full cmake support achieved (#1777, #1819, #1811, #1808, #1805, #1802, #1841, #1806)
• Add support for modules to have module specific global options (#1885)
• Improved MacOS support (#1862, #1864, #1865)
• Add new option to exclude directories in style-checker tool (#1834)
• Ivykis dependency updated to 0.42.2 release (#1711)
• Journald grammar, source and header files are part of dist tarball (#1852)
• Add valgrind support for unit tests (#1839)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:
Andras Mitzki, Antal Nemes, Balazs Scheidler, Björn Esser, Fabien Wernli, Gabor Nagy, Gergely Nagy,
Janos Szigetvari, Juhász Viktor, Laszlo Budai, Laszlo Szemere, László Várady, Orion Poplawski,
Attila Szalay, Shen-Ta Hsieh, Tamas Nagy, Peter Kokai, Norbert Takacs, Zoltan Pallagi.

syslog-ng-3.13.2

3.13.2

Fixes

• Missing manpages from release tarball (#1793)
• Package syslog-ng-mod-json is removed from (#1794)
• Drop syslog-ng-abi virtual packages (#1797)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:
Andras Mitzki, Gergely Nagy, Laszlo Budai, Laszlo Varady, Peter Czanik.

syslog-ng-3.13.1

3.13.1

Features

• Add app-parser() framework (automatic parsing of log messages) (#1689)
• Support microseconds in Riemann destination (#1710)
• Add osquery destination as an SCL plugin (#1728)
• Add network load balancer destination (#1706)
• Add possibility to only signal re-open of file handles (SIGUSR1) (#1530)
• It is possible from now to limit the number of registered dynamic counters (#1743)
• Add $(binary) template function (#1679)
• Add experimental transport for transferring messages in whole between syslog-ng instances (EWMM) (#1689)
• Docker based build and debian package generation (#1783)
• Add auto-parse(yes/no) to app-paser(), system() and default-network-drivers() (#1788)
• Add Graylog2 destination and $(format-gelf) template function (#1680)

Bugfixes

• Exit when a read fails on an included config file instead of
  starting up with an empty configuration. (#1721)
• Fix double free (#1720)
• Add missing discarded counter to groupingby (#1748)
• Fix a reference leak in Python destination (#1716)
• Fix timezone issue in snmptrapd parser (#1746)
• Fix potential crash in stdin driver (#1741)
• Fix a crash when initializing new config fails for socket with keep_alive off (#1723)
• Fix filter evaluation in case of contexts with multiple elements (#1718)
• Various grouping-by fixes (#1718)
• Fix potential use after free around dns-cache during shutdown (#1666)
• Fix access to indirect values within Java destination (#1732)
• Fix a crash in affile (#1725)
• Fix a memory leak (#1724)
• Fix a crash when getent is used empty group (#1691)
• Fix jvm-options() (#1704)
• Fix a crash in Python language binding (#1694)
• Fix a crash in afmongodb (#1765)
• Fix a memory leak in afmongodb (#1766)
• Fix name-to-GID calculation in the $(getent) template function (#1764)
• Fix a crash when redis is configured without the command() option (#1767)
• Fix a race condition in kv-parser() (#1789)

Other changes

• Cleanup diskq related warning messages (#1752)
• Provide tls block for tls options in amqp(), http(), riemann() destination drivers (#1715)
• It it possible from now to register blocks and generators as plugins (#1657)
• Drop compatiblity with configurations below 3.0 (#1709)
• Do not change permissions of a file by default (#1782)
• Allow source files to specify permissions locally (#1782)
• Minor performance improvement (#1729)
• The current config version can be queried with "--version" (#1740)
• Increase the performance of kv-parser() (#1789)

Notes to the developers

• Change configure default option for jsonc and mongoc from auto to internal (#1735)
• Disable ASLR when running unit tests (#1753)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:
Andras Mitzki, Antal Nemes, Attila Szalay, Balazs Scheidler, Gabor Nagy,
Jakub Jankowski, Janos Szigetvari, Laszlo Budai, Laszlo Varady, Laszlo Szemere,
Marton Illes, Mate Farkas, Peter Kokai, Pontus Andersson, Sam Stephenson,
Sebastian Roland, Viktor Juhasz, Zoltan Pallagi.

syslog-ng-3.12.1

3.12.1

Features

• HDFS: support macro in filename (#1638)
• HDFS: add append support (#1675)
• Java: allow to use sequence numbers in templates (#1628)
• TLS improvements (#1603, #1636)
  • Add PKCS 12 support with the new pkcs12-file() TLS option
  • startup time ssl-options() and peer-verify() check
  • startup time key_file, cert_file, ca_dir, crl_dir and cipher_suite check
  • ECDH cipher support (OpenSSL 1.0.1, 1.0.2, 1.1.0) with the ecdh-curve-list() option (only available >= 1.0.2)
    • for < 1.0.2, a hard-coded curve is used
    • for >= 1.0.2, automatic curve selection is used (the ecdh-curve-list() option can restrict this list)
  • DH cipher support with the dhparam-file() option
    • if the option is not specified, fallback RFC 3526 parameters are used
  • minor fixes
• stdin() source driver (#1605)
• Implement read_old_records option for systemd-journal source (#1642)
• Add tags-parser: a new module to parse $TAGS values (#1658)
• Add a Windows eventlog parser scl module (#1572)
• Add XML parser module (#1659, #1684)

Bugfixes

• Fix cannot parse ipv6 into hostname (#1617)
• Speedup add-contextual-data by making ordering optional (#1645)
• Fix monitor-method() option not working for wildcard-file() source (#1651)
• Sanitize SDATA keys in syslog-protocol messages to avoid generating non-valid messages (#1650, #1654)
• Fix memory leaks reported using Valgrind (#1649)
• Fix memory leak related to cloning pipes and reload (#1647)
• Fix getent protocol number returns incorrect value (#1665)
• Fix elasticsearch2 destination flush mechanism (#1668)
• Fix file destination related memory leak (#1685)
• Fix a possible memory leak around affile destination (#1685)

Other changes

• Improve syslog-ng debun functionality (#1633, #1641, #1663)
• Java: allow to set JVM options form global syslog-ng options (#1639)
• Do steps towards Python 3 support:
  • Fix string compatibility for Python 3 (#1632)
  • Improve Python version auto detection (#1660)
• HTTP destination: display verbose logs on debug level (#1526)
• Improvements for Solaris packing (#1664)

Notes to the Developers

• Update internal RabbitMQ (#1662)
• Update internal ivykis to v0.42 (#1566)
• Fix Travis and test related issues (#1566, #1644, #1674)
• Update docker images (#1637)
• Fix some clang compile errors (#1662)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:
Andras Mitzki, Antal Nemes, Attila Szalay, Balazs Scheidler, Gabor Nagy,
Gergely Orosz, Janos Szigetvari, Laszlo Budai, Laszlo Varady, Mate Farkas,
Marton Suranyi, Peter Kokai, Szilard Pfeiffer, Tamas Nagy, Zoltan Pallagi.

syslog-ng-3.11.1

3.11.1

Features

• Add geoip2 parser and template function.
  It is based on the libmaxminddb(MaxMindDB).
  It will replace the old geoip parser and template function,
  so they are deprecated from 3.11 (but still available).

• Add SSL support to AMQP.

• Add template option to apache-accesslog-parser.

• Add configurable event time to Riemann destination.

• Add drop-unmatched() option to dbparser.

• Add Ubuntu Xenial to the bundled docker images.

• Support multi-instance support for Solaris 10 and 11.

• Support multi-instance for systemd.

• Add configurable timeout to HTTP destination.

• Add prefix() option to cisco-parser.

Bugfixes

• Fix a memory usage counter underflow for threaded destination drivers
  and writers.

• Fix a potential crash in AMQP.

• Fix a potential crash during reload.

• Fix a reload/shutdown issue.
  Under heavy load, worker might never exit from the fetch loop from the
  queue.

• Fix a potential crash in afsocket destination during reload.

• Fix a counter registration bug.
  In some cases not all the required counters are registered.

• Fix a build issue on FreeBSD.

• Fix a memory leak in diskq plugin.

• Fix systemd-journal error codes validation.

• Fix a potential crash in diskq when it is used with file
  destination and the file is reaped.

• Fix a memory leak in HTTP destination

• Fix ENABLE_DEBUG in dbparser.

• Fix a unit tests that caused build issue on 32 bit platforms.

Other changes

• The eventlog library is part of syslog-ng from now.

• Improve error messages when the config cannot be initialized.

• Improve source suspended/resumed debug messages.

• Rename syslog-debun to syslog-ng-debun.

• Update manpages to v3.11

• Remove tgz2build directory.

Notes to the Developers

• Rewrite merge-grammar script from Perl to Python.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Antal Nemes, Attila Szalay, Balazs Scheidler, Fabien Wernli,
Gabor Nagy, Giuseppe D'Anna, Janos Szigetvari, Laszlo Budai, Laszlo Varady,
Lorand Muzamel, Mate Farkas, Noemi Vanyi, Peter Czanik, Tamas Nagy,
Tibor Bodnar, Tomasz Kazimierczak, Zoltan Pallagi

syslog-ng-3.10.1

3.10.1

Features

• Support https in http (curl) module

• Docker support : from now Dockerfile for CentOS7, Ubuntu Zesty and for
  Debian Jessie is part of our upstream

• Add --database parameter for geoip template function

• Metric improvements

  • add discarded messages for parsers
  • add matched/not matched counter for filters
  • add memory_usage counter to logqueue
  • add written counter
    • Written is a calculated counter which return the written messages
      by destinations. Written message is which was processed but not
      queued and not dropped. (written = processed - queued - dropped)
  • stats-counters: rename stored counter to queued
  • add global_allocated_logmsg_size counter for tracking memory logmsg
    related allocations

• Add snmp-parser (v1, v2)

  • parses snmptrapd log
  • The parsed information is available as key-value pairs, which can be
    used/serialized (macros, format-json, etc.) in the log path.
    If you want to send the message in a structured way, you can disable the
    default message generation with the generate-message(no) option.

• Add snmp-soure

  • available as an SCL block that containing a filesource and an SNMP parser
    modules: add snmptrapd parser

• Add osquery source

  • available as an SCL block
  • It reads the osquery log file and parses with the JSON parser,
    creating name-vaule pairs with an .osquery. prefix by default.

• Add cisco-parser

  • available as an SCL block

• Add wildcard filesource

• Add startdate template function

• Add $(basename) and $(dirname) template functions

• Add Kerberos support for HDFS destination

• Add AUTH support for redis destination

• Add map-value-pairs() parser

  • it can be used to map existing name-value pairs to a different set during
    processing, in bulk. Normal value-pairs expressions can be used, just
    like with value-pairs based destinations.

• Extend Python language binding by Python parser

• Add support for extract-stray-words() option in kv-parser()

  • stray words: those words that happen to be between key-value pairs and
    are otherwise not recognized either as keys nor as values.

• Add $(context-values) template function

• Add $(context-lookup) function

• Add list related template functions

  • $(list-head list ...) returns the first element (unquoted)
  • $(list-nth NDX list ...) returns the specific element (unquoted)
  • $(list-concat list1 list2 ...) returns a list containing the concatenated
    list
  • $(list-append list elem1 elem2 ...) returns a list, appending elem1,
    elem2 ...
  • $(list-tail list ...) returns a list containing everything except
    for the first element
  • $(list-slice FROM:TO list ...) returns a list containing the slice
    [FROM:TO), Python style slice
    boundaries are supported (e.g. negative)
  • $(list-count ...) returns the number of elements in list

• Add add query commands to syslog-ng-ctl

  • query list List names of counters which match the filter
  • query get Get names and values of counters which match the
    filter
  • query get --sum Get the sum of values of counters which match the
    filter

• Support multiple servers in elasticsearch2-http destination

• Implements elastic-v2 https in http mode

• Add getent module (ported from incubator)

  • This module adds $(getent) that allows one to look up various NSS based
    databases, such as passwd, services or protocols.

• Add support for IP_FREEBIND

Bugfixes

• Fix a libnet detection check error that caused problem configuring
  enable-spoof-source.

• Avoid warnings about _DEFAULT_SOURCE on recent glibc versions
  With the glibc on zesty, using _GNU_SOURCE and not defining _DEFAULT_SOURCE
  results in a warning, avoid that by defining _DEFAULT_SOURCE as well.

• Fix invalid database warning for geoip parser

• Fix prefix() default in systemd-journal for new config versions

• Fix a potential message loss in Riemann destination

• Fix a potential crash in the Riemann destination when the client is not
  connected to the Riemann server.

• Fix a possible add-contextual-data() related data loss in case of multiple
  reference to the same add-contextual-data parser in several logpaths.

• Fix dbparser deadlock

• Fix Python destination

  • open() was not called in every time_reopen()
  • python destination is not defined in stats output

• Fix processed stats counter for afsocket

• Fix stats source for pipes

  • Previously pipe source is shown as file

• Fix csv-parser multithreaded support
  In some cases (when csv-parser attached to network source), the parser
  randomly filled the column macros with garbage.

• Fix a message loss in case of filesource when syslog-ng was restarted and
  the log_msg_size > file size.

• Fix a potential crash in cryptofuncs

• Fix a potential crash in syslog-ng-ctl when no command line parameters was
  set.

• Fix token duplication in the output of '--preprocess-into'

• Fix UTF-8 support in syslog-ng-ctl

• Fix a potential crash during X.509 certificate validation.

• Fix a segfault in Python module startup

• Fix a possible endless reading loop issue in case of multi-line filesource.

• Fix soname for the http module from "curl" to "http"

• Avoid openssl 1.1.0 deprecated APIs
  When openssl is built with --api=1.1 disable-deprecated, use of deprecated
  APIs results in build failure.

Other changes

• Increase processed counter by queued counter after reload or restart when
  diskqueue is used otherwise the newly added written counter would underflow.

• Set the default time-zone to UTC for elasticsearch2
  Elasticsearch and Kibana use UTC internally.

• Add retries support for python destination

• Prefer server side cipher suite order

• Always include librabbitmq in the dist tarball

• Always include ivykis in the dist tarball

• Marking parse error locations with >@<.

• Default log_msg_size is increased to 64Kbyte from 8Kb

• Tons of syslog-debun improvements

• Exit with 0 return code when --help is specified for syslog-ng-ctl

• syslog-ng: make '--preprocess-into' foreground only

• Add debug messages on log_msg_set_value()

• Add more detail to filter evaluation related debug messages

Notes to the Developers

• Extract template perf test function to testlib

• Print a debug message when logmsg passed to the Python side

• Allow http module (curl) to be build with cmake

• astylerc: allow continuation lines to start until column 60

• Move kv-scanner under syslog-ng/lib

• scratch-buffers2: implement an alternative to current scratch buffers
  This new API is aimed a bit easier to use in situations where a throw away
  buffer is needed that will automatically be freed at the next message.
  It also gets does away with GTrashStack that is deprecated in recent glib
  versions.

• Several refactors in stats module.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Antal Nemes, Balazs Scheidler, eroen, Fabien Wernli, Gabor Nagy,
Gergely Nagy, Janos Szigetvari, Jason Hensley, Laszlo Varady, Laszlo Budai, Mate Farkas,
Noemi Vanyi, Peter Czanik, Peter Gervai, Todd C. Miller, Philip Prindeville,
Zoltan Pallagi

syslog-ng-3.9.1

3.9.1

Features

• Improve parsing performance in case of keep-timestamp(no), earlier the
  timestamp was parsed and then dropped, now we don't parse it, which is a
  2x performance improvement in reception speed.

• TLS based transports will publish the peer's certificate in a set of
  name-value pairs, as follows:

  • .tls.x509_cn - X.509 common name
  • .tls.x509_o - X.509 organization string
  • .tls.x509_ou - X.509 organizational unit

• Improve performance of the tcp() source, due to a bug, syslog-ng
  attempted to apply position tracking to messages coming over a TCP
  transport, which is used for file position tracking and causing
  performance degradation. This bug is fixed, causing performance to be
  increased. (#1195)

• Make it possible to configure the listen-backlog() for any stream based
  transports (unix-stream and tcp). Earlier this was hard-wired at 256
  connections, now can be tuned using an option. For example:

  tcp(port(6514) listen-backlog(2048));

• Add a groupunset() rewrite rule that pairs up with groupset() but instead
  of setting values it unsets them. (#1235)

• Add support for Elastic Shield (#1228) and SearchGuard (#1223)

• kv-parser() is now able to cope with unquoted values with an embedded
  space in them, it also trims whitespace from keys/values and is in
  general more reliable in extracting key-value pairs from arbitrary log
  messages.

• Improve performance for java based destinations. (#1243)

• Add prefix() option to add-contextual-data()

Bugfixes

• Fix a potential crash in the file destination, in case it is a template
  based filename and time-reap() is elapsed. (#1183)

• Fix a potential ACK problem within syslog-ng that can cause input windows
  to overflow queue sizes over time, effectively causing message drops that
  shouldn't occur. (#1230)

• Fix a heap corruption bug in the DNS cache, in case the maximum number of
  DNS cache entries is reached. (#1218)

• Fix timestamp for suppression messages. (#1233)

• Fix add-contextual-data() to support CRLF line endings in its CSV input
  files.

• Fixed key() option parsing in riemann() destinations.

• Find libsystemd-journal related functions in both libsystemd-journal.so
  and libsystemd.so, as recent systemd versions bundled all systemd
  related libs into the same library.

• Fixed the build-time detection of system-wide installed librabbitmq,
  libmongoc and libcap.

• Fix the file source to repeatedly check for unexisting files, as a bug
  caused syslog-ng to stop after two attempts previously. (#841)

• The performance testing tool "loggen" crashed if it was used to generate
  messages on multiple threads over TLS. This was now fixed. (#1182)

• Fix an issue in the syslog-parser() parser, so that timestamps parsed
  earlier in the log path are properly overwritten. Earlier a time-zone
  setting may have remained in the timestamp in case the first timestamp
  did contain a timezone and then the one parsed by syslog-parser() didn't.
  (#1206)

• Due to a compilation issue, tcp-keepalive-time(), tcp-keepalive-intvl() and
  tcp-keepalive-probes() were not working, now they are again. (#1214)

• The --disable-shm-counters option is now passed to mongo-c-driver to work
  around a minor security issue (#1219).

  https://jira.mongodb.org/plugins/servlet/mobile#issue/CDRIVER-1691/comment/1405406

• Fix compilation issues on FreeBSD. (#1252)

• Add support to month names in all caps in syslog timestamps. At least one
  device seems to generate these. (#1263)

• The options() option to java destination can now accept numbers and not
  just strings.

• Fix a memory leak in the java destination driver, that may affect java
  based destinations like ElasticSearch, Kafka & HDFS.

Other changes

• HDFS was updated to 2.7.3
• Elasticsearch was updated to 2.4.0
• Support was added for OpenSSL 1.1.x (#1281)

Notes to the Developers

• We started to standardize our tests on the criterion unit testing
  framework, please submit all new tests using this framework. Patches to
  convert existing ones are also welcome.
  https://github.com/Snaipe/Criterion
• We also added a configuration file for astyle and accompanying make
  targets to check/reformat the source code to meet syslog-ng's style.
• debian/ directory has been removed from the "master" branch and is now
  maintained in a separate "release" branch.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Lászlo Várady, 0xddaa, Balázs Scheidler, Tamás Nagy, László Budai,
Fabien Wernli, Viktor Juhász, Kyle Manna, Michael Wimpy, Noémi Ványi,
Attila Szalay, Tibor Bodnár, Zoltán Pallagi

syslog-ng-3.8.1

3.8.1

Library updates

• Kafka-client updated to version to 0.9.0.0
• Minimal required version of hiredis is set to 0.11.0 to avoid
  possible deadlocks
• Minimal version of libdbi is set to 0.9.0

New dependencies

• From now autoconf-archive package is a build-dependency.

Improvements and features

• Added the long-waited disk-buffer.
• date-parser ported from incubator to upstream
• New template functions: min, max, sum, average
• Added Apache-accesslog-parser
• Added loggly destination
• Added logmatic destination
• Added template function for supporting CEF.
• cURL-based HTTP destination driver added (implemented in C
  programming language)
• SELinux policy installer script now has support for Red Hat Enterprise Linux/CentOS/
  Oracle Linux 5, 6 and 7.
• Implemented add-contextual-data:
  With add-context-data syslog-ng can use an external database file to append
  custom name-value pairs on incoming logs (to enrich messages). The 'database'
  is actually a file that containing <selector, name, value> records.
  Currently only CSV format is supported.
  It is like geoip parser where the selector is $HOST, but in this case,
  the user can define the selector, and also the database contents.

Drivers

• Program destination/source drivers
  • Added inherit-environment configuration option to program source and
    destination. When it is set to true then the process will inherit the
    entire environment of the parent process.
  • Added keep-alive option to program destination (afprog).
    This option will control whether the destination program should be
    terminated at reload, or should be left running.
• Java drivers
  • HTTP destination
    • Added the ability to use templates in both url and message.
  • ElasticSearch Destination driver :
    • Support 2.2.x series of ElasticSearch (transport and node mode) .
    • Support Shield plugin for both ElasticSearch 1.x and ElasticSearch
      2.x .
    • Implemented new mode (HTTP) that can work with ElasticSearch 1.x,
      ElasticSearch 2.x, and even with Elastic 5. HTTP mode is based on
      a Java HTTP Rest client (Jest : https://github.com/searchbox-io/Jest).
      Note: make install will copy Jest library to the syslog-ng install
      directory.
• MongoDB destination driver
  • Replaced submodule limongo-client with mongo-c-driver.
  • Additional support for previous syntax used by libmongo-client
    before we started using mongo-c-driver and its URI syntax exclusively.
    Note that these are plainly translated to a connection URI without
    much sanity checking or preserving their former semantic meaning.
    So various aspects of the MongoDB connection like health checks, retries,
    error reporting and synchronicity will still follow the slightly altered
    semantics of mongo-c-driver.
• Riemann destination driver
  • Use cert-file() and key-file() options to match afsocket keywords as
    the same way as afsocket drivers use these options. The old one still
    work though.

Rewrite rules

• Introduced template options in rewrite rules.
• Added unset operation to make it possible to unset a specific name-value pair
  for a logmessage.

Parsers

• kvformat: make it possible to specify name-value separator
• linux-audit-scanner: recognize a0-a9* as fields to be decoded
  Argument lists are encoded in a0, a1, ... fields that can potentially be
  hex-encoded.
• csv-parser has been refactored, extended with new dialect and prefix options.
  Dialect is to convey CSV format information, instead of using flags
  Prefix option gets prefixed to all column names, just like with
  other parsers.

PatternDB

• added groupingby() parser that can perform simple correlation on
  log messages. In a way it is similar to the SQL GROUP BY operation, where
  an aggregate of a set of input records can be calculated.
  The major difference between SQL GROUP BY and groupingby() is that the
  first always operates on a enumerable list of records, whereas
  groupingby() works on a stream of data.
  A few use-cases where this can be useful:
  • Linux audit logs
  • postfix logs
• added create-context action
  Added a new possible action in the element, to create
  a new correlation context out of the current message and its associated
  context. This can be used to "split" a state.
• Added NLSTRING parser that captures a string until the
  following end-of-line. It can be used in patterns as: @NLSTRING:value@
  It doesn't expect any additional parameters. This makes it pretty easy to
  parse multi-line Windows logs.

Miscellaneous features

• syslog-debun (debug bundle script for syslog-ng) has been improved

Bugfixes

• geoip-parser: When default database if not specified, syslog-ng crashed.
• Added support for multiple drivers with the same name in syslog-ng config.
• Fixed aack counting logic for junctions that have branches that modify
  the LogMessage.
• Fixed a potential crash for code that uses log_msg_clear()
  in production (e.g. syslog-parser()).
• Fixed potential crash in reload logic
• system(): use string comparison instead of numeric in PID rewrite
  The meaning of the != operator has been fixed to refer to numeric comparison
  in @Version: 3.8, so make sure we are using string comparison.
• Support encoding on glib compiled with libiconv
• pdbtool: Fix the ordering of the debug-info list in PatternDB
• afprog: Don't kill our own process group
  If, for some reason, the program source/destination failed to set up its
  own process group, we need to make sure we do not run killpg() on that
  process group, as it would kill ourselves.
• Handle option names with hyphen (-) characters in java scls
• dnscache performance improved
  Instead of getting rid off the per-thread DNSCache when a worker thread
  exits, store them in a linked list and acquire them as a new thread starts.
  The set of cached hostnames are valuable as worker threads come and
  go (they exit after 10seconds of inactivity), but without this
  reusing of cache instances, our DNS cache is filled again and again.
• Fixed IPv6 parser in patterndb.
• Fixed journald program name flapping
• Fixed create-dirs() inheritance in file destinations
• Fixed pass-unix-credentials() global inheritance in afunix
  The global pass-unix-credentials option was not inherited in afunix-source
  if the options{}; block was positioned lower in the
  configuration file than the given module declaration.
• Fixed create-dirs() global inheritance in afunix
  When the global create-dirs option was set to yes, the local one was ignored.
• Fixed byteorder handling on bigendian systems in netmask6 filter
• Fixed flow-control issue when overflow queue is full
  (suspending source by setting the window size to 0).
• Log HTTP response error codes in HTTPDestination (Java).
• Fixed potential leaks related $(sanitize) argument parsing in basicfuncs.
• Fixed a memory leak in python debugger
• Fixed a use-after-free bug in templates.
• Fixed a memory leak around reload in netmask6 filter.
• Fixed a memory leak in LogProtoBufferedServer in case the
  encoding() option is used.
• configure: don't override $enable_python while executing pkg-config
• Fixed BSD timestamp parsing in syslog-format.
• Fixed a SIGPIPE bug in program destination.
• Error handling has been improved in AMQP destination.
• value-pairs performance improvements, memleak fixes
• Various issues around UTF-8 support fixed.
• Fixed integer overflow in numerical operations template function
• Fixed an integer underflow in afsocket.
• Fixed numerical comperisons issues around filters.
  There's a problem in straight fixing this issue though: anyone who used
  the numeric operators erroneously will have their behaviour changed, therefore
  this patch also adds a configuration update warning in case
  someone is using the wrong syntax.
• Fixed kernel log message time drift on Linux.
• Take CRLF sequences equivalent to an LF in patterndb.
  Windows logs contain embedded CRLFs which is difficult to match against
  from db-parser(), as we use a UNIX text file to store the patterns. Also,
  the fact that the input contains CRLF whereas our patterns only contain
  an LF makes it a very unintuitive non-match, which is difficult to debug.
• When syslog-ng failed to insert data into Redis, it has crashed.
• When device file is set as a file destination then syslog-ng will not try
  to change the permission of the device file.
• Various fixes around config file parsing:
  • in some circumstances syslog-ng crashed when the config
    file contained non-readable characters
  • fixed a memory leak
  • fixed memory leak around backtick substitution

Notes to the Developers

• copyright cleanup in source tree
• install tools and scl under a syslog-ng specific subdirectory
  These should never be installed in /usr/share directly, but rather under a
  subdirectory and as described in
  https://www.gnu.org/prep/standards/html_node/Directory-Variables.html
  we should do that right within the source and not rely on packaging tools
  to do it for us.This will trigger a required change in packaging scripts to
  avoid changing the --datadir, as the default of
  /usr/share should work out-of-the-box.
• Support for native-lanugage (compiled languages, like Rust) bindings.
  These bindings just forward the calls to the native side.
  This whole module compiles into a static library
  (libsyslog-ng-native-connector.a) which is linked to all external native
  modules. A native module defines the required functions
  (like native_parser_proxy_new()) so those symbols will be resolved.
  Some symbols have the visibility(hidden) attribute applied to them. Those
  symbols are defined by the other half of the native bindings, we only need
  their signature here. They are hidden because their definition is contained
  in other source files but we would like to keep...

syslog-ng-3.8.0beta2

3.8.0beta2

This is the second beta release for the 3.8.x series.

Changes compared to 3.8 (created by manually):

• add-contextual-data : more strict CSV parsing
  When a line in the CSV file containing more fields then required, syslog-ng won't start.
• add-contextual-data : in some circumstances syslog-ng crashed when the
  CSV file contained invalid data
• MacOSX support added to travis.yml
• FreeBSD 10.3 build issues fixed
• Oracle Solaris 11 build issues fixed
• logmsg serialization performance enhanced
• elastic-v2 and mode http added to the syslog-ng-mod-elastic Debian package
• ElasticSearch-v2: fixed missing 'path.home' issue

Changes compared to 3.7.x (automatically generated):

Note, that for beta release we generate the changes with
a tool. Final changelog will be more sophisticated (and will
include Credits section).

Bug Fixes

• Minor problems in kv-parser() Link László Várady Tamas Nagy
• Deprecate old configuration parameters in MongoDB destination Link Tibor Benke Tamas Nagy
• The output of pdbtool is scrambled Link László Várady László Várady
• @ in format json is absorbed Link pzoleex László Várady
• jni.h: No such file or directory Link czanik László Várady
• IPv6 Pattern Parser and trailing colons Link aslothinasuit László Várady
• 3.8 journal source problem Link czanik Tibor Benke
• Duplicate symbols (_TLSSslOptions and _last_parser) break build of syslog-ng 3.7.2 on OS X Link Douglas Carmichael Tamas Nagy
• Deadlock in redis destination Link suuuper Tibor Benke
• Eliminate compiler warnings and turn on -Werr Link Tibor Benke Tamas Nagy
• Deadlock with suppress option Link symphorien Balazs Scheidler

Enhancements

• support multiple drivers with the same name in syslog-ng config Link Budai Laszlo MÓZES Ádám István
• Support an alternative build system: CMake Link Tibor Benke Tibor Benke
• Verify that CFLAGS is propagated down to all of our sources Link Tamas Nagy Tamas Nagy
• Support encoding on glib compiled with libiconv Link Tamas Nagy Tamas Nagy
• Separator option for key-value parser Link Fabien Wernli Balazs Scheidler
• Copyright policy/checker Link Balazs Scheidler Tamas Nagy
• SCL for Loggly format and destination Link Robert Fekete Balazs Scheidler
• Please do not chown/chmod if the log file is a device like e.g. /dev/null Link Axel Beckert Zoltán FRIED

Fixed Issues

• Certain combination of configuration will sometimes generate incorrect unparseable JSON errors Link Allan Crooks Balazs Scheidler
• basicfuncs unit test seems to be buggy on 32bit systems Link Budai Laszlo László Várady
• Null pointer dereference when parsing malformed config Link Agostino Sarubbo Balazs Scheidler
• Aborting during configuration loading Link Tibor Benke Balazs Scheidler
• With an tag in patterndb.xml db-parser did not parse Link mitzkia Balazs Scheidler
• flapping ${PROGRAM} using systemd Link Fabien Wernli Tibor Benke
• Fix netmask6 filter on bigendian systems Link Zoltán FRIED Tibor Benke
• SIGABRT: using (") around macros in format-json type-hinting Link mitzkia Tibor Benke
• environment variables not passed on to gradle in 3.7.1 Link czanik Tibor Benke
• double-free because of the yydestruct Link Balazs Scheidler Tibor Benke
• rewrite rules don't honor time-zone Link Fabien Wernli Tibor Benke
• Integer overflow in numerical operations template function Link Fabien Wernli Tibor Benke
• distribute config.h Link Tibor Benke Tibor Benke
• lib/filter/filter-cmp.c:116: missing break ? Link dcb314 Balazs Scheidler
• plugin loading anomalies Link Tibor Benke Tibor Benke
• sigsegv in redis module when restart redis server Link Zoltán FRIED Zoltán FRIED
• 3.8 beta1 fails to compile on FreeBSD Link czanik Budai Laszlo
• Absence of clock_gettime() on OS X breaks build of lib/timeutils.c on 3.7.2 Link Douglas Carmichael László Várady
• mongodb: moving to official C mongo client library Link Budai Laszlo bkil-syslogng

Merged Pull Requests

• 'create-libsyslog-ng-so-symlink-with-cmake' : Merge branch 'create-libsyslog-ng-so-symlink-with-cmake' of git://github.com/ihrwein/syslog-ng Link MÓZES Ádám István
• 'f/fix-spelling-issues' : Merge branch 'f/fix-spelling-issues' Link Balazs Scheidler
• afstreams: include missing header (errno.h) Link Budai Laszlo
• F/freebsd10 java support Link Budai Laszlo
• elasticsearch_v2: set default path.home for node client Link Budai Laszlo
• add-contextual-data: build fixed on SLES-11 Link Budai Laszlo
• elastic-v2: fixed debian packaging Link Budai Laszlo
• FreeBSD compatibility of 3.8 - fixes #1138 Link Noémi Ványi
• F/logmsg serialize performance Link Balazs Scheidler
• Enable compatibility of make check on Mac OS X in Travis Link Tamas Nagy
• add-contextual-data: fixed double free bug when csv file contains inv… [Link](https://github.co...