Cherry-pick #11346 to 6.7: Don't use multiline options in Logstash module with json format

[ycombinator]

Cherry-pick of PR #11346 to 6.7 branch. Original message:

Multiline options were being applied to json log files with unexpected
results as several lines joined or lines lost.

[ruflin]

LGTM.

Can't 100% follow how this test magically works for both versions (if it does).

[ycombinator]

Can't 100% follow how this test magically works for both versions (if it does).

That's because it doesn't 🤦‍♂️. I had to hack my test_modules.py locally to make it work but obviously I didn't commit my hack so CI is complaining on exactly this test now. Let me see how it's working in master and then backport those changes as part of this PR as well.

[ycombinator]

@ruflin I backported these lines as part of this PR: https://github.com/elastic/beats/pull/9959/files#diff-bc529fe61de07bbc833c8b84fd86cd17R125. Now the tests should "magically" work for both versions: plain and json.

[ycombinator]

jenkins, test this

[ycombinator]

CI is failing for the libbeat build job. The same failure is happening in the 6.7 build job as well:

09:00:36 Step 9/15 : RUN mkdir -p ${KAFKA_LOGS_DIR} && mkdir -p ${KAFKA_HOME} && curl -s -o $INSTALL_DIR/kafka.tgz     "http://ftp.wayne.edu/apache/kafka/${KAFKA_VERSION}/kafka_2.11-${KAFKA_VERSION}.tgz" &&     tar xzf ${INSTALL_DIR}/kafka.tgz -C ${KAFKA_HOME} --strip-components 1
09:00:36  ---> Running in 4374aeb2bbe4
09:00:37 
09:00:37 gzip: stdin: not in gzip format
09:00:37 tar: Child returned status 1
09:00:37 tar: Error is not recoverable: exiting now
09:00:37 Removing intermediate container 4374aeb2bbe4
09:00:37 Service 'kafka' failed to build: The command '/bin/sh -c mkdir -p ${KAFKA_LOGS_DIR} && mkdir -p ${KAFKA_HOME} && curl -s -o $INSTALL_DIR/kafka.tgz     "http://ftp.wayne.edu/apache/kafka/${KAFKA_VERSION}/kafka_2.11-${KAFKA_VERSION}.tgz" &&     tar xzf ${INSTALL_DIR}/kafka.tgz -C ${KAFKA_HOME} --strip-components 1' returned a non-zero code: 2

There are no other failures in this PR's CI build. So I'm going to merge it.