About logstash #1048

nw-engineer · 2023-02-17T00:12:29Z

nw-engineer
Feb 17, 2023

There was a description that Logstash can be used with the latest version of manticoresearch.

Experimental: you can now execute Elasticsearch-compatible insert and replace JSON queries which enables using Manticore with tools like Logstash (version < 7.13), Filebeat and other tools from the Beats family. Enabled by default. You can disable it using SET GLOBAL ES_COMPAT=off.

What should I use for the logstash output plugin?
※If possible, it would be helpful if you could provide a sample definition that has been confirmed to work from input to output.

Also, will logstash version 7.13 or higher work without problems?
Thank you.

Nick-S-2018 · 2023-02-17T08:53:08Z

Nick-S-2018
Feb 17, 2023
Maintainer

You should use the 'elasticsearch' plugin with the following configuration:

output {
    elasticsearch {
 	index => "MANTICORE_TABLE_NAME"
        hosts => ["http://${MANTICORE_HOST}:${MANTICORE_PORT}"]
        ilm_enabled => false
        manage_template => false
    }
}

Note that now you must create your Manticore table before using Logstash.

Manticore doesn't support Logstash versions >= 7.13 yet, but we're working on this.
It'll also require the 'manticore-extra' package to work with the higher versions and to enable automatic table creation.

11 replies

nw-engineer Feb 18, 2023
Author

@sanikolaev
Thanks for the answer.

That error with "evenement" should have been already fixed in the latest dev version.

Hmm. So why did it fail to boot?
The version of buddy currently installed is as follows.
manticore-buddy-0.3.5_23021410.ed09f8f-1.noarch

sanikolaev Feb 18, 2023
Maintainer

Please upgrade your dev packages and try again. That was just a bug. Hopefully fixed now.

nw-engineer Feb 18, 2023
Author

I tried upgrading the target package.

[root@manticoresearch ~]# rpm -qa | grep manticore-buddy
manticore-buddy-0.3.5_23021808.ed09f8f-1.noarch
[root@manticoresearch ~]#
[root@manticoresearch ~]#
[root@manticoresearch ~]# tail -f /var/log/manticore/searchd.log
[Sat Feb 18 18:31:43.210 2023] [1384] listening on 10.2.0.17:9312 for sphinx and http(s)
[Sat Feb 18 18:31:43.210 2023] [1384] listening on 10.2.0.17:9306 for mysql
[Sat Feb 18 18:31:43.210 2023] [1384] listening on 10.2.0.17:9308 for sphinx and http(s)
[Sat Feb 18 18:31:43.559 2023] [1384] expression stack for creation is 208. Consider to add env MANTICORE_KNOWN_CREATE_SIZE=208 to store this value persistent for this binary
[Sat Feb 18 18:31:43.560 2023] [1384] expression stack for eval/deletion is 32. Consider to add env MANTICORE_KNOWN_EXPR_SIZE=32 to store this value persistent for this binary
[Sat Feb 18 18:31:43.567 2023] [1384] filter stack delta is 224. Consider to add env MANTICORE_KNOWN_FILTER_SIZE=224 to store this value persistent for this binary
[Sat Feb 18 18:31:43.611 2023] [1388] prereading 1 tables
[Sat Feb 18 18:31:43.611 2023] [1388] preread 1 tables in 0.000 sec
[Sat Feb 18 18:31:45.257 2023] [1384] [BUDDY] started '/usr/share/manticore/modules/manticore-buddy --listen=http://10.2.0.17:9312  --threads=8' at http://127.0.0.1:44909
[Sat Feb 18 18:31:45.257 2023] [1384] accepting connections

No more errors on the manticore side.
However, logstash now gives the following error:

[root@logstash001 logstash]# /usr/share/logstash/bin/logstash -f conf.d/file.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2023-02-18 18:35:44.541 [main] runner - Starting Logstash {"logstash.version"=>"7.12.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86_64]"}
[WARN ] 2023-02-18 18:35:44.683 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2023-02-18 18:35:45.003 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2023-02-18 18:35:45.451 [Converge PipelineAction::Create<main>] Reflections - Reflections took 17 ms to scan 1 urls, producing 23 keys and 47 values
[WARN ] 2023-02-18 18:35:45.639 [Converge PipelineAction::Create<main>] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2023-02-18 18:35:45.826 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://10.2.0.17:9308/]}}
[WARN ] 2023-02-18 18:35:45.897 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://10.2.0.17:9308/"}
[INFO ] 2023-02-18 18:35:45.980 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2023-02-18 18:35:45.981 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[ERROR] 2023-02-18 18:35:45.991 [[main]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"main", :exception=>#<TypeError: no implicit conversion of String into Integer>, :backtrace=>["org/jruby/RubyArray.java:952:in `fetch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/license_checker.rb:39:in `valid_es_license?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/license_checker.rb:17:in `appropriate_license?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:in `block in healthcheck!'", "org/jruby/ext/thread/Mutex.java:164:in `synchronize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:274:in `block in healthcheck!'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:265:in `healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:367:in `update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:83:in `update_initial_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:77:in `start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:303:in `build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:64:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:106:in `create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:102:in `build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:34:in `build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch.rb:270:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:585:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/file.conf"], :thread=>"#<Thread:0x3fd0eae4 run>"}
[INFO ] 2023-02-18 18:35:45.993 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[ERROR] 2023-02-18 18:35:46.007 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[INFO ] 2023-02-18 18:35:46.043 [LogStash::Runner] runner - Logstash shut down.

I also tried ubuntu (22.04 LTM), but the situation was the same. . .
Thank you for helping me so many times.

I kindly thank you.

nw-engineer Feb 18, 2023
Author

When I changed Logstash as follows, the error no longer appeared.

old) logstash-7.12.0-1.x86_64
new) logstash-oss-7.12.1-1.x86_64

[root@logstash001 logstash]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/file.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2023-02-18 23:08:27.525 [main] runner - Starting Logstash {"logstash.version"=>"7.12.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86_64]"}
[WARN ] 2023-02-18 23:08:27.700 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2023-02-18 23:08:28.008 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2023-02-18 23:08:28.457 [Converge PipelineAction::Create<main>] Reflections - Reflections took 19 ms to scan 1 urls, producing 23 keys and 47 values
[WARN ] 2023-02-18 23:08:28.679 [Converge PipelineAction::Create<main>] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2023-02-18 23:08:28.868 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://10.2.0.17:9308/]}}
[WARN ] 2023-02-18 23:08:28.937 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://10.2.0.17:9308/"}
[INFO ] 2023-02-18 23:08:29.018 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2023-02-18 23:08:29.019 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2023-02-18 23:08:29.029 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://10.2.0.17:9308"]}
[INFO ] 2023-02-18 23:08:29.069 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/file.conf"], :thread=>"#<Thread:0x2d13ce95 run>"}
[INFO ] 2023-02-18 23:08:29.444 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.37}
[INFO ] 2023-02-18 23:08:29.553 [[main]-pipeline-manager] file - No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_15940cad53dd1d99808eeaecd6f6ad3f", :path=>["/var/log/httpd/access_log"]}
[INFO ] 2023-02-18 23:08:29.560 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2023-02-18 23:08:29.588 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2023-02-18 23:08:29.595 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

However, an error occurs when data transfer is started.

[ERROR] 2023-02-18 23:09:35.969 [[main]>worker0] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff  {:code=>500, :url=>"http://10.2.0.17:9308/_bulk", :content_length=>407}
[ERROR] 2023-02-18 23:09:35.973 [[main]>worker2] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff  {:code=>500, :url=>"http://10.2.0.17:9308/_bulk", :content_length=>814}
[ERROR] 2023-02-18 23:09:35.976 [[main]>worker3] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff  {:code=>500, :url=>"http://10.2.0.17:9308/_bulk", :content_length=>326}
[ERROR] 2023-02-18 23:09:35.977 [[main]>worker1] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff  {:code=>500, :url=>"http://10.2.0.17:9308/_bulk", :content_length=>409}

Attached is the packet information when the internal error is issued.（And the log when "--logdebugvv" is enabled）
result_of_dump.txt
logdegvv.txt

Also, as far as I can confirm with "show variables", there is no item for "auto_schema", is this correct?

MySQL [(none)]> show variables;
+--------------------------+----------+
| Variable_name            | Value    |
+--------------------------+----------+
| autocommit               | 1        |
| auto_optimize            | 1        |
| optimize_cutoff          | 16       |
| collation_connection     | libc_ci  |
| query_log_format         | sphinxql |
| session_read_only        | 0        |
| log_level                | info     |
| max_allowed_packet       | 8388608  |
| character_set_client     | utf8     |
| character_set_connection | utf8     |
| grouping_in_utc          | 0        |
| last_insert_id           |          |
| pseudo_sharding          | 1        |
| secondary_indexes        | 1        |
| accurate_aggregation     | 0        |
| threads_ex_effective     |          |
| threads_ex               |          |
+--------------------------+----------+
17 rows in set (0.000 sec)

MySQL [(none)]> exit

nw-engineer Feb 18, 2023
Author

I checked if it is possible to insert without creating a table with the mysql command instead of Logstash.

[root@manticoresearch testindex001]# mysql -P 9306 -u root -h 10.2.0.17
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 68
Server version: 6.0.3 4f5acaf5d@230217 dev (columnar 2.0.1 37e6c01@230209) (secondary 2.0.1 37e6c01@230209) git branch HEAD (no branch)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> drop table testindex003;
Query OK, 0 rows affected (0.001 sec)

MySQL [(none)]>
MySQL [(none)]>
MySQL [(none)]>
MySQL [(none)]> show tables;
+--------------+------+
| Index        | Type |
+--------------+------+
| testindex001 | rt   |
+--------------+------+
1 row in set (0.000 sec)

MySQL [(none)]> insert into testindex003 (message, path, host) values('10.2.0.191 - - [19/Feb/2023:01:19:57 +0900] "GET /noindex/css/fonts/Light/OpenSans-Light.woff HTTP/1.1" 404 241 "http://10.2.0.15/noindex/css/open-sans.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"', '/var/log/httpd/access_log', 'logstash001');
Query OK, 1 row affected (0.005 sec)

MySQL [(none)]>
MySQL [(none)]> show tables;
+--------------+------+
| Index        | Type |
+--------------+------+
| testindex001 | rt   |
| testindex003 | rt   |
+--------------+------+
2 rows in set (0.000 sec)

MySQL [(none)]> select * from testindex003;
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+-------------+
| id                  | message                                                                                                                                                                                                                                                                        | path                      | host        |
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+-------------+
| 4973988392770469897 | 10.2.0.191 - - [19/Feb/2023:01:19:57  0900] "GET /noindex/css/fonts/Light/OpenSans-Light.woff HTTP/1.1" 404 241 "http://10.2.0.15/noindex/css/open-sans.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36" | /var/log/httpd/access_log | logstash001 |
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+-------------+
1 row in set (0.000 sec)

MySQL [(none)]>
MySQL [(none)]> show create table testindex003;
+--------------+-----------------------------------------------------------------------------+
| Table        | Create Table                                                                |
+--------------+-----------------------------------------------------------------------------+
| testindex003 | CREATE TABLE testindex003 (
id bigint,
message text,
path text,
host text
) |
+--------------+-----------------------------------------------------------------------------+
1 row in set (0.000 sec)

MySQL [(none)]>

Inserted without problems. . . . (increasingly mysterious)

tomatolog · 2023-02-20T09:04:46Z

tomatolog
Feb 20, 2023
Collaborator

for me

curl -sX POST "localhost:9312/_bulk" -H "content-type: application/x-ndjson" --data-binary @lstash-bulk2.log

with the following content lstash-bulk2.log failed with the error message

{"items":[{"index":{"_index":"testindex001","_type":"doc","_id":"0","status":400,"error":{"type":"mapper_parsing_exception","reason":"table 'testindex001' absent, or does not support INSERT"}}},{"index":{"_index":"testindex001","_type":"doc","_id":"0","status":400,"error":{"type":"mapper_parsing_exception","reason":"table 'testindex001' absent, or does not support INSERT"}}}],"errors":true,"took":1}

1 reply

nw-engineer Feb 20, 2023
Author

@tomatolog

I got the same result.

{"items":[{"index":{"_index":"testindex005","_type":"doc","_id":"0","status":400,"error":{"type":"mapper_parsing_exception","reason":"table 'testindex005' absent, or does not support INSERT"}}},{"index":{"_index":"testindex005","_type":"doc","_id":"0","status":400,"error":{"type":"mapper_parsing_exception","reason":"table 'testindex005' absent, or does not support INSERT"}}}],"errors":true,"took":1}

Nick-S-2018 · 2023-02-20T09:47:49Z

Nick-S-2018
Feb 20, 2023
Maintainer

Probably, there was some misunderstanding. In my first comment I meant we are now working on both the support of higher Logstash versions and auto table creation. Currently, installing manticore-extra won't help if you work with Logstash. This feature is to appear in the next update. It'll also be available in next dev packages.

1 reply

nw-engineer Feb 21, 2023
Author

It'll also be available in next dev packages.

Thank you for contacting us.
We are looking forward to the next update.

nw-engineer · 2023-07-02T10:04:04Z

nw-engineer
Jul 2, 2023
Author

Hello.

I used the latest development package.
I was able to confirm that data can be inserted from Logstash without having to define a schema in advance.

However, when I tried to send Paloalto Logs using the Logstash Filter file, I got an error.

[INFO ] 2023-07-02 18:55:07.383 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
Oct 09 10:19:15 SumPunFw07.sumotest.com 1,2019/10/09 10:19:15,001234567890002,TRAFFIC,drop,2304,2019/10/09 10:19:15,209.118.103.150,160.177.222.249,0.0.0.0,0.0.0.0,InternalServer,,,not-applicable,vsys1,inside,z1-FW-Transit,ethernet1/2,,All traffic,2019/10/09 10:19:15,0,1,63712,443,0,0,0x0,udp,deny,60,60,0,1,2019/10/09 10:19:15,0,any,0,0123456789,0x0,Netherlands,10.0.0.0-10.255.255.255,0,1,0,policy-deny,0,0,0,0,,SumPunFw07,from-policy,,,0,,0,,N/A,0,0,0,0,1202585d-b4d5-5b4c-aaa2-d80d77ba456e,0
{
                     "LogAction" => "All traffic",
             "SCTPAssociationID" => "0",
               "DestinationZone" => "z1-FW-Transit",
                     "BytesSent" => "60",
                       "Packets" => "1",
                    "SourceZone" => "inside",
                         "Bytes" => "60",
               "ParentStartTime" => nil,
                    "SCTPChunks" => "0",
               "ParentSessionID" => "0",
                      "SourceIP" => "209.118.103.150",
                    "TunnelType" => "N/A",
                "SequenceNumber" => "0123456789",
                   "RepeatCount" => "1",
    "DeviceGroupHierarchyLevel1" => "0",
                   "ActionFlags" => "0x0",
           "DestinationLocation" => "10.0.0.0-10.255.255.255",
               "PacketsReceived" => "0",
                    "@timestamp" => 2023-07-02T09:55:12.044Z,
                    "SourcePort" => "63712",
                         "Flags" => "0x0",
             "DestinationVMUUID" => nil,
              "NATDestinationIP" => "0.0.0.0",
                      "Protocol" => "udp",
                   "NATSourceIP" => "0.0.0.0",
                   "PacketsSent" => "1",
                  "SourceVMUUID" => nil,
                 "TunnelID_IMSI" => "0",
    "DeviceGroupHierarchyLevel2" => "0",
              "SessionEndReason" => "policy-deny",
                          "Type" => "TRAFFIC",
              "InboundInterface" => "ethernet1/2",
                    "FUTURE_USE" => "0",
                      "column67" => "0",
                   "ElapsedTime" => "0",
                 "DestinationIP" => "160.177.222.249",
             "OutboundInterface" => nil,
                          "host" => "manticore-logstash",
                      "@version" => "1",
                    "DeviceName" => "SumPunFw07",
                   "Application" => "not-applicable",
                    "SourceUser" => nil,
            "NATDestinationPort" => "0",
               "DestinationUser" => nil,
                        "Action" => "deny",
                 "BytesReceived" => "0",
                 "NATSourcePort" => "0",
                     "StartTime" => "2019/10/09 10:19:15",
    "DeviceGroupHierarchyLevel3" => "0",
                  "SerialNumber" => "001234567890002",
                      "RuleName" => "InternalServer",
                 "VirtualSystem" => "vsys1",
                     "SessionID" => "0",
               "DestinationPort" => "443",
    "DeviceGroupHierarchyLevel4" => "0",
               "MonitorTag_IMEI" => nil,
                      "column66" => "1202585d-b4d5-5b4c-aaa2-d80d77ba456e",
                      "Category" => "any",
            "SCTPChunksReceived" => "0",
                 "GeneratedTime" => "2019/10/09 10:19:15",
                  "ActionSource" => "from-policy",
                "SCTPChunksSent" => "0",
            "Threat_ContentType" => "drop",
                   "ReceiveTime" => "2019/10/09 10:19:15",
                "SourceLocation" => "Netherlands",
             "VirtualSystemName" => nil
}
[ERROR] 2023-07-02 18:55:12.289 [[main]>worker0] elasticsearch - An unknown error occurred sending a bulk request to Elasticsearch. We will retry indefinitely {:error_message=>"no implicit conversion of String into Integer", :error_class=>"TypeError", :backtrace=>["org/jruby/RubyArray.java:1487:in `[]'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/http_client.rb:158:in `block in join_bulk_responses'", "org/jruby/RubyArray.java:4493:in `any?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/http_client.rb:158:in `join_bulk_responses'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch/http_client.rb:146:in `bulk'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:271:in `safe_bulk'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:192:in `submit'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:160:in `retrying_submit'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.6-java/lib/logstash/outputs/elasticsearch.rb:303:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:295:in `block in start_workers'"]}

I changed the log to recognize everything as a string and it inserted fine.

[INFO ] 2023-07-02 18:58:01.451 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pi lines=>[]}
"Oct 09 10:19:15 SumPunFw07.sumotest.com 1","2019/10/09 10:19:15","001234567890002","TRAFFIC","drop","2304","2019/10/09 10:19:15" 209.118.103.150","160.177.222.249","0.0.0.0","0.0.0.0","InternalServer","","","not-applicable","vsys1","inside","z1-FW-Transit"," hernet1/2","","All traffic","2019/10/09 10:19:15","0","1","63712","443","0","0","0x0","udp","deny","60","60","0","1","2019/10/09  :19:15","0","any","0","0123456789","0x0","Netherlands","10.0.0.0-10.255.255.255","0","1","0","policy-deny","0","0","0","0","","Su unFw07","from-policy","","","0","","0","","N/A","0","0","0","0","1202585d-b4d5-5b4c-aaa2-d80d77ba456e","0"
{
                 "NATSourcePort" => "0",
                "SCTPChunksSent" => "0",
                    "@timestamp" => 2023-07-02T09:58:15.582Z,
                     "StartTime" => "2019/10/09 10:19:15",
                    "SCTPChunks" => "0",
                 "BytesReceived" => "0",
                      "column66" => "1202585d-b4d5-5b4c-aaa2-d80d77ba456e",
                 "GeneratedTime" => "2019/10/09 10:19:15",
            "Threat_ContentType" => "drop",
               "PacketsReceived" => "0",
              "SessionEndReason" => "policy-deny",
                    "SourceZone" => "inside",
                    "TunnelType" => "N/A",
                "SourceLocation" => "Netherlands",
                        "Action" => "deny",
                   "PacketsSent" => "1",
                   "NATSourceIP" => "0.0.0.0",
              "InboundInterface" => "ethernet1/2",
                 "DestinationIP" => "160.177.222.249",
                  "SerialNumber" => "001234567890002",
    "DeviceGroupHierarchyLevel4" => "0",
                "SequenceNumber" => "0123456789",
                      "Category" => "any",
    "DeviceGroupHierarchyLevel3" => "0",
             "SCTPAssociationID" => "0",
             "VirtualSystemName" => "",
            "SCTPChunksReceived" => "0",
                    "SourceUser" => "",
               "DestinationZone" => "z1-FW-Transit",
           "DestinationLocation" => "10.0.0.0-10.255.255.255",
              "NATDestinationIP" => "0.0.0.0",
               "ParentSessionID" => "0",
               "DestinationUser" => "",
                      "Protocol" => "udp",
                  "ActionSource" => "from-policy",
                    "SourcePort" => "63712",
             "DestinationVMUUID" => "",
               "ParentStartTime" => "",
                      "SourceIP" => "209.118.103.150",
                   "Application" => "not-applicable",
                      "column67" => "0",
                       "Packets" => "1",
                 "VirtualSystem" => "vsys1",
                    "DeviceName" => "SumPunFw07",
                    "FUTURE_USE" => "0",
                  "SourceVMUUID" => "",
                 "TunnelID_IMSI" => "0",
                     "LogAction" => "All traffic",
                   "RepeatCount" => "1",
               "DestinationPort" => "443",
                     "SessionID" => "0",
                   "ActionFlags" => "0x0",
                     "BytesSent" => "60",
                         "Bytes" => "60",
                          "Type" => "TRAFFIC",
                         "Flags" => "0x0",
    "DeviceGroupHierarchyLevel2" => "0",
                      "RuleName" => "InternalServer",
                      "@version" => "1",
                   "ReceiveTime" => "2019/10/09 10:19:15",
             "OutboundInterface" => "",
                          "host" => "manticore-logstash",
                   "ElapsedTime" => "0",
    "DeviceGroupHierarchyLevel1" => "0",
            "NATDestinationPort" => "0",
               "MonitorTag_IMEI" => ""
}

If you check on the server side, it is inserted correctly.

mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 2566
Server version: 6.0.5 1399e6f6b@230629 dev (columnar 2.0.5 106d244@230620) (secondary 2.0.5 106d244@230620) git branch master...origin/master

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> select * from paloaltoindex001;
+---------------------+------------------+-------------------+------------+---------------------+-------------+------------------+---------------------+-----------------+----------+----------------+----------------------------+------------+--------------------+-----------+---------------+-----------------+------------+-------+------------+-----------------+-----------------+-----------------+----------------------------+----------+----------+--------------+-----------------+-------+-------------+----------------+--------------------+----------------------------+------------------+---------+----------------+-------------------+---------------+--------------+------------+---------------+-----------+-------------------+---------+-------------+--------+---------------------+--------------------------------------+--------------------+-------------+------------+----------------+---------------+----------------------------+-----------------+------------+-----------------+-----------------+-------------+----------------+-------------+--------------------------+----------+-------------------------+-----------------+--------------------+-------------------+
| id                  | sessionendreason | virtualsystemname | sourceuser | receivetime         | natsourceip | natdestinationip | starttime           | parentsessionid | category | rulename       | devicegrouphierarchylevel3 | sctpchunks | host               | bytessent | tunnelid_imsi | destinationip   | tunneltype | bytes | future_use | parentstarttime | destinationuser | packetsreceived | devicegrouphierarchylevel4 | column67 | @version | actionsource | destinationport | flags | elapsedtime | sourcelocation | natdestinationport | devicegrouphierarchylevel2 | inboundinterface | packets | sctpchunkssent | sctpassociationid | bytesreceived | sourcevmuuid | sourcezone | virtualsystem | sessionid | destinationvmuuid | type    | repeatcount | action | generatedtime       | column66                             | threat_contenttype | packetssent | sourceport | application    | natsourceport | devicegrouphierarchylevel1 | sourceip        | devicename | monitortag_imei | serialnumber    | logaction   | sequencenumber | actionflags | @timestamp               | protocol | destinationlocation     | destinationzone | sctpchunksreceived | outboundinterface |
+---------------------+------------------+-------------------+------------+---------------------+-------------+------------------+---------------------+-----------------+----------+----------------+----------------------------+------------+--------------------+-----------+---------------+-----------------+------------+-------+------------+-----------------+-----------------+-----------------+----------------------------+----------+----------+--------------+-----------------+-------+-------------+----------------+--------------------+----------------------------+------------------+---------+----------------+-------------------+---------------+--------------+------------+---------------+-----------+-------------------+---------+-------------+--------+---------------------+--------------------------------------+--------------------+-------------+------------+----------------+---------------+----------------------------+-----------------+------------+-----------------+-----------------+-------------+----------------+-------------+--------------------------+----------+-------------------------+-----------------+--------------------+-------------------+
| 4974179136428834820 | policy-deny      |                   |            | 2019/10/09 10:19:15 | 0.0.0.0     | 0.0.0.0          | 2019/10/09 10:19:15 | 0               | any      | InternalServer | 0                          | 0          | manticore-logstash | 60        | 0             | 160.177.222.249 | N/A        | 60    | 0          |                 |                 | 0               | 0                          | 0        | 1        | from-policy  | 443             | 0x0   | 0           | Netherlands    | 0                  | 0                          | ethernet1/2      | 1       | 0              | 0                 | 0             |              | inside     | vsys1         | 0         |                   | TRAFFIC | 1           | deny   | 2019/10/09 10:19:15 | 1202585d-b4d5-5b4c-aaa2-d80d77ba456e | drop               | 1           | 63712      | not-applicable | 0             | 0                          | 209.118.103.150 | SumPunFw07 |                 | 001234567890002 | All traffic | 0123456789     | 0x0         | 2023-07-02T08:36:46.080Z | udp      | 10.0.0.0-10.255.255.255 | z1-FW-Transit   | 0                  |                   |
| 4974179136428834821 | policy-deny      |                   |            | 2019/10/09 10:19:15 | 0.0.0.0     | 0.0.0.0          | 2019/10/09 10:19:15 | 0               | any      | InternalServer | 0                          | 0          | manticore-logstash | 60        | 0             | 160.177.222.249 | N/A        | 60    | 0          |                 |                 | 0               | 0                          | 0        | 1        | from-policy  | 443             | 0x0   | 0           | Netherlands    | 0                  | 0                          | ethernet1/2      | 1       | 0              | 0                 | 0             |              | inside     | vsys1         | 0         |                   | TRAFFIC | 1           | deny   | 2019/10/09 10:19:15 | 1202585d-b4d5-5b4c-aaa2-d80d77ba456e | drop               | 1           | 63712      | not-applicable | 0             | 0                          | 209.118.103.150 | SumPunFw07 |                 | 001234567890002 | All traffic | 0123456789     | 0x0         | 2023-07-02T09:10:46.427Z | udp      | 10.0.0.0-10.255.255.255 | z1-FW-Transit   | 0                  |                   |
| 4974179136428834822 | policy-deny      |                   |            | 2019/10/09 10:19:15 | 0.0.0.0     | 0.0.0.0          | 2019/10/09 10:19:15 | 0               | any      | InternalServer | 0                          | 0          | manticore-logstash | 60        | 0             | 160.177.222.249 | N/A        | 60    | 0          |                 |                 | 0               | 0                          | 0        | 1        | from-policy  | 443             | 0x0   | 0           | Netherlands    | 0                  | 0                          | ethernet1/2      | 1       | 0              | 0                 | 0             |              | inside     | vsys1         | 0         |                   | TRAFFIC | 1           | deny   | 2019/10/09 10:19:15 | 1202585d-b4d5-5b4c-aaa2-d80d77ba456e | drop               | 1           | 63712      | not-applicable | 0             | 0                          | 209.118.103.150 | SumPunFw07 |                 | 001234567890002 | All traffic | 0123456789     | 0x0         | 2023-07-02T09:58:15.582Z | udp      | 10.0.0.0-10.255.255.255 | z1-FW-Transit   | 0                  |                   |
| 4974179136428834819 | policy-deny      |                   |            | 2019/10/09 10:19:15 | 0.0.0.0     | 0.0.0.0          | 2019/10/09 10:19:15 | 0               | any      | InternalServer | 0                          | 0          | manticore-logstash | 60        | 0             | 160.177.222.249 | N/A        | 60    | 0          |                 |                 | 0               | 0                          | 0        | 1        | from-policy  | 443             | 0x0   | 0           | Netherlands    | 0                  | 0                          | ethernet1/2      | 1       | 0              | 0                 | 0             |              | inside     | vsys1         | 0         |                   | TRAFFIC | 1           | deny   | 2019/10/09 10:19:15 | 1202585d-b4d5-5b4c-aaa2-d80d77ba456e | drop               | 1           | 63712      | not-applicable | 0             | 0                          | 209.118.103.150 | SumPunFw07 |                 | 001234567890002 | All traffic | 0123456789     | 0x0         | 2023-07-02T07:53:58.163Z | udp      | 10.0.0.0-10.255.255.255 | z1-FW-Transit   | 0                  |                   |
+---------------------+------------------+-------------------+------------+---------------------+-------------+------------------+---------------------+-----------------+----------+----------------+----------------------------+------------+--------------------+-----------+---------------+-----------------+------------+-------+------------+-----------------+-----------------+-----------------+----------------------------+----------+----------+--------------+-----------------+-------+-------------+----------------+--------------------+----------------------------+------------------+---------+----------------+-------------------+---------------+--------------+------------+---------------+-----------+-------------------+---------+-------------+--------+---------------------+--------------------------------------+--------------------+-------------+------------+----------------+---------------+----------------------------+-----------------+------------+-----------------+-----------------+-------------+----------------+-------------+--------------------------+----------+-------------------------+-----------------+--------------------+-------------------+
4 rows in set (0.004 sec)

MySQL [(none)]>

Is this "problem" already known and being fixed?
Thank you.

4 replies

tomatolog Jul 2, 2023
Collaborator

we have a ticket to allow string to integer conversion on data insert
I will inform you on push the fix into master

nw-engineer Jul 2, 2023
Author

thank you!
We are looking forward to hearing from you!

nw-engineer Jul 4, 2024
Author

@tomatolog
we have a ticket to allow string to integer conversion on data insert
I will inform you on push the fix into master

Do you have an estimated time for when this will be pushed to master?

sanikolaev Jul 4, 2024
Maintainer

@nw-engineer the likely related ticket is #2312

but it would be great if you prepared a minimal reproducible example to reproduce the issue you are experiencing to make sure it will be solved by #2312.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

About logstash #1048

{{title}}

Replies: 4 comments 17 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

About logstash #1048

nw-engineer Feb 17, 2023

Replies: 4 comments · 17 replies

Nick-S-2018 Feb 17, 2023 Maintainer

nw-engineer Feb 18, 2023 Author

sanikolaev Feb 18, 2023 Maintainer

nw-engineer Feb 18, 2023 Author

nw-engineer Feb 18, 2023 Author

nw-engineer Feb 18, 2023 Author

tomatolog Feb 20, 2023 Collaborator

nw-engineer Feb 20, 2023 Author

Nick-S-2018 Feb 20, 2023 Maintainer

nw-engineer Feb 21, 2023 Author

nw-engineer Jul 2, 2023 Author

tomatolog Jul 2, 2023 Collaborator

nw-engineer Jul 2, 2023 Author

nw-engineer Jul 4, 2024 Author

sanikolaev Jul 4, 2024 Maintainer

nw-engineer
Feb 17, 2023

Replies: 4 comments 17 replies

Nick-S-2018
Feb 17, 2023
Maintainer

nw-engineer Feb 18, 2023
Author

sanikolaev Feb 18, 2023
Maintainer

nw-engineer Feb 18, 2023
Author

nw-engineer Feb 18, 2023
Author

nw-engineer Feb 18, 2023
Author

tomatolog
Feb 20, 2023
Collaborator

nw-engineer Feb 20, 2023
Author

Nick-S-2018
Feb 20, 2023
Maintainer

nw-engineer Feb 21, 2023
Author

nw-engineer
Jul 2, 2023
Author

tomatolog Jul 2, 2023
Collaborator

nw-engineer Jul 2, 2023
Author

nw-engineer Jul 4, 2024
Author

sanikolaev Jul 4, 2024
Maintainer