How to parse custom log file?

[hightwoltt]

I want to use vector to read logs from the vmware-vmsvc-root.log file and convert them to json for sending to victoriametris logs.
But no data is being transmitted... Unfortunately, I could not find data on my problem or a similar problem in the documentation. What am I doing wrong? Please tell me how this functionality can be implemented??

my configuration:

# /etc/vector/vector.yaml

data_dir: /var/lib/vector

api:
   enabled: true

sources:

  # Getting vmware.log logs
  vmware_logs:
    type: file
    include:
      - /var/log/vmware-vmsvc-root.log


# Transform getting data
transforms:
  vmware_logs_transform:
    type: remap
    inputs:
      - vmware_logs
    source: |
      match = parse_regex!(.message, r'^\[(?P<timestamp>[^\]]+)\] \[ (?P<level>[^\]]+)\] \[(?P<service>[^\]]+)\] \[(?P<code>[^\]]+)\] (?P<message>.*)$')
      if !is_null(match){
        .timestamp = parse_timestamp!(match.timestamp, "%Y-%m-%dT%H:%M:%S.%3fZ")
        .level = match.level
        .service = match.service
        .code = match.code
        .message = match.message
      }


sinks:
  
  victorialogs_vmware:
    type: http
    inputs:
      - vmware_logs_transform
    uri: "http://192.168.1.3:9428/insert/jsonline"
    headers:
        AccountID: '0'
        ProjectID: '0'
        VL-Stream-Fields: source=vmware-vmswc-root.log,host=192.196.1.2,environment=production
        VL-Msg-Field: message.msg
        VL-Time-Field: timestamp
    compression: gzip
    encoding:
      codec: json
    healthcheck:
      enabled: false

  console_parse:
    type: console
    inputs:
      - vmware_logs_transform
    encoding:
      codec: json

[pront]

Hi @hightwoltt, here is a list of tools/techniques we use to diagnose issues:

• For manual tests, you can connect each source or transform to a console sink
• With vector tap you see the input and/or outputs of your components (caveat: sinks don't have outputs) - guide here
• vector top to see component metrics - docs here
• You can also add an internal_metrics source and manually inspect the metrics
• For your case, also make sure Vector has access to the file listed under include. I would start from the file source to ensure events are produced and take it from there.