OPCUA plugin fails to connect to OPCUA server from Codesys when running on groov RIO

[johnatark]

Relevant telegraf.conf

[agent]
  interval = "5s" #frequency telegraf will poll the opcua server
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 1000000
  collection_jitter = "0s"
  flush_interval = "30s" #frequency telegraf will push data to the database
  flush_jitter = "0s"
  precision = "1s"
  debug = true
#   quiet = false
#   logfile = ""
    ## Override default hostname, if empty use os.Hostname()
  hostname = ""
  ## If set to true, do no set the "host" tag in the telegraf agent.
  omit_hostname = true

# # Print outputs to stdout, useful for debugging
# [[outputs.file]]
#   files = ["stdout"]
#   data_format="json"

# Configurable HTTP health check resource based on metrics
[[outputs.health]]

[[outputs.postgresql]]
  connection = "postgres://${POSTGRES_USERNAME}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}/${POSTGRES_DB}?sslmode=${POSTGRES_SSL_MODE}"

  ## Postgres schema to use.
  schema = "metric_data"

  ## Templated statements to execute when creating a new table.
  create_templates = [
    '''CREATE TABLE {{ .table }} ({{ .columns }})''',
    '''SELECT create_hypertable({{ .table|quoteLiteral }}, 'time', chunk_time_interval => INTERVAL '7d')''',
    '''ALTER TABLE {{ .table }} SET (timescaledb.compress, timescaledb.compress_segmentby = 'controller_name')''',
    '''SELECT add_compression_policy({{ .table|quoteLiteral }}, INTERVAL '7d');''',
  ]

  ## Templated statements to execute when adding columns to a table.
  add_column_templates = [
    '''ALTER TABLE {{ .table }} ADD COLUMN IF NOT EXISTS {{ .columns|join ", ADD COLUMN IF NOT EXISTS " }}''',
  ]

  ## Enable & set the log level for the Postgres driver.
  log_level = "warn" # trace, debug, info, warn, error, none

  #drop all metrics intended for the metrics table
  # namedrop = ["metrics"]

###############################################################################
#                            PROCESSOR PLUGINS                                #
###############################################################################

# Process metrics using a Starlark script
[[processors.starlark]]
  ## The Starlark source can be set as a string in this configuration file, or
  ## by referencing a file containing the script.  Only one source or script
  ## should be set at once.

  ## Source of the Starlark script.
  script = "/etc/telegraf/script.star"

###############################################################################
#                            INPUT PLUGINS                                    #
###############################################################################
# Retrieve data from OPCUA devices that deliver constant data
[[inputs.opcua]]
  ## Metric name
  name = "opcua_0"
  #
  ## OPC UA Endpoint URL
  endpoint = "opc.tcp://182.234.251.111:4840"
  #
  ## Maximum time allowed to establish a connect to the endpoint.
  connect_timeout = "10s"
  #
  ## Maximum time allowed for a request over the established connection.
  request_timeout = "5s"

  # Maximum time that a session shall remain open without activity.
  # session_timeout = "20m"
  #
  ## Security policy, one of "None", "Basic128Rsa15", "Basic256",
  ## "Basic256Sha256", or "auto"
  security_policy = "None"
  #
  ## Security mode, one of "None", "Sign", "SignAndEncrypt", or "auto"
  security_mode = "None"
  #
  ## Path to cert.pem. Required when security mode or policy isn't "None".
  ## If cert path is not supplied, self-signed cert and key will be generated.
  # certificate = "/etc/telegraf/cert.pem"
  #
  ## Path to private key.pem. Required when security mode or policy isn't "None".
  ## If key path is not supplied, self-signed cert and key will be generated.
  # private_key = "/etc/telegraf/key.pem"
  #
  ## Authentication Method, one of "Certificate", "UserName", or "Anonymous".  To
  ## authenticate using a specific ID, select 'Certificate' or 'UserName'
  # auth_method = "Certificate"
  #
  ## Username. Required for auth_method = "UserName"
  # username = ""
  #
  ## Password. Required for auth_method = "UserName"
  # password = ""
  #
    ## Client trace messages
  ## When set to true, and debug mode enabled in the agent settings, the OPCUA
  ## client's messages are included in telegraf logs. These messages are very
  ## noisey, but essential for debugging issues.
  # client_trace = false
  #
  ## Option to select the metric timestamp to use. Valid options are:
  ##     "gather" -- uses the time of receiving the data in telegraf
  ##     "server" -- uses the timestamp provided by the server
  ##     "source" -- uses the timestamp provided by the source
  # timestamp = "gather"
  #
  ## Include additional Fields in each metric
  ## Available options are:
  ##   DataType -- OPC-UA Data Type (string)
  # optional_fields = ["DataType"]
  #
  ## Node ID configuration
  ## name              - field name to use in the output
  ## namespace         - OPC UA namespace of the node (integer value 0 thru 3)
  ## identifier_type   - OPC UA ID type (s=string, i=numeric, g=guid, b=opaque)
  ## identifier        - OPC UA ID (tag as shown in opcua browser)
  ## tags              - extra tags to be added to the output metric (optional); deprecated in 1.25.0; use default_tags
  ## default_tags      - extra tags to be added to the output metric (optional)
  ##
  # examples:
  
  ### Insert Metrics Below This Line
[[inputs.opcua.nodes]]  name = "OUTPUT/A1/PV/REAL"  namespace = "4"  identifier_type = "s"  identifier = "|var|Opto22-Cortex-Linux.Application.OUTPUT.A1.PV"  default_tags = {controller_name = "${CONTROLLER_NAME}", datatype = "REAL"}
[[inputs.opcua.nodes]]  name = "OUTPUT/A1/SP/REAL"  namespace = "4"  identifier_type = "s"  identifier = "|var|Opto22-Cortex-Linux.Application.OUTPUT.A1.SP"  default_tags = {controller_name = "${CONTROLLER_NAME}", datatype = "REAL"}

Logs from Telegraf

telegraf-1  | 2024-08-02T14:23:39Z I! Loaded inputs: opcua
telegraf-1  | 2024-08-02T14:23:39Z I! Loaded aggregators:
telegraf-1  | 2024-08-02T14:23:39Z I! Loaded processors: starlark
telegraf-1  | 2024-08-02T14:23:39Z I! Loaded secretstores:
telegraf-1  | 2024-08-02T14:23:39Z I! Loaded outputs: health postgresql
telegraf-1  | 2024-08-02T14:23:39Z I! Tags enabled:
telegraf-1  | 2024-08-02T14:23:39Z I! [agent] Config: Interval:5s, Quiet:false, Hostname:"", Flush Interval:30s
telegraf-1  | 2024-08-02T14:23:39Z D! [agent] Initializing plugins
telegraf-1  | 2024-08-02T14:23:39Z D! [inputs.opcua] Initialising OpcUAInputClient
telegraf-1  | 2024-08-02T14:23:39Z D! [inputs.opcua] Initialising OpcUAClient
telegraf-1  | 2024-08-02T14:23:39Z D! [inputs.opcua] Initialising node to metric mapping
telegraf-1  | 2024-08-02T14:23:39Z D! [agent] Connecting outputs
telegraf-1  | 2024-08-02T14:23:39Z D! [agent] Attempting connection to [outputs.health]
telegraf-1  | 2024-08-02T14:23:39Z I! [outputs.health] Listening on http://[::]:8080
telegraf-1  | 2024-08-02T14:23:39Z D! [agent] Successfully connected to outputs.health
telegraf-1  | 2024-08-02T14:23:39Z D! [agent] Attempting connection to [outputs.postgresql]
telegraf-1  | 2024-08-02T14:23:39Z D! [agent] Successfully connected to outputs.postgresql
telegraf-1  | 2024-08-02T14:23:39Z D! [agent] Starting service inputs
telegraf-1  | 2024-08-02T14:23:40Z D! [inputs.opcua] Connecting OPC UA Client to server
telegraf-1  | 2024-08-02T14:23:40Z D! [inputs.opcua] Configuring OPC UA connection options
telegraf-1  | 2024-08-02T14:23:40Z W! [inputs.opcua] Failed to load certificate: open /etc/telegraf/cert.pem: no such file or directory
telegraf-1  | 2024-08-02T14:23:40Z D! [inputs.opcua] security policy from configuration http://opcfoundation.org/UA/SecurityPolicy#None
telegraf-1  | 2024-08-02T14:23:40Z D! [inputs.opcua] User cares about both the policy (http://opcfoundation.org/UA/SecurityPolicy#None) and security mode (MessageSecurityModeNone)
telegraf-1  | 2024-08-02T14:23:40Z D! [inputs.opcua] Server has 2 endpoints
telegraf-1  | 2024-08-02T14:23:40Z D! [inputs.opcua] Evaluating endpoint opc.tcp://RIO4:4840, policy http://opcfoundation.org/UA/SecurityPolicy#None, mode MessageSecurityModeNone, level 0
telegraf-1  | 2024-08-02T14:23:40Z D! [inputs.opcua] Security policy and mode found. Using server endpoint opc.tcp://RIO4:4840 for security. Policy http://opcfoundation.org/UA/SecurityPolicy#None
telegraf-1  | 2024-08-02T14:23:40Z D! [inputs.opcua] Evaluating endpoint opc.tcp://182.234.251.7:4840, policy http://opcfoundation.org/UA/SecurityPolicy#None, mode MessageSecurityModeNone, level 0
telegraf-1  | 2024-08-02T14:23:40Z D! [inputs.opcua] Security policy and mode found. Using server endpoint opc.tcp://182.234.251.7:4840 for security. Policy http://opcfoundation.org/UA/SecurityPolicy#None
telegraf-1  | 2024-08-02T14:23:45Z D! [inputs.opcua] Previous collection has not completed; scheduled collection skipped
telegraf-1  | 2024-08-02T14:23:45Z W! [inputs.opcua] Collection took longer than expected; not complete after interval of 5s
telegraf-1  | 2024-08-02T14:23:45Z E! [inputs.opcua] Error in plugin: error in Client Connection: The operation timed out. StatusBadTimeout (0x800A0000)

System info

Telegraf v1.26 running in Docker on windows WSL. Connected to OPCUA server from Codesys running on a groov RIO that is available on a local ethernet network

Docker

services:
telegraf:
build:
context: .
volumes:
- ./telegraf.conf:/etc/telegraf/telegraf.conf:ro
- ./script.star:/etc/telegraf/script.star:ro
env_file:
- ./env/local.env
networks:
- reflex-docker
# depends_on:
# timescale:
# condition: service_healthy
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080"]
interval: 10s
timeout: 10s
retries: 3
restart: unless-stopped

networks:
reflex-docker:
driver: bridge
external: true

Steps to reproduce

1. Run a OPCUA server on the groov RIO
2. Connect to the RIO over ethernet (as a bonus, ensure you can ping the RIO, and that you can access the opcua server at ip:4840 in UAExpert)
3. Run Telegraf

Expected behavior

Telegraf should pull data from the opcua server and log it in my postgres database.

Actual behavior

Telegraf successully identifies the opcua server, which is clear because it able to locate its second endpoint. Telegraf then times out with an unkown error and is unable to get any data from the opcua server.

Additional info

I have an identical setup using Codesys on the groov Epic, which works perfectly. I have both devices connected on the same Ethernet network, if I change the ip address in my config file to the Epic's IP (and make no other changes) Telegraf successfully pulls data over opcua. From the log files Telegraf clearly is able to connect to the relevant OPCUA server as it is able to find it's other endpoint, but then fails for unkown reasons. I am able to successfully talk to the server on Rio over both UA Expert and Ignition, which is what leads me to believe it is a bug with Telegraf.

For a comparison here are the log messages when I talk to the server on the Epic instead of the Rio:

telegraf-1  | 2024-08-02T14:33:44Z I! Loaded aggregators:
telegraf-1  | 2024-08-02T14:33:44Z I! Loaded processors: starlark
telegraf-1  | 2024-08-02T14:33:44Z I! Loaded secretstores:
telegraf-1  | 2024-08-02T14:33:44Z I! Loaded outputs: health postgresql
telegraf-1  | 2024-08-02T14:33:44Z I! Tags enabled:
telegraf-1  | 2024-08-02T14:33:44Z I! [agent] Config: Interval:5s, Quiet:false, Hostname:"", Flush Interval:30s
telegraf-1  | 2024-08-02T14:33:44Z D! [agent] Initializing plugins
telegraf-1  | 2024-08-02T14:33:44Z D! [inputs.opcua] Initialising OpcUAInputClient
telegraf-1  | 2024-08-02T14:33:44Z D! [inputs.opcua] Initialising OpcUAClient
telegraf-1  | 2024-08-02T14:33:44Z D! [inputs.opcua] Initialising node to metric mapping
telegraf-1  | 2024-08-02T14:33:44Z D! [agent] Connecting outputs
telegraf-1  | 2024-08-02T14:33:44Z D! [agent] Attempting connection to [outputs.postgresql]
telegraf-1  | 2024-08-02T14:33:44Z D! [agent] Successfully connected to outputs.postgresql
telegraf-1  | 2024-08-02T14:33:44Z D! [agent] Attempting connection to [outputs.health]
telegraf-1  | 2024-08-02T14:33:44Z I! [outputs.health] Listening on http://[::]:8080
telegraf-1  | 2024-08-02T14:33:44Z D! [agent] Successfully connected to outputs.health
telegraf-1  | 2024-08-02T14:33:44Z D! [agent] Starting service inputs
telegraf-1  | 2024-08-02T14:33:45Z D! [inputs.opcua] Connecting OPC UA Client to server
telegraf-1  | 2024-08-02T14:33:45Z D! [inputs.opcua] Configuring OPC UA connection options
telegraf-1  | 2024-08-02T14:33:45Z W! [inputs.opcua] Failed to load certificate: open /etc/telegraf/cert.pem: no such file or directory
telegraf-1  | 2024-08-02T14:33:45Z D! [inputs.opcua] security policy from configuration http://opcfoundation.org/UA/SecurityPolicy#None
telegraf-1  | 2024-08-02T14:33:45Z D! [inputs.opcua] User cares about both the policy (http://opcfoundation.org/UA/SecurityPolicy#None) and security mode (MessageSecurityModeNone)
telegraf-1  | 2024-08-02T14:33:45Z D! [inputs.opcua] Server has 2 endpoints
telegraf-1  | 2024-08-02T14:33:45Z D! [inputs.opcua] Evaluating endpoint opc.tcp://Opto11:4840, policy http://opcfoundation.org/UA/SecurityPolicy#None, mode MessageSecurityModeNone, level 0
telegraf-1  | 2024-08-02T14:33:45Z D! [inputs.opcua] Security policy and mode found. Using server endpoint opc.tcp://Opto11:4840 for security. Policy http://opcfoundation.org/UA/SecurityPolicy#None
telegraf-1  | 2024-08-02T14:33:45Z D! [inputs.opcua] Evaluating endpoint opc.tcp://182.234.251.111:4840, policy http://opcfoundation.org/UA/SecurityPolicy#None, mode MessageSecurityModeNone, level 0
telegraf-1  | 2024-08-02T14:33:45Z D! [inputs.opcua] Security policy and mode found. Using server endpoint opc.tcp://182.234.251.111:4840 for security. Policy http://opcfoundation.org/UA/SecurityPolicy#None
telegraf-1  | 2024-08-02T14:33:47Z D! [inputs.opcua] Connected to OPC UA Server
telegraf-1  | 2024-08-02T14:33:48Z D! [processors.starlark] Processing data metric: Metric("opcua_0", tags={"controller_name": "Epic", "datatype": "REAL", "id": "ns=4;s=|var|Opto22-Cortex-Linux.Application.OUTPUT.A1.PV"}, fields={"OUTPUT/A1/PV/REAL": 0.0, "Quality": "OK (0x0)"}, time=1722609228000000000)
...

I am also talking with Opto22 (the company that makes both of these products, the Epic and the RIO) to see what differences there could be at a low level on their end, but they believe both systems should be identical in this regard.

[powersj]

Hi,

I am also talking with Opto22 (the company that makes both of these products, the Epic and the RIO) to see what differences there could be at a low level on their end

A couple things for you to try as well:

1. Telegraf v1.26 is pretty old and there have been some changes to the opcua plugin, including updating the underlying go opcua library that handles the connection. It would be really good to update to the the latest telegraf.
2. In the latest telegraf there is a client_trace option. Please set that to true, and keep your agent debug set to true. That will produce many more logs about the opcua library's connection and status.
3. Because this is timeout related, could you try setting request_timeout to something larger than the default of 5 seconds. Maybe 15s, just to eliminate if the device is slow at responding.

Thanks!

edit:

two upstream issues with the same error message:

• Potential Bug: StatusBadTimeout Error when attempting connection using numeric id gopcua/opcua#478 - looked to be around weird error message around registered reads using numeric ids
• The sequence number is unknown to the server. StatusBadSequenceNumberUnknown (0x807A0000) gopcua/opcua#270 - timeout related

[telegraf-tiger]

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Forums or provide additional details in this issue and reqeust that it be re-opened. Thank you!