Dashboards: Regression on InfluxQL query variable: found $, expected ) #80003

melroy89 · 2024-01-03T23:25:42Z

What happened?

On the latest Grafana version I have regression with InfluxDB with InfluxQL. path variable is not empty, but still gives an error in the query (however, this used to work fine.. ):

When manually selecting a path without the use of the variable it does work:

Console log:

XHRPOST
https://stats.melroy.org/api/ds/query?ds_type=influxdb&requestId=Q217
[HTTP/2 400  2ms]

	
results	Object { A: {…} }
A	Object { error: "InfluxDB returned error: error parsing query: found media, expected ; at line 1, char 122", status: 500, errorSource: "" }
error	"InfluxDB returned error: error parsing query: found media, expected ; at line 1, char 122"
errorSource	""
status	500

What did you expect to happen?

No errors on this query, it's a valid InfluxQL query.

Did this work before?

I use set this path as variable in InfluxQL together with the repeat feature, so I repeats for each of the selected mount/paths. This used to work fine on older Grafana versions.

How do we reproduce it?

Use a query variable called path with the query: SHOW TAG VALUES FROM "disk" WITH KEY = "path"
Use this query within InfluxDB version 1 using Grafana as shown above in the query
Turn ON repeat in the Panel settings for path variable as well.
Error: InfluxDB returned error: error parsing query: found $, expected ) at line 1, char 98

Is the bug inside a dashboard panel?

Key	Value
Panel	gauge @ 10.2.3
Grafana	10.2.3 (`1e84fed`) // Open Source

Panel debug snapshot dashboard

{
  "panels": [
    {
      "datasource": {
        "type": "grafana",
        "uid": "grafana"
      },
      "fieldConfig": {
        "defaults": {
          "mappings": [
            {
              "options": {
                "match": "null",
                "result": {
                  "text": "N/A"
                }
              },
              "type": "special"
            }
          ],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "orange",
                "value": 70
              },
              {
                "color": "red",
                "value": 85
              }
            ]
          },
          "unit": "percent"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 13,
        "w": 15,
        "x": 0,
        "y": 0
      },
      "id": 2,
      "links": [],
      "maxDataPoints": 100,
      "maxPerRow": 8,
      "options": {
        "reduceOptions": {
          "values": false,
          "calcs": [
            "lastNotNull"
          ],
          "fields": ""
        },
        "orientation": "horizontal",
        "showThresholdLabels": false,
        "showThresholdMarkers": true,
        "sizing": "auto",
        "minVizWidth": 75,
        "minVizHeight": 75
      },
      "pluginVersion": "10.2.3",
      "repeat": "path",
      "repeatDirection": "h",
      "targets": [
        {
          "refId": "A",
          "datasource": {
            "type": "grafana",
            "uid": "grafana"
          },
          "queryType": "snapshot",
          "snapshot": []
        }
      ],
      "title": "Reproduced with embedded data",
      "type": "gauge",
      "scopedVars": {
        "path": {
          "text": "/",
          "value": "/",
          "selected": true
        }
      },
      "description": ""
    },
    {
      "gridPos": {
        "h": 7,
        "w": 9,
        "x": 15,
        "y": 0
      },
      "id": 5,
      "options": {
        "content": "<table width=\"100%\">\n    <tr>\n      <th width=\"2%\">Panel</th>\n      <td >gauge @ 10.2.3</td>\n    </tr>\n    <tr>\n      <th>Queries</th>\n      <td>A[influxdb]</td>\n    </tr>\n    \n    <tr><th>Data</th><td>Error 0 frames, 0 fields, 0 rows </td></tr>\n    \n    <tr>\n      <th>Grafana</th>\n      <td>10.2.3 (1e84fede543acc892d2a2515187e545eb047f237) // Open Source</td>\n    </tr>\n  </table>",
        "mode": "html"
      },
      "title": "Debug info",
      "type": "text"
    },
    {
      "id": 6,
      "title": "Original Panel JSON",
      "type": "text",
      "gridPos": {
        "h": 13,
        "w": 9,
        "x": 15,
        "y": 7
      },
      "options": {
        "content": "{\n  \"datasource\": {\n    \"uid\": \"a312d509-775e-4c6d-88d4-c6acdf164ea1\",\n    \"type\": \"influxdb\"\n  },\n  \"fieldConfig\": {\n    \"defaults\": {\n      \"mappings\": [\n        {\n          \"options\": {\n            \"match\": \"null\",\n            \"result\": {\n              \"text\": \"N/A\"\n            }\n          },\n          \"type\": \"special\"\n        }\n      ],\n      \"thresholds\": {\n        \"mode\": \"absolute\",\n        \"steps\": [\n          {\n            \"color\": \"green\",\n            \"value\": null\n          },\n          {\n            \"color\": \"orange\",\n            \"value\": 70\n          },\n          {\n            \"color\": \"red\",\n            \"value\": 85\n          }\n        ]\n      },\n      \"unit\": \"percent\"\n    },\n    \"overrides\": []\n  },\n  \"gridPos\": {\n    \"h\": 4,\n    \"w\": 6,\n    \"x\": 0,\n    \"y\": 9\n  },\n  \"id\": 318,\n  \"links\": [],\n  \"maxDataPoints\": 100,\n  \"maxPerRow\": 8,\n  \"options\": {\n    \"reduceOptions\": {\n      \"values\": false,\n      \"calcs\": [\n        \"lastNotNull\"\n      ],\n      \"fields\": \"\"\n    },\n    \"orientation\": \"horizontal\",\n    \"showThresholdLabels\": false,\n    \"showThresholdMarkers\": true,\n    \"sizing\": \"auto\",\n    \"minVizWidth\": 75,\n    \"minVizHeight\": 75\n  },\n  \"pluginVersion\": \"10.2.3\",\n  \"repeat\": \"path\",\n  \"repeatDirection\": \"h\",\n  \"targets\": [\n    {\n      \"datasource\": {\n        \"type\": \"influxdb\",\n        \"uid\": \"a312d509-775e-4c6d-88d4-c6acdf164ea1\"\n      },\n      \"dsType\": \"influxdb\",\n      \"groupBy\": [\n        {\n          \"params\": [\n            \"$__interval\"\n          ],\n          \"type\": \"time\"\n        },\n        {\n          \"params\": [\n            \"null\"\n          ],\n          \"type\": \"fill\"\n        }\n      ],\n      \"hide\": false,\n      \"limit\": \"10\",\n      \"measurement\": \"disk\",\n      \"orderByTime\": \"ASC\",\n      \"policy\": \"default\",\n      \"refId\": \"A\",\n      \"resultFormat\": \"time_series\",\n      \"select\": [\n        [\n          {\n            \"params\": [\n              \"used_percent\"\n            ],\n            \"type\": \"field\"\n          },\n          {\n            \"params\": [],\n            \"type\": \"last\"\n          }\n        ]\n      ],\n      \"tags\": [\n        {\n          \"key\": \"host\",\n          \"operator\": \"=~\",\n          \"value\": \"/^$hostname$/\"\n        },\n        {\n          \"condition\": \"AND\",\n          \"key\": \"path::tag\",\n          \"operator\": \"=~\",\n          \"value\": \"/^$path$/\"\n        }\n      ]\n    }\n  ],\n  \"title\": \"Percentaged Used ($path)\",\n  \"type\": \"gauge\",\n  \"scopedVars\": {\n    \"path\": {\n      \"text\": \"/\",\n      \"value\": \"/\",\n      \"selected\": true\n    }\n  },\n  \"description\": \"\"\n}",
        "mode": "code",
        "code": {
          "language": "json",
          "showLineNumbers": true,
          "showMiniMap": true
        }
      }
    },
    {
      "id": 3,
      "title": "Data from panel above",
      "type": "table",
      "datasource": {
        "type": "datasource",
        "uid": "-- Dashboard --"
      },
      "gridPos": {
        "h": 7,
        "w": 15,
        "x": 0,
        "y": 13
      },
      "options": {
        "showTypeIcons": true
      },
      "targets": [
        {
          "datasource": {
            "type": "datasource",
            "uid": "-- Dashboard --"
          },
          "panelId": 2,
          "withTransforms": true,
          "refId": "A"
        }
      ]
    }
  ],
  "schemaVersion": 37,
  "title": "Debug: Percentaged Used ($path) // 2024-01-04 00:23:04",
  "tags": [
    "debug",
    "debug-gauge"
  ],
  "time": {
    "from": "2024-01-03T17:23:04.615Z",
    "to": "2024-01-03T23:23:04.615Z"
  }
}

Environment (with versions)?

Grafana: 10.2.3 (recent version seems to cause issues!)
OS: Ubuntu Server 22.04 LTS
Browser: Firefox

Grafana platform?

A package manager (APT, YUM, BREW, etc.)

Datasource(s)?

Telegraf

Related PRs

I'm just guessing here. But this could be regression of either: #77917 or #77799 (or even maybe maybe: #78834) or just something else.. I'm again not sure.

The text was updated successfully, but these errors were encountered:

melroy89 · 2024-01-03T23:28:48Z

Ps. Also the gauge panels that did work with Grafana 10.2.2 are now too big to fit / display (this use to fit without any issues, but for some reason the upgrade caused the gauge to no longer fit the content). See:

Regression of: #76216 ?

leeoniya · 2024-01-04T01:10:45Z

probably fixed by #79926 and #79940

are you able to try a nightly build?

https://grafana.com/grafana/download/nightly?edition=oss

melroy89 · 2024-01-04T03:19:17Z

probably fixed by #79926 and #79940

are you able to try a nightly build?

https://grafana.com/grafana/download/nightly?edition=oss

What about my main issue with the influxql issue?

EDIT: I could run the nightly build, I prefer not to do that, I now use APT for the updates. I'll wait for a new release.

leeoniya · 2024-01-04T03:47:41Z

^ cc @itsmylife

phil-or · 2024-01-04T15:52:23Z

I get the same InfluxQL Query error with 2.10.3

lischetzke · 2024-01-05T09:38:15Z

Hey, a temporary fix for the initial issue would be to modify the query and how the variable is used from "/^$hostname$/" to "/^${hostname:regex}$/". That seemed to fix it for me

gtk-grafana · 2024-01-05T15:56:45Z

I am able to reproduce this issue locally on 10.2 with the setup above. Enabling influxdbBackendMigration breaks dashboards with regex variables:

But when the flag is not enabled things work as expected:

Still seeing the bug on current main as well, and can verify that disabling the flag on main prevents the bug from displaying.
Still working on root cause

gtk-grafana · 2024-01-05T16:48:44Z

Looks like the problem here is that we're not properly formatting regex tag values on the backend for queries that are created in the visual builder. We're currently formatting regex tag values as '/^/var/log/host$/', you'll note that in influxdb regex tags aren't wrapped with single quotes, but with the regex delimiter /, also other instances of / are not escaped.

mbentley · 2024-01-05T16:51:16Z

That's what I was seeing as well @gtk-grafana. Just cross posting what I had shared in another issue:

I am not sure if this is exactly related but I also have noticed that I have variables which are no longer working as expected when I upgrade to Grafana 10.2.x and it also seems to be related to improperly escaped characters. I am on 10.2.3 right now and I have a variables where there are characters that need to be escaped but they are not. If I roll back to 10.1.6, they function as expected. I can see in the InfluxDB logs what Grafana is querying:

10.1.6 properly escaped:
ts=2024-01-04T15:43:34.381012Z lvl=info msg="executing new query" log_id=0mJG1EDW000 query="SELECT last(\"total\") FROM \"trivy\" WHERE (\"image\" =~ /^mbentley\\/acme\\.sh:latest-amd64$/ AND \"image\" =~ /^mbentley\\/.*/) AND time >= now() - 7d and time <= now() GROUP BY time(2h), \"image\" fill(none)"
10.2.3, improperly escaped:
ts=2024-01-04T15:47:35.364305Z lvl=info msg="executing new query" log_id=0mJG1EDW000 query="SELECT last(\"total\") FROM \"trivy\" WHERE (\"image\" =~ /^mbentley/acme.sh:latest-amd64$/ AND \"image\" =~ /^mbentley\\/.*/) AND time >= 1703778455227ms and time <= 1704383255227ms GROUP BY time(2h), \"image\" fill(none) ORDER BY time ASC"
An example of an item that is failing as a variable is mbentley/acme.sh:latest-amd64

Here is my query:
SELECT last("total") FROM "trivy" WHERE ("image" =~ /^$IMAGE$/ AND "image" =~ /^mbentley\/.*/) AND $timeFilter GROUP BY time($__interval), "image" fill(none)
*edit: I found #79692 which mentions a feature I can disable which resolves my issue.

melroy89 · 2024-01-05T17:21:43Z

Set influxdbBackendMigration = false in /etc/grafana/grafana.ini could potentially workaround the issue?

gtk-grafana · 2024-01-05T17:39:23Z

@melroy89 yes disabling the flag is going to be the best workaround for now. We'll update this issue when a fix has been merged.
e.g. setting influxdbBackendMigration = false in your custom.ini

pprkut · 2024-01-08T12:54:31Z

I don't think the issue is limited to regex values. I'm getting the same/similar error with a variable used for a simple exact match. There seems to be an escaping mismatch between encoding and decoding.

I have a query

SELECT "level_name", "message", "file", "line", "exception", "stacktrace" FROM "3m"."php_log" WHERE "environment" = '$Environment' AND "hostname" = '$Host' AND "request_id" = '$RequestID' AND $timeFilter

which gets sent as JSON like

SELECT \"level_name\", \"message\", \"file\", \"line\", \"exception\", \"stacktrace\" FROM \"3m\".\"php_log\" WHERE \"environment\" = 'acceptance' AND \"hostname\" = 'www\\.example\\.com' AND \"request_id\" = 'bbb43433dfb840fe8c58b88c7577d59a' AND $timeFilter

The hostname here isn't (un)escaped properly.

influxdbBackendMigration = false didn't help, but downgrading to 10.2.2 did

Not sure it's the same as or related to #77465, since that one mentions an issue when upgrading from 10.1.x, while it's working for me with 10.2.2

iBug · 2024-01-09T14:28:41Z

"mountpoint" =~ /^$mountpoint$/ broke for me as well and I found "mountpoint" = '${mountpoint:raw}' as a satisfactory workaround.

The Gauge display also broke for me but changing the Orientation from Horizontal to Auto fixed it.

gtk-grafana · 2024-01-09T20:16:46Z

@pprkut Good catch, yes there appears to be two different (but similar) problems which is causing some confusion.

Regex variables are not escaping values containing the regex delimiter / in the query editor.
#79368
#80161

The workaround for this issue is to disable the influxdbBackendMigration flag.

AND there is a regression in 10.2.3 that isn't related to the backend migration, but variable interpolation for raw influxdb queries are being escaped when they weren't in 10.2.2.

#79692
#80242

The fix for this issue is to revert back to 10.2.2, although I can verify @iBug's workaround above does work, but I don't want to recommend that folks change their dashboards because of an unintentional change like this. I would expect we can release/backport a fix out for both of these in 10.3.1 as we just missed the 10.3 cutoff.

Thanks to everyone for your reporting/investigating/patience

dmayan-ss · 2024-01-23T18:42:37Z

Hi,

The issue is still happening to me on 10.3.1. @iBug workaround still works.

Thanks,

itsmylife · 2024-01-25T11:53:07Z

The PR #80971 is going to resolve all interpolation issues. It is targeting 10.4.x.
I just need to run a few more tests to make sure it resolves the problem %100.
Thank you for your patience.

melroy89 · 2024-01-25T14:54:00Z

The PR #80971 is going to resolve all interpolation issues. It is targeting 10.4.x. I just need to run a few more tests to make sure it resolves the problem %100. Thank you for your patience.

Thank you for fixing all the regression issues.

gtk-grafana added the datasource/InfluxDB label Jan 5, 2024

This was referenced Jan 5, 2024

Grafana 10.2.3 is breaking dashboard (No data) #79692

Closed

Dashboards: influxdbBackendMigration FF enabled - Breaks Template Variables with special characters #79368

Closed

gtk-grafana mentioned this issue Jan 5, 2024

InfluxDB: regex tag values not being properly formatted/escaped #80096

Closed

3 tasks

devopstt mentioned this issue Jan 9, 2024

Query: Forward slashes in variables are not escaped #80161

Closed

gtk-grafana mentioned this issue Jan 9, 2024

InfluxDB: Regression interpolating variables in code editor - single variables being escaped in 10.2.3, but not in 10.2.2 #80242

Closed

itsmylife self-assigned this Jan 20, 2024

itsmylife mentioned this issue Jan 22, 2024

InfluxDB: Fix template variable interpolation #80971

Merged

3 tasks

itsmylife closed this as completed in #80971 Jan 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dashboards: Regression on InfluxQL query variable: found $, expected ) #80003

Dashboards: Regression on InfluxQL query variable: found $, expected ) #80003

melroy89 commented Jan 3, 2024 •

edited

Loading

melroy89 commented Jan 3, 2024 •

edited

Loading

leeoniya commented Jan 4, 2024

melroy89 commented Jan 4, 2024 •

edited

Loading

leeoniya commented Jan 4, 2024

phil-or commented Jan 4, 2024

lischetzke commented Jan 5, 2024 •

edited

Loading

gtk-grafana commented Jan 5, 2024

gtk-grafana commented Jan 5, 2024

mbentley commented Jan 5, 2024

melroy89 commented Jan 5, 2024 •

edited

Loading

gtk-grafana commented Jan 5, 2024 •

edited

Loading

pprkut commented Jan 8, 2024

iBug commented Jan 9, 2024

gtk-grafana commented Jan 9, 2024 •

edited

Loading

dmayan-ss commented Jan 23, 2024

itsmylife commented Jan 25, 2024

melroy89 commented Jan 25, 2024

Dashboards: Regression on InfluxQL query variable: found $, expected ) #80003

Dashboards: Regression on InfluxQL query variable: found $, expected ) #80003

Comments

melroy89 commented Jan 3, 2024 • edited Loading

What happened?

What did you expect to happen?

Did this work before?

How do we reproduce it?

Is the bug inside a dashboard panel?

Environment (with versions)?

Grafana platform?

Datasource(s)?

Related PRs

melroy89 commented Jan 3, 2024 • edited Loading

leeoniya commented Jan 4, 2024

melroy89 commented Jan 4, 2024 • edited Loading

leeoniya commented Jan 4, 2024

phil-or commented Jan 4, 2024

lischetzke commented Jan 5, 2024 • edited Loading

gtk-grafana commented Jan 5, 2024

gtk-grafana commented Jan 5, 2024

mbentley commented Jan 5, 2024

melroy89 commented Jan 5, 2024 • edited Loading

gtk-grafana commented Jan 5, 2024 • edited Loading

pprkut commented Jan 8, 2024

iBug commented Jan 9, 2024

gtk-grafana commented Jan 9, 2024 • edited Loading

dmayan-ss commented Jan 23, 2024

itsmylife commented Jan 25, 2024

melroy89 commented Jan 25, 2024

melroy89 commented Jan 3, 2024 •

edited

Loading

melroy89 commented Jan 3, 2024 •

edited

Loading

melroy89 commented Jan 4, 2024 •

edited

Loading

lischetzke commented Jan 5, 2024 •

edited

Loading

melroy89 commented Jan 5, 2024 •

edited

Loading

gtk-grafana commented Jan 5, 2024 •

edited

Loading

gtk-grafana commented Jan 9, 2024 •

edited

Loading