project id not accepted

[symbiont-ji]

Bug Report

Trying to set up the bigquery-grafana data source on a GCE instance. The instance has the right scopes and is running under a service account with the right roles. To verify, I run this query from a shell, and I get back the expected results:

bq query 'select * from billing.gcp_billing_export_v1_XXXXX limit 10' (obviously I'm masking my real billing account!).

So this is not a simple case of bad permissions.

Expected Behavior

Proper installation of the datasource

Actual Behavior

No matter what I enter as project id, or even if I leave it blank, I get back this screen

It's as if it is not accepting the project at all, but neither is it getting it from the metadata server.

curl -H 'Metadata-Flavor: Google' http://169.254.169.254/computeMetadata/v1/project/project-id gives the correct project id, of course.

More information:

$ curl -H 'Metadata-Flavor: Google' http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/scopes
https://www.googleapis.com/auth/bigquery
https://www.googleapis.com/auth/cloud-platform


## Steps to Reproduce the Problem


## Specifications

- Version: doitintl-bigquery-datasource @ 2.0.1
- Platform: GCE, Ubuntu 20.04LTS
- Grafana Version: Version 7.4.3 (commit: 010f20c1c8, branch: HEAD)

[codeblooded]

I can confirm that we are experiencing this same issue using the plugin with GCE default service accounts.

[symbiont-ji]

FWIW, my workaround was to just create something random (I forget what), then go into /var/lib/grafana/grafana.db with sqlite3 and edit the corresponding field: (some lines removed and replaced with ... for clarity):

# sqlite3 /var/lib/grafana/grafana.db
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> SELECT name FROM data_source;
Google BigQuery
...

sqlite> UPDATE data_source SET json_data = '{"authenticationType":"gce","flatRateProject":"prj-8225-dev","processingLocation":"US","queryPriority":"INTERACTIVE"}' WHERE name = 'Google BigQuery';

[codeblooded]

@symbiont-ji You are awesome. Thanks for the workaround. The issue must be in not properly persisting the flatRateProject.

[codeblooded]

The workaround just stopped working on me. I checked the database, and it hasn't changed. Weird. My queries look like this:

{
  "request": {
    "data": {
      "priority": "INTERACTIVE",
      "location": "US",
      "query": "#standardSQL\nSELECT\n `metadata`.`created` AS time,\n  `clientStats`.`cqPollCount`\nFROM `<example set>`\nWHERE\n  `metadata`.`created` BETWEEN TIMESTAMP_MILLIS (1572391292226) AND TIMESTAMP_MILLIS (1577344892226)\nGROUP BY 1,2 \nORDER BY 1 LIMIT 1338",
      "useLegacySql": false,
      "useQueryCache": true
    },
    "method": "POST",
    "url": "api/datasources/proxy/1/bigquery/v2/projects/undefined/queries",
    "hideFromInspector": false
  },
  "response": {
    "error": {
      "code": 403,
      "message": "Access Denied: Project project_id: \"undefined\"\n\n: User does not have bigquery.jobs.create permission in project undefined.",
      "errors": [
        {
          "message": "Access Denied: Project project_id: \"undefined\"\n\n: User does not have bigquery.jobs.create permission in project undefined.",
          "domain": "global",
          "reason": "accessDenied"
        }
      ],
      "status": "PERMISSION_DENIED"
    }
  }
}

[ofir5300]

Managed to finally solve it. Will be released on plugin version 2.0.2

[MohamedMSaeed]

@ofir5300 Hi,

I'm facing the same issue here. Just checked the fix date and it seems that v2.0.2 zip file doesn't have the fix. How can I use the latest version that has the fix?

Note: I following from your docs:

...
plugins:
  - https://github.com/doitintl/bigquery-grafana/archive/2.0.2.zip;doit-bigquery-datasource

grafana.ini:
  plugins:
    allow_loading_unsigned_plugins: doitintl-bigquery-datasource
...

Thanks 😄