[FEAT] Add Instructions for Bring-your-own-InfluxDB with restricted access token

[jnaav]

I want to use my existing influxdb in the same server, but it is not working and I need some help.

This is my compose file:

 scrutiny:
   image: ghcr.io/analogj/scrutiny:master-omnibus
   container_name: scrutiny
   cap_add:
     - SYS_RAWIO
   volumes:
     - /docker/scrutiny:/opt/scrutiny/config
     - /run/udev:/run/udev:ro
   ports:
     - 8017:8080
   devices:
     - /dev/sda:/dev/sda
     - /dev/sdb:/dev/sdb
     - /dev/sdc:/dev/sdc
     - /dev/sdd:/dev/sdd
     - /dev/sde:/dev/sde
   restart: unless-stopped   

This is my scrutiny.yaml file

log:
  file: ""
  level: INFO
notify:
  urls: []
web:
  database:
    location: /opt/scrutiny/config/scrutiny.db
  influxdb:
    bucket: scrutiny
    host: MY-SERVER-LOCAL-IP
    org: MYORG
    port: "8086"
    retention_policy: true
    token: MY-TOKEN
  listen:
    basepath: ""
    host: 0.0.0.0
    port: "8080"
  src:
    frontend:
      path: /opt/scrutiny/web

I have configured a new bucket in my influxdb instance called scrutiny and a new API token with read and write permissions in that bucket.

In logs, there is this error:

panic: organization 'MYORG' not found

goroutine 1 [running]:
github.com/analogj/scrutiny/webapp/backend/pkg/web/middleware.RepositoryMiddleware(0x129f920, 0xc000408088, 0x12a4b00, 0xc000471650, 0x129f9a0)
	/go/src/github.com/analogj/scrutiny/webapp/backend/pkg/web/middleware/repository.go:14 +0xe6
github.com/analogj/scrutiny/webapp/backend/pkg/web.(*AppEngine).Setup(0xc000405ad0, 0x12a4b00, 0xc000471650, 0x14)
	/go/src/github.com/analogj/scrutiny/webapp/backend/pkg/web/server.go:26 +0xd8
github.com/analogj/scrutiny/webapp/backend/pkg/web.(*AppEngine).Start(0xc000405ad0, 0x0, 0x0)
	/go/src/github.com/analogj/scrutiny/webapp/backend/pkg/web/server.go:97 +0x234
main.main.func2(0xc00040f400, 0x4, 0x6)
	/go/src/github.com/analogj/scrutiny/webapp/backend/cmd/scrutiny/scrutiny.go:112 +0x198
github.com/urfave/cli/v2.(*Command).Run(0xc0004737a0, 0xc00040f280, 0x0, 0x0)
	/go/pkg/mod/github.com/urfave/cli/v2@v2.2.0/command.go:164 +0x4e0
github.com/urfave/cli/v2.(*App).RunContext(0xc000484000, 0x128e820, 0xc000130010, 0xc000126020, 0x2, 0x2, 0x0, 0x0)
	/go/pkg/mod/github.com/urfave/cli/v2@v2.2.0/app.go:306 +0x814
github.com/urfave/cli/v2.(*App).Run(...)
	/go/pkg/mod/github.com/urfave/cli/v2@v2.2.0/app.go:215
main.main()
	/go/src/github.com/analogj/scrutiny/webapp/backend/cmd/scrutiny/scrutiny.go:137 +0x65a

It seems it cannot found my organization? The name is correct, I have other services working fine with that influxedb instance.

I am missing something?

[AnalogJ]

First thing to note: the omnibus image already includes a influxdb instance/service. If you want to run Scrutiny with an existing/external InfluxDB service, you'll probably want to use the separate -weband -collector images.
See: https://github.com/AnalogJ/scrutiny/blob/master/docker/example.hubspoke.docker-compose.yml

Regarding the panic: organization 'MYORG' not found error: Are you sure you're running InfluxDB 2.2? or are you running an older version?

[jnaav]

Ok, I changed to use web and collector images.

Web container still not working, this is what I see in the logs:

time="2022-05-14T06:18:07Z" level=info msg="Trying to connect to scrutiny sqlite db: /opt/scrutiny/config/scrutiny.db\n"
time="2022-05-14T06:18:07Z" level=info msg="Successfully connected to scrutiny sqlite db: /opt/scrutiny/config/scrutiny.db\n"
time="2022-05-14T06:18:07Z" level=debug msg="InfluxDB url: http://MY-SERVER-LOCAL-IP:8086"
panic: organization 'MYORG' not found

The influxdb url is correct and the organization exists. Running InfluxDB v2.2.0

Why it cannot connect?

[AnalogJ]

That error message is in the influxdb client SDK/library, not code under my control. Weird, I'm not sure why it would be throwing an error, unless the Org doesnt exist, or the token doesnt have access to the Org.

How did you create your token? is it namespaced to a different organization?

[jnaav]

Okey, I think I found my problem.

In influxDB, I created a token for scrutiny with write/read permissions in scrutiny bucket and that did not work.

Then, I tried with admin token and it worked, and I also noticed that it created three new buckets (monthly, weekly and yearly) so I created a new token with write/read permissions in all that buckets...but it did not worked again.

So, it only works with admin token, not sure why need admin permission it needs, but I think it would be better if it is possible to scrutiny to only need permissions in scrutiny buckets and not admin permssions.

[AnalogJ]

@goliath888 Scrutiny also registers 3 background tasks for down-sampling. Can you check if that's the reason why it needs admin permissions?

[jnaav]

How can I check that?

EDIT: The problem is that in influxDB I only can create two types of token:

• All access API tocken: like the admin token
• Read/Write API token: Only referes to bucket

So I cannot give some permissions one by one to try to find which one is the cause of not working.

[AnalogJ]

Looks like the UI is limited in which resources you can scope your token to. however it looks like the API allows you to add additional resource scopes.

https://docs.influxdata.com/influxdb/cloud/api/#operation/PostAuthorizations

[jnaav]

After a few tries, I got it.

The API Token needs read permission in "orgs" and "tasks". Maybe also write for tasks to create the task the first time, not sure of that.

[AnalogJ]

@goliath888 would you be willing to share your curl command/token auth payload? I'd like to add it to the troubleshooting guide.

Thanks!

[jnaav]

Sure, not problem. I did it with postman, but adapting to curl should be easy (but a very long command):

POST:
http://INFLUXDB-URL:INFLUXDB-PORT/api/v2/authorizations

Headers:
Authorization: Token YOUR-TOKEN

Body:

{
  "description": "scrutiny",
  "orgID": "YOUR-ORG-ID",
  "permissions": [
        {
            "action": "read",
            "resource": {
                "type": "orgs"
            }
        },
        {
            "action": "read",
            "resource": {
                "type": "tasks"
            }
        },
     ....
     ....
     ....
  ]
}

In permissions you also need to add scrutiny, scrutiny_weekly, scrutiny-monthly and scrutiny_yearly read and write like this:

        {
            "action": "read",
            "resource": {
                "type": "buckets",
                "id": "ID",
                "orgID": "ORG-ID",
                "name": "scrutiny",
                "org": "YOUR-ORG"
            }
       },

To make it easy, I just create a new api token from the UI for scrutiny. Then, get that with the api, copy all and make a new one token with the api and add the new permissions (I think you cannot edit the one you create, just create a new one and delete the older).

I

[theryecatcher]

Thank you both @AnalogJ for the brilliant app - something that I was looking for and @goliath888 describing what you did to resolve the issue. I also faced the same thing with an existing influxdb instance. Followed the same process above, created the buckets and tasks using the admin token and scoped it down later with the above template.

Not sure if we would want to expand the config more to support multiple buckets in the config file to help with the retention policies with

• I did specify the retention to be false in config in hopes that it would not create the additional buckets and tasks but still it failed with org not found
• I tried giving it only read org access but still failed on writing to the buckets api, seems like it is trying to create the retention buckets in any case.)

[AnalogJ]

Let me reopen this issue (and rename it) to keep track of this request.

[AnalogJ]

Hey @theryecatcher @goliath888 @thvd

I added steps in the InfluxDB troubleshooting guide that walk you though the steps to create a scope restricted InfluxDB API token for scrutiny.

https://github.com/AnalogJ/scrutiny/blob/beta/docs/TROUBLESHOOTING_INFLUXDB.md#bring-your-own-influxdb

Can you take a look and test them if possible?

[jnaav]

Hi @AnalogJ,

My previous token stop working, so I just tried following your steps and generate a new token and everything is working fine again ;)

[AnalogJ]

Awesome, that's great to hear. Yeah, I added the ability to "update" InfluxDB task scripts, which means that your existing token would have failed unless you gave it write permissions to the tasks scope.

Glad to hear my instructions worked for you!