Role of `API_KEY` variable for ingester and how to have multiple diode client

[dga-nagra]

TL;DR

• What is the use of the ingester's API_KEY variable, it is not used anywhere
• Can I set up a single Diode server for multiple clients (e.g. use tokens generated from Netbox) ?
  The token the client is using is passed to Netbox with an environment variable. Therefore, we can only have one token and therefore only one client (or multiple ones sharing the same secret which is not good).

Detailed question

Hi,

I am trying to configure Diode in the same docker compose as Netbox.
I see that there is a RECONCILER_API_KEY variable set to CHANGE_.ME

https://github.com/netboxlabs/diode/blob/develop/diode-server/docker/sample.env#L2

diode/diode-server/docker/docker-compose.yaml

Line 39 in 3f4235f

- API_KEY=${RECONCILER_API_KEY}

I went in all repositories and I don't find the mention to this variable, even in the Go code:

diode/diode-server/ingester/component_test.go

Line 37 in 3f4235f

func setupEnv(redisAddr string) {

Is this variable needed? I thought that this would be used by the clients, but from netbox-learning I am supposed to use DIODE_API_KEY
If I understand correctly:

• the variables are here to define the communication ingester -> reconciler <-> Netbox.
• DIODE_API_KEY is shared by Netbox, the reconciler but also the client
• Since the client uses DIODE_API_KEY, and since this value is defined on startup on diode and netbox, this means we can only ever have only one DIODE_API_KEY even if we were to spawn multiple diode instances since the token is still passed to netbox.
• If we need to rotate the token, we need to restart both netbox and diode and also update the value on the client(s).

Please, correct me if I am wrong. Thank you in advance for your help.

[mfiedorowicz]

Hi @dga-nagra,

What is the use of the ingester's API_KEY variable, it is not used anywhere

It's a leftover from one of very first iterations, not used and is going to be removed.

Can I set up a single Diode server for multiple clients (e.g. use tokens generated from Netbox) ?

You can utilise diode server from multiple clients, but at this stage of the project we use single shared DIODE_API_KEY across all these clients.

the variables are here to define the communication ingester -> reconciler <-> Netbox.

That's correct.

DIODE_API_KEY is shared by Netbox, the reconciler but also the client

DIODE_API_KEY is used for ingestion (SDK/client -> diode-ingester) only. All three API keys are stored/managed in the NetBox, diode-reconciler has it as well currently as we didn't add pulling most of these keys via API from NetBox yet, but aiming to simplify the process, especially in scenarios like API keys rotation, etc.

Since the client uses DIODE_API_KEY, and since this value is defined on startup, this means that either all clients uses the same key, or we need one Diode server per client ?

As mentioned above, all clients using same DIODE_API_KEY (per diode <--> netbox) 👍

[dga-nagra]

Hi @mfiedorowicz and thank you a lot for taking the time to respond!

• Noted for API_KEY
• The DIODE_API_KEY is referenced in netbox.env file: https://github.com/netboxlabs/diode/blob/develop/diode-server/docker/netbox/env/netbox.env#L41 This is why I assumed it was used by netbox.
  Maybe this must be removed as well ?
  One other reason that made me believe that is that on Netbox, you get these 3 users created:
  
  I assumed (but did not check that) that the netbox-diode-plugin created these users. I don't understand the need for DIODE_TO_NETBOX and NETBOX_TO_DIODE when DIODE user is already being created

In my opinion, it is really important that the data being ingested uses the user's permissions and that every user can manage their own tokens. But I understand the limitations, especially, since it uses a redis queue, you cannot store the user credential for later use or give the user a permission denied when ingesting since it will only be known during the reconciler step. If you need help and have a roadmap/feature list/... I would be glad to contribute.

Thank you again for your responses.
Have a nice day.