Setting and using SELDON_ENVIRONMENT for Request Logging to use one ELK Cluster for multiple Environment

[os-virtualretail]

MAIN MOTIVATION: Having 1 ELK Cluster for different ENVIRONMENTs ( dev/test/staging/prod)

Can we put a prefix on index to Request-Logger's ELK Indices

Right now it is like

inference-log-seldon-<NAMESPACE>-<SELDONDEPLOYMENT>-<MODELNAME>

what i want is something like

inference-log-<SELDON_ENVIRONMENT>-seldon-<NAMESPACE>-<SELDONDEPLOYMENT>-<MODELNAME>
 

as i create user based on that and have 1 ELK cluster that dev/staging/prod will log to this cluster. For me to create user via Kibana DevTools: ( I define Index Pattern to access to role).

POST _security/role/logstash_writer-dev
{
  "cluster": ["manage_index_templates", "monitor", "manage_ilm"], 
  "indices": [
    {
      "names": [ "k8s_cluster-dev-*" , "inference-log-dev-*" ], 
      "privileges": ["write","create","delete","create_index","manage","manage_ilm"]  
    }
  ]
}



POST _security/user/elastic-seldon-dev
{
  "password" : "PASS",
  "roles" : [ "logstash_writer-dev"],
  "full_name" : "Internal Logstash User for Seldon"
}

I can provide prefix to Fluentd part like:

fluentd-values.yaml: ( via elasticsearch.logstash.prefix

elasticsearch:
  hosts: ["df65d4b2795d481dbe235733219ca518.eu-central-1.aws.cloud.es.io:9243"]
  logstash:
    prefix: k8s_cluster-dev
  auth:
    enabled: true
    user: elastic-seldon-dev
    password: <PASSSWORD>
  sslVerify: false
  scheme: https