Feature/ldap

[AyadiAmen]

What this PR does / why we need it:

Which issue this PR fixes

• fixes [cetic/nifi] External secure does not work #45

This pull request adds the connection to ldap server for authentication.

Special notes for your reviewer:

Please check the comments in mentioned issue ( #45 ) for a better understand of all the changes and configurations.

Checklist

[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

• DCO signed
• Chart Version bumped
• Variables are documented in the README.md

[banzo]

@AyadiAmen can you test that this feature is disabled by default via the values.yaml flag?

[AyadiAmen]

@banzo i'll test that later today and i'lll let you know asap.

[iammoen]

Does the readiness probe still work with this? We have been going down the path of ldap auth and found that the initial admin identity needs to be the node in order to use the secured nifi api.

[AyadiAmen]

@iammoen yes it works, after the last commit you can pass your “Initial Admin Identity” in the variable auth.ldap.admin in thevalues.yaml file . after deployment there will be two users in the users.xml file: the node identity and what you pass in auth.ldap.admin. The latter will have the necessary permissions to create and add policies as he is the “Initial Admin Identity”.

[alexnuttinck]

Hello @AyadiAmen, thanks for your work on this feature :) ! Can you see my comments before we can merge?
When it is fixed, I will try your new feature on my laptop.

[alexnuttinck]

The release name can be retrieved by .Release.Name so the user doesn't need to add it in values file.

[alexnuttinck]

Why the namespace name is needed?

[alexnuttinck]

Also I propose to add ldap in a sub-section as it's one of the possible auth method. You can also say that it's mandatory for nifi to enable the ssl to use an auth method.

[alexnuttinck]

It should be possible to pass all these values via the values.yaml. You can nevertheless set ldap.enabled to false and add default values for the ldap propreties.

[alexnuttinck]

All these values should be pass via the values.yaml

[alexnuttinck]

by passing nifi-0 it will not work for a cluster of nifi. you can retrieve this value with hostname -f, and the variable should be injected in the propreties file.

[alexnuttinck]

via values.yaml

[alexnuttinck]

Not acceptable! we can't use root user

[alexnuttinck]

use hostname -f in your scripts, so the user do not need to use these variables.

[alexnuttinck]

should be false by default?

[alexnuttinck]

sould be usable even if we don't use ldap auth

[alexnuttinck]

-> NodePort

[AyadiAmen]

there's an issue to access nifi when it runs in a secured cluster, accessing it with NodePort is not possible for and there might be a solution with LoadBalener.

[iammoen]

Should this variable be lower camel case like all the other ones?

[alexnuttinck]

I confirm @AyadiAmen SSL, UserIdentity, AuthStrategy, IdentityStrategy, AuthExpiration should be lower case.

[iammoen]

was also thinking that maybe this should be a different ldap user besides the one for initial admin? Since this one will be used to bind to the ldap server it would probably be a service account of some kind whereas the initial admin would be likely to be a proper user?

[makeacode]

Not to hijack this PR, but but i've also created my own PR #76 that configures TLS support (using nifi-toolkit) and OIDC authentication (tested using adfs).

[alexnuttinck]

@makeacode once #76 merged, we can use this PR to work on the ldap auth :)

[makeacode]

theoretically once TLS is enabled the auth should be pretty simple....theoretically.

[piotron]

Practically it's too quite simple. there are few quirks(like adding proper permissions manually in UI) but this may be because of our configuration changes and nature of hybrid nifi cluster.
However we managed to spawn today secure Nifi cluster with ldap integration.

[alexnuttinck]

Practically it's too quite simple. there are few quirks(like adding proper permissions manually in UI) but this may be because of our configuration changes and nature of hybrid nifi cluster.
However we managed to spawn today secure Nifi cluster with ldap integration.

Thanks for your feedback @piotron!

Could you share your configuration setup? It would help to improve the Readme part/section about ldap integration with nifi.

[alexnuttinck]

We setup nifi with ldap and it seems to work: we can see that nifi and openldap communicate together in their respective logs. But we face a new error. We can't reach the UI as described in #72... Any ideas?

[alexnuttinck]

Current work is on branch feature: https://github.com/cetic/helm-nifi/tree/feature/atuh

[piotron]

@alexnuttinck you have this issue with browser OR nifi/minifi agents(We noticed that MiNiFi agent's cannot proxy through loadbalancer properly and had to do workaround with proxy setup)

Have you set up webProxyHost properly(including nifi listening port nifi.test:9443 ?)
if your netries for httpsPort are like this(both)

httpsPort: 9443

then your entry for webProxyHost also should contain domain:port because statefulset does not set it

Other case your UI cannot open port 443 because of security settings. Ports below 1024 needs to have specific capabilities to open ports for non-root user.

[alexnuttinck]

@alexnuttinck you have this issue with browser OR nifi/minifi agents(We noticed that MiNiFi agent's cannot proxy through loadbalancer properly and had to do workaround with proxy setup)

Have you set up webProxyHost properly(including nifi listening port nifi.test:9443 ?)
if your netries for httpsPort are like this(both)

httpsPort: 9443

then your entry for webProxyHost also should contain domain:port because statefulset does not set it

Other case your UI cannot open port 443 because of security settings. Ports below 1024 needs to have specific capabilities to open ports for non-root user.

@piotron thanks for the reply!

This is a browser issue. Ok, I didn't setup the port for the webProxyHost, only the domain. Let's have a try.

[AyadiAmen]

Thank you @alexnuttinck and @piotron ,

i tried nifi.test:9443 but it didn't work, i even tried to pass nifi.test:9443 in webProxyHost and still not working, the only way i can reach the UI now is via forwarding the port 9443 and using localhost.

[piotron]

@AyadiAmen could you past whole list of allowed domains? it should show all allowed ports and combinations for access.

[AyadiAmen]

@piotron

Valid host headers are [empty] or:
127.0.0.1
127.0.0.1:9443
localhost
localhost:9443
[::1]
[::1]:9443
nifi-0.nifi-headless.default.svc.cluster.local
nifi-0.nifi-headless.default.svc.cluster.local:9443
172.17.0.9
172.17.0.9:9443
nifi.test

[piotron]

@AyadiAmen and your webProxyHost is exactly nifi.test:9443 in values.yml? which branch are you using for this test?

[AyadiAmen]

@piotron i tried with webProxyHost: nifi.test, webProxyHost: nifi.test:9443 and webProxyHost: nifi.test, nifi.test:9443 with the branches feature/atuh and feature/ldap

[piotron]

@AyadiAmen if possible please share values.yml(ommiting internal data ofc) you used, and overrides.
From what I see it's somehow stripping secure port, and this may be caused by some other flag then.

[AyadiAmen]

@piotron

---
# Number of nifi nodes
replicaCount: 1

## Set default image, imageTag, and imagePullPolicy.
## ref: https://hub.docker.com/r/apache/nifi/
##
image:
  repository: apache/nifi
  tag: "1.11.4"
  pullPolicy: IfNotPresent

  ## Optionally specify an imagePullSecret.
  ## Secret must be manually created in the namespace.
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ##
  # pullSecret: myRegistrKeySecretName

securityContext:
  runAsUser: 1000
  fsGroup: 1000

sts:
  # Parallel podManagementPolicy for faster bootstrap and teardown. Default is OrderedReady.
  podManagementPolicy: Parallel
  AntiAffinity: soft
  hostPort: null
  pod:
    annotations:
      security.alpha.kubernetes.io/sysctls: net.ipv4.ip_local_port_range=10000 65000
      #prometheus.io/scrape: "true"      

## Useful if using any custom secrets
## Pass in some secrets to use (if required)
# secrets:
# - name: myNifiSecret
#   keys:
#     - key1
#     - key2
#   mountPath: /opt/nifi/secret

## Useful if using any custom configmaps
## Pass in some configmaps to use (if required)
# configmaps:
#   - name: myNifiConf
#     keys:
#       - myconf.conf
#     mountPath: /opt/nifi/custom-config


properties:
  # use externalSecure for when inbound SSL is provided by nginx-ingress or other external mechanism
  externalSecure: false
  isNode: true
  httpPort: null
  httpsPort: 9443
  webProxyHost: nifi.test
  clusterPort: 6007
  clusterSecure: true
  needClientAuth: false
  provenanceStorage: "8 GB"
  siteToSite:
    port: 10000
  authorizer: managed-authorizer
  # use properties.safetyValve to pass explicit 'key: value' pairs that overwrite other configuration
  safetyValve:
    #nifi.variable.registry.properties: "${NIFI_HOME}/example1.properties, ${NIFI_HOME}/example2.properties"
    nifi.web.http.network.interface.default: eth0
    # listen to loopback interface so "kubectl port-forward ..." works
    nifi.web.http.network.interface.lo: lo

## Include additional libraries in the Nifi containers by using the postStart handler
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/
# postStart: /opt/nifi/psql; wget -P /opt/nifi/psql https://jdbc.postgresql.org/download/postgresql-42.2.6.jar

# Nifi User Authentication
auth:
  admin: CN=admin, OU=NIFI
  SSL:
    keystorePasswd: env:PASS
    truststorePasswd: env:PASS
  ldap:
    enabled: true
    host: ldap://openldap:389
    searchBase: --
    admin: --
    pass: --
    searchFilter: (objectClass=*)
    userIdentityAttribute: cn
    authStrategy: SIMPLE 
    identityStrategy: USE_DN
    authExpiration: 12 hours

  oidc:
    enabled: false
    discoveryUrl:
    clientId:
    clientSecret:
    claimIdentifyingUser: email

## Expose the nifi service to be accessed from outside the cluster (LoadBalancer service).
## or access it from within the cluster (ClusterIP service). Set the service type and the port to serve it.
## ref: http://kubernetes.io/docs/user-guide/services/
##

# headless service
headless:
  type: ClusterIP
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"

# ui service
service:
  type: LoadBalancer
  httpPort: 8080
  httpsPort: 443
  annotations: {}
    # loadBalancerIP:
    ## Load Balancer sources
    ## https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service
    ##
    # loadBalancerSourceRanges:
    # - 10.10.10.0/24
    ## OIDC authentication requires "sticky" session on the LoadBalancer for JWT to work properly...but AWS doesn't like it on creation
    # sessionAffinity: ClientIP
    # sessionAffinityConfig:
    #   clientIP:
    #     timeoutSeconds: 10800

  # Enables additional port/ports to nifi service for internal processors
  processors:
    enabled: false
    ports:
      - name: processor01
        port: 7001
        targetPort: 7001
        #nodePort: 30701
      - name: processor02
        port: 7002
        targetPort: 7002
        #nodePort: 30702

## Configure Ingress based on the documentation here: https://kubernetes.io/docs/concepts/services-networking/ingress/
##
ingress:
  enabled: true
  annotations: {}
  tls: []
  hosts: [nifi.test]
  path: /
  # If you want to change the default path, see this issue https://github.com/cetic/helm-nifi/issues/22

# Amount of memory to give the NiFi java heap
jvmMemory: 2g

# Separate image for tailing each log separately and checking zookeeper connectivity
sidecar:
  image: busybox
  tag: "1.32.0"

## Enable persistence using Persistent Volume Claims
## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
##
persistence:
  enabled: false

  # When creating persistent storage, the NiFi helm chart can either reference an already-defined
  # storage class by name, such as "standard" or can define a custom storage class by specifying
  # customStorageClass: true and providing the "storageClass", "storageProvisioner" and "storageType".
  # For example, to use SSD storage on Google Compute Engine see values-gcp.yaml
  #
  # To use a storage class that already exists on the Kubernetes cluster, we can simply reference it by name.
  # For example:
  # storageClass: standard
  #
  # The default storage class is used if this variable is not set.

  accessModes:  [ReadWriteOnce]
  ## Storage Capacities for persistent volumes
  configStorage:
    size: 100Mi
  authconfStorage:
    size: 100Mi
  # Storage capacity for the 'data' directory, which is used to hold things such as the flow.xml.gz, configuration, state, etc.
  dataStorage:
    size: 1Gi
  # Storage capacity for the FlowFile repository
  flowfileRepoStorage:
    size: 10Gi
  # Storage capacity for the Content repository
  contentRepoStorage:
    size: 10Gi
  # Storage capacity for the Provenance repository. When changing this, one should also change the properties.provenanceStorage value above, also.
  provenanceRepoStorage:
    size: 10Gi
  # Storage capacity for nifi logs
  logStorage:
    size: 5Gi

## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #  cpu: 100m
  #  memory: 128Mi
  # requests:
  #  cpu: 100m
#  memory: 128Mi

logresources:
  requests:
    cpu: 10m
    memory: 10Mi
  limits:
    cpu: 50m
    memory: 50Mi

nodeSelector: {}

tolerations: []

initContainers: {}
  # foo-init:  # <- will be used as container name
  #   image: "busybox:1.30.1"
  #   imagePullPolicy: "IfNotPresent"
  #   command: ['sh', '-c', 'echo this is an initContainer']
#   volumeMounts:
#     - mountPath: /tmp/foo
#       name: foo

extraVolumeMounts: []

extraVolumes: []

## Extra containers
extraContainers: []

terminationGracePeriodSeconds: 30

## Extra environment variables that will be pass onto deployment pods
env: []

# ca server details
# Setting this true would create a nifi-toolkit based ca server
# The ca server will be used to generate self-signed certificates required setting up secured cluster
ca:
  ## If true, enable the nifi-toolkit certificate authority
  enabled: true
  persistence:
    enabled: true
  server: ""
  port: 9090
  token: sixteenCharacters
  admin:
    cn: admin

# ------------------------------------------------------------------------------
# Zookeeper:
# ------------------------------------------------------------------------------
zookeeper:
  ## If true, install the Zookeeper chart
  ## ref: https://github.com/kubernetes/charts/tree/master/incubator/zookeeper
  enabled: true
  ## If the Zookeeper Chart is disabled a URL and port are required to connect
  url: ""
  port: 2181

# ------------------------------------------------------------------------------
# Nifi registry:
# ------------------------------------------------------------------------------
registry:
  ## If true, install the Nifi registry
  enabled: true
  url: ""
  port: 80
  ## Add values for the nifi-registry here
  ## ref: https://github.com/dysnix/charts/blob/master/nifi-registry/values.yaml

[piotron]

@AyadiAmen please try this file

---
# Number of nifi nodes
replicaCount: 1

## Set default image, imageTag, and imagePullPolicy.
## ref: https://hub.docker.com/r/apache/nifi/
##
image:
  repository: apache/nifi
  tag: "1.11.4"
  pullPolicy: IfNotPresent

  ## Optionally specify an imagePullSecret.
  ## Secret must be manually created in the namespace.
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ##
  # pullSecret: myRegistrKeySecretName

securityContext:
  runAsUser: 1000
  fsGroup: 1000

sts:
  # Parallel podManagementPolicy for faster bootstrap and teardown. Default is OrderedReady.
  podManagementPolicy: Parallel
  AntiAffinity: soft
  hostPort: null
  pod:
    annotations:
      security.alpha.kubernetes.io/sysctls: net.ipv4.ip_local_port_range=10000 65000
      #prometheus.io/scrape: "true"      

## Useful if using any custom secrets
## Pass in some secrets to use (if required)
# secrets:
# - name: myNifiSecret
#   keys:
#     - key1
#     - key2
#   mountPath: /opt/nifi/secret

## Useful if using any custom configmaps
## Pass in some configmaps to use (if required)
# configmaps:
#   - name: myNifiConf
#     keys:
#       - myconf.conf
#     mountPath: /opt/nifi/custom-config


properties:
  # use externalSecure for when inbound SSL is provided by nginx-ingress or other external mechanism
  externalSecure: false
  isNode: true
  httpPort: null
  httpsPort: 9443
  webProxyHost: nifi.test:9443
  clusterPort: 6007
  clusterSecure: true
  needClientAuth: false
  provenanceStorage: "8 GB"
  siteToSite:
    port: 10000
  authorizer: managed-authorizer
  # use properties.safetyValve to pass explicit 'key: value' pairs that overwrite other configuration
  safetyValve:
    #nifi.variable.registry.properties: "${NIFI_HOME}/example1.properties, ${NIFI_HOME}/example2.properties"
    nifi.web.http.network.interface.default: eth0
    # listen to loopback interface so "kubectl port-forward ..." works
    nifi.web.http.network.interface.lo: lo

## Include additional libraries in the Nifi containers by using the postStart handler
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/
# postStart: /opt/nifi/psql; wget -P /opt/nifi/psql https://jdbc.postgresql.org/download/postgresql-42.2.6.jar

# Nifi User Authentication
auth:
  admin: CN=admin, OU=NIFI
  SSL:
    keystorePasswd: env:PASS
    truststorePasswd: env:PASS
  ldap:
    enabled: true
    host: ldap://openldap:389
    searchBase: --
    admin: --
    pass: --
    searchFilter: (objectClass=*)
    userIdentityAttribute: cn
    authStrategy: SIMPLE 
    identityStrategy: USE_DN
    authExpiration: 12 hours

  oidc:
    enabled: false
    discoveryUrl:
    clientId:
    clientSecret:
    claimIdentifyingUser: email

## Expose the nifi service to be accessed from outside the cluster (LoadBalancer service).
## or access it from within the cluster (ClusterIP service). Set the service type and the port to serve it.
## ref: http://kubernetes.io/docs/user-guide/services/
##

# headless service
headless:
  type: ClusterIP
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"

# ui service
service:
  type: LoadBalancer
  httpPort: 8080
  httpsPort: 9443
  annotations: {}
    # loadBalancerIP:
    ## Load Balancer sources
    ## https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service
    ##
    # loadBalancerSourceRanges:
    # - 10.10.10.0/24
    ## OIDC authentication requires "sticky" session on the LoadBalancer for JWT to work properly...but AWS doesn't like it on creation
    # sessionAffinity: ClientIP
    # sessionAffinityConfig:
    #   clientIP:
    #     timeoutSeconds: 10800

  # Enables additional port/ports to nifi service for internal processors
  processors:
    enabled: false
    ports:
      - name: processor01
        port: 7001
        targetPort: 7001
        #nodePort: 30701
      - name: processor02
        port: 7002
        targetPort: 7002
        #nodePort: 30702

## Configure Ingress based on the documentation here: https://kubernetes.io/docs/concepts/services-networking/ingress/
##
ingress:
  enabled: true
  annotations: {}
  tls: []
  hosts: [nifi.test]
  path: /
  # If you want to change the default path, see this issue https://github.com/cetic/helm-nifi/issues/22

# Amount of memory to give the NiFi java heap
jvmMemory: 2g

# Separate image for tailing each log separately and checking zookeeper connectivity
sidecar:
  image: busybox
  tag: "1.32.0"

## Enable persistence using Persistent Volume Claims
## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
##
persistence:
  enabled: false

  # When creating persistent storage, the NiFi helm chart can either reference an already-defined
  # storage class by name, such as "standard" or can define a custom storage class by specifying
  # customStorageClass: true and providing the "storageClass", "storageProvisioner" and "storageType".
  # For example, to use SSD storage on Google Compute Engine see values-gcp.yaml
  #
  # To use a storage class that already exists on the Kubernetes cluster, we can simply reference it by name.
  # For example:
  # storageClass: standard
  #
  # The default storage class is used if this variable is not set.

  accessModes:  [ReadWriteOnce]
  ## Storage Capacities for persistent volumes
  configStorage:
    size: 100Mi
  authconfStorage:
    size: 100Mi
  # Storage capacity for the 'data' directory, which is used to hold things such as the flow.xml.gz, configuration, state, etc.
  dataStorage:
    size: 1Gi
  # Storage capacity for the FlowFile repository
  flowfileRepoStorage:
    size: 10Gi
  # Storage capacity for the Content repository
  contentRepoStorage:
    size: 10Gi
  # Storage capacity for the Provenance repository. When changing this, one should also change the properties.provenanceStorage value above, also.
  provenanceRepoStorage:
    size: 10Gi
  # Storage capacity for nifi logs
  logStorage:
    size: 5Gi

## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #  cpu: 100m
  #  memory: 128Mi
  # requests:
  #  cpu: 100m
#  memory: 128Mi

logresources:
  requests:
    cpu: 10m
    memory: 10Mi
  limits:
    cpu: 50m
    memory: 50Mi

nodeSelector: {}

tolerations: []

initContainers: {}
  # foo-init:  # <- will be used as container name
  #   image: "busybox:1.30.1"
  #   imagePullPolicy: "IfNotPresent"
  #   command: ['sh', '-c', 'echo this is an initContainer']
#   volumeMounts:
#     - mountPath: /tmp/foo
#       name: foo

extraVolumeMounts: []

extraVolumes: []

## Extra containers
extraContainers: []

terminationGracePeriodSeconds: 30

## Extra environment variables that will be pass onto deployment pods
env: []

# ca server details
# Setting this true would create a nifi-toolkit based ca server
# The ca server will be used to generate self-signed certificates required setting up secured cluster
ca:
  ## If true, enable the nifi-toolkit certificate authority
  enabled: true
  persistence:
    enabled: true
  server: ""
  port: 9090
  token: sixteenCharacters
  admin:
    cn: admin

# ------------------------------------------------------------------------------
# Zookeeper:
# ------------------------------------------------------------------------------
zookeeper:
  ## If true, install the Zookeeper chart
  ## ref: https://github.com/kubernetes/charts/tree/master/incubator/zookeeper
  enabled: true
  ## If the Zookeeper Chart is disabled a URL and port are required to connect
  url: ""
  port: 2181

# ------------------------------------------------------------------------------
# Nifi registry:
# ------------------------------------------------------------------------------
registry:
  ## If true, install the Nifi registry
  enabled: true
  url: ""
  port: 80
  ## Add values for the nifi-registry here
  ## ref: https://github.com/dysnix/charts/blob/master/nifi-registry/values.yaml

and call https://nifi.test:9443

[AyadiAmen]

@piotron still not accessible :/

[piotron]

@AyadiAmen this may be wild guess now but try to disable ingress and return it to default. because now this is only thing that differs.

[AyadiAmen]

@piotron and access it how without ingress ? i'll try anyway but i think the ingress should the solution here

[piotron]

@AyadiAmen Maybe Ingress is a way to go but for now we managed to get it to work with separate LoadBalancer service this way it presents with it's own certificates.

Our setup is K3S + Metallb

[makeacode]

@AyadiAmen I'm definitely not a nifi expert but when i was working on adding TLS support it seemed that nifi wants/needs to do the termination itself and it cannot be done at the ingress level (unless ingress can do some trickery i'm not aware of). I used EKS for my development/testing and exposed nifi through a LoadBalancer and set the url i wanted to access it by to webProxyHost. After that I just added a dns CNAME to point the webProxyHost value to the load balancer url so all the values would match up.

[jrote1]

What is this status of this ldap support as the currently implementation is a bit limiting especially when upgrading the chart as it wipes my configuration

[alexnuttinck]

Done in #107. I close this PR.