NiFi v2 Behind Openshift Route

[greg-pendlebury]

Perhaps this is just a request for guidance (if I am wrong), but I can't work out how to gracefully get a NiFi v2 container online behind an Openshift Route because of the "Invalid SNI" issue.

This issue (697) gave some hints using an ingress, but I haven't found a way using routes.

From tracing through the current solution what I think is happening is:

• The operator creates a StatefulSet which forces NODE_ADDRESS to exist during startup, but builds the value itself based on a internal address.
• This is overwriting attempts to set NODE_ADDRESS using a ConfigMap.
• The nifi.properties loads NODE_ADDRESS into both nifi.cluster.node.address and nifi.web.https.host, with the whole file being drawn from a ConfigMap.
• Attempts to edit nifi.properties in the ConfigMap are overwritten by the operator.
• Attempts to edit the StatefulSet to adjust the CLI setting NODE_ADDRESS are overwritten by the operator.
• TBF I expected those just noting them as opposed to... attempts to add new environment variables into the ConfigMap are not overwritten.

Would it be as simple as allowing the nifi.web.https.host to be overwritten by a new variable, like PUBLIC_ADDRESS that we can set in the ConfigMap? I gather this is what is causing Jetty to reject the traffic originating from the public route.

For NiFi v1.27.0, using a pass-through route 'just worked', but either upgrading or a fresh install of v2 are all failing. Unclear to me why the internal pod wants to perform this validation... and TBH I would love to disable it, but maybe it adds value. My last gasp is going to track down an admin that knows cert-fu and trying to switch to a Re-Encrypt route, but not looking forward to that.

[maltesander]

Hi @greg-pendlebury,

not sure i completely follow, but could you please have a look at https://docs.stackable.tech/home/stable/nifi/usage_guide/security/#host-header-check?

Especially the part nifi.web.proxy.host=*?

Otherwise could you provide the errors from the nifi pods and other possible logs / events?

Malte

[greg-pendlebury]

Thanks @maltesander , I have tried those proxy settings and they weren't what I was looking for. TBH my hunch that nifi.web.https.host would solve my problem was also a dead end (I managed to overwrite using the 'config override' I found elsewhere in the doco).

I am specifically referring to the same issue noted in 697 where NiFi v2+ (or Jetty v10+) is very strict with regards to the SSL handshake. Because the cert generated inside the pod(? or somewhere else in the operator) only uses SANs that are either 1) internal cluster addresses ( various *.svc.cluster.local) or 2) the non-FQDN of the worker node (most network segments couldn't even resolve that DNS) or 3) the HostIP of the worker node, no client that initiates an external connection to the cluster using a more polished public address can talk directly to NiFi.

The pod logs something very minimal about the failed connection, but I think it is Jetty rejecting things, not NiFi, and the client receives the "Invalid SNI" error. All my research suggests everyone migrating to v2+ is challenged by this with docker. I have got v2 working perfectly using the host IP as the public access pathway (just to test).

The heart of the above issue (I think) is Openshift routes using 'passthrough' (i.e. don't mess with the SSL, the pod can handle this) worked fine for v1 NiFi but cannot for v2... at least all the dead ends I tried Yesterday are suggesting that. I suspect I could switch to 're-encrypt', but then I need to lift all the SSL details out of the pod and build them into the route config... academically doable, but I was already concerned when I saw the Host IP on the cert SANs because it of course means the pod is inherently fragile and couldn't handle being moved to a new worker node, unless the cert got re-issued... which then means the cert inside the route config would be stale and I would be chasing my own tail. Maybe... I am off in theory-land on this one. And the third option for route (edge termination) didn't look like something supported either; but I didn't explore it much either, I want https working internally.

Eventually I gave up Yesterday when I was considering implementing your cert-manager to manage the SSL outside the pod (I guess that is what it does) but an admin shut me down because they want to use the Openshift cert-manager they are already controlling (and that the routes leverage). We decided we would come back and re-assess the operator once v2 is a bit more mature. I was noticing the docker image is something like 3 weeks old at this stage and your team has only likely just hit the first set of teething issues to bed this down.

[maltesander]

Thank you @greg-pendlebury for the detailed steps!

Regarding the SSL/DNS issues i woud ping @Jimvin to maybe have some insights.

Now i have to throw some "experimentals" at you :)

• You can combine the Cert manager and the Stackable "cert" manager https://docs.stackable.tech/home/stable/secret-operator/cert-manager/.
• Yes, our NiFi v2 is currently experimental, we did not spend much time on investigating any problems. This is for users to have a preview or eventually test update steps, but it is not ready for production yet

Hopefully this is resolved until the next release, but i cannot promise anything.

Cheers,
Malte

[greg-pendlebury]

Thanks @maltesander , unfortunately I am not going to get time to re-visit this until next week, but I will try and see what I can do by wiring up the cert-manager.

[gitmedur]

Seconding this. Hit the SNI issue on OpenShift..

[aaschofield1974]

The issue boils down to the fact that you cannot bypass the SNI checks (I've tried every Java option I can find). The latest Jetty requires the routes hostname to be in the SAN certificate that's generated. You can see that by looking at the statefulSet that's generated by the operator:

     - ephemeral:
          volumeClaimTemplate:
            metadata:
              annotations:
                secrets.stackable.tech/backend.autotls.cert.lifetime: 1d
                secrets.stackable.tech/class: tls
                secrets.stackable.tech/format: tls-pkcs12
                secrets.stackable.tech/format.compatibility.tls-pkcs12.password: secret
                secrets.stackable.tech/scope: service=nifi-cluster,service=nifi-cluster-reporting-task,node,pod
              creationTimestamp: null
            spec:
              accessModes:
              - ReadWriteOnce
              resources:
                requests:
                  storage: "1"
              storageClassName: secrets.stackable.tech
              volumeMode: Filesystem
        name: keystore

Then read this: https://docs.stackable.tech/home/stable/secret-operator/scope/ which explains the scope settings. You can see this reflected in the certificates that are generated:

#5: ObjectId: 2.5.29.17 Criticality=true
SubjectAlternativeName [
  DNSName: nifi-cluster.NAMESPACE.svc.cluster.local
  DNSName: nifi-cluster-reporting-task.NAMESPACE.svc.cluster.local
  DNSName: NODEHOSTNAME
  IPAddress: NODEIP
  DNSName: nifi-cluster-node-default.NAMESPACE.svc.cluster.local
  DNSName: nifi-cluster-node-default-0.nifi-cluster-node-default.NAMESPACE.svc.cluster.local
]

Really the secrets operator needs to include support for Openshift routes (a route=ROUTENAME option would seem appropriate. I've worked around this by creating a pre hook which generates a new set of certificates, inserts these into a secret and then mounts this as an extraVolume. I then use the configOverrides to set the nifi.security.* options.

[nightkr]

First of all: If you're generating your own certificate then you can use the k8sSearch backend instead of injecting your own hooks. That secret could then be fed from cert-manager, or wherever else. As @aaschofield1974 touched upon, for your use-case you would probably want to podOverride secrets.stackable.tech/scope to at least remove the node scope.

Looking forward, for SDP 25.7 (#784) we're working on migrating nifi-operator to rely on our listener-operator instead, which is then responsible for discovering the addresses (including for certificates, via the listener-volume= scope). It doesn't currently support Route either (stackabletech/listener-operator#303), but it provides a path towards addressing the problem across the platform.

[EagleIJoe]

I have it running fine with an OpenShift route. The clue is the following:

spec:
  httpHeaders:
    actions:
      response: null
      request:
        - name: host
          action:
            type: Set
            set:
              value: internal-hostname

This resolves the SNI error. I setup TLS as re-encrypt.

You can use the cert-utils-operator to inject the destination CA from a secret.

[akyolog]

Thanks @EagleIJoe. It works by me. It was very time consuming until solving the problem.