Problem on F5 vip configuration for webMethods kubernetes cluster

Product/components used and version/fix level:

webMethods containers 10.15, downloaded from SAG container repo
containers.softwareag.com
webMethods helm charts mirrored and customized from (recently merged from mirrored repo)

Detailed explanation of the problem:

We are trying to create a VIP and DNS entry but we don’t know the settings for F5. We tried to build a generic VIP, configured ingress using helm values. For integration server, we are able to see some response from the pods, but not the desired ones (which having the admin UI). For UM server, we are able to get some response from nirvana document API (http://hostname:9000) but enterprise manager doesn’t connect.

I am working together with F5 admins and Platform Engineering team and acting as a communication layer between them is really difficult. PE team says that, F5 team needs to create a datagroup and they need to add ingress IPs to the nodes, which we did but it didn’t look like it was necessary and also it didn’t work. If my understanding is correct datagroup is only used for filtering the requests. This is a POC environment and we already have a working DNS/iRules in higher environments. But the iRules there are really complicated and I really want to keep it simple for POC env. I don’t want to have anything complicated for POC environment, it will increase the number of possible failures.

So I kept discussing this problem with them over weeks and we still don’t have a working VIP. PE team says that we don’t need any product specific ports, they claim 443 is enough. F5 team says that it won’t redirect anything to a specific port if you don’t add that as a node and we tried both and neither worked.

Currently I have 2 ingress IP’s configured for port 443 as nodes in the generic VIP, this one redirects traffic using paths. I can get response from this VIP but only the http requests are returning a response. And I have a dedicated UMserver VIP, which has both ingress nodes added and doesn’t use the path itself. We are trying to make UMserver work first because its ingress configuration is pretty simple but it uses a different protocol. I can get some response from this VIP but nothing um related.

ingress part of the helm values for dedicated VIP

ingress:
  # -- Enable or disable Ingress, default is disabled. On enabling Ingress, only the first `-0` replica is currently supported. 
  enabled: true
  defaultHostname: wm-poc-um
  className: ""
  annotations: 
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/x-forwarded-prefix: /wm-poc-um
  hosts:
    # -- Hostname of Ingress. By default the defaultHostname is used. For more complex rules or addtional hosts, you will need to overwrite this section.
    - host: "apps-umserver-wm.*****"
      # --  Address the backend
      paths:
        # -- Path to address the backend
        - path: /(/|$)(.*) # this part and the annotations are recommended to be added by the PE team
          # -- Path type to address the backend
          pathType: Prefix
          # -- Port of service
          port: 9000
  # -- TLS of Ingress
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

Ingress values for the generic vip

ingress:
  # -- Enable or disable Ingress, default is disabled. On enabling Ingress, only the first `-0` replica is currently supported. 
  enabled: true
  defaultHostname: wm-poc-um
  className: ""
  annotations: 
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/x-forwarded-prefix: /wm-poc-um
  hosts:
    # -- Hostname of Ingress. By default the defaultHostname is used. For more complex rules or addtional hosts, you will need to overwrite this section.
    - host: "apps-wm-review.*****"
      # --  Address the backend
      paths:
        # -- Path to address the backend
        - path: /wm-poc-um(/|$)(.*)
          # -- Path type to address the backend
          pathType: Prefix
          # -- Port of service
          port: 9000
  # -- TLS of Ingress
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

I have already verified the pods are running and I can get response within the kubernetes cluster but I could only check http:// response using curl, not nsp:// because I don’t have a enterprise manager container that I can use right now.

Questions are,
how should we configure F5 vip?
Should we add nodes with port 9000 for um for example or would port 443 will be enough?
Do we need to set datagroups for sure?
Do we definately need an iRule?
Is there something required that I am missing?
And how can I set a VIP for ingress with the least config/work?

I can share screen shots from F5 as well but those pages are really long and they also have too much confidential and unrelated data. If you need a spesific part for troubleshooting I can mask it and share here as well.

Response of the browser from http call.

Response of enterprise manager.

We checked the ingress logs and we haven’t see any connection attempt when using nsp protocol, so it looks like F5 doesn’t forward traffic when using nsp protocol but we don’t see anything related to nsp or http in F5 configuration. Only protocol specific configuration we have found is the monitoring.

Any help is appreciated.