Running behind traefik reverse proxy

[morphZ]

Hi everyone,

I'm trying to run the omada controller behind a traefik reverse proxy. Traefik works with SSL-Termination, meaning that it handles the TLS layer on the connection with the client browser, but uses plain http when connecting to the backend (omada in this case).

Problem is: omada always redirects http requests to https and the respective port (8043 as default). When the client browser executes the redirect, it cannot connect to port 8043 on the reverse proxy (which is the correct behaviour).

A similar issue has been addressed in this post but I think the workaround proposed there (a) wouldn't work with traefik and (b) somehow renders one of the main purposes of a reverse proxy mood (-> using a centralized SSL infrastructure for all services).

Here is my docker-compose.yml part:

  omada:
    container_name: omada
    image: mbentley/omada-controller:3.2
    ports:
      - "29811-29813:29811-29813"
      - "29810:29810/udp"
    networks:
      - t2_proxy
    environment:
      TZ: UTC
      MANAGE_HTTP_PORT: 8088
      MANAGE_HTTPS_PORT: 8043
      SHOW_SERVER_LOGS: "true"
      SHOW_MONGODB_LOGS: "false"
      SSL_CERT_NAME: "tls.crt"
      SSL_KEY_NAME: "tls.key"
    volumes:
      - omada-data:/opt/tplink/EAPController/data
      - omada-work:/opt/tplink/EAPController/work
      - omada-logs:/opt/tplink/EAPController/logs
    restart: unless-stopped
    labels:
      traefik.enable: "true"
      traefik.http.routers.omada-rtr.entrypoints: https
      traefik.http.routers.omada-rtr.rule: Host(`omada.$DOMAINNAME0`)
      traefik.http.routers.omada-rtr.middlewares: chain-no-auth@file
      traefik.http.services.omada-svc.loadbalancer.server.port: 8088

I think the best solution would be to introduce a environment variable to the omada docker image, e.g. NO_HTTPS_REDIRECT. When set to true, the omada web server should not redirect to https and 8043, but serve everything via http and 8088. Would that be feasable?

Has anyone made omada work with traefik?

Let me know, if you need any more logs or data to assess.

Thanks in advance
morphZ

[gca3020]

I have this in my nginx proxy config for my omada service. Not sure if you can get traefik to do something similar:

# Add a proxy redirect because omada expects it and will redirect to the ssl port if
# it's not explicitly in the URL
proxy_redirect https://$http_host:$upstream_port/login https://$http_host/login;
proxy_buffering off;

# Add a few more headers to make Omada happy
proxy_set_header        X-Real-IP                 $remote_addr;
proxy_set_header        X-Forwarded-For           $proxy_add_x_forwarded_for;
proxy_set_header        Cookie                    $http_cookie;
proxy_set_header        X-Forwarded-Proto         $scheme;

proxy_set_header        Upgrade                   $http_upgrade;
proxy_set_header        Connection                "upgrade";

[mbentley]

Looking at the thread you linked, those connector properties in 3.2 are the equivalent of the values in the manage.*.port and portal.*.port settings in omada.properties in 4.x and 5.x. When the 3.2 image was the latest, I didn't have the modification of the property file in it's entrypoint as I wasn't yet aware of that being a thing and I only added it to the 4.x image and then I didn't backport the feature since it wasn't the same in 3.x and 4.x.

@gca3020 looks to be on to something. The Upgrade and Connection lines are going to be there to support websockets. I would imagine that Traefik has support for websockets and should be fairly straight forward to enable. Not sure about the proxy_redirect equivalent though.

[gca3020]

Possibly redirectregex? https://doc.traefik.io/traefik/middlewares/http/redirectregex/

[nickmaleao]

The web behavior of V5 is different from the V4, I noticed that after I upgraded to the V5 the URI of the http request has changed and now includes the ID of the omadac.

I'm also using Traefik but with a combination of static and dynamic configs. I've the Omadac in an LXC container but I'm using the Traefik in docker.

The http request to the Omadac needs to have the Host header with the required https port, and the URI of the http request must be rewritten to include the omada instance ID.

Here's the toml config I'm using, you can adapt it to docker labels.

[http.routers]
  [http.routers.wlc1]
    # By default, routers listen to every entry points
    rule = "Host(`wlc1.local.domain.com`)"
    service = "wlc1"
    entrypoints = ["web"]
    middlewares = ["wlc1-https"]

  [http.routers.wlc1-sec]
    # By default, routers listen to every entry points
    rule = "Host(`wlc1.local.domain.com`)"
    service = "wlc1"
    entrypoints = ["websecure"]
    middlewares = ["wlc1-redirectregex", "wlc1-header"]
    [http.routers.wlc1-sec.tls]

[http.services]
  [http.services.wlc1.loadBalancer]
   serversTransport = "wlc1-transp"
    [[http.services.wlc1.loadBalancer.servers]]
      url = "https://192.168.1.40:8043/"

[http.middlewares]
  [http.middlewares.wlc1-https.redirectscheme]
    scheme = "https"
    permanent = true
  [http.middlewares.wlc1-header.headers]
    [http.middlewares.wlc1-header.headers.customRequestHeaders]
        Host = "wlc1.local.domain.com:8043"
  [http.middlewares.wlc1-redirectregex.redirectRegex]
        regex = "^https:\\/\\/([^\\/]+)\\/?$"
        replacement = "https://$1/b0f63ab3712f12c80658ece29d8ed9dc/login"

I tried to do a regex rule to automatically get the Omada id from the http response but didn't find a way to do it.

You can get the Omadac ID from the logs, cat /opt/tplink/EAPController/logs/server.log | grep OmadacVO (main] [] c.t.s.o.s.t.OmadacInitTask(149): succeed get default omadac OmadacVO(id=b0f63ab3712f12c80658ece29d8ed9dc, name=wlc1)
)

[mbentley]

The only other ways I have found to get the controller ID are:

From the Location: header on the login redirect:

$ curl -skI "https://omada.casa.mbentley.net/login" | grep "^Location: " | awk -F '/' '{print $4}'
f2c92c2a9b8fb0a427393f413bb16354

From the API:

$ curl -s "https://omada.casa.mbentley.net/api/info" | jq -r .result.omadacId
f2c92c2a9b8fb0a427393f413bb16354

[morphZ]

@everyone: Thank you for your replies and hints. I'm going to be offline for the next week and try them afterwards. Nonetheless it would be great and much more straight forward if the redirect could be disabled. CU

…

On Wed, 19 Jan 2022 at 16:53, Matt Bentley ***@***.***> wrote: The only other ways I have found to get the controller ID are: From the Location: header on the login redirect: $ curl -skI "https://omada.casa.mbentley.net/login" | grep "^Location: " | awk -F '/' '{print $4}' f2c92c2a9b8fb0a427393f413bb16354 From the API: $ curl -s "https://omada.casa.mbentley.net/api/info" | jq -r .result.omadacId f2c92c2a9b8fb0a427393f413bb16354 — Reply to this email directly, view it on GitHub <#168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACUSYTH7HGIVECKC7VC4MEDUW3NAJANCNFSM5L4NPPUA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ilarrain]

I was able to achieve reverse proxying by setting "HTTPS Port for Controller Management:" in the web-UI to 443, then recreating the container with the host in a different port than the container -p 8043:443, and MANAGE_HTTPS_PORT=443 (this config seems to not have any effect on an already configured installation, hence the change in the UI first).

I'm using Nginx Proxy Manager, not traefik, but it worked without any further configuration there (listens on port 443 and connect to the container in port 8043).

Of course, I can't login directly on port 8043 anymore, but that is expected.

[sphoenixp]

How do you add devices over the internet to your self-hosted controller?

[mbentley]

You would need to use their cloud service to enable remote access, have a VPN back to the location where the controller is running, or expose the controller on the internet (very much not recommended).

[mbentley]

I am going to close this for now since it seems like there are a few patterns that could work for people; feel free to reopen if you wish. If someone did want to submit a PR with a section on reverse proxy details for the README, we can add that.

[milch]

Just to add to this, the missing piece for me here was that my Omada server's SSL certificate was self-signed and therefore untrusted by Traefik. I had to add the following lines to my traefik.yml to enable Traefik to accept untrusted SSL certificates:

# Allow backends to use insecure SSL
serversTransport:
  insecureSkipVerify: true

If you were using a configuration similar to @nickmaleao's, but got 500 errors, this might solve it. Other than that my configuration is very similar, just translated to use labels instead

[reey]

For anyone interested in the docker labels to get this working:

    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.omada.rule=Host(`omada.example.com`)"
      - "traefik.http.routers.omada.entrypoints=websecure"
      - "traefik.http.routers.omada.tls.certresolver=myresolver"
      - "traefik.http.routers.omada.tls.domains[0].main=*.example.com"
      - "traefik.http.services.omada.loadbalancer.server.port=8043"
      - "traefik.http.routers.omada.service=omada"
      - "traefik.http.services.omada.loadbalancer.server.scheme=https"
      - "traefik.http.routers.omada.middlewares=omada-middlewares"
      - "traefik.http.middlewares.omada-middlewares.chain.middlewares=omada-redirect@docker,omada-headers@docker"
      - "traefik.http.middlewares.omada-headers.headers.customrequestheaders.Host=omada.example.com:8043"
      - "traefik.http.middlewares.omada-headers.headers.customresponseheaders.Host=omada.example.com"
      - "traefik.http.middlewares.omada-redirect.redirectregex.regex=^https:\\/\\/([^\\/]+)\\/?$$"
      - "traefik.http.middlewares.omada-redirect.redirectregex.replacement=https://$$1/<controllerId>/login"

replace all occurrences of omada.example.com with your domain and <controllerId> with your controllerId.

[Anzic23]

change in the UI first

Can you tell me how you changed the port in the UI? I did the same but the controller won't let me set it to less than 1024. I would really appreciate it if you could share screenshots of the Nginx Proxy Manager settings as I also use it for my containers. But I can't seem to get it to work with this particular container.

[sarvesh-lad]

proxy_redirect https://$http_host:$upstream_port/login https://$http_host/login;

I dont think this is the case with newer releases since the controller ID is added between the host and login

[SlothCroissant]

Wanted to pass an FYI, this is my currently-working config as of V5 and Traefik v2:

http:
  routers:
    rtr-omada:
      entryPoints:
        - websecure
      rule: "Host(`omada.url.com`)"
      service: svc-omada
      middlewares:
        - mid-omada-redirectRegex
        - mid-omada-headers
      tls:
        certResolver: letsencrypt
  services:
    svc-omada:
      loadBalancer:
        servers:
        - url: "https://omada.lan:8043"
  middlewares:
    mid-omada-redirectRegex:
      redirectRegex:
        regex: "^https:\\/\\/([^\\/]+)\\/?$"
        replacement: "https://$1/controller_id/login"
    mid-omada-headers:
      headers:
        customRequestHeaders:
          host: "omada.url.com:8043"
        customResponseHeaders:
          host: "omada.url.com

Replace controller_id, omada.url.com and omada.lan with your respective values of course, along with general Traefik basics like certResolver, entryPoints, etc.

Also unsure if it matters, but I have omada.url.com specified in Settings > Controller > Access Config > Controller Hostname/IP.

[tehniemer]

Sorry to necro post, but where do I find controller_id in the settings?

[SlothCroissant]

Sorry to necro post, but where do I find controller_id in the settings?

Not near my controller at the moment, but it’s in the URL when you’re in the controller UI. It’s a string of random characters, iirc.

[jenn0pal]

Hi! sorry for necro post, how can I convert this to docker-compose labels?

[tehniemer]

Hi! sorry for necro post, how can I convert this to docker-compose labels?

I ended up just doing this and letting traefik handle SSL.

[stuckj]

Just to add to this, the missing piece for me here was that my Omada server's SSL certificate was self-signed and therefore untrusted by Traefik. I had to add the following lines to my traefik.yml to enable Traefik to accept untrusted SSL certificates:

# Allow backends to use insecure SSL
serversTransport:
  insecureSkipVerify: true

If you were using a configuration similar to @nickmaleao's, but got 500 errors, this might solve it. Other than that my configuration is very similar, just translated to use labels instead

^^^
This was the problem for me. Simply setting the backend service scheme to https and port to 443 and making sure traefik skipped HTTPS verification when talking to a backend service fixed it. Makes sense since the backend service in this case (and probably most cases) is a self-signed certificate. NOTE: That is a global configuration for traefik, NOT a configuration per service. It only skips verification when talking to a backend service, not at the front edge. No funky regex rewrite middleware for controller id or login needed.

[mbseid]

This thread was very helpful for me. Thank you. I'm using Traefik in k3s, so I had to make a few adjustments. Here is my Kubernetes config in case anyone wants it. There are two services, one for http and a second for the ports. The ports are exposed through the k3s Load Balancer. Note, it's also using a SSL cert too.

apiVersion: v1
kind: Service
metadata:
  name: omada-controller-http
  labels:
    app: omada-controller
spec:
  selector:
    app: omada-controller
  ports:
    - name: manage-http
      protocol: TCP
      port: 8088
      targetPort: manage-http
    - name: manage-https
      protocol: TCP
      port: 8043
      targetPort: manage-https
    - name: portal-https
      protocol: TCP
      port: 8843
      targetPort: portal-https
---
apiVersion: v1
kind: Service
metadata:
  name: omada-controller-ports
  labels:
    app: omada-controller
spec:
  selector:
    app: omada-controller
  type: LoadBalancer
  ports:
    - name: app-discovery
      protocol: UDP
      port: 27001
      targetPort: app-discovery
    - name: discovery
      protocol: UDP
      port: 29810
      targetPort: discovery
    - name: manager-v1
      protocol: TCP
      port: 29811
      targetPort: manager-v1
    - name: adopt-v1
      protocol: TCP
      port: 29812
      targetPort: adopt-v1
    - name: upgrade-v1
      protocol: TCP
      port: 29813
      targetPort: upgrade-v1
    - name: manager-v2
      protocol: TCP
      port: 29814
      targetPort: manager-v2
    - name: transfer-v2
      protocol: TCP
      port: 29815
      targetPort: transfer-v2
    - name: rtty
      protocol: TCP
      port: 29816
      targetPort: rtty
---
apiVersion: traefik.containo.us/v1alpha1
kind: ServersTransport
metadata:
  name: omada-controller-transport
  namespace: default
spec:
  insecureSkipVerify: true
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: omada-controller
spec:
  tls:
    secretName: homelab-ssl-cert
  entryPoints:
    - web
    - websecure
  routes:
    - match: Host(`{{host_name}}`)
      kind: Rule
      services:
        - name: omada-controller-http
          scheme: https
          port: manage-https
          serversTransport: omada-controller-transport
      middlewares:
        - name: redirect-https
     ```

[Wielewout]

Thanks @mbseid! Maybe for completeness sake the redirect-https middleware:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: redirect-https
spec:
  redirectScheme:
    scheme: https
    port: "{{ traefik_websecure_port }}"
    permanent: false

[mikki8]

is there a easy to spin up docker compose example with traefik and lets encrypt ?

[IetIesAai]

in system settings, I turned off "Redirect HTTP to HTTPS" + restarted the container, and that allows me to use my standard traefik config.

[tehniemer]

This was the simple fix for me as well.

[Shadoweee77]

Does anyone here have a current (v5/v6) compose file that uses traefik labels?

[IetIesAai]

you can no longer turn off the redirect to https, so I do it this way:
it can't be done with just labels though, some options needed for allowing it to use a selfsigned cert need to be in files.

in omada compose file:

    container_name: omada
    image: mbentley/omada-controller:latest-chromium
    restart: always
    stop_grace_period: 60s
    networks:
   [your net work config here]
    environment:
      - PUID=xxx
      - PGID=xxx
      - MANAGE_HTTP_PORT=8088
      - MANAGE_HTTPS_PORT=8043
      - PORTAL_HTTP_PORT=8088
      - PORTAL_HTTPS_PORT=8843
      - PORT_APP_DISCOVERY=27001
      - PORT_ADOPT_V1=29812
      - PORT_UPGRADE_V1=29813
      - PORT_MANAGER_V1=29811
      - PORT_MANAGER_V2=29814
      - PORT_DISCOVERY=29810
      - PORT_TRANSFER_V2=29815
      - PORT_RTTY=29816
      - SHOW_SERVER_LOGS=true
      - SHOW_MONGODB_LOGS=true
      - TZ=Europe/Brussels
    volumes:
      - xxxxx/data:/opt/tplink/EAPController/data
      - xxxxx/logs:/opt/tplink/EAPController/logs
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.omada.rule=Host(`omada.[hostname]`)"
      - "traefik.http.routers.omada.entrypoints=websecure"
      - "traefik.http.routers.omada.tls.certresolver=letsencrypt"  
      - "traefik.http.services.omada.loadbalancer.server.port=8043"
      - "traefik.http.services.omada.loadbalancer.serversTransport=skipverifytransport@file"
      - "traefik.http.services.omada.loadbalancer.server.scheme=https" 
      - "traefik.docker.network=[network]"

I replaced some parts in this example with xxxxxx and [ ] - replace those for your situation

The hsts-on@file and skipverifytransport@file refer to the following section in traefik-options.yml, which I declare in the compose file for traefik itself - put the file in the folder you mount to /config in the traefik container:

in the traefik compose:

    command:
      - '--providers.file.filename=/config/traefik-options.yml'

in traefik-options.yml:

  serversTransports:
    skipverifytransport:
      insecureSkipVerify: true

[YetAnotherBugHunter]

Got it workin with Traefik, omada-controller:6.2.14.11 and lets encrypt handling the certificate nonsense! :)

omada-controller:
    image: mbentley/omada-controller:6.2.14.11
    container_name: omada-controller

      - traefik.enable=true
      - traefik.docker.network=traefik
      # Controller
      - traefik.http.routers.omada-controller.rule=Host(`<FQDN>`) # Replace <FQDN> with the FQDN of your instance
      - traefik.http.routers.omada-controller.entrypoints=https
      - traefik.http.routers.omada-controller.tls=true
      - traefik.http.routers.omada-controller.tls.certresolver=letsencrypt
      - traefik.http.routers.omada-controller.service=omada-controller
      - traefik.http.routers.omada-controller.middlewares=omada-controller-forwarded
      - traefik.http.middlewares.omada-controller-forwarded.headers.customrequestheaders.X-Forwarded-Proto=https
      - traefik.http.middlewares.omada-controller-forwarded.headers.customrequestheaders.X-Forwarded-Port=443
      - traefik.http.middlewares.omada-controller-forwarded.headers.customrequestheaders.Host=controller.omada.cidercreekranch.net:8043
      - traefik.http.services.omada-controller.loadbalancer.server.port=8043
      - traefik.http.services.omada-controller.loadbalancer.server.scheme=https
      - traefik.http.services.omada-controller.loadbalancer.serverstransport=omada-controller-transport@file`
    
Also needed to create a dynamic TOML file saved as traefik-dynamic.toml into the config/dynamic directory with the following content:

[http.serversTransports]
  [http.serversTransports.omada-controller-transport]
    insecureSkipVerify = true

In my setup Traefik is in its own container, so I need to restart its stack.

There may a simpler way of arriving a the same outcome but I'm leaving well enough alone.