[Help]: Does this ring any bells? "unable to find valid certification path to requested target"

[ovizii]

Controller Version

5.15.20.20

Describe Your Issue or Question

After successfully running for more than a year, even after many updates, the controller suddenly is no longer being reported as online in the TP-Link Cloud.

I don't see much in the controller's logs so I thought I'd ask if this is any indication as this is the only thing looking like an error in the logs.?

omada-controller  | 05-25-2025 14:47:14.408 INFO [Thread-9] [] c.t.e.c.c.CloudClient(): Connect service server automatically, ConnectionType is PERSISTENT_CONNECTION.
omada-controller  | 05-25-2025 14:47:14.408 INFO [Thread-9] [] c.t.e.c.c.q(): set connection status: CONNECTING
omada-controller  | 05-25-2025 14:47:14.525 INFO [Thread-9] [] c.t.e.c.c.q(): Exception occurs when connecting service server:javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
omada-controller  | 05-25-2025 14:47:14.525 INFO [Thread-9] [] c.t.e.c.c.q(): set connection status: DISCONNECTED_NORMAL
omada-controller  | 05-25-2025 14:50:07.429 WARN [quartzScheduler_Worker-1] [] c.t.s.c.s.c.TaskExecutorService(): receive scheduled event with identity (log_device_status_push, null) but did not execute because corresponding handler log_device_status_push doesn't exist

Expected Behavior

I'd expect my controller to show as ONLINE:

Steps to Reproduce

Sorry, but no idea, fishing for help here :-)

How You're Launching the Container

services:

# https://github.com/mbentley/docker-omada-controller

  omada-controller:
    image: mbentley/omada-controller:5.15
    container_name: omada-controller
    hostname: omada-controller
    restart: "no"
    networks:
      vlan_MGM:
        ipv4_address: 192.168.98.3
      traefik_omada:
    ulimits:
      nofile:
        soft: 4096
        hard: 8192
    stop_grace_period: 60s
    volumes:
      - ./data:/opt/tplink/EAPController/data
      - ./logs:/opt/tplink/EAPController/logs
    cpus: 2
    mem_limit: 3G
    memswap_limit: 4G
    environment:
      - TZ=Europe/City
      - PGID=1000
      - PUID=1000
      - SHOW_SERVER_LOGS=true
      - SHOW_MONGODB_LOGS=false
    labels:
      # enable Traefik
      - traefik.enable=true
      # Network
      - traefik.docker.network=traefik_omada
      # Router
      - traefik.http.routers.omada.tls=true
      - traefik.http.routers.omada.entrypoints=websecure
      - traefik.http.routers.omada.rule=Host(`omada01.mydomain.tld`)
      - traefik.http.routers.omada.middlewares=crowdsec@file,secHeaders@file
      - traefik.http.routers.omada.service=omada
     # Service
      - traefik.http.services.omada.loadbalancer.server.port=8043
      - traefik.http.services.omada.loadbalancer.server.scheme=https

Container Logs

I can provide the full output if this looks like it could help. Currently, I just want to figure out if it's something with this container or something on my system causing problems.

MongoDB Logs

No response

Additional Context

I have recently cleaned up zones, VLANs, DHCP on my firewall, just need a heads-up if the above error from the log looks like I caused that or if others also have that.

[mbentley]

Hmm, that's odd. I just setup a clean install of a controller, connected it to the cloud and it worked. My current controller I actually use in my house isn't functioning with the cloud for other reasons. It seems like it's getting some sort of SSL error - that error seems to be indicating that the SSL cert isn't trusted by the controller (in this case the java trust store). Not sure if they had some sort of issue with their SSL cert for https://omada.tplinkcloud.com/ as I believe that's the endpoint it hits but it might be worth checking again or trying to unbind and re-bind to the cloud.

[ovizii]

Thanks for having a look. If you have any other ideas, please let me know. Meanwhile, I've been searching the internet but haven't found anything similar yet.

There is an additional problem here. I cannot log in if I access the controller locally, neither directly via IP nor via reverse proxy. I've had this issue for a while but as long as I could access and configure it via the cloud, I didn't mind.

Now I am stuck wit ha working controller but if I want to reconfigure anything on it, I'd have to start from scratch due to not being able to log in.

Accessing its GUI and entering my credentials end in this error: Checking with the browser debug tools, all is well.

And this shows in the container's logs at the same time:

omada-controller | 05-26-2025 19:10:39.064 INFO [https-jsse-nio-8043-exec-8] [] c.t.s.o.a.d.SpeedUpLoginController(): omadacId=a5210d42e158323b4c9c2093f74dbe1b check login by TP CLOUD response:OperationResponse(errorCode=-7119, msg=Operation failed. Please check your network connection., result=null, auditLogReturn=null, auditLogReturnList=null)

I have no idea what to do. I have turned off IDS and IPS and any filtering rules as well as DNS based ad-blocking. I inserted a new Allow Any rule in my FW just for testing this.
I entered the controller's container and since there is no ping or curl or nc available, I tried accessing some TP link website and a few other test candidates with wget and I get a 200 response, so all is well.

docker compose exec --user=omada omada-controller bash
omada@omada-controller:/opt/tplink/EAPController/lib$
omada@omada-controller:/opt/tplink/EAPController/lib$ wget --server-response -O /dev/null https://omada.tplinkcloud.com/
--2025-05-26 19:16:36--  https://omada.tplinkcloud.com/
Resolving omada.tplinkcloud.com (omada.tplinkcloud.com)... 3.160.246.65, 3.160.246.26, 3.160.246.12, ...
Connecting to omada.tplinkcloud.com (omada.tplinkcloud.com)|3.160.246.65|:443... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Content-Type: text/html
  Content-Length: 2720
  Connection: keep-alive
  Last-Modified: Thu, 15 May 2025 02:49:26 GMT
  Server:
  Content-Security-Policy: default-src 'self' https://*.tplinkcloud.com/; script-src 'self' 'unsafe-eval' https://www.paypal.com/ https://www.paypalobjects.com/ https://js.stripe.com 'sha256-7W9UiBaYGlOHpT1aQBLegqffUVHbYq6/ZAb+ErjUb40=' 'sha256-VGQ8jNTL2g0e8wPwOgyCQJDqhuRgfV7gRYexcBkBe4Y=' 'sha256-+9dQLByJ0rMH7ojZkdfnL0p0S0pZqKxWTlh7vSa+FMg=' 'sha256-x2jgB1zBLi30IsfY+VNgWjwBGeHPJxOSrzl+IdsT6k0=' 'sha256-0AHZXO4clnpdcxqdmASPBEp4JCIrtaxIX/mUuL1kzZw=' 'sha256-lfXlPY3+MCPOPb4mrw1Y961+745U3WlDQVcOXdchSQc=' 'sha256-VohL58KrXs+2LMTip+0SXk2/JwaZNxG/j9hadIPKc2E='; style-src 'self' 'unsafe-inline'; connect-src 'self' https://*.cloudfront.net/ https://*.tplinkcloud.com/ https://*.tplinkcloud.com:8843/ http://*.tplinkcloud.com:8088/ wss://*.tplinknbu.com/ ws://localhost:*/ wss://*.tplinkcloud.com/ https://*.paypal.com/ https://api.stripe.com https://*.tiles.mapbox.com https://api.mapbox.com/ https://events.mapbox.com https://*.amazonaws.com/ data:; frame-src 'self' OmadaSightWebPlayer: vigiwebplayer: https://*.tplinkcloud.com/ https://js.stripe.com https://hooks.stripe.com https://*.paypal.com/ data: blob:; img-src 'self' https://*.cloudfront.net/ https://*.tplinkcloud.com/ https://*.amazonaws.com/ https://*.mzstatic.com/ https://play-lh.googleusercontent.com/ https://kcart.alipay.com/ data: blob:; child-src blob:; worker-src https://*.tplinkcloud.com blob:; media-src https://*.tplinkcloud.com/ https://*.cloudfront.net/ https://*.amazonaws.com/ blob:; object-src 'self' data: blob:
  Strict-Transport-Security: max-age=63072000; includeSubdomains; preload
  X-Content-Type-Options: nosniff
  X-Frame-Options: SAMEORIGIN
  X-XSS-Protection: 1; mode=block
  Referrer-Policy: strict-origin-when-cross-origin
  CloudFront-Viewer-Country: PL
  Date: Mon, 26 May 2025 16:16:37 GMT
  Cache-Control: no-cache
  ETag: "0145eb74c854b92c964345c68f41a567"
  Vary: Accept-Encoding
  X-Cache: RefreshHit from cloudfront
  Via: 1.1 3b1578e183b63617888fbfb492903da4.cloudfront.net (CloudFront)
  X-Amz-Cf-Pop: OTP50-P2
  X-Amz-Cf-Id: jCvkLDVwThWtokF_JsMeeHUlMucwQPpDWQgH-UGtpLBU7FY4ladviQ==
Length: 2720 (2.7K) [text/html]
Saving to: '/dev/null'

/dev/null                                 100%[=====================================================================================>]   2.66K  --.-KB/s    in 0s

2025-05-26 19:16:36 (78.9 MB/s) - '/dev/null' saved [2720/2720]

[mbentley]

Any chance you've tried to just restart the container at this point? I've seen some odd issues people will have with the last few versions of the controller but restarting seems to help. One thing being an issue with the controller going through a lot more open files than normally expected but I see your ulimit is adjusted and these are more like network related errors than open files. Otherwise, also trying to re-create the container (compose down/up) just to see if something got borked in the container itself.

I don't know of if there is a way to do a sort of curl/wget test equivalent in java as it certainly has it's own unique environment vs using CLI tools where issues wouldn't show up there.

[ovizii]

Thanks for your feedback and suggestions. I've solved it. Will leave detailed explanation in case anyone faces something similar.

I use the Sophos XG firewall as a VM appliance.
There was an active feature, which was not set up with any blocking rules, simply inspecting encrypted traffic, but it caused me not to be able to access the local interface by interfering with encrypted traffic.

So I could now access my controller locally, but it was disconnected from the cloud.

Turns out the controller connects to the cloud via port 80, and I had only allowed outgoing connections using port 443. So I made an exception to let omada controller connect to the tplink cloud via port 80. Still feels weirdly insecure. Not sure what traffic goes through port 80, I might stop using the cloud and access it locally so I can close port 80.

[ovizii]

Damn, I couldn't disconnect the controller from the cloud, so I deleted the organization from TPLINK's cloud interface, now I can't log in with my cloud credentials and the local credentials don't work anymore. A total nightmare this Omada stuff ;-)

Time to restore the controller backup from last night.

[ovizii]

Btw. I reset my credentials like this:

enter container
docker exec -ti omada-controller bash
install mongo clients
apt update && apt install -y mongodb-clients
connect to mongod
mongo --port 27217
connect to db
use omada
find the _id of my user
db.tenant.find({}, {name:1, email:1, password:1}).pretty()
the known pw hash for "password" is $shiro1$SHA-256$500000$$Z85mqKxm1Lt0NJRw9jUlw3AzDQxrMHQWebk1kNb4pSM=

update password

db.iam_user.update(
  { _id: ObjectId("<user_id_from_tenant>") },
  { $set: { password: "$shiro1$SHA-256$500000$$Z85mqKxm1Lt0NJRw9jUlw3AzDQxrMHQWebk1kNb4pSM=" } }
)

Logged in with my username and password and reset it to a safe pw.

[mbentley]

This is awesome - I am going to make sure to add this to the README somewhere. Thanks for sharing!

[ovizii]

You're welcome. It should be 99% accurate. I found 3 different tutorials explaining the same thing but for different controller versions, so I fed all that to ChatGPT, tested its reply, it worked, so I summarized it. Should be error free.

And thanks for this amazing image you have put together and made available!

[lyledouglass]

For anyone that comes and finds this solution - mongodb-clients has been deprecated - you can instead run

Install mongosh package
sudo apt-get install -y mongodb-mongosh

Connect to MongoDB
mongosh --port 27217

Rest of instructions are the same