maintenance daemon error when use_secondary_nodes=always

[Ping (Pookping)]

Hello all. It seems Citus can use secondary nodes (aka standby nodes, right?) to achieve read-only on replicas. If I am correct, I have managed to get it to work with citus.use_secondary_nodes='always'. However, the maintenance daemon errors. Am I doing something wrong? Am I misunderstanding the intent of use_secondary_nodes?

I can trace the error to the maintenance functions, but I don't know how to change the daemon to ignore the secondary nodes.

Thank you in advance for your help.

Here's the log errors from Citus coordinator.

2023-08-06T05:06:08.481021460Z 2023-08-06 05:06:08.480 UTC [1255] LOG:  starting maintenance daemon on database 16384 user 10
2023-08-06T05:06:08.481078901Z 2023-08-06 05:06:08.480 UTC [1255] CONTEXT:  Citus maintenance daemon for database 16384 user 10
2023-08-06T05:06:08.484165240Z 2023-08-06 05:06:08.484 UTC [1255] ERROR:  writing to worker nodes is not currently allowed
2023-08-06T05:06:08.484193264Z 2023-08-06 05:06:08.484 UTC [1255] DETAIL:  citus.use_secondary_nodes is set to 'always'
2023-08-06T05:06:08.484205611Z 2023-08-06 05:06:08.484 UTC [1255] CONTEXT:  Citus maintenance daemon for database 16384 user 10
2023-08-06T05:06:08.486284136Z 2023-08-06 05:06:08.486 UTC [1] LOG:  background worker "Citus Maintenance Daemon: 16384/10" (PID 1255) exited with exit code 1
2023-08-06T05:06:08.922388401Z 2023-08-06 05:06:08.922 UTC [1256] LOG:  starting maintenance daemon on database 16384 user 10
2023-08-06T05:06:08.922430871Z 2023-08-06 05:06:08.922 UTC [1256] CONTEXT:  Citus maintenance daemon for database 16384 user 10
2023-08-06T05:06:08.925600746Z 2023-08-06 05:06:08.925 UTC [1256] ERROR:  writing to worker nodes is not currently allowed
2023-08-06T05:06:08.925620097Z 2023-08-06 05:06:08.925 UTC [1256] DETAIL:  citus.use_secondary_nodes is set to 'always'
2023-08-06T05:06:08.925624592Z 2023-08-06 05:06:08.925 UTC [1256] CONTEXT:  Citus maintenance daemon for database 16384 user 10
2023-08-06T05:06:08.927713455Z 2023-08-06 05:06:08.927 UTC [1] LOG:  background worker "Citus Maintenance Daemon: 16384/10" (PID 1256) exited with exit code 1

Here's my pg_dist_nodes:

athena=# select * from pg_dist_node;
 nodeid | groupid |          nodename           | nodeport | noderack | hasmetadata | isactive | noderole  | nodecluster | metadatasynced | shouldhaveshards
--------+---------+-----------------------------+----------+----------+-------------+----------+-----------+-------------+----------------+------------------
      1 |       0 | xxxxxxxxxxxxx_citus_master  |     5432 | default  | t           | t        | primary   | default     | t              | f
      2 |       1 | xxxxxxxxxxxxx_citus_worker1 |     5432 | default  | t           | t        | primary   | default     | t              | t
      3 |       2 | xxxxxxxxxxxxx_citus_worker3 |     5432 | default  | t           | t        | primary   | default     | t              | t
      4 |       1 | xxxxxxxxxxxxx_citus_worker2 |     5432 | default  | f           | t        | secondary | default     | f              | t
      5 |       2 | xxxxxxxxxxxxx_citus_worker4 |     5432 | default  | f           | t        | secondary | default     | f              | t
(5 rows)
```