nixos/traefik: update and revamp module

[TheRealGramdalf]

Updated the Traefik module to modernize it. Primary changes:

• Removed useEnvSubst in favor of environmentFile
• Rename *ConfigOptions and *ConfigFile to *.settings and *.file
• Rename dynamic to routing and static to install to remain consistent with upstream
• Introduce a new method of routing configuration, routing.extraFiles
• Update the NixOS tests to cover more cases
• Added documentation

Note: I have been running an older variant of this PR on my server for the past two years without issue. It (primarily the extraFiles functionality) has worked seamlessly ever since, surviving many updates and continuing to proxy a slew of services. I have just updated the module to the version in this PR, and have not run into any issues. See the comment below.

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.
• Follows the automation/AI policy.

Full changelog

The following changes have been implemented:

• Increase UDP buffer sizes to the recommended modern value, ~7.5MB
  • Changed to use mkDefault to match the kubo and caddy modules
• Add the module refactor note to notable changes (the refactor doesn't actually break configs, except for envsubst)
  • Add removal of useEnvSubst as a breaking change
• Add redirects to doc/redirects.json
• Updated the NixOS tests to cover more cases
  • A VM is used to test the docker provider
  • The rest of the tests use systemd-nspawn containers, which are much faster. Multiple tests are run in parallel, using different configuration methods
• Added documentation to the NixOS manual
  • I did my best to make this conform to the style guide, but it's by no means perfect. Suggestions are welcome
• Added mkRenamed/mkRemoved option modules
• Fixed the enable option description, which incorrectly called Traefik a web server
• Updated the systemd service definition
  • See: https://github.com/traefik/traefik/blob/master/contrib/systemd/traefik.service
  • Fixed the description, which incorrectly called Traefik a web server
  • Added a unitConfig.documentation link
  • Set the service type to notify, allowing for more accurate unit status
  • Set ProtectSystem = "strict"
  • Rename LimitNPROC to TasksMax
    • Upstream this is set to 1, not 64 - I haven't tested if that works or not. Upstream systemd service hasn't been updated in 6 years
  • Protect cgroups and kernel tunables
• Add/use the cfg.user option
• Only create user/group if it is the default value
• Added meta attributes to the module
• Renamed references to static and dynamic configuration to the new install and routing terminology
• Use settings and file instead of <method>ConfigOptions and <method>ConfigFile
  • Renamed all references in nixpkgs to the new values
• Use pkgs.formats.json instead of toml
• Add more information to cfg.user and cfg.group (matching the redis server options)
• Add/use supplementaryGroups to give access to the docker daemon instead of setting the primary group
  • An assertion has been added to prevent this from happening, as it poses a security risk
• Add instructions to the environmentFiles for usage in place of envSubst
• Options directly under routing or install could be considered "vendored" options - they add extra glue to make the module feel better to use
• Wherever reasonable, internal logic ("glue") have been exposed through defaultText or readOnly options
  • install.settings now houses most of the mkIf logic that adds the file provider
  • In order to make this work, install.settings is a submodule with freeformtype.
    • install.file now contains the logic to generate the install configuration from install.settings
    • empty lists ([]), attribute sets ({}), or null values (null) are now filtered out in the generation logic to allow definition of typed options with default values
• This replaces the following logic, which was only exposed through the let ... in block at the top of the module - it was not really documented anywhere

  dynamicConfigFile =
    if cfg.dynamicConfigFile == null then
      format.generate "config.toml" cfg.dynamicConfigOptions
    else
      cfg.dynamicConfigFile;

  staticConfigFile =
    if cfg.staticConfigFile == null then
      format.generate "config.toml" (
        recursiveUpdate cfg.staticConfigOptions {
          providers.file.filename = "${dynamicConfigFile}";
        }
      )
    else
      cfg.staticConfigFile;

  finalStaticConfigFile =
    if cfg.environmentFiles == [ ] then staticConfigFile else "/run/traefik/config.toml";

• if services.traefik.routing.extraFiles are declared, routing.dir is not set, and routing.settings is, automatically merge contents of files.settings into routing.settings. Modules can declare traefik.enable options as routing.extraFiles, and it will work with either routing.extraFiles or routing.settings
  • A new readOnly option, routing.settingsDrv, is used to merge everything together.

What remains to be done:

• Optional: In addition to the nixosTests, traefik provides unit tests (see: https://doc.traefik.io/traefik/contributing/building-testing/#testing). These could be added to the package directly (as part of the check phase)

Final Checks - just before merge

• Ensure that options referenced in documentation (option descriptions, manuals) accurately reflect the final option trees
• Probably a good idea: Manually test migration from a config with the old system to the new module
• Probably a good idea: test the new system with an actual config (I can do this since I'm already using the option interface from the original PR, which is largely unchanged). This should hopefully catch any issues with a more complex system that the tests may not have
• Possibly a good idea: write a brief history of this and previous PRs for historical purposes
• Thorough review by a third party (not previously involved with this PR) as a sanity check (once everything is ready to merge)

[RafaelKr]

Could you please elaborate why you're using those default options which introduce so much evaluation logic (you call it "glue") on the option level instead of having "dumb" options and merge conditionally on the config level?

Don't get me wrong, this is a serious question as I tried to wrap my head around it since the #490985 PR and just don't understand the reason behind it.

To me it feels like it makes things much more complicated than they need to be. And it feels unintuitive to my understanding of options as the declaration- and config as the evaluation-layer.

I think all that glue could be moved to the config layer and it would simplify reading the module code a lot.
Also using the module would be more intuitive, especially because you could drop your converge logic which filters out {} (empty attrsets), which is valid traefik syntax and is used in even the most basic examples in their documentation. So "translating" existing traefik config to their nixos module counterpart would be more or less 1:1.

[TheRealGramdalf]

Could you please elaborate why you're using those default options which introduce so much evaluation logic (you call it "glue") on the option level instead of having "dumb" options and merge conditionally on the config level?

It's for a few reasons, but part of why I took this approach is because Traefik is somewhat unique in the way it handles configuration - it has several different providers, and each can provide configuration in one way or another. All of these providers, however, eventually boil down to the install configuration.

Since install configuration methods are mutually exclusive, adding a dynamic file provider is not just as simple as passing --providers.file.filename=/path/to/file.yaml - it must be part of the install file instead. This means merging with the user's configuration, which is typically done in the config = {...} section.

Stepping back from the development perspective, think about an end user. Imagine the user doesn't know how to read Nix source code yet, and is migrating from another platform like docker. If you were that user, would you prefer to:

• Read the NixOS manual to learn what sections of the install configuration the module touches, and in which circumstances
• Learn to read the source code of the module
• ...Or simply look at install.settings.providers.file.filename on https://search.nixos.org, and see that the default value is something along the lines of json.generate { } routing.file

While this is a cherry-picked scenario, I've spent enough time learning enough different things in life from scratch to know that having the information right there can make all the difference. Is it more work on the part of the module maintainers? Yes. But I think it's better for the maintainer to put in a little extra work so that every single person who ends up using the module has a better experience.

Don't get me wrong, this is a serious question as I tried to wrap my head around it since the #490985 PR and just don't understand the reason behind it.

No worries. I realize this approach is slightly unorthodox, but I genuinely think it is a better way of doing it.

To me it feels like it makes things much more complicated than they need to be. And it feels unintuitive to my understanding of options as the declaration- and config as the evaluation-layer.

Not sure I can add much here that I haven't said elsewhere.

I think all that glue could be moved to the config layer and it would simplify reading the module code a lot.

It would, but as I mentioned above that's not necessarily a good thing.

Also using the module would be more intuitive, ...

At risk of repeating myself, using the module might be more intuitive if you can read the source code.

...especially because you could drop your converge logic which filters out {} (empty attrsets), which is valid traefik syntax and is used in even the most basic examples in their documentation.

(Warning: pseudocode below)

I'm genuinely curious; which examples are you referring to? I checked while I was developing this, and couldn't find anything indicating this was the case. In my time using Traefik with docker, I can only ever recall one instance of using {} - which was for the experimental setting enabling http3 (I think - it may have been some other experimental setting though). The only reason I remember that is because I found it odd that an empty attribute set meant something, and that experimental.http3 = true also worked just as well. I think the reason for that was because defining any of the settings underneath like experimental.http3.cachesize would implicitly enable http3.

Said option has since been removed (it is now enabled by default), and I have never encountered a situation in which an empty value has been required.

So "translating" existing traefik config to their nixos module counterpart would be more or less 1:1.

This is one of the other reasons I took a different approach - the extraFiles interface is intended to somewhat mimic the docker provider, which uses labels. Each service can have its own submodule, which keeps configuration segmented.

This provides a number of benefits when actually using Traefik as a proxy - each service has its own file, which can be inspected, copied and modified imperatively, all while (hopefully) seamlessly meshing with the rest of NixOS. Declarative is great, but the reason the routing provider exists is so that changes in it don't require a restart of the primary daemon. My implementation takes it a step further by not requiring a nixos-rebuild - if you need to quickly fix something but rebuilding the whole NixOS configuration takes time (e.g. a package needs to be compiled), you can do so. Need to take down a specific service for security purposes? Delete the symlink. The changes will sync back up when you rebuild the system next, after you've hardened the proxy configuration.

I do admit that this is a potential downside, but unless there is significant use of empty values I don't think it's an issue. If so, there's some trickery I can pull to differentiate empty option defaults from empty values set by the user

---

Misc benefits:

• Less need for documentation of how options merge, everything is visible on search.nixos.org
• Almost all merge logic can be easily tweaked by the end user, without waiting for a PR to be drafted and reviewed. Just set the option which contains the logic
• All merge logic is handled cleanly, with each option depending on the next until it hits --configfile. Without this approach, it would be a bunch of boolean logic with a few arbitrary conditions. This way it's nearly impossible for that logic to fail in a way that prevents a configuration from working properly, and there are very few "implementation details" to worry about

[RafaelKr]

I didn't forget you, just didn't have time to answer yet! Thank you very much for the detailed answer.

[TheRealGramdalf]

I have removed the following assertions:

{
        # TODO ensure this works with install.settings being a submodule
        assertion =
          opt.install.file.highestPrio != defaultOptPrio
          -> opt.install.settings.highestPrio == defaultOptPrio;
        message = ''
          The 'services.traefik.install.file' and 'services.traefik.install.settings' options are mutually exclusive.
          It is recommended to use 'settings'.
        '';
      }
      (
        let
          isEmpty = a: (a == { } || a == [ ] || a == null);
        in
        {
          assertion =
            (opt.install.file.highestPrio != defaultOptPrio)
            -> (builtins.all isEmpty [
              cfg.routing.extraFiles
              cfg.routing.dir
              cfg.routing.file
              cfg.routing.settings
            ]);
          message = ''
            None of the routing configuration options may be used if Traefik is being managed imperatively.
            The following options have non-default values:
              - ${
                concatMapStringsSep "\n  - " (str: "'services.traefik.routing.${str}'") (
                  filter (attr: !(isEmpty cfg.routing."${attr}")) [
                    "extraFiles"
                    "dir"
                    "file"
                    "settings"
                  ]
                )
              }
          '';
        }
      )

And the now-unused let bindings:

inherit (lib)
    concatMapStringsSep
    filter
  opt = options.services.traefik;
  defaultOptPrio = (lib.mkOptionDefault { }).priority;

...Since I realized they unintentionally prevented the case of a user overriding the merge logic by setting the relevant option. These could be reintroduced as warnings or have an acknowledgeWarnings option added to have the upside of catching configuration mistakes while allowing full control if needed.

[TheRealGramdalf]

I didn't forget you, just didn't have time to answer yet! Thank you very much for the detailed answer.

All good! Are you able to comment on the empty attribute set question?

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/7429

[TheRealGramdalf]

I have updated the module I use on my server from an older version of this PR, which I have been running for the past two years. The migration worked flawlessly, though it didn't hit the mkRenamedOptionModules since the previous version was also using install.settings equivalents.

To use the module from this PR, import it with the following changes.
Unless you use useEnvSubst, no changes should be required to your configuration.

meta = {
    maintainers = with lib.maintainers; [
      jackr
      therealgramdalf
    ];
-    doc = ./traefik.md;
+    # Fix eval error on local copy
+    #doc = ./traefik.md;
  };

let
  ...
in
{
+  # Replace the upstream module
+  disabledModules = [ "services/web-servers/traefik.nix" ];
}

With this, I have manually tested a few things:

• Replacing extraFiles symlinks with the file from the store works, given the correct permissions (see docs)
• systemd-tmpfiles will cull files matching the prefix and replace them with the new symlink (see docs)
  • Tested with systemd-tmpfiles --dry-run --remove --create --prefix /path/to/routing/dir (I tested without --dry-run, added here for safety)
• Traefik will not load .json files from routing.dir, but it is perfectly fine with .yml files with json contents
• Traefik will accept a file with a .json suffix for the install configuration (since the full path is specified). This likely works the same for routing.file, the NixOS tests should cover that case
• My ~22 services proxied through Traefik, configured through routing.extraFiles

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-in-distress/3604/143

[TheRealGramdalf]

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 553100 --package nixosTests.traefik
Commit: 69bfcab00ba57d5f9a5fd8e14a8b411e2a6fa1bf

---

x86_64-linux

✅ 1 test built:

• nixosTests.traefik

[RafaelKr]

Sorry for the delay!

I'm genuinely curious; which examples are you referring to?

These are the usages of empty curly braces I found:

• api: {} to enable the API/dashboard (docs)
• http3: {} on an entrypoint (docs). This is the current form. The removed one is the traefik v2 experimental.http3 flag you were thinking of
• metrics.prometheus: {} (docs)
• ping: {} to enable the /ping health-check endpoint (docs)
• providers.docker: {} to enable the Docker provider with defaults (docs)
• tls: {} on a router to enable TLS with defaults (docs)

From my understanding the thing you most care about is end user usability. And you're right with that. Also I agree, that I'd like to work through the search.nixos.org options.

Filtering {} out at generation time folds "on with defaults" into "unset". You try to handle this case with your install.settings note that is asking the users to write attr = true instead of attr = {}, but that's a module-specific conversion you have to discover in the docs / on search.nixos.org. IMO it's easy to miss and adds mental overhead versus translating an existing Traefik config as-is. The worst thing though, it is silent and thus hard to debug. You would only notice it if your config doesn't behave as expected or if you look at the effective config file.

Not a blocker, I just think it hurts usability and you could easily work around it by moving the
option merging logic out of the options.services.traefik.install.file.default to a variable
which is defined above each options and config. You can see what I mean at installFile in my commit RafaelKr@4ad173f

This would simplify your implementation a lot as you can easily differentiate the user defined options from the default options.

Less need for documentation of how options merge, everything is visible on search.nixos.org

Unfortunately my time is limited today, so I'll just drop a discoverability comparison of your option interface and mine.
It's using https://github.com/NuschtOS/search (which, unlike search.nixos.org, unfortunately doesn't allow to search inside the option description).

• Yours at 4ff7df0: https://rafaelkr.github.io/nixos-search/pr553100-vs-attrtag/options?scope=PR%20%23553100%20(4ff7df0e)
• Mine at 7c01005: https://rafaelkr.github.io/nixos-search/pr553100-vs-attrtag/options?scope=attrTag%20(7c01005a)

Mine still contains the plugin logic which was part of #490985, so the comparison is not fully clean.

...Or simply look at install.settings.providers.file.filename on https://search.nixos.org, and see that the default value is something along the lines of json.generate { } routing.file

I don't know if this was possible with an earlier version of your PR, but now install.settings.providers.file.filename doesn't lead me to an option anymore, which would let me find the json.generate logic. Or are you referring to routing.file which gives me a hint about routing.settingsDrv which shows me the (pretty hard to read) settingsDrv logic?

All in all this comes down to two design styles: I model each file-provider scenario as its own typed option. routing.provider.file (Nix-generated), routing.provider.externalFile (a file you manage yourself), routing.provider.directory (a watched dir) and let the type pick exactly one. Yours does it with routing.file and routing.dir and an assertion. Both work, I just find the explicit split easier to read and discover. Curious what you think.

[TheRealGramdalf]

These are the usages of empty curly braces I found:

* `api: {}` to enable the API/dashboard ([docs](https://doc.traefik.io/traefik/reference/install-configuration/api-dashboard/))

* `http3: {}` on an entrypoint ([docs](https://doc.traefik.io/traefik/reference/install-configuration/entrypoints/)). This is the current form. The removed one is the traefik v2 `experimental.http3` flag you were thinking of

* `metrics.prometheus: {}` ([docs](https://doc.traefik.io/traefik/reference/install-configuration/observability/metrics/))

* `ping: {}` to enable the `/ping` health-check endpoint ([docs](https://doc.traefik.io/traefik/reference/install-configuration/observability/healthcheck/))

* `providers.docker: {}` to enable the Docker provider with defaults ([docs](https://doc.traefik.io/traefik/reference/install-configuration/providers/docker/))

* `tls: {}` on a router to enable TLS with defaults ([docs](https://doc.traefik.io/traefik/routing-configuration/http/tls/overview/))

I did a search on github and it seems there are people using empty attribute sets in their configurations (these results do not include { })

From my understanding the thing you most care about is end user usability. And you're right with that. Also I agree, that I'd like to work through the search.nixos.org options.

Yes. I would like the module to be powerful, readable, and maximally flexible - Traefik is used in some pretty wacky setups, and I would like the module to support as many of those as possible on the configuration layer - let the user tweak things, don't let what the module does get in the way.

Filtering {} out at generation time folds "on with defaults" into "unset". You try to handle this case with your install.settings note that is asking the users to write attr = true instead of attr = {}, but that's a module-specific conversion you have to discover in the docs / on search.nixos.org. IMO it's easy to miss and adds mental overhead versus translating an existing Traefik config as-is. The worst thing though, it is silent and thus hard to debug. You would only notice it if your config doesn't behave as expected or if you look at the effective config file.

I do agree it does hurt usability. I personally think this is just bad design on the part of Traefik (or perhaps a consequence of yaml), but I don't think it's likely to change.

Not a blocker, I just think it hurts usability and you could easily work around it by moving the option merging logic out of the options.services.traefik.install.file.default to a variable which is defined above each options and config.

The other approaches I've considered:

• Add warnings to detect uses of empty attribute sets for the known valid uses you mentioned. Manual, but it would get the job done without major complexity
• Modify the merge logic/type of the submodule
  • Set the freeformType such that it does not allow empty attribute sets/lists/nulls. If the freeform type doesn't apply to explicitly typed options, then exclusivity can be guaranteed
  • The cleanest solution I've been able to theorize is filtering empty attribute sets only if they have the mkOptionDefault priority. I'm not sure if this is actually possible or not, since you would need access to the valueMeta or similar
  • It might be possible to filter only the options = {} portion of the submodule, such as with apply. I haven't explored that very thoroughly
  • This might be relevant?

    nixpkgs/lib/types.nix

    Lines 231 to 235 in 99b0af3

    	# Whether this type has a value representing nothingness. If it does,
    	# this should be a value of the form { value = <the nothing value>; }
    	# If it doesn't, this should be {}
    	# This may be used when a value is required for `mkIf false`. This allows the extra laziness in e.g. `lazyAttrsOf`.
    	emptyValue ? { },

The thing I dislike about a variable is that it has to be either documented manually or the user needs to be able to read the source code. More info below.

...Or simply look at install.settings.providers.file.filename on https:/search.nixos.org, and see that the default value is something along the lines of json.generate { } routing.file

I don't know if this was possible with an earlier version of your PR, but now install.settings.providers.file.filename doesn't lead me to an option anymore, which would let me find the json.generate logic. Or are you referring to routing.file which gives me a hint about routing.settingsDrv which shows me the (pretty hard to read) settingsDrv logic?

I must have accidentally changed that. I'll add that back in on my next round of fixes.

All in all this comes down to two design styles: I model each file-provider scenario as its own typed option. routing.provider.file (Nix-generated), routing.provider.externalFile (a file you manage yourself), routing.provider.directory (a watched dir) and let the type pick exactly one. Yours does it with routing.file and routing.dir and an assertion. Both work, I just find the explicit split easier to read and discover. Curious what you think.

While it doesn't make as big of a difference for the file provider alone, my approach makes a little more sense when Traefik plugins are taken into account. Having that logic accessible in the option docs makes it much more extensible, and allows other modules like pangolin to do their thing without extra hacks and workarounds. This PR is more about laying the groundwork for that to be possible, since implementing it all at once made review/merging harder.

[TheRealGramdalf]

Added an assertion preventing empty attribute sets from being used in the config. This should notify users that there is an issue while allowing the filter logic to remain functional, since empty attribute sets are only exposed once the null value is removed.

This could still be improved to show offending locations, but I have implemented it as a simple check for now.