nixos/traefik: refactor, add plugins option, add documentation

[SigmaSquadron]

wip-traefik-plugins -> master

Closes #436807

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes (PENDING BRANCH-OFF)
  • Package update: when the change is major or breaking.
• NixOS Release Notes (PENDING BRANCH-OFF)
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

---

Add a 👍 reaction to pull requests you find important.

[SigmaSquadron]

[TheRealGramdalf]

This note holds true for the static settings, but this one should be is mutually exclusive, not has precedence over.

[TheRealGramdalf]

It doesn't look like anything official has happened yet, but schemastore.org seems to have a schema for the static and dynamic file provider: https://www.schemastore.org/traefik-v3.json https://www.schemastore.org/traefik-v3-file-provider.json

I don't know that I would rely on that since it's out of tree (and thus not possible to change based on cfg.package), but since at the very least a base is there it might be worth trying to get traefik to adopt the schema and keep it updated as options change.

[SigmaSquadron]

@TheRealGramdalf @jackrosenberg:

I have an idea below for a redesign of the dynamic configuration loader. If we can define multiple files, then it'd be a good idea to just have extraFiles and nothing else.

TODO

• Example for in-line plugin in the option's example attr.
• Reference the docs in the plugin option's description.
• Change IP address in example.
• Expose the finalStaticFile builder in an internal option.
• Ensure that only one of static.file or static.settings is set.
• Verify the Traefik schema like what the udev module does.
• Re-think the stateful data directory and how it interacts with the traefik user.
  • Maybe we can use ReadWritePaths or something to ensure traefik.service always has access to it, even if the user is different.

Dynamic config refactor

• Make extraFiles be the only way to influence Traefik's dynamic config.
• Turn extraFiles into the dynamic.files.<filename>.{file,settings} interface and always have the dynamic config folder. This will look something like this:

  {
    services.traefik.dynamic = {
      watch = false; # This would be true by default, since we're specifying a real directory.
      directory = "/var/lib/traefik/dynamic";
      files = {
        "10-pangolin-routes".settings = { ... };
        "11-my-custom-routes".file = "/run/user/1000/traefik-routes";
      };
    };
  }

• Only one of dynamic.files.<filename>.settings or dynamic.files.<filename>.file must be set per filename.
  • Question: would it be better to have an assertion or a simple either option type?
• dynamic.watch is false if the folder is in the Nix store.
• dynamic.directory is null by default, signalling that it's a store path, but if set by the user, then:
  • Copy files to the user-specified mutable folder, otherwise symlinkJoin.
    • How do we copy/link files? Should we use a separate service? Tmpfiles?
• Import deprecated options.

[jackrosenberg]

I will have time for this in the weekend, will take a look then!

[TheRealGramdalf]

Again I'll have to follow up on this when I'm at my laptop, but I'd really prefer not to force extraFiles. dynamic.dir and dynamic.file are basically just mapped to options in the static config, so it doesn't make sense to force users to a single method. There's already emphasis on extraFiles` as the recommended method, but users shouldn't be forced to change their methods because of it. Some people might want to keep using a config file from a dockerized installation, or a directory full of them - it's supported upstream, so why arbitrarily limit that?

Couple other things for the todo list:

• we need to ensure that dynamic.file is accessible, I believe only dynamic.diris added to readonlypaths at the moment.
• Writing some nixos tests to verify that configurations are loaded correctly might be a good idea. They may not be as useful on an ongoing basis (the file provider is unlikely to change significantly), but it may be helpful to have the test framework in place in case bug reports start coming in once this gets merged.
  • I'm thinking just a simple proxy and then query the API to ensure the config is loaded
• Again regarding the different config methods, it might be helpful to create a compatibility matrix so we're all on the same page about what assertions to make and edge cases we need to catch regarding dynamic, static, and plugins

Other random thought:

• dynamic.settings should probably be set to a path under /run in the static config so it can be reloaded similar to extraFiles, and just symlink the store path there

[SigmaSquadron]

Some people might want to keep using a config file from a dockerized installation, or a directory full of them - it's supported upstream, so why arbitrarily limit that?

No, no. We're not limiting anything. I'm saying that if we have an option that can accept an infinite number of files and settings that will be transpiled into a config file by Nix, we don't need separate options to configure a random separate configuration file that is not as extensible.

If a user has a single non-declarative file they want to add to the config, they simply use dynamic.files."10-my-config".file = "/path/to/file";. If they have a whole directory full of them, all they need to do is set dynamic.directory = "/path/to/dir";.

This has the benefit of working with mixed setups, so you can set the directory to a directory full of config files, add a new dynamic.files."11-extra-file".file from elsewhere that will be linked to your directory, and use the dynamic.files."12-nix-config".settings to add more settings using Nix while also keeping everything that's not configured declaratively.

Writing some nixos tests to verify that configurations are loaded correctly might be a good idea. They may not be as useful on an ongoing basis (the file provider is unlikely to change significantly), but it may be helpful to have the test framework in place in case bug reports start coming in once this gets merged.

• I'm thinking just a simple proxy and then query the API to ensure the config is loaded

I would welcome contributions on this front. NixOS tests are not my strong suit.

Again regarding the different config methods, it might be helpful to create a compatibility matrix so we're all on the same page about what assertions to make and edge cases we need to catch regarding dynamic, static, and plugins

The only thing to consider is that if you're using static.file, all other options (dynamic.*, plugins and static.settings) are disabled. Everything else works together.

dynamic.settings should probably be set to a path under /run in the static config so it can be reloaded similar to extraFiles, and just symlink the store path there

I'm not sure what you mean by that. dynamic.files."name".settings is an attribute set that generates a file using the JSON builder. I haven't figured out how we're going to merge random non-declarative files with declaratively-generated files in an user-specified directory, but symlinkJoin is probably the easiest way if there's only settings and no directory, and systemd tmpfiles if random files or custom directories are involved.

[adamcstephens]

Could this be split into multiple, smaller PRs? For example the plugin capability should be mostly standalone from what I can tell.

[adamcstephens]

I'm not sure if this changed or each plugin may be different, but this postFetch failed for me as the zip file did not have the expected structure. I was using https://plugins.traefik.io/plugins/66b63d12d29fd1c421b503f5/oidc-authentication

This fixed it:

Suggested change

	export tmpdir=$(mktemp -d)
	mv $out/${moduleName}@v${version}/* $out/${moduleName}@v${version}/.* $tmpdir
	rm -rf $out/${provider}
	mkdir -p "$out/src/${moduleName}"
	mv -t "$out/src/${moduleName}" $tmpdir/* $tmpdir/.*
	mkdir -p "$out/src/${provider}/${owner}"
	mv $out/${owner}-${plugin}-* "$out/src/${moduleName}"

[SigmaSquadron]

I've split the refactor and other non-plugin commits into #482512