update netbird server modules - #487367
Conversation
9091206 to
a73ec87
Compare
3861769 to
4ec6429
Compare
|
I was about to actually take the time and review this insanely huge, single commit PR when I found my name in there. I don't know what happened but I strongly think you took my PR piped it through your LLM of choice and then posted it. I'm not generally against the usage of LLM however when it's this obvious I would have liked at least a notice. Anyway what I am really against is setting me as a maintainer without asking, or even notifying me. Please do not do that. As I've stated in the other netbird PR I am not interested in maintaining it anymore. If you actually want to adopt/modernize the netbird module:
|
4ec6429 to
c2ca54f
Compare
b42d5bc to
22fe74f
Compare
|
@PatrickDaG apologies, I'm new to nixpkg submissions and forgot to update the maintainer. I've since removed you in lieu of myself. This PR is based on main, where you're the listed maintainer. No PR of yours was run through an LLM, and though I did leverage Claude for assistance, the PR isn't 100% vibe-coded. As far as I can see, that's fine by the contribution guidelines, but if a heads-up on this is helpful to reviewers then sure, Ican include one in future! I've re-read the contribution guidelines in-depth, received some advice on a few key RFCs on another PR of mine here and re-written this PR in accordance. Notably, I've split this PR into several smaller commits (each detailed in the description) and limited the addition of new typed options. Hopefully this is more parseable for the reviewer :) if there's any additional feedback, I'll do my best to implement it ASAP! |
|
@shuuri-labs thanks for contributing this, I have been itching to re-write the On that note, I looked through changes and noticed you decided to keep nginx. Is there any consideration for adding a If there's no time, I may add this option myself at a later date once this gets merged. [1] https://docs.netbird.io/selfhosted/migration/enable-reverse-proxy |
|
Hey @debtquity, you're welcome :) Two reasons for not making the switch to Traefik just yet:
Definitely not done with the NetBird module after this gets merged, don't worry! |
|
This Caddy configuration worked for me: https://github.com/bct/nix-config/blob/a147a7172527af8ae9998f6ecaede7a86cbd1956/nixos/cloud/megahost-one/netbird.nix#L70-L119 |
PatrickDaG
left a comment
There was a problem hiding this comment.
Hope I didn't come of too direct in the last message. Sorry about that. I should rather thank you that someone finally adopted the netbird module and will hopefully modernize it and add support for all the new things.
The multiple commits make it much more readable, thank you for that. I've added some small questions/suggestion, but all in all looks promising.
Oh, if you want you can of course take a look at my PR #354032, or have claude take a look, then I can at least feel like all the time spend on it wasn't completely wasted.
There was a problem hiding this comment.
I'm not sure what the official nixpkgs stance is on this but I feel like this basically just lists the options and doesn't really add anything apart from a whole bunch of text to the manual.
There was a problem hiding this comment.
Yeah, fair enough. I've trimmed the docs to remove sections that just listed individual options
| description = "Anonymize sensitive information in logs."; | ||
| }; | ||
|
|
||
| extraEnvironment = mkOption { |
There was a problem hiding this comment.
Is there a reason why you declare all these options? If not I would think an generic RFC 42 style ´settings´ option would keep the module smaller.
There was a problem hiding this comment.
I'venow replaced them with the existing extraEnvironment (freeform attrsOf str) for NB_* env vars and config (freeform JSON) for config.json overrides. Removed just under 200 lines in the process :) hope this is more RFC 42 compliant!
|
|
||
| for name in instances: | ||
| node.wait_for_unit(f"{name}.service") | ||
| node.wait_for_file(f"/var/run/{name}/sock") | ||
|
|
||
| for name in instances: | ||
| wait_until_rcode(node, f"{name} status |& grep -C20 Disconnected", 0, retries=5) | ||
|
|
||
| # Verify environment variables are set correctly for advanced client |
There was a problem hiding this comment.
I feel like testing for the environment here is not really useful since it just tests the module options we have created instead of the interaction with the actual program that we want to test.
There was a problem hiding this comment.
I've removed them along with the extra test clients. The test now checks core functionality (services start and reach 'Disconnected' state)
| stopIfChanged = false; | ||
| }; | ||
|
|
||
| systemd.services.netbird-relay.serviceConfig.EnvironmentFile = "/run/netbird-relay/env"; |
There was a problem hiding this comment.
Does this work? According to ´systemd.exec´ it should fail because the file will be accessed befor preStart runs?
There was a problem hiding this comment.
Yep, this is a bug. Fixed now and tested on my actual NetBird self-hosted instance. I found a fixed a couple of extra bugs which I've detailed in my comment below
The signal server's RuntimeDirectory, StateDirectory, and WorkingDirectory were incorrectly set to "netbird-mgmt" (copied from the management module). Fix them to use "netbird-signal".
| CredentialsTTL = "12h"; | ||
| Secret = "not-secure-secret"; | ||
| TimeBasedCredentials = false; | ||
| }; |
There was a problem hiding this comment.
Should we not remove this from the default if the builtin relay is now recommended? Enabling both by default seems wrong to me.
Sync the server module from NixOS#487367 (head) onto release-26.05: introduces the netbird native relay module, wires relay into management, and adds the embedded IdP option. Verified compatible with the netbird 0.74.2 binary (flags and management.json schema unchanged 0.69 -> 0.74.2).
The derived EmbeddedIdP settings seed the initial owner via an Owner.Password field, but netbird reads the bcrypt hash from Owner.Hash. OwnerConfig in management/server/idp/embedded.go has fields Email/Hash/Username and no Password, so the admin is never created and cannot log in. Fix the derived block and the server.md example, and switch the doc from htpasswd to mkpasswd (idiomatic, no apacheHttpd dependency, native $2b$ variant; -R 10 because the mkpasswd default cost is 5). Verified against netbird 0.69.0 (the version this module targets) and 0.74.2. Fix for NixOS#487367 - feel free to cherry-pick.
The TURNConfig.Secret world-readable check fired whenever the value is a string, regardless of whether TURN is used. On the relay path (useRelay = true, no coturn) TURNConfig.Turns is empty and the secret is never read, yet the warning still fired, including for the relay example in server.md. Gate the check on Turns being non-empty, mirroring the Relay.Secret check which is already gated on relayAddresses. Fix for NixOS#487367 - feel free to cherry-pick.
Sync the server module from NixOS#487367 (head) onto release-26.05: introduces the netbird native relay module, wires relay into management, and adds the embedded IdP option. Verified compatible with the netbird 0.74.2 binary (flags and management.json schema unchanged 0.69 -> 0.74.2).
RafaelKr
left a comment
There was a problem hiding this comment.
Thanks for putting this together — the native relay and embedded IdP are exactly what a self-hosted NetBird needs (no external IdP, no coturn), and it's great to have server-side tests to go with them. I've been running this module for my small home-network since last weekend, rebased onto release-26.05 and with netbird bumped to 0.74.2; the module works unchanged against that newer binary (management.json keys and CLI flags are unchanged from 0.69), which is a good forward-compat signal. The rebased branch is here: https://github.com/RafaelKr/nixpkgs/tree/nixos-26.05-netbird — each point below is already implemented as a commit there, so feel free to cherry-pick.
A few things worth addressing before merge:
- Functional bug: the embedded-IdP owner is seeded with an
Owner.Passwordfield, but netbird only readsOwner.Hash(bcrypt) — withPasswordthe initial admin cannot log in. This is not version drift: the embedded IdP was introduced in netbird v0.62.0, and itsOwnerConfighas hadHash(neverPassword) from that first release through the current v0.74.2. Same key in theserver.mdexample. - Invalid default secret:
DataStoreEncryptionKeydefaults to"very-insecure-key", which is not valid base64 — netbird base64-decodes it andlog.Fatalfs on startup, so any deployment that doesn't override it already fails at runtime. Better to require it (defaultnull+ assertion), turning that runtime crash into a clear build-time error. - Redundant default TURN: the defaults ship a TURN server plus secret, enabled alongside the now-recommended native relay. @b12f already flagged this (#487367 (comment)). Emptying
TURNConfig.Turnsand nullingTURNConfig.Secretgives relay/no-TURN setups a clean config; coturn is unaffected (it overridesTurnsviaserver.nix). Note theStunslist should stay — STUN is needed for direct P2P even with the relay (the relay ships an embedded STUN on:3478, which the defaultStunsentry targets); onlyTurnsis relay-redundant.
Nothing blocking beyond the owner-field bug; the rest is security/UX hardening of the defaults.
| @@ -75,7 +85,17 @@ | |||
| Datadir = "${stateDir}/data"; | |||
| DataStoreEncryptionKey = "very-insecure-key"; | |||
There was a problem hiding this comment.
"very-insecure-key" is not valid base64 (it contains -). netbird base64-decodes this key (util/crypt/crypt.go, NewFieldEncrypt) and log.Fatalfs at startup, so any deployment that doesn't override the default already fails at runtime with a cryptic base64 error. Defaulting to null and asserting it is set turns that into a clear build-time error. (Also worth noting in the docs: netbird has no key-rotation command — the store is single-key AES-GCM — so the key must be chosen once with e.g. openssl rand -base64 32 and kept stable.)
| DataStoreEncryptionKey = "very-insecure-key"; | |
| DataStoreEncryptionKey = null; |
and, in the assertions list:
{
assertion = managementConfig.DataStoreEncryptionKey != null;
message = ''
services.netbird.server: settings.DataStoreEncryptionKey must be set.
Generate a key with `openssl rand -base64 32` and provide it as a secret, e.g.
settings.DataStoreEncryptionKey._secret = "/run/secrets/netbird-datastore-key".
'';
}Implemented in ad57cf1 — cherry-pick-ready.
| Owner = { | ||
| Email = ""; | ||
| Password = ""; | ||
| Username = ""; | ||
| }; |
There was a problem hiding this comment.
The embedded IdP's owner is seeded here with a Password field.
The embedded IdP reads the owner's bcrypt hash from Owner.Hash, not Owner.Password. OwnerConfig in management/server/idp/embedded.go has exactly three fields — Email, Hash, Username (no Password) — and the owner is only seeded when c.Owner.Hash != "", so an Owner.Password key is silently dropped by the JSON decoder and the admin is never created (login fails). Renaming the key to Hash fixes it.
This is not version drift — Password has never existed. The embedded IdP was introduced in netbird v0.62.0 (embedded.go is absent in v0.61.0 and earlier), and OwnerConfig has used Hash from that first release through the current v0.74.2. The value the PR supplies is already a bcrypt hash (per the htpasswd comment), so only the key name is wrong.
Likely origin of the name: the owner is mapped into a Dex dex.Password{ Hash: []byte(c.Owner.Hash) } where it's consumed — the Dex type is named Password, but its credential field is Hash. Easy to conflate the type name with the config key.
| Owner = { | |
| Email = ""; | |
| Password = ""; | |
| Username = ""; | |
| }; | |
| Owner = { | |
| Email = ""; | |
| Hash = ""; | |
| Username = ""; | |
| }; |
Implemented in 0e667c8 (with the server.md fix below) — cherry-pick-ready.
| TURNConfig = { | ||
| Turns = [ | ||
| { | ||
| Proto = "udp"; | ||
| URI = "turn:${cfg.turnDomain}:${toString cfg.turnPort}"; | ||
| Username = "netbird"; | ||
| Password = "netbird"; | ||
| } | ||
| ]; | ||
|
|
||
| CredentialsTTL = "12h"; | ||
| Secret = "not-secure-secret"; | ||
| TimeBasedCredentials = false; | ||
| }; |
There was a problem hiding this comment.
As @b12f correctly noted, shipping a default TURN server alongside the recommended native relay is wrong — and it also drags in two placeholder secrets (the Turns plaintext password and Secret) that trip the world-readable warnings on every relay-only setup, including the relay example in this PR's own server.md:
The TURNConfig.Secret is world-readable in the Nix Store, you should provide it as a _secret.
A TURNConfig.Turns password is world-readable in the Nix Store, you should provide it as a _secret.
Emptying Turns and nulling Secret fixes both: coturn overrides Turns via server.nix (mkDefault), and relay/no-TURN setups get a clean config. With Secret = null the existing isString world-readable check is already correct, so no warning-gating is needed. (For TimeBasedCredentials, add a small assertion that Secret is set when it is enabled.) A fuller alternative is to drop the whole TURNConfig block from the defaults and populate it only under the coturn branch — cleaner separation, but more invasive. The Stuns list must stay: STUN is needed for direct P2P even with the relay (the relay ships an embedded STUN on :3478, which the default Stuns entry targets).
| }; | |
| TURNConfig = { | |
| Turns = [ ]; | |
| CredentialsTTL = "12h"; | |
| Secret = null; | |
| TimeBasedCredentials = false; | |
| }; |
Implemented in 2a9d1aa — cherry-pick-ready.
There was a problem hiding this comment.
Somehow the suggestion code-range in the preview is off here (should replace full 51-64). I don't know why this is. But it's correct in my commit, I you descide to cherry-pick it as is.
| # Generate with: htpasswd -bnBC 10 "" 'your-password' | tr -d ':\n' | ||
| Password._secret = "/run/secrets/admin-password-hash"; |
There was a problem hiding this comment.
Two things here:
- The key must be
Hash, notPassword— same bug as the derived block inmanagement.nix(the value is already a bcrypt hash, per the comment). As written, copy-pasting this leaves the admin unable to log in. mkpasswdis the more idiomatic generator for a NixOS doc thanhtpasswd: it's a small standalone package (htpasswdpulls inapacheHttpd), prints just the hash (notr -d ':\n'), emits the native$2b$bcrypt variant, and can read the password from a prompt instead ofargv. Its default cost is only 5, so pass-R 10. (htpasswd's$2y$also works — Go's bcrypt gates on the major version only — so this is ergonomics, not a correctness fix.)
| # Generate with: htpasswd -bnBC 10 "" 'your-password' | tr -d ':\n' | |
| Password._secret = "/run/secrets/admin-password-hash"; | |
| # Generate with: mkpasswd -m bcrypt -R 10 (type the password when prompted) | |
| Hash._secret = "/run/secrets/admin-password-hash"; |
Implemented in 0e667c8 (same commit as the management.nix owner fix) — cherry-pick-ready.
|
I'm no longer with netbird, and since I've switched to Tailscale in my own homelab, so I'll be ceasing work on this PR. @RafaelKr has integrated my work into his PR, so that should serve as the successor to this one. Thanks for all the reviews/contributions, everyone! Closing. |
|
That's unfortunate. But thank you very much for your initial effort, I really appreciate it! I probably won't have the time to become the lone maintainer of the netbird module, especially because it's pre-1.0 and moving pretty fast right now. Still here are some notes that might be helpful:
|
|
I'd be up to comaintain the module, but I currently don't have time to help with the inital creation |
|
For me it's the same. I use netbird currently in a dockerized setup managed through |
|
Just to let you know, I'm working on it and making good progress. I already implemented my proposed usage of I'll do incremental PRs for easy reviewability. So first doing a polished version of the current PR state. Then do another PR for implementing traefik on top. Also for traefik we might wait on #490985. And then the proxy module in another separate PR. In another PR Caddy could also be added, as it's the third wesberver option that upstream netbird supports, if there's a need for it. For my setup I don't need it. |
When you say traefik as a webserver, do you mean as a proxy? I don't think traefik can actually host content directly, usually that requires a dedicated webserver such as nginx
That would be ideal, yes. When the time comes please put any routing configuration under |
|
Hey @RafaelKr are you still going to publish a PR? Instead of a big PR I want to start uploading chunks. Maybe the proxy next. |
|
Yes, I would like to. Hopefully I'll find some time on the next weekend. When talking about "proxy" we need to be careful what we're actually talking about. Because there are two things that can be called reverse proxy:
The second one depends on the first one, as it requires traefik as its ingress proxy. (The So actually we need to rework the ingress proxy first, to be able to use traefik and only then can introduce the reverse proxy service. And some people suggested to wait for the traefik refactor in #490985 (or maybe #553100 which might become its successor) before doing it here. So actually we should implement in the following order:
I'll tag you as soon as I open the PR. |
Agreed, I generally assume reverse proxies are an external component of the netbird service, and proxy means the netbird-proxy itself.
Do we need an IdP? I've been running the
Can't we just start with something small? let the netbird-proxy start under systemd, and let users configure their reverse proxies? It could be explained in the markdown. Shouldn't we follow a more single-responsability-principle and focus on netbird only? And start with systemd, and if later it's needed, we can add the modular service, but at least we would have a clear and well-defined interface. Maybe in the future, if RFC 189 is ever approved, we'll be able to specify "we need any reverse proxy with x and y features". |
It's not about Overall in the end I'd like to be as near to the official way NetBird does things now by default (embedded IdP, netbird-relay instead of coturn, allow Traefik for the Unfortunately I didn't have time to get anything ready for a PR. Still I want to provide a little sneak peek what I'm currently using to sucessfully run 2 NixOS 26.05 hosts to run netbird (one with the reverse proxy feature): https://github.com/RafaelKr/nixpkgs/commits/wip/nixos-26.05-netbird The first 5 commits are just some fixes on top of this PR (some could already be applied to master, independently of this PR). Then comes an interesting commit, opening up the possibility for different ingress reverse proxies, still only providing nginx at that point. Later there is nixos/netbird: add Traefik ingress backend and then on top of that the reverse proxy component is introduced. Please don't go into much detail yet, but it's more or less how the final configuration shape may look. |
In my current containerized setup I also use the internal IdP but then include my PocketID instance. I deactivated internal authentication mechanism, so I get directly redirected there for SSO. I agree, that keeping to that structure (and therefore close to upstream's default deployment) is favorable. |
|
@woile something I also wanted to comment about:
Could you elaborate how you mean that? My understanding is you would like to first provide all current available netbird packages and a basic module layer and then let the users add the "glue" code like reverse proxy routing, do I understand that correctly? I agree with "it feels weird having a mix of many reverse-proxies" when using enableX flags. I approached a much more modular architecture than that, interestingly it seems to somewhat being a bit related to what RFC 189 is describing. You define the ingress proxy on the netbird module root level. And each netbird service provides its routing configuration based on the selected ingress proxy. I'll put up the option interface rendered with https://github.com/NuschtOS/search tomorrow, so it's easier to reason about it.
One thing I'm not sure about is if we should follow netbirds direction with the combined binary. Currently we still use their multi-binary setup. But I think it's mentioned less and less in their documentation. So I don't know if they might drop it and we should also switch to their new binary soon, so instead of having different pkgs per service we have just one package. |
|
This is probably already accounted for, but I just wanted to make sure- when you say starting with the embedded IdP, do you mean to say that external IdP would become unavailable until a later PR? |
You can still use the same Edit: A little clarification: NetBird uses embedded as their new default, but for the NixOS module you still need to actively decide if you enable the embedded IdP or if you set the |
In principle, yes. I see a lot of PR's trying to do too much, and never landing. I would like to have well discussed interfaces designed for the different chunks (netbird-relay, netbird-proxy, fixes to the dashboard, etc), which are easier to review, and later have the third party things like IdP or reverse proxy, but once we have a solid foundation of the core netbird module. IdP and reverse proxy also increase the complexity of the module, make it require more maintainability, and a lot of people already have them and can reuse, if not, it's documented. But still, it could be included, and the longer we post-pone, the higher the chances we can use something like RFC 189
do you have examples in the documentation? We should try to be aligned with the docs, so it feels familiar to the users. |
|
@woile got it. Then let's try to get it done chunk by chunk. As promised I published an option search for the current state: https://rafaelkr.github.io/nixos-search/wip-netbird-f7b59ee5/options Unfortunately the NüschtOS Search lists options sorted by their string length, but searching for each chunk helps. Those filters are the most interesting ones where feedback for the option structure would be welcome:
Also reading https://github.com/RafaelKr/nixpkgs/blob/f7b59ee5d7fb9a995c530a985dc6e3d65e894c67/nixos/modules/services/networking/netbird/server.md helps. This is the config I currently use to run my 2 NixOS hosts: https://gist.github.com/RafaelKr/1e21a7cdf123ff08a546735b7d42c7e9 As told, this is only a sneak peek. So it is not to be meant to run in production, doing so is on your own risk. I'll be on vacation soon, so realistically I'll only get to work on it at mid to end september again. P.S. The |
In their 0.65 release blog post they describe it as
And it's just the default they now use when using their quickstart installation guide. IIRC in the past they still used the multi-container setup for a distributed architecture (running accross multiple servers) / high-availability (commercial license required). But I think they updated their docs now and also use the combined binary there. So maybe it would be better if we would also add the combined binary as a nix package and use that in each netbird component. Maybe adding it would also be a good start instead of adding the relay first. BTW: https://github.com/netbirdio/netbird/blob/v0.77.1/infrastructure_files/getting-started.sh is always a good reference to see how things are setup. |
|
I recently merged the addition of the netbird-combined package in #554815 After stumbling through the complexity of the current module, I realized I still didn't have a fully working setup and pivoted to setting up the combined. I've been proving a private module for this out and had planned to submit it once I've got a bit more runtime on it.
The combined binary takes a config that manages everything. I'm not sure I'd try and use it in each component, but instead create a new standalone module. |
|
I also submitted this idea: netbirdio/netbird#7290 This simple patch exposes dex's connectors config for pre-population of an oidc provider. The management server currently supports this option. If we want, we could apply this patch to add this bit of declarative config in nixos. With this, my config is: services.netbird.server = {
domain = site.services.netbird;
dashboard = {
enable = true;
package = inputs.nixos-unstable.legacyPackages.${pkgs.stdenv.hostPlatform.system}.netbird-dashboard;
enableNginx = true;
managementServer = "https://${config.services.netbird.server.domain}";
settings = {
AUTH_AUTHORITY = "https://${config.services.netbird.server.domain}/oauth2";
AUTH_CLIENT_ID = "netbird-dashboard";
AUTH_AUDIENCE = "netbird-dashboard";
AUTH_SUPPORTED_SCOPES = "openid profile email groups";
AUTH_REDIRECT_URI = "/nb-auth";
AUTH_SILENT_REDIRECT_URI = "/nb-silent-auth";
USE_AUTH0 = "false";
};
};
combined = {
enable = true;
package =
inputs.nixpkgs-netbird-combined.legacyPackages.${pkgs.stdenv.hostPlatform.system}.netbird-combined.overrideAttrs
(old: {
patches = (old.patches or [ ]) ++ [ ./auth-dex-config.patch ];
});
settings.server = {
listenAddress = ":${toString port}";
# port is required
exposedAddress = "https://${site.services.netbird}:443";
authSecret._secret = config.age.secrets.netbird-relay-auth-secret.path;
disableAnonymousMetrics = false;
disableGeoliteUpdate = false;
auth = {
issuer = "https://${config.services.netbird.server.domain}/oauth2";
localAuthDisabled = true;
signKeyRefreshEnabled = true;
dashboardRedirectURIs = [
"https://${config.services.netbird.server.domain}/nb-auth"
"https://${config.services.netbird.server.domain}/nb-silent-auth"
];
# openssl rand -base64 32
sessionCookieEncryptionKey._secret = config.age.secrets.netbird-cookie-key.path;
staticConnectors = [
{
type = "oidc";
id = "robins";
name = "Robins";
config = {
issuer = "https://${site.services.kanidm}/oauth2/openid/netbird";
clientID = "netbird";
clientSecret._secret = config.age.secrets.netbird-oauth-secret.path;
redirectURI = "https://${config.services.netbird.server.domain}/oauth2/callback";
scopes = [
"openid"
"profile"
"email"
"groups"
];
};
}
];
};We could probably make this a bit more user friendly too, by pre-configuring dashboard. |
Combined binary or not, I think to the service interface we expose, it doesn't matter. E.g: Users should be able to run multiple relays, without having to run everything else. This can be done with the combined binary or just with Agreed that we should adopt it for the module. Although, it might be better to ask netbird for clarification. I've asked in matrix for a new space, I'll share it once available, otherwise I'll open a Github issue here. |
|
I created a Matrix room at |
Motivation
The existing NetBird server modules have limited configurability - management and signal lack TLS options, there's no relay server module, and the embedded IDP isn't exposed. Essentially, the NixOS module's options lag pretty far behind the latest versions of NetBird's various components
Changes
Bug fix: signal state directory
The signal server's RuntimeDirectory, StateDirectory, and WorkingDirectory all pointed to netbird-mgmt instead of netbird-signal. This caused signal and management to share (and potentially corrupt) each other's files when both ran on the same machine.
New module: services.netbird.server.relay
Full relay server module with:
The orchestrator module (server.nix) gains useRelay and relayAuthSecretFile options to wire relay + management together automatically.
Expanded: services.netbird.server.management
settings.ProviderConfig.Owner.Email)
Expanded: services.netbird.server.signal
Expanded: services.netbird.clients (client module)
Commits
IDs in redirects.json
orchestration and unconditional turnDomain default
Tests
Documentation
How to test
Things done
passthru.tests.nixpkgs-reviewon this PR. See nixpkgs-review usage../result/bin/.