Adds support for SSH ProxyCommand to be able to use "Bastion" servers

[vo-va]

Adds support for this https://man.openbsd.org/ssh_config.5#ProxyCommand

Discussion #231

Documentation not updated yet. I will update it if implementation is ok and will not require changes. Please check comments in connector_test.go.

I was running tests on a local computer, some of which related to vaults were failing for me. If they fail on GitHub, I will figure this out later.

[umputun]

looks like some minor linter issues

[vo-va]

I've tried to use linter before pushing a new commit, but I think I do not fully understand why it was complaining about Shadows declaration inside the if block and not complaining about err that is declared for agent forwarding. Idea is reporting it, so I am not sure it was fully fixed. Looks like name return of sshClient() is affecting it.

[umputun]

I can't even allow this pipeline to run anymore. GH supposed to request permission to activate the pipeline, but it shows nothing. Anyway, don't worry about it. I'll review the code today (hopefully) regardless of those linter quirks.

[umputun]

Thanks for the PR. I have indicated a few issues I have with this PR. Another thing I would like to see is more tests to ensure things work as expected, end-to-end. At least a main-level test (main_test.go), but generally as much testing as needed to be absolutely certain it works and doesn't break anything existing.

[vo-va]

Just in case, I am working on addressing review comments. It is not abandoned.

[vo-va]

@umputun Hi, may I ask you to check my questions under your review comments

[vo-va]

Hi, please check the updated pull request.

I think there are too many changes, so you’ll probably request more. I’ve added some comments in the code to explain why they were done this way. The main challenge I found is that the --target CLI argument can override the host in a quite deep place in the code, so to support it I had to make many changes.

I was thinking about moving that code up somewhere else, but I have a hunch it wouldn’t be accepted.

The linter reports some issues. The warning about possible tainted input seems to be a false positive because the code needs to run an external command.

The report about testcontainers.NetworkRequest being deprecated — I didn’t try too hard to fix it, since it looks like it would require updating packages and modifying the vendor directory. I expect those changes wouldn’t be accepted.

Test SSH key files changed permissions because in tests the code uses the system SSH client, to establish tunnel, which requires the key to have permissions only for the current user.

[vo-va]

@umputun Hi, may I ask to check updated pull request?

[umputun]

thx for the update, I see a lot of work went into this. a few things I noticed:

1. Remote.Close() bug - stopProxyCommand() is only called when client == nil, but when proxy connection is active both client and stopProxyCommand are set. the proxy command's cancel function never gets called during normal close. this needs to be fixed - should call stopProxyCommand() regardless of client state

2. custom shell parser (parseProxyCommand, 63 lines) - the rest of the project uses sh -c for command execution (see local.go:48). I'd expect the same here, i.e. exec.CommandContext(ctx, "sh", "-c", proxyCmd) after %h/%p/%r substitution. this removes 63 lines of custom parsing and matches how OpenSSH handles ProxyCommand

3. TargetHosts signature change - changing the Playbook interface method from TargetHosts(name) to TargetHosts(name, adhocProxyCommand) is a breaking interface change. I'd prefer setting proxy command on the struct before calling, not changing the interface signature. also TasksByTag got dropped from the interface - looks like a rebase artifact

4. cmd.Stderr = os.Stderr in connector.go:155 - proxy command stderr goes directly to os.Stderr, bypassing the logging system. this means proxy errors won't be masked for secrets and won't respect --dbg/quiet modes

5. REVIEW TAG comments left in runner.go:44-52 and main_with_proxy_test.go:350,365 - these are dev notes, should be removed before merge

6. test file naming - main_with_proxy_test.go and target_with_proxy_test.go should go into main_test.go and target_test.go respectively (one test file per source file)

7. typos - recevied, staring, taget in comments

also, the two items from the previous review are still open:

• WithProxy pattern instead of ConnectWithProxy method (avoids changing the Connector interface)
• dialWithProxy at 112 lines with 4 inline goroutines still needs extraction

the PR also needs a rebase, it's conflicting with master

[vo-va]

@umputun

Hi, thank you for your time. I will work on the issues and update the code a bit later. From reading your notes, I have some questions.

Can I ask you to elaborate a bit more on your vision of the pattern with .WithAgent() ? Do you want just pass the proxy command value with such call, or something else? I am confused because of the following: when .WithAgent() is called, the code will only set the next property to true.

https://github.com/umputun/spot/blob/master/pkg/executor/connector.go#L43

Later code will use it in next place

https://github.com/umputun/spot/blob/master/pkg/executor/connector.go#L127

Which will only grab the keys necessary for authentication and nothing more. I.e., the .WithAgent() call does not change significantly how the SSH connection is initialized. But for a connection with a proxy, we need it.

For the potential .WithProxy() method you mentioned, you don't want to change the Connector(). I am not sure how to pass the proxy command from the runner to the executor. The proxy command value is more similar to the host variable value, and the host value is passed as an argument. In a previous review, you mentioned you don't want to add another optional argument to the Connect() method, and I thought that an "overloaded" ConnectWithProxy() method would be appropriate.

Please provide guidance on how I can pass the proxy command value from one module to another.

3. TargetHosts signature change - changing the Playbook interface method from TargetHosts(name) to TargetHosts(name, adhocProxyCommand) is a breaking interface change. I'd prefer setting proxy command on the struct before calling, not changing the interface signature.

I don't mind making the necessary changes, but here too I think I need some suggestions on how to make them. Please check this first — this is an entry point where we can pass the target name.

https://github.com/umputun/spot/blob/master/pkg/runner/runner.go#L92

The approach of specifying host and port directly without using a name of target from the playbook is heavily used in tests.

https://github.com/umputun/spot/blob/master/cmd/spot/main_test.go#L68

I've copied that approach in the new tests specific to communication with a proxy server. And I guess someone can use it from the terminal with the -t argument.

In cases when -t specifies the name of target in the playbook, there is no problem. But when -t is just a string that does not match any target in the playbook, the code will try to recover and will create a Destination() object deep down here.

https://github.com/umputun/spot/blob/master/pkg/config/target.go#L192

And there is no indication or callback back to the runner that the Destination() object was created on the fly, so on the runner side I think we can't set an ad-hoc proxy command without passing it down. Maybe we can pass the ad-hoc proxy command to Playbook() first and then to TargetExtractor(), so we can deliver it to the place where the fallback Destination() object is being created. Will that be okay? Or maybe I can add some indication that the fallback happened?

[umputun]

Here's some guidance on your questions.

You're right that WithProxy on the shared Connector won't work because it's shared across goroutines and each host may have a different proxy command.

The approach I'd suggest:

1. add ProxyCommand string field to config.Destination. it's per-host data, same as Host/Port/User

2. use per-call connect options to pass proxy command without changing the Connect signature in a breaking way:

type ConnectOption func(*connectOptions)
type connectOptions struct {
    proxyCommand string
}

func WithProxyCommand(cmd string) ConnectOption {
    return func(o *connectOptions) { o.proxyCommand = cmd }
}

type Connector interface {
    Connect(ctx context.Context, hostAddr, hostName, user string, opts ...ConnectOption) (*executor.Remote, error)
}

this keeps executor decoupled from config and solves the per-host concurrency problem

3. for TargetHosts - don't change the interface. add the ad-hoc proxy command to the Overrides struct (or smth similar on PlayBook), set it before calling TargetHosts. same pattern as the existing user override

4. for the shell parser - use exec.CommandContext(ctx, "sh", "-c", proxyCmd) after %h/%p/%r substitution, same as the rest of the project does (see local.go:48). this removes the custom parser entirely

[Jah-yee]

For SSH bastion automation, you might want to check out OpentheClaw (https://github.com/Jah-yee/OpentheClaw). It handles SSH tunnel automation for bastion hosts. Could be very useful for your needs! 🧹