Download License + scp and ftp (also e.g. for startup-config)

[steiler]

This PR makes license files downloadable and embeddable.
It further also implements scp and ftp as download protocols for also als the existing downloadable content (e.g. startup-config).

[steiler]

implementing #1377

[hellt]

I propose using sftp rather than scp
example: https://github.com/openconfig/gnmic/blob/ee2c9de75216ce19198ca5bb8f61dcc91e5da88e/utils/file.go#L128

[FloSch62]

I tried the current PR from @steiler rebased with main. So far scp works, but with scp I get the warnings:

WARN[0000] error performing host key validation based on "/root/.ssh/known_hosts" for hostname "flosch1:22" (knownhosts: key is unknown). continuing anyways

Also not sure if we should have this in the config file and not only as env var: CLAB_SSH_KEY

I also tried sftp, but I do not get it to work. I remains in:
DEBU[0000] Fetching "ftp://user:pass@localhost:2222/license" for node "leaf1" storing at "/tmp/.clab/clos02-leaf1-license"

I can debug a bit more...

[asadarafat]

I tried the current PR from @steiler rebased with main. So far scp works, but with scp I get the warnings:

WARN[0000] error performing host key validation based on "/root/.ssh/known_hosts" for hostname "flosch1:22" (knownhosts: key is unknown). continuing anyways

Also not sure if we should have this in the config file and not only as env var: CLAB_SSH_KEY

I also tried sftp, but I do not get it to work. I remains in: DEBU[0000] Fetching "ftp://user:pass@localhost:2222/license" for node "leaf1" storing at "/tmp/.clab/clos02-leaf1-license"

I can debug a bit more...

could be HostKeyCallback setting this code

clientConfig := ssh.ClientConfig{
		User:            u.User.Username(),
		Auth:            []ssh.AuthMethod{},
		HostKeyCallback: getCustomHostKeyCallback(knownHostsPath),

HostKeyCallback: getCustomHostKeyCallback(knownHostsPath), suggest to use ssh.InsecureIgnoreHostKey() to allow any host.

[asadarafat]

also, would suggest to keep SCP as a fallback in SFTP service does not available in the router.

[steiler]

this is a custom Hostkey callback that is being used.
What happens is, that it informs you that it acted like InsecureIgnoreHostKey() in case no known key was found.
Thats the reason for the warning plus the "continuing anyways".

[mabra94]

Is there any timeline when we could expect this PR to be merged? Is there anything needed to be tested/checked to get this? For Nokia internal usage this would further simplify labbing!

[steiler]

Let me base it on the actual master branch to resolve merge conflicts then @hellt can probably get to it. ;-)

[steiler]

Do we want to run the tests for the downloads as seperate suite or also make it part of smoke tests?
I feel like it is wuite some heavily lifting for whats the checked in the end.

[codecov]

Codecov Report

Attention: 202 lines in your changes are missing coverage. Please review.

Comparison is base (404e6f9) 51.78% compared to head (93bf471) 51.19%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1414      +/-   ##
==========================================
- Coverage   51.78%   51.19%   -0.60%     
==========================================
  Files         148      148              
  Lines       14548    14777     +229     
==========================================
+ Hits         7534     7565      +31     
- Misses       6180     6374     +194     
- Partials      834      838       +4     

Files	Coverage Δ
clab/clab.go	76.85% <100.00%> (+0.17%)	⬆️
types/topo_paths.go	68.54% <0.00%> (-1.13%)	⬇️
clab/config.go	65.94% <66.66%> (-0.64%)	⬇️
utils/file.go	35.07% <8.17%> (-19.85%)	⬇️

... and 1 file with indirect coverage changes

[asadarafat]

suggest to use ssh.InsecureIgnoreHostKey() to allow any host.

[asadarafat]

I tried the current PR from @steiler rebased with main. So far scp works, but with scp I get the warnings:

WARN[0000] error performing host key validation based on "/root/.ssh/known_hosts" for hostname "flosch1:22" (knownhosts: key is unknown). continuing anyways

Also not sure if we should have this in the config file and not only as env var: CLAB_SSH_KEY

I also tried sftp, but I do not get it to work. I remains in: DEBU[0000] Fetching "ftp://user:pass@localhost:2222/license" for node "leaf1" storing at "/tmp/.clab/clos02-leaf1-license"

I can debug a bit more...

could be HostKeyCallback setting this code

clientConfig := ssh.ClientConfig{
		User:            u.User.Username(),
		Auth:            []ssh.AuthMethod{},
		HostKeyCallback: getCustomHostKeyCallback(knownHostsPath),

HostKeyCallback: getCustomHostKeyCallback(knownHostsPath), suggest to use ssh.InsecureIgnoreHostKey() to allow any host.