Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement new build flag --docker-host #988

Merged
merged 2 commits into from
Feb 19, 2021

Conversation

matejvasek
Copy link
Contributor

@matejvasek matejvasek commented Dec 14, 2020

Summary

Introducing new build flag: --docker-host.
The purpose of the flag is to expose custom docker deamon to build
container. This is useful when running docker daemon on non standard
location e.g. "tcp://localhost:1234", or if resulting image should be
stored in different daemon than you are running locally.

@matejvasek
Copy link
Contributor Author

Using TCP via --net=host and envvar might be advantageous when using on system with SELinux since SELinux doesn't like mounting things into container without proper policies set.

@matejvasek
Copy link
Contributor Author

This is also of interest for #966.

@matejvasek
Copy link
Contributor Author

Docker doc also mentions fd:// protocol but I never used it.

@codecov
Copy link

codecov bot commented Dec 15, 2020

Codecov Report

Merging #988 (fcd3103) into main (650ce05) will increase coverage by 0.03%.
The diff coverage is 97.62%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #988      +/-   ##
==========================================
+ Coverage   80.35%   80.38%   +0.03%     
==========================================
  Files         130      130              
  Lines        7964     7976      +12     
==========================================
+ Hits         6399     6411      +12     
  Misses       1147     1147              
  Partials      418      418              
Flag Coverage Δ
os_linux 79.70% <97.06%> (+0.02%) ⬆️
os_macos 77.15% <97.06%> (+0.04%) ⬆️
os_windows 80.30% <97.62%> (+0.03%) ⬆️
unit 79.75% <97.06%> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@jromero
Copy link
Member

jromero commented Dec 18, 2020

Hi @matejvasek, thanks for this PR. Just wanted to give you an update and heads up. Given the holiday's it'll probably take a little bit longer to get through the process. I'm setting up an environment to test this. I have a few comments I'll add to the PR but it's great to see this feature finally coming in.

Adding some context for others, pack does support DOCKER_HOST for managing the containers it uses to execute the lifecycle (aka build process). The problem this PR is attempting to address is the fact that we currently blindly try to mount the docker socket when --publish is false. This ultimately fails when the lifecycle tries to use the daemon since it doesn't have the DOCKER_HOST information. In other words, this PR provides the DOCKER_HOST information to the lifecycle.

@jromero
Copy link
Member

jromero commented Dec 19, 2020

Update: So I ran into a case where this conflicts with the use of Docker Desktop. Currently the mount /var/run/docker.sock:/var/run/docker.sock works when the remote docker daemon is "Docker Desktop" because it mounts the internal socket from the VM of the daemon host (not the client).

The proposed solution in this PR currently adds complexity that isn't quite a drop-in replacement. For example, if you use a tcp://docker:1234 remote daemon and docker hostname is local to the client via /etc/hosts the lifecycle container wouldn't be able to resolve the hostname.

I do wonder if this solution could be optionally applied via detecting the host daemon process (docker desktop vs podman) or (less preferable) by using an optional flag/configuration element. Overall, I think it's going in the right direction but we'll want to make sure it's not disruptive out of the gate.

@dfreilich dfreilich added this to the 0.16.0 milestone Dec 22, 2020
@dfreilich dfreilich added the type/enhancement Issue that requests a new feature or improvement. label Jan 4, 2021
Copy link
Member

@dfreilich dfreilich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the PR! I have a few reservations about the PR as is:

  1. Tests added for these cases would help the robustness
  2. We'd probably want to thread the logger through the provider, so we don't have to write directly to errorWriter

I'm happy to take care of both of those in a polish PR, but I do want to clarify the specific logical questions I had – notably, which OS's we think we need to set host (so we can have an appropriate comment clarifying what we're doing, and setting provider.ctrConf.Env even for non linux environments.

Definitely hoping to get this in in the next few days!

internal/build/phase_config_provider.go Outdated Show resolved Hide resolved
internal/build/phase_config_provider.go Outdated Show resolved Hide resolved
internal/build/phase_config_provider.go Outdated Show resolved Hide resolved
internal/build/phase_config_provider.go Outdated Show resolved Hide resolved
internal/build/phase_config_provider.go Outdated Show resolved Hide resolved
@matejvasek
Copy link
Contributor Author

The proposed solution in this PR currently adds complexity that isn't quite a drop-in replacement. For example, if you use a tcp://docker:1234 remote daemon and docker hostname is local to the client via /etc/hosts the lifecycle container wouldn't be able to resolve the hostname.

@jromero I know that's problematic, on Linux this can be solved by using --network=host I don't know how to fix that on other OSes.

@dfreilich dfreilich requested a review from micahyoung January 6, 2021 11:12
@dfreilich dfreilich modified the milestones: 0.16.0, 0.17.0 Jan 6, 2021
Copy link
Member

@micahyoung micahyoung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To summarize my thinking, I love the idea of being able to specify an alternative to the default bind-mounted socket. I'm a bit concerned with this implementation right now because it's very unencrypted-TCP-on-Linux oriented and breaks the currently working, TCP-over-mTLS which is platform agnostic and much safer for general use.

I wonder, what if this were done with pack build option like --docker-host=
which, if specified, would always disable the default binding and sets a "universal" docker host for pack and lifecycle, supporting values like:

Use-case 1: localhost-over-unencrypted-TCP

Just like pack's current behavior (respect DOCKER_HOST) but also copy it to down to lifecycle

export DOCKER_HOST=tcp://localhost:2375
pack build my-app --docker-host=inherit --network=host

# equivalent
pack build my-app --docker-host=tcp://localhost:2375 --network=host

Use-case 2: Non-standard socket location

This would bind-mount anything with a prefix of unix: (or npipe: for Windows) and use it as pack and lifecycle's DOCKER_HOST (though maybe using the standard socket path in the container). This should enable rootless docker (which as of last year, did not work due to the hard-coded host socket path)

pack build my-app --docker-host=unix:///run/user/1001/docker.sock

Use-case 3: mTLS-secured daemon

Parse any TCP URI for optional cert options (syntax could be tweaked but a cert paths would also set DOCKER_TLS_VERIFY and DOCKER_CERT_PATH for lifecycle). More details on use-case.

pack build my-app --docker-host=tcp://my-secure-host:2376?tlscertpath=/docker-certs --volume $HOME/.docker:/docker-certs

We could start with just Use-case 1 which could expand into the other cases with future contributions.

@matejvasek
Copy link
Contributor Author

Thanks for the feedback. I hope I'll get back to this next week.

@dfreilich dfreilich modified the milestones: 0.17.0, 0.18.0 Feb 2, 2021
@dfreilich
Copy link
Member

Hey @matejvasek ! Just wanted to check on in this – this is something we're definitely interested in enabling, and it would be great to get this done. Do you think you'll have some time for this over the next few weeks? If not, I know @micahyoung had mentioned that he had some ideas for doing it as well.

@matejvasek matejvasek force-pushed the envvar-aware branch 2 times, most recently from 4591339 to 87a9460 Compare February 15, 2021 11:01
@matejvasek
Copy link
Contributor Author

matejvasek commented Feb 15, 2021

@dfreilich sorry for not getting to this earlier, I had to fix some podman issues to make this feature useful for me. I've just pushed new commit adding new docker-host flag as @micahyoung suggested. Tests are TODO. It should support first two use-cases, but not the third one.

@matejvasek matejvasek changed the title Make pack aware of DOCKER_HOST envvar Add docker-host flag Feb 15, 2021
@matejvasek matejvasek force-pushed the envvar-aware branch 2 times, most recently from ae711d6 to 5cc674e Compare February 15, 2021 17:37
@matejvasek matejvasek requested a review from dfreilich February 15, 2021 17:47
@matejvasek matejvasek requested a review from a team as a code owner February 16, 2021 15:25
@@ -143,6 +145,13 @@ func buildCommandFlags(cmd *cobra.Command, buildFlags *BuildFlags, cfg config.Co
cmd.Flags().StringArrayVar(&buildFlags.EnvFiles, "env-file", []string{}, "Build-time environment variables file\nOne variable per line, of the form 'VAR=VALUE' or 'VAR'\nWhen using latter value-less form, value will be taken from current\n environment at the time this command is executed\nNOTE: These are NOT available at image runtime.\"")
cmd.Flags().StringVar(&buildFlags.Network, "network", "", "Connect detect and build containers to network")
cmd.Flags().BoolVar(&buildFlags.Publish, "publish", false, "Publish to registry")
cmd.Flags().StringVar(&buildFlags.DockerHost, "docker-host", "inherit",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies, but I realize I missed one aspect of the workflow in my earlier comment, that the previous behavior needs to be the new default (namely, when --docker-host is not provided, don't pass through DOCKER_HOST env vars into the container and always using a bind mount). Unfortunately, inherit isn't sufficient for a backward compatible default.

I was able to fix this in your PR by setting the default value for the flag to "" instead of "inherit". I'd be happy with this change for now and if needs any further cleanup, I can do another pass when I add in the TLS behavior.

@matejvasek
Copy link
Contributor Author

@micahyoung this is odd build / test (windows-lcow) fails after minor change.

@matejvasek
Copy link
Contributor Author

@micahyoung could you please try restarting the action?

@matejvasek
Copy link
Contributor Author

I'll have to squash the commits anyway.

@micahyoung
Copy link
Member

Thank you for the quick implementation and changes. Looking forward to using the feature. It appears the last run went green as well.

@matejvasek
Copy link
Contributor Author

squashed

Copy link
Member

@dfreilich dfreilich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much for doing this, this looks really good, and really close! I'm sorry for sending this back, just a few small things I'd like to see added to this.

internal/build/phase_config_provider.go Show resolved Hide resolved
internal/build/phase_config_provider.go Outdated Show resolved Hide resolved
internal/build/phase_config_provider.go Show resolved Hide resolved
internal/build/phase_config_provider.go Outdated Show resolved Hide resolved
internal/build/phase_test.go Outdated Show resolved Hide resolved
internal/build/phase_test.go Outdated Show resolved Hide resolved
internal/build/lifecycle_execution_test.go Outdated Show resolved Hide resolved
internal/build/lifecycle_execution_test.go Outdated Show resolved Hide resolved
@matejvasek
Copy link
Contributor Author

@dfreilich PTAL

Copy link
Member

@dfreilich dfreilich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thank you so much for this, this ended up being considerably more complex than we initially thought, and I appreciate all the work that went into this!

@micahyoung I'll give you time to check it out as well (if you have the time).

Introducing new build flag: --docker-host.
The purpose of the flag is to expose custom docker deamon to build
container. This is useful when running docker daemon on non standard
location e.g. "tcp://localhost:1234", or if resulting image should be
stored in different daemon tha you are running locally.

Signed-off-by: Matej Vasek <[email protected]>
@matejvasek matejvasek changed the title Add docker-host flag Implement new build flag --docker-host Feb 17, 2021
@matejvasek
Copy link
Contributor Author

@micahyoung what is your use-case for this? Are you pushing images into remote daemon?

@dfreilich dfreilich mentioned this pull request Feb 18, 2021
2 tasks
Copy link
Member

@micahyoung micahyoung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Great contribution, many thanks!

@micahyoung
Copy link
Member

@micahyoung what is your use-case for this? Are you pushing images into remote daemon?

Yeah, I am using Docker Desktop Windows daemons (and recently Docker Desktop MacOS/ARM64 daemons) on local/remote VMs mostly, with the daemon API either unencrypted or secured with mTLS, but running pack commands from my non-Windows Workstation. I develop buildpacks and stacks mostly for Windows OCI images (and contribute support for this on pack/lifecycle). Having various daemons (Linux/Windows, ARM/AMD64, podman/docker) and just swapping the DOCKER_HOST makes it easy to try out various combinations of things against lots of different daemons:

My current workflow, before this feature, was based on swapping around the DOCKER_HOST/DOCKER_CERT_PATH:

  • Targeting Windows daemon insecurely on local VM

    source ~/.docker/docker-windows-local/env.sh
    
    docker build ...
    
    # contents of env.sh
    export DOCKER_HOST=tcp://<local VM IP>:2375
    unset DOCKER_TLS_VERIFY
    unset DOCKER_CERT_PATH
    
  • Targeting Windows daemon securely over an untrusted network:

    source ~/.docker/docker-windows-gcp/env.sh
    
    docker build ...
    
    # contents of env.sh
    export DOCKER_HOST=tcp://<windows daemon IP>:2376
    export DOCKER_TLS_VERIFY=1
    export DOCKER_CERT_PATH=~/.docker/docker-windows-gcp/  # self-signed client/server certs
    

With the functionality of this feature, I should be able to now do this from the command line:

  • Targeting Windows daemon on local VM

    docker build --docker-host=tcp://<local VM IP>:2375 ...
    
  • Targeting Windows daemon on an untrusted network:
    ... until, the tls* options for --docker-host=tcp://..., are supported, I'll still need to connect to secured TLS daemons with:

    source ~/.docker/docker-windows-gcp/env.sh
    
    docker build --docker-host=""  ...  # or unset
    

    ... but eventually, with TLS cert options (syntax TBD)

    docker build --docker-host=tcp://<windows daemon IP>:2376 --docker-host-options=tlscertpath=~/.docker/docker-windows-gcp,tlsverify=1
    

    There's a few details here that may be tricky: how to copy in certs in a way that only lifecycle can access them but not buildpacks (maybe root/Administrator-only read permissions?). Or perhaps an option to only use --docker-host="tcp://..."
    and certs for pack, but continue to bind-mount socket in the container for lifecycle.

But overall, this feature is a great step to making the implicit DOCKER_HOST functionality that I've been relying on become more explicit, while enabling more rootless daemon options which is awesome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement Issue that requests a new feature or improvement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants