vagrant-lxc nested in lxc #339
Comments
|
Sanity check: Does CircleCI uses docker under the hood to run builds? |
|
@fgrehm yes I believe it is a docker in docker scenario. What does that mean for us then? |
|
I actually remember running lxc inside docker at some point but I cant remember what I've done. You might get an insight from https://github.com/jpetazzo/dind |
|
Thanks, will have a look! It's docker-in-docker definitely, here's the reference: |
|
Quick heads up: got (a really fast btw) response from @notnoopci from CircleCI: @fgrehm have not digged any further into vagrant-lxc though. The mknod looks suspicious, but I have no idea on how to get around it. |
|
Cool - I'll take a closer look when I get a chance this week. But let me jot some thoughts to provide some context first: CircleCI builds use lxc containers actually and we rely on nested lxc support (i.e. lxc-in-lxc) for docker support. LXC should work also "out-of-the-box". We also run on unprivileged lxc containers, where root-in-container != root-on-host, and we apply further apparmor restrictions to make the system secure, and restricting mknod operations. That may clash with privileged container starting, if it mounts /dev device files (e.g. /dev/null) with mknod. The suggested solution is to use bind-mounts when running in unprivileged-containers instead. In ordinary circumstances, lxc detects it's running on unprivileged container and does the right thing by using bind-mounts. If vagrant-lxc uses a custom lxc config files (or it imports a privileged-container config file), the lxc mount settings will need to be tweaked to use bind-mount. In our containers, the configurations we applied to Docker default to make it work were: There are some additional minor complexities, if you are downloading a container tarball that contains some device files (e.g. tarball contains /dev/null) and some security extended attributes; but I can expand on that another time :). I will dig a bit further during the week. Would love to have vagrant-lxc support :). |
|
@notnoopci thanks for the explanation! I guess it's coming from |
|
Just tried to get a bit further with this, so I edited
What I did was:
However, no matter what how I edited the I even put some total nonsense into the config file but still got the same output. Looks like I'm editing the wrong file? Any more ideas @fgrehm @notnoopci ? |
|
@fgrehm @notnoopci any chance you might have a short timeslot to take a look at this? I'm really stuck with my limited lxc knowledge at the moment... :-/ |
|
@tknerr Would you be able to set up a VirtualBox Vagrant VM that reproduces the problem? That might make things easier to debug ;-) |
|
@fgrehm not sure if I can reproduce the CircelCi environment :-/ I have an ubuntu virtualbox VM where vagrant-lxc works flawlessly. However, it does not work in the dockerized environment of a CircleCI build container, as mentioned above. @notnoopci do you have a vagrant / virtualbox environment to simulate the dockerized environment of a CircleCi build container at hand that we could use for debugging? The way I debugged it on CirlceCI was enabling SSH access which gives you ssh access and keeps the container running for 30 minutes. It would require you to set up a CircelCI account, but once you have that you could just
This would run the high-level integration tests, and you should see the errors stated above. From there on I went more low-level with the plain lxc commands but this is where I got stuck due to lack of experience with lxc... |
|
Okay. I have this working! 😀 Ubuntu Trusty running on CircleCI (Ubuntu Precise) via vagrant-lxc All scripts/files are available at https://gist.github.com/rcoup/36c75555683ac9db2309 Changes to
|
|
Awesome! Unfortunately as of now I won't be able to move forward with that 😞 If you or anyone else is able to put up a PR and is able to get some people to test the changes I'd be more than happy to provide commit access to the project in order to keep the fire burning! |
|
@fgrehm fair enough. I'll need to dig into vagrant & vagrant-lxc and see if I can figure out a good approach. Doesn't seem like just changing the default templates is a great idea, would be better to feature-detect or use Vagrantfile to specify the various options (I know |
|
Awesome stuff @rcoup! You are both way more experts in LXC than I am, so I can't offer much help in that space, but I would definitely be in for testing this stuff. Concerning the basesbox: also not much experience with packaging my own, but some auto detection would be ideal (i.e. not a separate basebox for CircleCI) |
|
@tknerr in terms of boxes, currently what you need to do is:
|
|
For the record, CircleCI builds run in pure LXC containers not Docker. |
|
Hey, sorry for the silence here but this project is looking for maintainers 😅 As per #499, I've added the |
Yesterday I found out about http://circleci.com and that they support running docker containers as part of their build. Within a few hours I got a test-kitchen suite with the kitchen-docker driver up and running:
So I thought if docker runs then LXC should work too!
However, I'm a bit stuck with getting vagrant-lxc up and running up there. Here's what I got so far:
This is the relevant part from the logs I guess:
I'm not really fond of LXC, but I ssh'ed into the box (yes you can do that on circleci) and mimicked some of the commands that vagrant-lxc does.
This is what I got:
Do you see any chance of getting this working on circleci?
It would be much more than awesome if that worked!
The text was updated successfully, but these errors were encountered: