auto-discover apt sources from /etc/apt/sources.list.d/

[timrobotson]

I'm using the package raspicam_node which comes from the ubiquity robotics apt repository https://packages.ubiquityrobotics.com/ubuntu/ubiquity.

It built and ran fine, but when running colcon bundle it failed to find the package from apt throwing the following error:

Bundling workspace...
Reading package lists... Done
Building dependency tree... Done
Checking if dependency tarball exists...
Starting >>> object_tracker_robot
loading view [*default*] with rospkg loader
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 255, in __getitem__
    return self._weakref[key]
  File "/usr/lib/python3.5/weakref.py", line 131, in __getitem__
    o = self.data[key]()
KeyError: 'ros-kinetic-raspicam-node'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 259, in __getitem__
    rawpkg = self._cache[key]
KeyError: 'ros-kinetic-raspicam-node'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/colcon_core/executor/__init__.py", line 91, in __call__
    rc = await self.task(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/colcon_core/task/__init__.py", line 92, in __call__
    return await task_method(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/colcon_ros_bundle/task/ros_bundle.py", line 70, in bundle
    package_name)
  File "/usr/local/lib/python3.5/dist-packages/colcon_bundle/installer/apt.py", line 154, in add_to_install_list
    logger.info(self._cache[name].versions)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 261, in __getitem__
    raise KeyError('The cache has no package named %r' % key)
KeyError: "The cache has no package named 'ros-kinetic-raspicam-node'"
[110.747s] ERROR:colcon.colcon_core.executor.sequential:Exception in job execution 'object_tracker_robot': "The cache has no package named 'ros-kinetic-raspicam-node'"
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 255, in __getitem__
    return self._weakref[key]
  File "/usr/lib/python3.5/weakref.py", line 131, in __getitem__
    o = self.data[key]()
KeyError: 'ros-kinetic-raspicam-node'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 259, in __getitem__
    rawpkg = self._cache[key]
KeyError: 'ros-kinetic-raspicam-node'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/colcon_core/executor/sequential.py", line 49, in execute
    loop.run_until_complete(future)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/usr/local/lib/python3.5/dist-packages/colcon_core/executor/__init__.py", line 91, in __call__
    rc = await self.task(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/colcon_core/task/__init__.py", line 92, in __call__
    return await task_method(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/colcon_ros_bundle/task/ros_bundle.py", line 70, in bundle
    package_name)
  File "/usr/local/lib/python3.5/dist-packages/colcon_bundle/installer/apt.py", line 154, in add_to_install_list
    logger.info(self._cache[name].versions)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 261, in __getitem__
    raise KeyError('The cache has no package named %r' % key)
KeyError: "The cache has no package named 'ros-kinetic-raspicam-node'"

The reason for this is colcon bundle has it's own list of apt sources in /etc/apt/sources.list.d (https://github.com/colcon/colcon-bundle/blob/8eb65de65d3cdd601119528331e84849b6ecba74/colcon_bundle/installer/assets/xenial.sources.list) and this list doesn't contain the ubiquty robotics repository.

I think colcon bundle should automatically pull in any custom defined apt repositories in /etc/apt/sources.list.d/ (or similar location on other OS's) so users aren't confused about this in the future.

[tfoote]

"Discovering" the repository won't actually let you resolve this dependency. Those repositories listed are the ones that are required for using the rosdistro and are expected to be already available on the target platforms, the ubiquity repository is not expected to be available on the target platform and the error that you're getting is a precursor to the bundle failing to install on the other end because the target system won't have the ubiquity repositories setup and thus won't be installable from the bundle generically.

[timrobotson]

Doesn't colcon bundle take all the apt dependencies required and add them to the zip so they don't need to be installed from apt on the robot? Why can't the packages be used/installed from the bundle generically? Isn't that the point of bundling?

The idea is that I want to be able to setup Ubiquity Robots repository on my development machine, install raspicam, then bundle and have raspicam added to that bundle so it can be used with the robot without having to install it from apt.

This should at least give a more helpful error explaining to the user why they can't bundle the package they need.

[tfoote]

Why can't the packages be used/installed from the bundle generically? Isn't that the point of bundling?

There's a significant difference between bundling content into user space and installing system dependencies. I would be quite unhappy if I used a bundle and it injected a system package onto my system that's not in my apt sources. Versus it just installing things that I've already declared trusted.

Note that I believe that it only persists the apt package names and not the actual binaries. And this also means that you don't have to do things like a recursive dependency walk to install packages. For example if raspicam-node had a dependency on libraspicam also available in the custom apt sources, that would also need to be included into the bundle. And any other recursive dependency that isn't known to be available on the target system. And since the target system is undefined you might then need to actually include all recursive dependencies. And then you're looking a lot more like a container than a bundle of a workspace with a list of known dependencies.

The idea is that I want to be able to setup Ubiquity Robots repository on my development machine, install raspicam, then bundle and have raspicam added to that bundle so it can be used with the robot without having to install it from apt.

I would agree with this approach if it setup a full chroot and didn't install the packages onto the system. But again this is getting towards what a container provides.

This should at least give a more helpful error explaining to the user why they can't bundle the package they need.

Collecting the known sources.list and displaying it when the bundle install fails might make sense to give the user the option to add an apt source. Alternatively it could also build the cache when generating the bundle and warn or error that the apt source is missing.

It also appears that there is an option for you to add an apt source when installing you could use:

colcon-bundle/colcon_bundle/installer/apt.py

Line 50 in cecf38c

parser.add_argument(

that would probably get you past your specific problem. And since you're explicitly setting it at installation time it's not unexpected behavior to install from that source.