Docker for our ROS robotic overlords

The typical robotic stack is based on the Robotic Operating System (ROS) whose version is linked to a specific Ubuntu LTS distribution, e.g. Indigo-14.04 and Kinetic-16.04 to mention the most recent ones. When working with recent laptops it is possibly that even with a kernel update (like Linux 4.4) it is not possible to have a good working machine.

This post aims at providing some basic indications for using the Docker container system for creating a ROS setup with a different Linux distribution together with GPU support, e.g. running ROS Indigo inside Ubuntu 16.04. Docker is used in the industry for isolation and easy deployment, and in this case it solves the compatibility problem with minimal performance penalty.

We are using this approach internally at PERCRO laboratory of SSSA for working with research robots developed in RAMCIP and ReMeDi, and also with the Baxter robot from Rethink Robotics. The post provides a basic recap also on Docker for the minimal elements useful for pursuing this approach.

Docker

Docker is based on the concept of Container and Images. A container is an isolated execution environment running a Linux user software different from the hosting machine while sharing the Linux kernel. It is lightweight and efficient and provides several levels of isolation between the hosting system and the contained ones.

When running a named container (--name XYZ) it is better to use --rm
Containers are started up from Images that are lightweight providers of the Linux filesystem. Images can be created by extending existing images and adding files and operations as specified in the dockerfile, used by the docker build command. The common image to all the activity discussed here is ubuntu:14.04 (188MB) as provided in the Docker hub repository.

Docker relies heavily on the Union File System (UFS) that is a mechanism for layering multiple filesystems one over the other.
When a Container is running, it is possible to commit its content to a storage mechanism for later use in another docker run command.

docker run -ti IMAGENAME :
docker ps 
docker ps -a
docker images
docker rm
docker rmi

Docker for ROS

There are some existing pre-build images in Docker Hub as follows with detailed instructions here https://hub.docker.com/_/ros/.

• ros:indigo-ros-core (over ubuntu:trusty, 188MB) 813MB
• ros:indigo == ros:indigo-ros-base (over core) 827MB
• ros:indigo-robot (over base) 1GB
• ros:indigo-perception (over base) 2GB

The alternative to these prepared Docker images is to prepare the Docker image by hand as follows. Replace the macros UBUNTUNAME and ROSNAME with the correct versions (e.g. trusty indigo):

sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $UBUNTUNAME main" > /etc/apt/sources.list.d/ros-latest.list'
apt-get update
apt-get -q -y --force-yes  install ros-$ROSNAME-desktop-full
apt-cache search ros-$ROSNAME
rosdep init
rosdep update
apt-get -q -y --force-yes install python-rosinstall

#User integration
echo "source /opt/ros/$ROSNAME/setup.bash" >> ~/.bashrc
source ~/.bashrc

The docker hub page above contains some discussions about network isolation with the docker instance, in our case we are simply using the hosted solution.

Other notes can be found here: http://wiki.ros.org/docker/Tutorials/Docker

Working with Dockerized ROS

After having prepared or downloaded the Docker image it is advised to have the Catkin workspace and any variable elements in the hosting machine so that the Docker image can be considered frozen. This just requires to perform:

-v ramcipws:/root/ramcipws

In addition to user workspace (ros_ws or other) it is advised to map /root/.ros containing log data and debugging information.

Using Host ROS Core from Guest

Sometimes we are interested running ROS Core inside the Host machine and access it in the Guest. This works also across different ROS versions under the assumption that messages have the same signature.

The configuration is similar to multiple ROS machine with single ROS Core sharing ROS_HOSTNAME and ROS_MASTER_URI. Much depends on the type of network configuration used in Docker:

• Docker bridge (not for Docker for Mac): host sees bridge0 interface, guest sees docker0 so that they can share the same network
• Docker host: they are in the same network

Dy default roslaunch system tries to launch things from the ROS Core where the master roslaunch is present, but if you need to roslaunch from the Guest there are two options: specify machine tag or run a Guest roslaunch and use it in the invocation.

Running a new roslaunch is:  roslaunch server http://localhost:50153/

Running a launch file with the local roslaunch: roslaunch -u URI
Where URI is the one used above

Ports used in ROS:

• 50153 for ROS Launcher Server
• 11311 for ROS Core
• Any port for data exchange over Service and Topics

Check also http://wiki.ros.org/ROS/NetworkSetup

Baxter Robot

Following the instruction on the official website it is easy to prepare a Dockerfile with the SDK. I have packed them into a Dockerfile for building a U1404_Baxter image containing both the real and simulated system.

Available on github: https://github.com/eruffaldi/linuxsetup/blob/master/dockerbaxter/Dockerfile 

Additional packages have been added to our setup for supporting some more activities (e.g. VISP and Moveit). We could remake the above to inherit from ros:indigo-robot instead of ubuntu 14.04.

GPU Support

The access to GPU is fundamental for enabling advanced computations with pointclouds or Deep Learning. Thanks to the fact Docker for Linux shares the same kernel of the hosting operating system it is just necessary to map the GPU devices and key driver files into the hosted Linux and it works. The important aspect is that the hosted Linux has the same version of the drivers of the hosting Linux. Here we are looking only at NVidia.

Recently Nvidia has provided a tool for simplifying the sharing of the GPU called nvidia-docker that at the time of writing has to be installed from sources. Than after that you just need to:

nvidia-docker volume setup

Provided that nvidia-docker has done its job it can be used in plugin mode or manually. In plugin mode it provides a Web server that returns a string to be passed to docker, or alternatively to run nvidia-docker for run.

Finally the manual mode is something similar to this (assuming NVidia version 361.42):

export MYNVIDIA="-e LD_LIBRARY_PATH=/usr/local/nvidia/lib64 --volume=nvidia_driver_361.42:/usr/local/nvidia:ro --device=/dev/nvidia0 --device=/dev/nvidiactl --device=/dev/nvidia-uvm"

Typically it requires to install the driver inside the container but without the kernel-modules:

wget http://it.download.nvidia.com/XFree86/Linux-x86_64/${NVIDIA_VERSION}/${NVIDIA_DRIVER_FILE}

chmod +x ./${NVIDIA_DRIVER_FILE}

./${NVIDIA_DRIVER_FILE} -s -N --no-kernel-module

Note that some CUDA versions try to load the modules and the modprobe could failing, requiring the modprobe.dep.bin.

More notes about ROS and X11 can be found here:

http://wiki.ros.org/docker/Tutorials/Hardware%20Acceleration 

X11 Support

The access to X11 is documented in many examples online. Here I am proposing to use the direct usage in which the hosted Linux uses directly the hosting Linux by providing the following piece of string to Docker, to be added to the other NVidia specific.

-e QT_X11_NO_MITSHM=1 -v /tmp/.X11-unix -e DISPLAY 

This has been tested with RViz and Gazebo. If a working graphical shell is needed from the hosted Linux it is possible to install xfce4 and run "startxfce4". This will create a shell that overlap over the existing Ubuntu shell allowing to run application from both systems, with separate menus.
Support for Kinect

In many ROS setups there is the need for a Kinect. In this case it is just simple as mapping these device and the associated udev rules into the docker image at startup:

-v /dev/bus/usb
-v /etc/udev/rules.d

Mapping the /dev/bus/usb folder exposes all the USB bus and it not wanted in some times. I have prepared a scripts that mounts only Microsoft devices mapusbkinect.sh (gist):

#!/bin/bash
IFS=$'\n'
a=''
for i in $( IFS=$'\n' lsusb | grep 045e: ); do
IFS=' :' read -r -a array <<< "$i"
a="${a} --device /dev/bus/usb/${array[1]}/${array[3]}"
done
echo $a

This can be used as $(mapusbkinect.sh).

Additional instruction for build and setup Kinect 1 in general over Linux can be found here and here.

Docker for ROS and Mac OSX

The above instructions can be used for accessing ROS from OSX without any major difference except for the aspect of GPU that unfortunately cannot be yet mapped.

Sharing of X11 is possible with the limitation that 3D does not work well due to the limitations of XQuartz (OpenGL 1.4) and the lack of X11 Direct Rendering. The best approach at the moment is to use RViz or Gazebo from outside and let's connect them via ros master.

For other details about Docker X11 and OSX you can check this post.

ATTENTION: with Docker for Mac there is so far no possibilit of using USB devices, and, moreover, bridged networking is limited