Debugging node issues using CoreOS toolbox

You might need to install additional packages or tools on Container-Optimized OS for certain tasks, such as debugging. Although Container-Optimized OS does not include a package manager, you can use the pre-installed CoreOS Toolbox utility to install any additional packages or tools you require. Using /usr/bin/toolbox is the preferred method for installing and running one-off debugging tools.

/usr/bin/toolbox essentially provides you a shell in a Debian chroot-like environment. When you invoke /usr/bin/toolbox, it runs following commands:

  1. docker pull and docker create to set up the environment. These are only run the first time you invoke /usr/bin/toolbox.
  2. systemd-nspawn to run the given command or (in absence of any command) provides you a shell

toolbox has some other properties to keep in mind:

  • Invoking toolbox after the first invocation does not require a working Docker daemon, nor does it incur any network/disk overhead.
  • The toolbox environment is set up once for each user invoking it. Running sudo toolbox sets it up for root user.
  • The toolbox environment is created under /var/lib/toolbox and is persistent across reboots.
  • You can access sections of the root filesystem, such as user home directories, from inside the toolbox environment.

Customizing toolbox for your deployment

You can customize the Docker image that toolbox uses, as well as the paths available to toolbox in the root filesystem. These settings are located in the file /etc/default/toolbox. The default /etc/default/toolbox file typically resembles the following:

TOOLBOX_BIND="--bind=/:/media/root/ --bind=/mnt/disks/:/media/root/mnt/disks/ --bind=/var/:/media/root/var/ --bind=/home:/media/root/home/"
  • The TOOLBOX_DOCKER_IMAGE and TOOLBOX_DOCKER_TAG variable specify the Docker image to be used. The default comes with some of the common tools like the gcloud command-line tool pre-installed.
  • The TOOLBOX_BIND variable specifies the paths from rootfs to be made available inside the toolbox environment.

To change the default settings, modify the /etc/default/toolbox file, or specify new values for the variables in ${HOME}/.toolboxrc for the appropriate user as follows:

    echo "TOOLBOX_DOCKER_IMAGE=fedora" > "${HOME}/.toolboxrc"
    echo "TOOLBOX_DOCKER_TAG=latest" >> "${HOME}/.toolboxrc"

Installing and running tools from toolbox

Once you've invoked the toolbox utility to start the shell, you can use apt-get inside the resulting container to install packages. For example:

# Inside the toolbox shell
apt-get update && apt-get install -y htop psmisc
pstree -p

You can also use a shorthand notation to invoke tools in toolbox. For example, to install and run the strace utility to trace execution of a running process:

toolbox apt-get install -y strace
toolbox strace -p `pidof docker`

To run the pre-installed gcloud command-line tool, make sure your instance has sufficient scopes to access the various APIs.

# Inside the toolbox shell
which gcloud

# View installed components
gcloud components list

Your current Cloud SDK version is: 134.0.0
The latest available version is: 141.0.0
Was this page helpful? Let us know how we did:

Send feedback about...

Container-Optimized OS