Controlling Docker from Within A Docker Container

I’ve been tinkering with a project to interact with the Docker-Engine api using docker-py. The catch is that the program is running inside a docker container.

Modify /lib/systemd/system/docker.service to fix the Docker daemon to a TCP port.

First create a loopback IP address

sudo ip addr add 10.254.254.254/32 dev lo

By default the Unix socket used by Docker is inaccessible from inside the container for this reason we need to use TCP.

Modify the ExecStart line to remove the unix socket and instead replace it with the address of your loopback and a port of your choosing. I used 2376 because that is what Docker uses on Windows where the unix sockets are not available.

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket firewalld.service
Requires=docker.socket

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H 10.254.254.254:2376
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
ExecReload=/bin/kill -s HUP $MAINPID
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target

Reload systemd daemons to pull in the new changes to the unit file.

sudo systemctl daemon-reload

Start Docker with the new settings.

sudo systemctl stop docker.service
sudo systemctl start docker.service

Modify your environment variables so you don’t have to use the -H argument everytime you call a docker cli command.

export DOCKER_HOST=10.254.254.254:2376
echo "export DOCKER_HOST=10.254.254.254:2376" >> ~/.bashrc

Run a new docker image in the new environment.

docker run -itd --name=test ubuntu /bin/bash
docker exec -itd ubuntu /usr/bin/apt-get update -y 
docker exec -itd ubuntu /usr/bin/apt-get install python python-pip -y
docker exec -itd ubuntu /usr/local/bin/pip install docker

Create a Test Script to Try In the Container

#!/usr/bin/python
import docker
import pprint

# PrettyPrint Setup
pp = pprint.PrettyPrinter(indent=4)

# DOCKER API Setup
#  NORMAL SETUP
#client = docker.from_env()
#low_level_client=docker.APIClient(base_url='unix://var/run/docker.sock')
#  Custom SETUP
client = docker.DockerClient(base_url='tcp://10.254.254.254:2376')
low_level_client=docker.APIClient(base_url='tcp://10.254.254.254:2376')

container_list=client.containers.list()
pp.pprint(container_list)
for container in container_list:
    docker_container=low_level_client.inspect_container(container_id)
    pp.pprint(docker_container)

Run The Test Script

chmod +x ./test.py
docker cp ./test.py test:/root/test.py
docker exec -it test /root/test.py

Security: Don’t forget to secure your newly exposed port with IPtables rules!

sudo iptables -t filter -A INPUT -i eth0 -p tcp -m tcp --dport 2376 -j DROP
Advertisements

Migrating From Machine to Machine with Smokeping (and Armbian)

After having purchased an Orange Pi I have been looking for services that it can host on my network. One of the first services that come to mind for me is Smokeping. I love Smokeping it does one thing, and it does it well a true embodiment of the Unix Philosophy.

Enough glorification of Smokeping how do we get it running.

In this case I wasn’t starting from scratch. I already have an installation running on a Raspberry Pi elsewhere in my network. My first attempt had me copying all the files in the /etc/smokeping directory over to the Orange Pi directly. However this did not work. What I came to find is that the RRD files are architecture-specific, as it turns out, while both the Raspberry Pi and Orange Pi are ARM-based, they are not the same version of ARM and hence the RRD files are not directly compatible.

Starting fresh is no good here because I have years of Smokeping data in my existing install that I don’t want to lose; so how to migrate that data.

Scouring some obscure Smokeping mailing lists I was able to put together this procedure.

NOTE: I’m assuming preshared keys have already been setup between the root account of the old and new machines.

#######################
# On the New Machine
####################### 
sudo su
# Install Smokeping
apt-get install smokeping rrdtool sendmail -y
systemctl stop smokeping
#######################
# On the Old Machine
#######################
sudo su

echo <<EOT >/tmp/smokeping_backup.sh
#!/bin/bash

NEWMACHINE="192.168.1.100"

cd /var/lib/smokeping
echo "Stoping SMOKEPING"
service smokeping stop

echo "GENERATING XML..."
rm -v ./*/*.xml
for f in ./*/*.rrd; do echo ${f}; rrdtool dump ${f} > ${f}.xml; done

echo "SENDING FILES TO NEW MACHINE..."
scp -rv ./* root@$NEWMACHINE:/var/lib/smokeping/
scp -v /etc/smokeping/config.d/General root@$NEWMACHINE:/etc/smokeping/config.d/General
scp -v /etc/smokeping/config.d/Targets root@$NEWMACHINE:/etc/smokeping/config.d/Targets
scp -v /etc/smokeping/config.d/Probes root@$NEWMACHINE:/etc/smokeping/config.d/Probes

echo "CLEANING-UP AND RESTARTING SMOKEPING..."
rm -v ./*/*.xml
service smokeping start

echo "DONE!"
EOT

chmod +x /tmp/smokeping_backup.sh
/tmp/smokeping_backup.sh
#######################
# On the New Machine
#######################
sudo su
cat <<EOT > /tmp/smokeping_restore.sh 
#!/bin/bash

# convert XML to RRD
cd /var/lib/smokeping

systemctl stop smokeping

echo "REMOVING ANY EXISTING RRD FILES..."
rm -v ./*/*.rrd

echo "CONVERTING XML FILES BACK INTO RRD..."
for f in ./*/*.xml; do echo ${f}; rrdtool restore $f `echo $f |sed s/.xml//g`; done

echo "REMOVING Interim XML FILES..."
rm -v ./*/*.xml
sleep 1

echo "CHANGING OWNERSHIP ON SMOKEPING FILES..."
chown -v smokeping:www-data /var/lib/smokeping/*/*.rrd
chmod 755 -Rv /var/lib/smokeping

echo "STARTING SMOKEPING."
systemctl restart apache2
systemctl start smokeping

echo "DONE!"
EOT

chmod +x /tmp/smokeping_restore.sh
/tmp/smokeping_restore.sh

This can convert the RRD files to an intermediary XML format that can then be converted back to RRD on the migration target, on Armbian, even with Smokeping installed, rrdtool itself was not installed. After installing rrdtool I operated on the subdirectories full of RRD files in the /var/lib/smokeping/ directory. Once converted to XML I moved the XML files in place on the new machine and converted them back to RRD.

These scripts were used on my machines, hopefully they can help you too!

 

Fixing LLDPd Default PortID Configuration

LLDPd is a great program written by Vincent Bernat which implements 802.1ab for unix based OSes.

https://vincentbernat.github.io/lldpd/

Debian and Ubuntu offer this lovely utility right in their default software repositories meaning that is only an apt-get install away. One caveat is that the default configuration provides the MAC address of the interface instead of the interface name which is unusual compared to the implementation of LLDP in use on most network OS vendors like Cumulus, Cisco, Juniper, Arista etc…

Here is what is sent by default as seen by the far side of the link:

-------------------------------------------------------------------------
LLDP neighbors:
-------------------------------------------------------------------------
Interface: swp4, via: LLDP, RID: 2, Time: 0 day, 00:00:12
 Chassis: 
 ChassisID: mac 52:54:00:b9:e3:98
 SysName: server1
 SysDescr: Ubuntu 16.04 LTS Linux 4.4.0-22-generic #40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016 x86_64
 TTL: 120
 MgmtIP: 192.168.121.53
 MgmtIP: fe80::5054:ff:feb9:e398
 Capability: Bridge, off
 Capability: Router, off
 Capability: Wlan, off
 Capability: Station, on
 Port: 
 PortID: mac 44:38:39:00:00:06
 PortDescr: eth1
 PMD autoneg: supported: yes, enabled: yes
 Adv: 10Base-T, HD: yes, FD: yes
 Adv: 100Base-TX, HD: yes, FD: yes
 Adv: 1000Base-T, HD: no, FD: yes
 MAU oper type: 1000BaseTFD - Four-pair Category 5 UTP, full duplex mode
-------------------------------------------------------------------------

Luckily there are knobs to change that behavior. The configuration below will set LLDPd to provide a more similar configuration like a network device might expect.

sudo apt-get install lldpd
sudo bash -c "echo 'configure lldp portidsubtype ifname' > /etc/lldpd.d/port_info.conf"
sudo systemctl restart lldpd.service

 

Here is the net result of the changed configuration:

-------------------------------------------------------------------------
Interface: swp4, via: LLDP, RID: 2, Time: 0 day, 00:00:21
 Chassis: 
 ChassisID: mac 52:54:00:b9:e3:98
 SysName: server1
 SysDescr: Ubuntu 16.04 LTS Linux 4.4.0-22-generic #40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016 x86_64
 TTL: 120
 MgmtIP: 192.168.121.53
 MgmtIP: fe80::5054:ff:feb9:e398
 Capability: Bridge, off
 Capability: Router, off
 Capability: Wlan, off
 Capability: Station, on
 Port: 
 PortID: ifname eth1
 PortDescr: eth1
 PMD autoneg: supported: yes, enabled: yes
 Adv: 10Base-T, HD: yes, FD: yes
 Adv: 100Base-TX, HD: yes, FD: yes
 Adv: 1000Base-T, HD: no, FD: yes
 MAU oper type: 1000BaseTFD - Four-pair Category 5 UTP, full duplex mode
-------------------------------------------------------------------------

Securely Wiping a Hard Drive

When getting rid of a hard drive, I, like everyone else like to be secure about it. After collecting my data I usually like to overwrite the drive with garbage. In the past I used to just use the basic DD approach to zero the drive out.

dd if=/dev/zero of=/dev/sd<DRIVE>

This works fine but I started hearing rumors of being able to recover data from a zeroed out drive. Indeed this is partially true. Zeroing out is probably sufficient in most cases.

Ideally, I would write data out from /dev/random or /dev/urandom (whatever your system has) but the amount of entropy that is harnessed here is not enough to saturate the write speed of the drive meaning that it will take forever a very long time. Never the less I was curious to find out about another option to wipe a drive.

This approach uses OpenSSL with seed data from /dev/urandom. Supposedly it is possible to generate about 1.5gbps of garbage data with this technique… I’ll never know though because the write speed of my drive is nowhere near that.

Command:

Use the following command to randomize the drive/partition using a randomly-seeded AES cipher from OpenSSL (displaying the optional progress meter with pv):

# openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt </dev/zero \
    | pv -bartpes <DISK_SIZE> | dd bs=64K of=/dev/sd"X"

where the (optional) total disk size in bytes (DISK_SIZE) may be obtained via:

# blockdev --getsize64 /dev/sd"X"
250059350016

Sidenote: I love the use of PV here, this is an underloved utility that is truly awesome; I’ve only ever used it in one other place. TARing a remote file over SSH for delivery on my local machine (shown below).

ssh –c blowfish user@host  "tar cjpf - /home/user/file" | pv | cat > ./file.tar.bz2

Should I Leave The Window Open?

As the days start to get cooler again in North Carolina I’m always left with this conundrum… should I leave the window open or not?

On one hand I could feel a pleasant draft that cools me and my wife down all evening without having to pay for air conditioning, on the other hand I could end up waking up to a soaked wall as it’s rained in the middle of the night or the air conditioning has kicked on and I’m pumping lovely cool air that I’ve just paid for directly out the open window.

#firstworldproblems…. I know but it is never the less something I think about. So what can be done here? I’ve got a Nest thermostat and am a reasonably competent scripter.

Here is what I put together, a quick webpage that you can hit from any device on the home network and displays the following info…

selection_064

Turns out there are a few libraries that help us get off to a useful start.

  • weather-cli — this one allows us to pull real time hourly weather info for the next 24 hours based on precise location.
  • flask — flask can be used to quickly serve up a webpage in python
  • python-nest — this is used to get/set data with Nest thermostats.

Interestingly, python-nest and the nest-API  provides some hourly weather info back in a pretty useful form. However it is missing one pivotal piece of data… which is the actual weather conditions for those individual hours, is it raining? is it sunny? that very important datapoint is not present and pretty much the primary reason I’m using weather-cli here.

There’s not a ton of documentation on weather-cli but for the few things that I wanted to know that weren’t documented, there’s always the source-code, and the code was helpful here.

Installing Software

sudo apt-get install python-pip
sudo pip install pip --upgrade
sudo pip install setuptools --upgrade
sudo pip install  weather-cli flask python-nest

Setting Up Weather-CLI

To use weather-cli, you’re going to need a Forecast.io API key, register here to obtain one. With the basic free plan, you have 999 API calls per day for free, enough to check the weather a couple hundred times per day which is more than enough for me.

To get weather-cli set up we need to use the setup argument.

pi@underhousepi ~/scripts/weather_predictor $ sudo weather-cli --setup

Enter your forecast.io api key (required):xxxxxxxxxxxxxxxxx
Warning:
The script will try to geolocate your IP.
If you fill the latitude and longitude, you avoid the IP geolocation.
Use it if you want a more acurated result, use a different location,
or just avoid the IP geolocation.
Check your latitude and longitude from http://www.travelmath.com/

Enter the latitude (optional): 36.982917
Enter the longitude (optional): -75.402301
generating config file...
setup complete

Once setup is complete we can simply call weather-cli and it will return data for us based on the location we used for initial setup.

The rest is writing a quick flask app. Here is the code for that in the gist below. You’ll notice the flask app calls a template. I’ve placed that index.html template in a templates directory as shown below.

https://gist.github.com/ericpulvino/b7753c1f598642f3bbd64895553232db

Note that I’m also calling the nest api to query the target temperature of the nest device and using that as a basis for whether or not the window open. You could hard-code that value alternatively if you don’t have a nest or for testing purposes. I also have more than one nest in my home so in line 29 I’m calling devices[1] to get to query the second nest in my environment. You may want to change this to devices[0] or another value based on your output from the python-nest api’s documentation page.

user@raspi4 ~/scripts/window_open $ tree
.
├── templates
│   └── index.html
└── window_open.py

Here is the code for the template too.

https://gist.github.com/ericpulvino/8196a057493dec94d0f48c495ad5f706

Hope this might save you a few minutes in your search for cool air at an affordable price.

Handling SSH Protocol Links in Chrome (on Linux)

As a network engineer, I have been annoyed by not being able to click on SSH links in webpages for years while running Linux.

After digging in a bit I was able to find this solution which works for SSH in Chrome.

I’m running Ubuntu 16.04 in my setup.

My SSH URL links look like this:

ssh:///user@host:port

Here is the process I used to get these links working.

#Check which handler is setup for SSH
xdg-mime query default x-scheme-handler/ssh

#Write code for new handler
cat << EOF > ~/.local/share/applications/ssh.desktop
[Desktop Entry]
Version=1.0
Name=SSH Launcher
Exec=bash -c '(URL="%U" HOST=\$(echo \${URL:6} | cut -d ":" -f1) PORT=$(echo \${URL:6} | cut -d ":" -f2); ssh \$HOST -p \$PORT); bash'
Terminal=true
Type=Application
Icon=utilities-terminal
EOF

#Apply New handler for SSH
xdg-mime default ssh.desktop x-scheme-handler/ssh

#Confirm new handler has been applied
xdg-mime query default x-scheme-handler/ssh

Manually Installing Plugins in Vagrant

It looks like there have been some changes lately in Vagrant from v 1.7.4 –> 1.8.4 that allow me to no longer install plugins locally.

I kept getting failures that looked like this:

vagrant plugin install ./vagrant-libvirt-0.0.33.gem 
Installing the './vagrant-libvirt-0.0.33.gem' plugin. This can take a few minutes...
Bundler, the underlying system Vagrant uses to install plugins,
reported an error. The error is shown below. These errors are usually
caused by misconfigured plugin installations or transient network
issues. The error from Bundler is:

Could not find gem 'vagrant-libvirt (= 0.0.33)' in any of the gem sources listed in your Gemfile or available on this machine.

Warning: this Gemfile contains multiple primary sources. Using `source` more than once without a block is a security risk, and may result in installing unexpected gems. To resolve this warning, use a block to indicate which gems should come from the secondary source. To upgrade this warning to an error, run `bundle config disable_multisource true`.Warning: this Gemfile contains multiple primary sources. Using `source` more than once without a block is a security risk, and may result in installing unexpected gems. To resolve this warning, use a block to indicate which gems should come from the secondary source. To upgrade this warning to an error, run `bundle config disable_multisource true`.

So based on my read of https://github.com/mitchellh/vagrant/issues/5643 there is a work around to host a local gem server to make the install proceed that way and sure enough it works. Here are the steps for that workaround as it pertains to the vagrant-libvirt plugin for Vagrant.

sudo apt-get install ruby-dev zlib1g-dev

#Download and build the Vagrant-libvirt plugin
git clone https://github.com/vagrant-libvirt/vagrant-libvirt.git
cd vagrant-libvirt/
gem build vagrant-libvirt.gemspec

#workaround for Local Gem Install Failure
#https://github.com/mitchellh/vagrant/issues/5643

#Install it locally
sudo gem install ./vagrant-libvirt*.gem

#Serve the locally installed gems on localhost:8808
sudo gem server & 

#Install the vagrant plugin while pointing at a local gemserver
vagrant plugin install vagrant-libvirt --plugin-source http://localhost:8808

# Turn off the gem server
kill %1