Preparing a new RAID Drive for Insertion

Today I had the pleasure of fixing my deficient RAID array. With the new drive slotted and identified in position /dev/sdc it was time to create a GPT partition table on the drive and a single primary partition.

blackbox# parted /dev/sdc
GNU Parted 2.3
Using /dev/sdc
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel GPT
Warning: The existing disk label on /dev/sdc will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? Yes
(parted) mkpart primary 2048s 100%
(parted) q
Information: You may need to update /etc/fstab.

Unmount the filesystem, as a safety precaution.

root@blackbox:/home/eric# sudo umount /media/Manta_Array/
root@blackbox:/home/eric# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] 
md0 : active raid6 sdh1[1] sdg1[7] sdb1[6] sda1[5]
 8790400512 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/4] [UU_UU]
unused devices: <none>

At this point I’ve already confirmed that the partition table on all the RAID devices is consistent/identical by using the following commands.

parted /dev/sda print

parted /dev/sdb print

parted /dev/sdg print

parted /dev/sdh print

I’ve also already removed the errant drive before the last boot via

mdadm --manage /dev/md0 --remove /dev/sdc1

Now it’s time to add in the new drive.

root@blackbox:/home/eric# mdadm --manage /dev/md0 --add /dev/sdc1
mdadm: added /dev/sdc1
root@blackbox:/home/eric# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] 
md0 : active raid6 sdc1[8] sdg1[7] sdb1[6] sda1[5] sdh1[1]
 8790400512 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/4] [UU_UU]
 [>....................] recovery = 0.0% (401324/2930133504) finish=730.0min speed=66887K/sec
unused devices: <none>

As you can see there is a good bit of synchronization left to do here before the new drive can be put to use.


Meet Charles Clos: The Father of Clos Networks

Around Cumulus Networks we’re constantly talking about this thing called a Clos network. If you don’t know much about what those are, check out the following descriptions from Network World and Wikipedia:–what-s-old-is-new-again.html

They were a concept initially envisioned in the 1950’s by a gentleman working at Bell Labs named Charles Clos. Not a ton is known about Mr. Clos aside from the fact that his seminal paper  in “The Bell System Technical Journal ( Volume: 32, Issue: 2, March 1953 )” A study of non-blocking switching networks went on to make waves in the Data Center Computer Networking space many years later as it turns out non-blocking telephone networks have a lot to do with modern Data Center design pillars.

It bothered me that were no images to be found anywhere on the internet of Mr. Clos. So one day I decided I would get this mystery solved. I reached out to Bell Labs with an e-mail explaining my academic interest in Mr. Clos and asking for a picture. Before the end of that very day I received a response containing this image.

Internet, I would like to introduce you to Mr. Charles Clos…

10232017 - Scan - 171347
Description: three dimensional chart showing relationship of dial tone service marker occupancy and register occupancy in the #5 crossbar system l t r –mr r i wilkinson mrs r d leonard and mr c clos on the right.
Date: May 19, 1949

So there he is on the right, Charles Clos. I can sleep a little easier tonight putting a face to this much revered gentleman in the Data Center networking space.

Making Realistic Tombstones

Halloween is my favorite holiday of the year. To celebrate the occasion I try something a bit different each year. This year I had seen some interesting tutorials on making tombstones out of 4×8 foot 2 inch thick foam board. These days my biggest issue with all projects is having the time to complete them; knowing this I started this one in July.


The first step which I do not have a picture of was to cut the profile out for the tombstones. In our case we found that a single 4×8 sheet had enough material to construct about two whole tombstones. We did multiple layers, my spire tombstone was three layers thick while my wife’s traditional tombstone was two layers thick. Each of the main pieces was also surrounded by a base that was two layers thick as well.

At this point the time had come to put them together. For this I carved a C-channel in between the layers and adhered a piece of PVC pipe with construction adhesive in the channel. The PVC pipe would serve as a guide for the rebar which will hold these stones firmly in the ground.

Notice I left about 1/2″ of PVC pipe exposed. This was to be made flush by a bottom base layer made of wood. I used construction adhesive again to glue all of the layers together.


Here you can see I’ve cut a bottom sheet of MDF and used a paddle bit to make allowances for the extra 1/2 inch of PVC pipe to sit flush with the bottom edge. I used a heavy dose of construction adhesive again in between where the PVC conduit pokes through the bottom plate.

In the background of this photo you can also see the latex-based Drylock paint that was used as the primer coat for all the tombstones. I lathered this stuff on thick. You can get the white color from Home Depot for about $23/gallon and they’ll even throw some gray pigment in there too if you ask.

You can also see the 2 foot epoxy-coated rebar pieces I picked up from Home Depot as well. These should work well as they’ll provide a little extra protection from rust.


At this point I used the Stanley Sur-form shaper to even out all the edges and make the layers look as one. I also used a Dremel tool with the 565-02 attachment to make some nice even cuts to carve out the epitaphs for the tombstones. You can use a program like rasterizer to make images or words large enough to cover your tombstone.

With epitaphs carved and everything looking smooth, it was time to put on the first 2 layers of Drylock. You can see the entire family enjoyed this.


Here is a glamour shot of the first two coats of Drylock complete on the stones. I also slathered Drylock on the underside as well.


From here I just bought a few darker colors to fill-in the insets and reliefs in the tombstone to make it look a little more realistic. You can get by with a sample can from Home Depot and that should provide enough color for 1-2 tombstones.

Last step is to dig these bad boys into the ground and enjoy!


Adding New Fonts in Bulk to Ubuntu 16.04

This process should also work for 12.04 and 14.04.

Create a new directory under /usr/share/fonts

sudo mkdir /usr/share/fonts/opentype/newfonts

Place all OTF or TTF files in that directory.

Run the font-caching utility to fix permissions on these new fonts and make them available to applications immediately.

sudo chmod -R 655 /usr/share/fonts
sudo fc-cache -fv

Other Methods

There are other methods available on modern Ubuntu as well. For individual fonts you can just double-click on them and click the “install” option in the upper right. Or use a purpose-built program like font-manager.

sudo apt-get install font-manager

Building FRRouting for PowerPC on Debian Wheezy

Tried to do this to modernize the routing software running on an older whitebox which was built on the PowerPC architecture.

One of the challenges on these platforms aside from the PPC arch is the limited space. I found my switch did not have enough hard disk space to complete the build. My answer was to use a USB stick to provide additional disk space to complete the build. At the completion of the build my build directory consumed ~214 MB so plan accordingly if your switch does not have sufficient on-board space.

Assume ROOT for all commands unless otherwise stated.

I mounted my USB stick to –> /mnt/USB

mkdir /mnt/USB
# Use Fdisk to confirm USB device.
fdisk -l 
mount /dev/sda1 /mnt/USB

Add the sources

cat << EOT >> /etc/apt/sources.list
deb wheezy main contrib non-free
deb-src wheezy main contrib non-free

deb wheezy/updates main contrib non-free
deb-src wheezy/updates main contrib non-free

deb wheezy-updates main contrib non-free
deb-src wheezy-updates main contrib non-free

deb wheezy-backports main non-free contrib

Add the Prereq packages

apt-get install git autoconf automake libtool make gawk libreadline-dev texinfo dejagnu pkg-config libpam0g-dev bison flex python-pytest libc-ares-dev python3-dev libjson-c-dev build-essential fakeroot devscripts


Install some out of Repo Prereqs from Source as shown in the Ubuntu 12.04 LTS build guide

Install newer bison from Ubuntu 14.04 package source:

mkdir builddir
cd builddir
tar -jxvf bison_3.0.2.dfsg.orig.tar.bz2 
cd bison-3.0.2.dfsg/
tar xzf ../bison_3.0.2.dfsg-2.debian.tar.gz 
sudo apt-get build-dep bison
debuild -b -uc -us
cd ..
sudo dpkg -i ./libbison-dev_3.0.2.dfsg-2_amd64.deb ./bison_3.0.2.dfsg-2_amd64.deb 
cd ..
rm -rf builddir

Install newer version of autoconf and automake:

tar xvf autoconf-2.69.tar.gz
cd autoconf-2.69
./configure --prefix=/usr
sudo make install
cd ..

tar xvf automake-1.15.tar.gz
cd automake-1.15
./configure --prefix=/usr
sudo make install
cd ..

Add frr groups and user

sudo groupadd -g 92 frr
sudo groupadd -r -g 85 frrvty
sudo adduser --system --ingroup frr --home /var/run/frr/ \
   --gecos "FRR suite" --shell /sbin/nologin frr
sudo usermod -a -G frrvty frr

Download Source, configure and compile it

git clone frr
cd frr
./configure \
    --prefix=/usr \
    --enable-exampledir=/usr/share/doc/frr/examples/ \
    --localstatedir=/var/run/frr \
    --sbindir=/usr/lib/frr \
    --sysconfdir=/etc/frr \
    --enable-pimd \
    --enable-watchfrr \
    --enable-ospfclient=yes \
    --enable-ospfapi=yes \
    --enable-multipath=64 \
    --enable-user=frr \
    --enable-group=frr \
    --enable-vty-group=frrvty \
    --enable-configfile-mask=0640 \
    --enable-logfile-mask=0640 \
    --enable-rtadv \
    --enable-fpm \
    --with-pkg-git-version \
make install

Most guides would end here but there’s a bit more required to get FRR functioning.

Create empty FRR configuration files

sudo install -m 755 -o frr -g frr -d /var/log/frr
sudo install -m 775 -o frr -g frrvty -d /etc/frr
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/zebra.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/bgpd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/ospfd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/ospf6d.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/isisd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/ripd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/ripngd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/pimd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/ldpd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/nhrpd.conf
sudo install -m 640 -o frr -g frrvty /dev/null /etc/frr/vtysh.conf

Install the init.d service

sudo install -m 755 tools/frr /etc/init.d/frr
sudo install -m 644 tools/etc/frr/daemons /etc/frr/daemons
sudo install -m 644 tools/etc/frr/daemons.conf /etc/frr/daemons.conf
sudo install -m 644 -o frr -g frr tools/etc/frr/vtysh.conf /etc/frr/vtysh.conf

Enable your Routing Daemons

cat << EOT > /etc/frr/daemons

Start FRR

service frr start
service frr status

Enable FRR At boot time for subsequent reboots

sudo update-rc.d frr defaults

Fix Exit Scripts

 sed -i 's/ip route flush proto ripng/ip route flush proto 190 \# ripng/' /usr/lib/frr/frr
 sed -i 's/ip route flush proto bgp/ip route flush proto 186 \# bgp/' /usr/lib/frr/frr
 sed -i 's/ip route flush proto isis/ip route flush proto 187 \# isis/' /usr/lib/frr/frr
 sed -i 's/ip route flush proto ospf/ip route flush proto 188 \# ospf/' /usr/lib/frr/frr
 sed -i 's/ip route flush proto rip/ip route flush proto 189 \# rip/' /usr/lib/frr/frr
 sed -i 's/ip route flush proto static/ip route flush proto 191 \# static/' /usr/lib/frr/frr

Hopefully that should do it for you. Now the next step is figuring out how to build a proper deb from the source. I’ll leave that process for next time 🙂

Securely Wiping a Hard Drive

When getting rid of a hard drive, I, like everyone else like to be secure about it. After collecting my data I usually like to overwrite the drive with garbage. In the past I used to just use the basic DD approach to zero the drive out.

dd if=/dev/zero of=/dev/sd<DRIVE>

This works fine but I started hearing rumors of being able to recover data from a zeroed out drive. Indeed this is partially true. Zeroing out is probably sufficient in most cases.

Ideally, I would write data out from /dev/random or /dev/urandom (whatever your system has) but the amount of entropy that is harnessed here is not enough to saturate the write speed of the drive meaning that it will take forever a very long time. Never the less I was curious to find out about another option to wipe a drive.

This approach uses OpenSSL with seed data from /dev/urandom. Supposedly it is possible to generate about 1.5gbps of garbage data with this technique… I’ll never know though because the write speed of my drive is nowhere near that.


Use the following command to randomize the drive/partition using a randomly-seeded AES cipher from OpenSSL (displaying the optional progress meter with pv):

# openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt </dev/zero \
    | pv -bartpes <DISK_SIZE> | dd bs=64K of=/dev/sd"X"

where the (optional) total disk size in bytes (DISK_SIZE) may be obtained via:

# blockdev --getsize64 /dev/sd"X"

Sidenote: I love the use of PV here, this is an underloved utility that is truly awesome; I’ve only ever used it in one other place. TARing a remote file over SSH for delivery on my local machine (shown below).

ssh –c blowfish user@host  "tar cjpf - /home/user/file" | pv | cat > ./file.tar.bz2

Installing A New Car Stereo in 2007-2009 Hyundai Elantra

This guide should apply to all Hyundai Elantras from 2007-2009.

From stock to Awesome!

This slideshow requires JavaScript.

Parts required:

Materials Required:

  • Soldering Iron and Solder
  • Electrical Tape


Here is the starting point.

Step 1). Loosen the Surrounding Fascia

Showing the fascia that is going to be removed.

Note the location of the gray clips below on the backside of the fascia, prying up around these points easiest. I had the most luck working on the lower left and right corners as a starting point.

This picture shows the bac side of the fascia. You can see the 8 gray clips along the top and bottom. At this point I had lost two of them in the dash in the lower right corner of the picture.

Step 2). Remove Wire Harnesses from Fascia

Once the fascia itself is loose the 3 cables that connect to it need to be removed. Start by depressing the clip which connects to the back of the hazard lights. Shown in the photo below.


You can see the wire harness connecting to the back of the hazard lights here.

Next remove the two wiring harnesses that connect to the clock (black wires) and passenger airbag sign (yellow wires) above the radio.


This picture looks up from the bottom of the radio with the fascia loosened. Notice the two wire harnesses above the radio.

Step 3). Remove the Stock Radio

Unscrew the four screws which surround the unit.

Here is the radio with the fascia removed.

Disconnect the wire harness and the antenna cable from the back of the radio.

Looking at the cables from the right side of the radio.

Step 4). Prepare the Adapter Cable

At this point I took the Metra Wire Harness adapter cable and started soldering the wires to the new wiring harness provided by the new headunit. Which wires will need to actually be connected will vary based on the capabilities of your headunit.

When looking at the photo below you can see the adapter cable beneath the stock wiring harness that stretches over the black adapter which plugs into the back of the new headunit. I took the liberty of wrapping all my wires in the harness together with a little electrical tape but you don’t HAVE to do that, it’s just a nice touch.

Showing the stock wiring harness and the adapter from metra along with the wiring harness provided by the headunit.

Step 5). Install the New Headunit

Connect the wiring harness adapter from metra into the stock wire harness. Then screw in your mounting bracket from metra using the four screw which previously held the old radio in place.

I choose to mount my new head unit using the ISO mount technique which is a bit simpler than the classic DIN method (which employs the metallic cage surrounding the headunit which needs to have some pins bent down to hold it in place). Using the ISO technique requires the removal of the DIN cage from the new headunit and attaching the side rails included in the metra mounting bracket kit. At this point, connect the cables to the back of the new headunit and slide it into place.

I choose to run an additional microphone cable that was included with my headunit for handsfree calling but don’t have any pictures of that step.

At this point you can replace the fascia and then add the trim bracket which surrounds the headunit.