Meet Charles Clos: The Father of Clos Networks

Around Cumulus Networks we’re constantly talking about this thing called a Clos network. If you don’t know much about what those are, check out the following descriptions from Network World and Wikipedia:

https://www.networkworld.com/article/2226122/cisco-subnet/clos-networks–what-s-old-is-new-again.html

https://en.wikipedia.org/wiki/Clos_network

They were a concept initially envisioned in the 1950’s by a gentleman working at Bell Labs named Charles Clos. Not a ton is known about Mr. Clos aside from the fact that his seminal paper  in “The Bell System Technical Journal ( Volume: 32, Issue: 2, March 1953 )” A study of non-blocking switching networks went on to make waves in the Data Center Computer Networking space many years later as it turns out non-blocking telephone networks have a lot to do with modern Data Center design pillars.

It bothered me that were no images to be found anywhere on the internet of Mr. Clos. So one day I decided I would get this mystery solved. I reached out to Bell Labs with an e-mail explaining my academic interest in Mr. Clos and asking for a picture. Before the end of that very day I received a response containing this image.

Internet, I would like to introduce you to Mr. Charles Clos…

10232017 - Scan - 171347
Description: three dimensional chart showing relationship of dial tone service marker occupancy and register occupancy in the #5 crossbar system l t r –mr r i wilkinson mrs r d leonard and mr c clos on the right.
Date: May 19, 1949

So there he is on the right, Charles Clos. I can sleep a little easier tonight putting a face to this much revered gentleman in the Data Center networking space.

Advertisements

Installing the Newest Ifupdown2 on an OrangePi Nano

I’ve been pretty enamored with my OrangePi nanos since I first got one. So enamored in fact that I’m up to owning 5 of them now doing all matters of tasks. Being a network person I wanted to make sure I had some of the best interface configuration software available installed so naturally I wanted Ifupdown2.

The OrangePi nano runs a debian based version of software called Armbian which is truly awesome software. It has been stripped down and customized for specific devices to the point that it is a work of art. Since it is debian based it has access to ifupdown2 natively right in the repos. The only problem with that version is that it is outdated being from the November 2015 timeline. So I want to install the latest and greatest Ifupdown2….

From my armbian device:

# Install the newest version in the standard repo
sudo apt-get update -y
sudo apt-get install ifupdown2 -qy

# Now install the newest version of ifupdown2 directly from the debian repos
wget -O /root/ifupdown2.deb http://ftp.us.debian.org/debian/pool/main/i/ifupdown2/ifupdown2_1.0~git20170314-1_all.deb && \
dpkg -i /root/ifupdown2.deb && \
rm -rfv /root/ifupdown2.deb

sudo apt-cache policy ifupdown2 | grep Installed
echo "Output above should say: \"Installed: 1.0~git20170314-1\""

# Overwrite NMCLI tool to control the Eth0 interface with Ifupdown2
echo "[keyfile] unmanaged-devices=interface-name:eth0" | sudo tee -a /etc/NetworkManager/NetworkManager.conf

echo "### Before Change ###"
sudo nmcli dev status
sudo systemctl stop NetworkManager; sudo systemctl start NetworkManager

echo "### After Change ###"
sudo nmcli dev status
echo "Eth0 should now show as \"unmanaged\" according to the output above."

 

Using Active/Backup Bonding (mode 1) with Ifupdown2

Ifupdown2 is a very useful interface configuration utility with tons of enhancements over the stock utility ifupdown. It was built with a specific initial use-case in mind which is for use on network operating systems (NOS) like Cumulus Linux. Cumulus requires LACP support as the primary bonding method. Other modes like active-backup (mode 1) were not initially fully implemented if ifupdown2. This is changing however; CM-14985 brings support for the bond-primary keyword and will be present in the next release of Cumulus Linux and the next version of Ifupdown2.

To hold you over until then here’s a workaround I’ve been using on my server at home running Ifupdown2 for performing active/backup bonding. Writing the sys file directly can provide the same behavior.

auto lo
iface lo inet loopback

auto enp4s0
iface enp4s0
 alias Motherboard Ethernet 
 mtu 9194

auto enxf01e341f95
iface enxf01e341f95
 alias USB3 Ethernet
 mtu 9194

auto bond0
iface bond0
 alias ActiveBackup Uplink
 bond-mode active-backup
 bond-slaves enxf01e341f95 enp4s0
 address 192.168.1.10/24
 gateway 192.168.1.1
 mtu 9194
 pre-up echo enp4s0 > /sys/class/net/bond0/bonding/primary

Building FRRouting for PowerPC on Debian Wheezy

Tried to do this to modernize the routing software running on an older whitebox which was built on the PowerPC architecture.

One of the challenges on these platforms aside from the PPC arch is the limited space. I found my switch did not have enough hard disk space to complete the build. My answer was to use a USB stick to provide additional disk space to complete the build. At the completion of the build my build directory consumed ~214 MB so plan accordingly if your switch does not have sufficient on-board space.

Assume ROOT for all commands unless otherwise stated.

I mounted my USB stick to –> /mnt/USB

mkdir /mnt/USB
# Use Fdisk to confirm USB device.
fdisk -l 
mount /dev/sda1 /mnt/USB

Add the sources

cat << EOT >> /etc/apt/sources.list
deb http://httpredir.debian.org/debian/ wheezy main contrib non-free
deb-src http://httpredir.debian.org/debian/ wheezy main contrib non-free

deb http://security.debian.org/ wheezy/updates main contrib non-free
deb-src http://security.debian.org/ wheezy/updates main contrib non-free

deb http://httpredir.debian.org/debian/ wheezy-updates main contrib non-free
deb-src http://httpredir.debian.org/debian/ wheezy-updates main contrib non-free

deb http://ftp.debian.org/debian/ wheezy-backports main non-free contrib
EOT

Add the Prereq packages

apt-get install git autoconf automake libtool make gawk libreadline-dev texinfo dejagnu pkg-config libpam0g-dev bison flex python-pytest libc-ares-dev python3-dev libjson-c-dev build-essential fakeroot devscripts

 

Install some out of Repo Prereqs from Source as shown in the Ubuntu 12.04 LTS build guide

Install newer bison from Ubuntu 14.04 package source:

mkdir builddir
cd builddir
wget http://archive.ubuntu.com/ubuntu/pool/main/b/bison/bison_3.0.2.dfsg-2.dsc
wget http://archive.ubuntu.com/ubuntu/pool/main/b/bison/bison_3.0.2.dfsg.orig.tar.bz2
wget http://archive.ubuntu.com/ubuntu/pool/main/b/bison/bison_3.0.2.dfsg-2.debian.tar.gz
tar -jxvf bison_3.0.2.dfsg.orig.tar.bz2 
cd bison-3.0.2.dfsg/
tar xzf ../bison_3.0.2.dfsg-2.debian.tar.gz 
sudo apt-get build-dep bison
debuild -b -uc -us
cd ..
sudo dpkg -i ./libbison-dev_3.0.2.dfsg-2_amd64.deb ./bison_3.0.2.dfsg-2_amd64.deb 
cd ..
rm -rf builddir

Install newer version of autoconf and automake:

wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz
tar xvf autoconf-2.69.tar.gz
cd autoconf-2.69
./configure --prefix=/usr
make
sudo make install
cd ..

wget http://ftp.gnu.org/gnu/automake/automake-1.15.tar.gz
tar xvf automake-1.15.tar.gz
cd automake-1.15
./configure --prefix=/usr
make
sudo make install
cd ..

Add frr groups and user

sudo groupadd -g 92 frr
sudo groupadd -r -g 85 frrvty
sudo adduser --system --ingroup frr --home /var/run/frr/ \
   --gecos "FRR suite" --shell /sbin/nologin frr
sudo usermod -a -G frrvty frr

Download Source, configure and compile it

git clone https://github.com/frrouting/frr.git frr
cd frr
./bootstrap.sh
./configure \
    --prefix=/usr \
    --enable-exampledir=/usr/share/doc/frr/examples/ \
    --localstatedir=/var/run/frr \
    --sbindir=/usr/lib/frr \
    --sysconfdir=/etc/frr \
    --enable-pimd \
    --enable-watchfrr \
    --enable-ospfclient=yes \
    --enable-ospfapi=yes \
    --enable-multipath=64 \
    --enable-user=frr \
    --enable-group=frr \
    --enable-vty-group=frrvty \
    --enable-configfile-mask=0640 \
    --enable-logfile-mask=0640 \
    --enable-rtadv \
    --enable-fpm \
    --with-pkg-git-version \
    --with-pkg-extra-version=-MyOwnFRRVersion   
make
make install

Most guides would end here but there’s a bit more required to get FRR functioning.

Create empty FRR configuration files

sudo install -m 755 -o frr -g frr -d /var/log/frr
sudo install -m 775 -o frr -g frrvty -d /etc/frr
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/zebra.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/bgpd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/ospfd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/ospf6d.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/isisd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/ripd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/ripngd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/pimd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/ldpd.conf
sudo install -m 640 -o frr -g frr /dev/null /etc/frr/nhrpd.conf
sudo install -m 640 -o frr -g frrvty /dev/null /etc/frr/vtysh.conf

Install the init.d service

sudo install -m 755 tools/frr /etc/init.d/frr
sudo install -m 644 tools/etc/frr/daemons /etc/frr/daemons
sudo install -m 644 tools/etc/frr/daemons.conf /etc/frr/daemons.conf
sudo install -m 644 -o frr -g frr tools/etc/frr/vtysh.conf /etc/frr/vtysh.conf

Enable your Routing Daemons

cat << EOT > /etc/frr/daemons
zebra=yes
bgpd=yes
ospfd=no
ospf6d=no
ripd=no
ripngd=no
isisd=no
pimd=no
ldpd=no
nhrpd=no
eigrpd=no
babeld=no
EOT

Start FRR

service frr start
service frr status

Enable FRR At boot time for subsequent reboots

sudo update-rc.d frr defaults

Fix Exit Scripts

 sed -i 's/ip route flush proto ripng/ip route flush proto 190 \# ripng/' /usr/lib/frr/frr
 sed -i 's/ip route flush proto bgp/ip route flush proto 186 \# bgp/' /usr/lib/frr/frr
 sed -i 's/ip route flush proto isis/ip route flush proto 187 \# isis/' /usr/lib/frr/frr
 sed -i 's/ip route flush proto ospf/ip route flush proto 188 \# ospf/' /usr/lib/frr/frr
 sed -i 's/ip route flush proto rip/ip route flush proto 189 \# rip/' /usr/lib/frr/frr
 sed -i 's/ip route flush proto static/ip route flush proto 191 \# static/' /usr/lib/frr/frr

Hopefully that should do it for you. Now the next step is figuring out how to build a proper deb from the source. I’ll leave that process for next time 🙂

Fixing LLDPd Default PortID Configuration

LLDPd is a great program written by Vincent Bernat which implements 802.1ab for unix based OSes.

https://vincentbernat.github.io/lldpd/

Debian and Ubuntu offer this lovely utility right in their default software repositories meaning that is only an apt-get install away. One caveat is that the default configuration provides the MAC address of the interface instead of the interface name which is unusual compared to the implementation of LLDP in use on most network OS vendors like Cumulus, Cisco, Juniper, Arista etc…

Here is what is sent by default as seen by the far side of the link:

-------------------------------------------------------------------------
LLDP neighbors:
-------------------------------------------------------------------------
Interface: swp4, via: LLDP, RID: 2, Time: 0 day, 00:00:12
 Chassis: 
 ChassisID: mac 52:54:00:b9:e3:98
 SysName: server1
 SysDescr: Ubuntu 16.04 LTS Linux 4.4.0-22-generic #40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016 x86_64
 TTL: 120
 MgmtIP: 192.168.121.53
 MgmtIP: fe80::5054:ff:feb9:e398
 Capability: Bridge, off
 Capability: Router, off
 Capability: Wlan, off
 Capability: Station, on
 Port: 
 PortID: mac 44:38:39:00:00:06
 PortDescr: eth1
 PMD autoneg: supported: yes, enabled: yes
 Adv: 10Base-T, HD: yes, FD: yes
 Adv: 100Base-TX, HD: yes, FD: yes
 Adv: 1000Base-T, HD: no, FD: yes
 MAU oper type: 1000BaseTFD - Four-pair Category 5 UTP, full duplex mode
-------------------------------------------------------------------------

Luckily there are knobs to change that behavior. The configuration below will set LLDPd to provide a more similar configuration like a network device might expect.

sudo apt-get install lldpd
sudo bash -c "echo 'configure lldp portidsubtype ifname' > /etc/lldpd.d/port_info.conf"
sudo systemctl restart lldpd.service

 

Here is the net result of the changed configuration:

-------------------------------------------------------------------------
Interface: swp4, via: LLDP, RID: 2, Time: 0 day, 00:00:21
 Chassis: 
 ChassisID: mac 52:54:00:b9:e3:98
 SysName: server1
 SysDescr: Ubuntu 16.04 LTS Linux 4.4.0-22-generic #40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016 x86_64
 TTL: 120
 MgmtIP: 192.168.121.53
 MgmtIP: fe80::5054:ff:feb9:e398
 Capability: Bridge, off
 Capability: Router, off
 Capability: Wlan, off
 Capability: Station, on
 Port: 
 PortID: ifname eth1
 PortDescr: eth1
 PMD autoneg: supported: yes, enabled: yes
 Adv: 10Base-T, HD: yes, FD: yes
 Adv: 100Base-TX, HD: yes, FD: yes
 Adv: 1000Base-T, HD: no, FD: yes
 MAU oper type: 1000BaseTFD - Four-pair Category 5 UTP, full duplex mode
-------------------------------------------------------------------------

Handling SSH Protocol Links in Chrome (on Linux)

As a network engineer, I have been annoyed by not being able to click on SSH links in webpages for years while running Linux.

After digging in a bit I was able to find this solution which works for SSH in Chrome.

I’m running Ubuntu 16.04 in my setup.

My SSH URL links look like this:

ssh:///user@host:port

Here is the process I used to get these links working.

#Check which handler is setup for SSH
xdg-mime query default x-scheme-handler/ssh

#Write code for new handler
cat << EOF > ~/.local/share/applications/ssh.desktop
[Desktop Entry]
Version=1.0
Name=SSH Launcher
Exec=bash -c '(URL="%U" HOST=\$(echo \${URL:6} | cut -d ":" -f1) PORT=$(echo \${URL:6} | cut -d ":" -f2); ssh \$HOST -p \$PORT); bash'
Terminal=true
Type=Application
Icon=utilities-terminal
EOF

#Apply New handler for SSH
xdg-mime default ssh.desktop x-scheme-handler/ssh

#Confirm new handler has been applied
xdg-mime query default x-scheme-handler/ssh

Udev Network Interface Renaming with no Reboot

While working on the Topology_Converter for work I came upon several lessons with Udev. The topology_converter project essentially takes input (from a graphiviz file) and builds a network topology with proper interface names. In order to make the interface names work there is a script which spits out udev rules.

Writing Udev Rules

With Udev you can rename interfaces using a number of parameters which are defined in rules. Rules should be stuck in the “/etc/udev/rules.d/70-persistent-net.rules” file to follow convention but you could technically stick them anywher in the rules.d directory.

To see all of the possible criteria that can be matched upon for a given network interface, use the command below replacing “eth0” with your interface of choice.

udevadm info -a -p /sys/class/net/eth0

Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.

looking at device '/devices/pci0000:00/0000:00:19.0/net/eth0':
 KERNEL=="eth0"
 SUBSYSTEM=="net"
 DRIVER==""
 ATTR{addr_assign_type}=="0"
 ATTR{addr_len}=="6"
 ATTR{address}=="54:ee:75:22:3d:70"
 ATTR{broadcast}=="ff:ff:ff:ff:ff:ff"
 ATTR{carrier}=="0"
 ATTR{carrier_changes}=="1"
 ATTR{dev_id}=="0x0"
 ATTR{dev_port}=="0"
 ATTR{dormant}=="0"
 ATTR{duplex}=="unknown"
 ATTR{flags}=="0x1003"
 ATTR{gro_flush_timeout}=="0"
 ATTR{ifalias}==""
 ATTR{ifindex}=="2"
 ATTR{iflink}=="2"
 ATTR{link_mode}=="0"
 ATTR{mtu}=="1500"
 ATTR{netdev_group}=="0"
 ATTR{operstate}=="down"
 ATTR{proto_down}=="0"
 ATTR{speed}=="-1"
 ATTR{tx_queue_len}=="1000"
 ATTR{type}=="1"

looking at parent device '/devices/pci0000:00/0000:00:19.0':
 KERNELS=="0000:00:19.0"
 SUBSYSTEMS=="pci"
 DRIVERS=="e1000e"
 ATTRS{broken_parity_status}=="0"
 ATTRS{class}=="0x020000"
 ATTRS{consistent_dma_mask_bits}=="64"
 ATTRS{d3cold_allowed}=="1"
 ATTRS{device}=="0x15a2"
 ATTRS{dma_mask_bits}=="64"
 ATTRS{driver_override}=="(null)"
 ATTRS{enable}=="1"
 ATTRS{irq}=="56"
 ATTRS{local_cpulist}=="0-3"
 ATTRS{local_cpus}=="0f"
 ATTRS{msi_bus}=="1"
 ATTRS{numa_node}=="-1"
 ATTRS{subsystem_device}=="0x2227"
 ATTRS{subsystem_vendor}=="0x17aa"
 ATTRS{vendor}=="0x8086"

looking at parent device '/devices/pci0000:00':
 KERNELS=="pci0000:00"
 SUBSYSTEMS==""
 DRIVERS==""

</code>

You can see there are quite a few options to match on. When remapping physical interfaces on Linux, I strongly recommend adding the match for PCI to make sure this interface is mapped to the PCI bus in some way. The concern when not using the PCI match (as shown below) is that if these physical interfaces are to take part in bridges or bonds with vlans or sub interfaces…. in that case your bridge or bond may inherit mac addresses from a physical interface and there will be a collision in the renaming process which means your interfaces may be left named “renameXX” or something like that.

Here are some sample Udev rules for a given series of interface renaming operations.

#### UDEV Rules (/etc/udev/rules.d/70-persistent-net.rules) ####
ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="44:38:39:00:00:1a", NAME="swp2", SUBSYSTEMS=="pci" 
ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="44:38:39:00:00:12", NAME="swp1", SUBSYSTEMS=="pci" 
ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="44:38:39:00:00:49", NAME="swp48", SUBSYSTEMS=="pci" 
ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="44:38:39:00:00:42", NAME="eth0", SUBSYSTEMS=="pci" 
ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="08:00:27:8a:39:05", NAME="vagrant", SUBSYSTEMS=="pci"

Applying the New Rules

Now that you’ve written the new rules it’d be nice to apply them without having to reboot.

EDIT: In Ubuntu 16.04 you have another option.

systemctl restart systemd-udev-trigger.service

 

That can be a little complicated and is totally disruptive to networking traffic likely on all interfaces but the procedure looks like this:

  1. Detect the driver used by each interface that requres a remap. The easiest way is to use
    $ ethtool -i eth0
    driver: e1000e
    version: 3.2.6-k
    firmware-version: 0.2-4
    expansion-rom-version: 
    bus-info: 0000:00:19.0
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: yes
    supports-register-dump: yes
    supports-priv-flags: no

    You could potentially use this technique:

    $ basename $(readlink /sys/class/net/eth0
    /device/driver/module)

    or this one:

    $ basename $(readlink /sys/class/net/+interface+/device/driver)

    YMMV  depending on the driver in use.

  2. Remove the driver that is shared/used by each interface that is to be remapped (other interfaces that are using that driver may get caught in the crossfire here).
    $ modprobe -r e1000e
  3. Run the following command to detect the newly installed rules
    $ udevadm control --reload-rules
  4. Apply the new rules with the last command
    $ udevadm trigger

Applying the new rules with the trigger operation will also reinitialize the driver that you’ve previously removed.

Presto you’re done.