Prologin’s 2014 setup¶
Authors:
Marin Hannache
Paul Hervot
Pierre Bourdon
Rémi Audebert
Overview¶
We had 2 rooms:
Pasteur, ~ 96 machines, 95% of them working
Masters, ~ 42 machines, 90% of them working
We did not setup an alternate cluster because we had enough computing power with all the i7.
Electrical setup¶
We asked an electrician to setup Masters for ~40 machines.
Hardware setup¶
All machines were of i7 generation.
We used the machines already in Pasteur.
We moved machines from SM14 into Masters, used screens, keyboards and mice from the storage room of the Bocal. Some SM14 machines were used a server and stored in Pasteur. Each machine we took from SM14 had to put back exactly at the same location se labeled accordingly.
We bought 10 500Go disks, 2 for each RHFS in RAID 1. The EPITA ADM lent us 4 1To racks that we used for the other servers: gw, web, misc1 and misc2.
Note
We bought the same brand for all the disks, it is not a good idea to do that. If a disk from the batch is faulty, then it is pretty certain that the other are too. We should have bought disks from different manufacturers.
Network setup¶
Pasteur¶
There were 7 switches in Pasteur and 4 boxes to hold them. Each switch’s uplink was wired directly back to The Bocal but they could not setup a proper VLAN so they brought a 24 port gibabit switch, removed uplinks from every switch and wired them to this one.
Masters¶
We borrowed a 48 port Gigabit switch (HP Procurve 2848) from the LSE and satreix lent us his little 16 port Gbit switch. 3/4 of the room was on the 48 port switch and 1/4 was on the other one.
The link between Pasteur and Masters was done by a custom cable setup by the Bocal.
Wifi for organizers¶
We used a TP-LINK 703n and bridged the WLAN and LAN.
MAC addresses for the organizers’ machines were added to mdb with an IP on the services range.
Services organization¶
GW:
bind
dhcpd
firewall
mdb
netboot
udb
postgresql database for hfs
Web:
concours
postgresql database for concours
redmine
map
misc1:
minecraft
collectd
graphite
dj_ango
RHFS:
rhfs01 (pasteur)
rhfs23 (pasteur)
rhfs45 (pasteur)
rhfs67 (masters)
The gate lock¶
There should be another article on the subject.
Issues encountered during the event¶
Bad network setup¶
We asked for the network to be setup such as all links were on the same VLAN and no dhcp server. Our gateway were to route the packets to the Bocal’s gateway.
Instead, no VLAN was setup, all uplinks were disconnected and all the switches were connected to another Gigabit switch. Because we wanted to have an uplink, we had to add another nic to our gateway and connect it to another network, then route the packets from one interface to another.
Some of the iptables rules we used are in the cookbook.
Switch failure¶
4~6 hours after the beginning of the event a switch suddenly stopped forwarding packets. After quick checks we diagnosed a hardware problem, and asked the contestants to go to another spot in the machine room.
We rebooted the switch and disconnected every cable from it and started looking for the one that was giving us trouble. At some point it started to work again, and did not fail thereafter. The only cables we did not connect were the uplink, the IP phone and a strange PoE camera.
Services misconfigurations¶
mdbDNS misconfiguration: a machine was inserted with a bad hostname (it contained a ‘
_
’), causing bind to fail reading the configuration file.mdb and DHCP misconfiguration: the MAC address of a machine is used as the primary key, modifying it is like creating another entry in the table. For mdb is added another machine with the same hostname but with another MAC address.
Fix: Remove the offending entry from the database.
Ethernet flow control¶
One RHFS was flooding the neighboors with pause packets, causing the NBD/NFS to be really slow and make the machines freeze.
Fix: ethtool --pause autoneg off rx off rx off
References:
Bad NTP server¶
We did not edit ntp configuration on the rfs root so it was trying to contact
0.pool.ntp.org
instead of gw.prolo
.
Fix: pssh on all machines “ntpdate gw && find /home -print0 | xargs -0 touch”
Cookbook¶
Here are the tools, techniques, and knowledge we used to setup and run everything.
LLDP¶
The switches broadcasted LLDP packets to every machines connected to them. It contains, among other things, the name of the switch and the port to wich the link is connected. We used those packets to know where each machine was connected, and select the closest RHFS.
Note
Not all the switches sent those packets.
Reloading LVM/RAID devices¶
# If using LVM, remove logical volumes
$ dmsetup remove /dev/mapper/<NAME>
# Deactivate MD device
$ mdadm --stop /dev/mdXXX
# Scan for hotplugged/swapped disks
$ for d in /sys/class/scsi_host/host*/scan; do echo '- - -' > $d; done
# Rescan for RAID devices
$ mdadm --assemble --scan
iptables and ipset¶
We used ipset to implement ip-based filtering.
Sample usage:
$ ipset -! create allowed-internet-access bitmap:ip range 192.168.0.0/23
$ ipset add allowed-internet-access 192.168.0.42
$ ipset flush allowed-internet-access
# Allow packets having src in the set
$ iptables -A FORWARD -m set --match-set allowed-internet-access src -j ACCEPT
Sample rules:
# Rewrite packets going out of interface lan
$ iptables -t nat -A POSTROUTING -o lan -j MASQUERADE
# Allow packets coming from 192.168.1.0/24 to go out
$ iptables -A FORWARD -s 192.168.1.0/24 -j ACCEPT
# Black list a set of IP to access port 80
$ iptables -A INPUT -i lan -p tcp --destination-port 80 -m set --match-set allowed-internet-access src -j DROP
# Allow packets in an already established connection
$ iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
Eggdrop’s latency fixes¶
By default eggdrop added fakelag to the motus modules, we removed it by patching the binary at runtime.