Matt McCarrell - MP451 (May 2009 - Dec. 2009)
Projects
Dom0s
One of my major projects last semester was building the new Xen dom0s that were built for hosting VMs in the labs. At the end of last semester, I discovered various hard drives issues on each of the new systems (DOA, SMART errors, IDE). After discovering these problems, I convinced everyone that the best course of action was to replace the drives with some new SATA drives. Afterwards, I rebuilt the systems and proceeded to migrate all of the VMs off of our old setup over to the new systems. The transition to the new systems went fairly smoothly and only resulted in a few hours of downtime for most of the VMs. The new systems have been working fairly well except that I did experience some random crashes on both xen1 and xen2 a few times but haven't experience one in about 3 months or so.
Mirror
One of my other major projects last semester was getting the new mirror server up and running. Shortly after last semester ended, I finished syncing all of the things that we mirrored at the time and moved the new system into production. So far, everything has been working fairly well but I have had to make several changes over the past 6 months. I have contacted several other mirror admins (RIT, Utah, Georgia Tech) and have set up agreements with them to mirror a lot of our content from their mirrors due to our bandwidth limitations on CU's commodity Internet connection. Through these agreements, I have added things like Debian, Gentoo, Knoppix, Mozilla, OpenOffice.org, and the Ubuntu ISOs to our mirror. Our mirror is even listed as an official mirror for several things, including CentOS, Ubuntu, Fedora, and GNU. I also recently made an official page/site for mirror to provide more information than a basic directory listing and I've added directories to host things like the Xen Live CDs and some Chromium OS VMs that Todd has created. In addition, I'm always making sure OIT gives us as much outbound bandwidth as possible (currently 50M to the Internet & 20M to the I2) and I've written a script that emails me details about daily changes in directory size changes and posts the output to a page on mirror.
Animal / Backups
Rebuilding animal was also one of my other major projects over the last 6 months. Originally, animal use to be the primary system that held all of our VMs; however, animal is now used primarly for backups since all of the VMs are now stored locally on each dom0. It took about two weeks back in May for me to figure out how to get the RAID units configured for RAID 10 and to get CentOS installed. After that, getting backups set up was just a matter of writting a few scripts to perform the backups on the remote systems and prepare the directories on animal. I also decided to set up another system (PLBackup1) and have it reside down in the CS Projects Lab to hold an extra copy of everything that is on animal. Recently (this week), I had to rebuild animal (with help from Zach & Todd) due to a failure with some of the drives in the raid unit. This caused us to not have daily backups for about two weeks but everything should be good to go when backups run tonight.
Generic VMs
Another major project that I'm still working on is a set of Generic VMs which are used to produce new VMs on our dom0s in the labs. So far, I've really enjoyed working on this a lot because I got to write some scripts that use a variety of commands to produce new VMs based of a generic version of a preconfigured VM. I currently have developed a Ubuntu Generic VM that has already been used to create several new VMs (web2, git, print, etc.) and I keep it fairly up-to-date. I'm still in the process of creating a CentOS Generic VM; however, I have managed to created the base VM and all that really remains is for me to write the duplication script. I also need to clean up and simplify the scripts a bit but that likely won't happen for a bit. I'm also considering making a generic Debian VM since it would be fairly trivial since I could likely use the Ubuntu script.
Web & MySQL VMs
Another one of my smaller projects has been working on the web1, web2, and mysql VMs. For web1, I've mainly just been doing routine maintenance and adding sites like the Admin. Portal as needed. In the case of web2, I actually set up this VM over the summer to support running sites which require Ubuntu for the host OS, require a newer version of PHP, or those which might be considered a security risk. As of right now, web2 is running the StatusNet for the labs and supports the lab build repository that Mark is using for the COSI lab build. The mysql VM is even getting used as well and currently supports the StatusNet and Gitorious setups in the labs.
Server Management
In regards to general management of servers in the labs, I haven't done too much per say (excluding items above). I've mainly just been making sure that the systems I maintain get patched when new updates are released (see StatusNet) by watching the CentOS and Ubuntu mailing lists. I've also done some other things like updating the hostnames on the CentOS systems, updating the Server Room Shutdown Procedure, and have been attempting to maximize our uptime. Currently, we have an uptime of 160 days on systems like Righteous, Web1, Docs, and others.
Network Status Server
Lately, I haven't done a ton of work on the Network Status Server. I've mainly just been adding new systems to Nagios and Cacti as they come online; however, I did write up some generic instructions for monitoring a remote system with SNMP and I do have some instructions that I need to type up for installing Cacti. I also did write a nagios plugin to check the current running kernel version on each system since it's nice to know which systems are running which kernel without having to log into each of them.
DNS
DNS in the labs has been pretty much the same except for adding a few entries and rebuilding the VM last semester. For the future, I'm planning on redoing DNS (yes again sadly) but as two separate VMs running BIND and possibly with one of the VMs running down in the projects lab. I'm mainly doing this so we can provide authorative answers for DNS queries and to provide redundancy since this will allow DNS to still function in the event that one of the VMs is down. Currently, I've created two development VMs and two production VMs for this purpose and will start setting up BIND on the development VMs sometime this month.
Central Authentication
After the DNS changes, I'm planning on finally implementing a central authentication systems for almost everything in the labs. This should simplify user management and allow us to easily grant people access to systems. Currently, I've created two development VMs and two production VMs for this purpose and will start setting up CentOS Directory Server on the development systems after the new DNS setup is active.
Emeeting
I'm hoping to set up dimdim on the emeeting VM sometime in the near future. This would allow us to have remote meetings as needed and would allow us to make our weekly presentations available to everyone on the web, in real time. At this time, I have a VM created and just need to try installing dimdim again since my first attempt failed horribly.
StatusNet
StatusNet (http://status.cslabs.clarkson.edu/) has somewhat become one of my projects. I originally was encouraging someone else to maintain the install and promote it but the original maintainer hasn't been active in a few months. I haven't really done much with it due to lack of time but I've been actively using it to post status updates for things I'm working on.
Misc.
I've also done a lot of other minor things over the past 6-7 months. I reorganized the wiki into different categories back in August and I've been changing content like crazy. I also made a new template on the wiki to organize all of the physical machines. Also, I recovered a number of old pictures from the old COSI site and posted them back up on the site. I also set up a request email from lab requests to get routed to the correct people without someone needing to know who the maintainer of something is. I'm sure there is more but I never seem to be able to remember everything.