Monday, 8 October 2012

Building a bioinformatics server on a budget in 2012


Introduction

As more labs get into the "next generation sequencing" game, they are finding themselves unwittingly entering into the "high performance computing" game too. Some lucky labs will be able to access a local or national HPC service, or collaborate with those who do, or pay for access to one like Amazon EC2. But ultimately any bioinformatician will want (and need) a "general server" to experiment, develop, and test on. This is particularly true in terms of large, fast storage which is usually in short supply or costly in HPC environments.

Details


With the small lab in mind, here I outline how, for about A$2500/US$2600/2000), you can build your own server with 6/12 cores, 64GB RAM, and 12TB RAID storage from affordable commodity parts:

  • CPU
    • Intel CORE i7 3930K - LGA 2011 
    • $580
    • The Intel CPUs are about 40% faster for the same clock rate than the equivalent AMD. This is a 3.2 Ghz 6 core CPU (12 threads) and can go to 3.8 Ghz when only using some of the cores. It has good RAM bandwidth too.
  • CPU Cooler
  • Motherboard
    • Gigabyte GA-X79-UP4 Intel Mainboard - LGA 2011 
    • $269
    • This motherboard was chosen as it takes the new LGA2011 CPU, has 8 DIMM slots, and 8 SATA ports, including 6.0Gbsp SATA3 ones which we will want for the SSD boot disk described later. We will save money by using Linux software RAID.
  • RAM
  • Root disk
    • OCZ Agility 3 Series 120GB 2.5" SSD Sata 6Gbs 
    • $170 (2 drives)
    • For the boot/root disk, we will use 120GB SSD instead of mechanical hard disk, for speed. Two disks in RAID1 (mirror) configuration. This would store the operating system, and any important databases eg. BLAST indices, SQL etc.
  • Data disk
    • Seagate Barracuda 3TB 
    • $924 (6 drives)
    • In bioinformatics you can never have enough disk space! Here I suggest using six 3TB 7200rpm drives in RAID6 arrangement for a total of 12TB storage. Reading will be fast, but writing a bit slower. We need to compromise on a budget.
  • Case
    • Antec P280 
    • $135
    • To hold all this stuff, this case is great value. It has good ventilation, which is crucial for keeping all the disks and the CPU cool.
  • Power supply
    • Antec TruePower New Series 650W 
    • $120
    • The case doesn't come with a power supply. I've chosen this one because it has 9 SATA power connectors for the disks, and some unneeded cable sets can be removed in a modular fashion.
  • Operating system
    • Ubuntu Server
    • $0
    • Ubuntu Server edition is pre-tuned for server settings as opposed to interactive desktop use. Because it is Debian based, you have access to far more packaged bioinformatics applications than Centos, including those in the BioLinux project.
  • Miscellaneous
    • Extra 120mm cooling fans (for the hard disk zones)
    • Extra SATA3 cables (ones with locking clips)
    • CPU thermal paste (the grey stuff)
    • PCIe graphics card (for the screen, any old one will do)


Conclusion

For about A$2500/US$2600/2000 and a little bit of Linux know-how, you can build your own "high-end" server. It is missing some of the features of commercial servers from Dell etc (eg. hardware RAID controller, IPMI remote management, hot-swap disks) but it is much more affordable.


11 comments:

  1. Then add a remote KVM to save trips to the server room: http://www.kvms.com/Product/KN1000.aspx Although, it blows out the budget a bit...

    ReplyDelete
  2. If you're building a budget server, you probably don't have a server room. It will be under your student's desk, a friendly walk down the hall :-)

    ReplyDelete
  3. Thanks Torst. Nice post. Maintenance requirements? Six-monthly replacement of that thermal grease to keep those CPUs chilly?

    ReplyDelete
  4. ...with an Aldi fan to cool it

    ReplyDelete
  5. Do you have any views about using ECC vs non-ECC memory in these clusters. Seems that for having ECC memory you need to go for professional servers.

    ReplyDelete
  6. Some consumer motherboards do support ECC memory, but you tend to need a server motherboard, such as those made by Intel, SuperMicro and Tyan. But these usually also require REGISTERED ECC memory. These are electrically different to regular memory, but allow the use of more DIMMs as they can handle longer physical distances from the memory controller (as I understand it!). They also have higher latency. This post is all about budget; so if your budget server is behaving oddly, run memtest+86 and replace the faulty DIMMs. Hopefully, if the fans/cooling are set up correctly, you shouldn't have any issues using non-ECC RAM.

    ReplyDelete
  7. how do you feel about the motherboard:
    http://www.centrecom.com.au/catalog/z77x-ud5h-gigabyte-z77x-ud5h-intel-mainboard-1155-p-57437.html?sort=2a

    ReplyDelete
    Replies
    1. If you can only afford lower-end LGA1155 CPUs and only need 32GB RAM maximum, then I don't have any issues with it.

      Delete
    2. Thank you Torsten,
      I seem to have pasted the wrong link. I apologize for that. I was in fact looking for this motherboard that seems to allow for a bit more DRAM

      http://www.centrecom.com.au/catalog/x79a-gd658d-x79a-gd65-intel-mainboard-2011-p-55118.html?sort=2a

      what do you think?

      Omar

      Delete
  8. Looks good! Just make sure you check the RAM you buy is compatible with the motherboard. The MSI web site will have tables of RAM it has tested.

    ReplyDelete