Skip to main content

Posts

Showing posts from February, 2013

Using chef to build out a Hadoop cluster

After not posting for a while, I have about 3-4 posts that I'd like to get out there. The first is about using chef to build a Hadoop cluster. Chef  is a configuration management tool that allows one to automate the process of provisioning servers. I had to create a Hadoop cluster of 4-5 servers and I wanted to use this opportunity to automate the process with chef. I had to perform a series of the same steps on these Linux nodes: Install ruby and chef Install Sun Java Install VMware Tools Install NTP Add its hostname to a shared /etc/hosts file Configure passwordless ssh login Installing Chef and Ruby I followed the steps in this link . The first step is to sign up for a Hosted Chef account on the  Opscode  site. An account is free for 5 nodes or less. Perform the following steps: Create a new organization Select "Generate knife config" to download knife.rb Select "Regenerate validation key" to download (validator).pem Click on your acc