Monday, May 16, 2011

Building a Hadoop cluster

I've recently had to build a Hadoop cluster for a class in information retrieval. My final project involved building a Hadoop cluster.

Here are some of my notes on configuring the nodes in the cluster.

These links on configuring a single node cluster and multi node cluster were the most helpful.

I downloaded the latest Hadoop distribution then moved it into /hadoop. I had problems with this latest distribution (v.21) so I used v.20 instead.

Here are the configuration files I changed:

core-site.xml:
  
    fs.default.name
    hdfs://master:9000
  
  
    hadoop.tmp.dir
    /hadoop/tmp
    A base for other temporary directories.
  

hadoop-env.sh:
# Variables required by Mahout
export HADOOP_HOME=/hadoop
export HADOOP_CONF_DIR=/hadoop/conf
export MAHOUT_HOME=/Users/rpark/mahout
PATH=/hadoop/bin:/Users/rpark/mahout/bin:$PATH

# The java implementation to use.  Required.
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home

hdfs-site.xml:
  
    dfs.replication
    3
  

mapred-site.xml:
  
    mapred.job.tracker
    master:9001
  

masters:
master

slaves:
master
slave1
slave2
slave3
slave4

Be sure to enable password-less ssh between master and slaves. Use this command to create an SSH key with an empty password:
ssh-keygen -t rsa -P ""

Enable password-less ssh login for the master to itself:
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

Then copy id_rsa.pub to each slave and do the same with each slave's authorized_keys file.

I ran into a few errors along the way. Here is an error that gave me a lot of trouble in the datanode logs:
2011-05-08 01:04:30,032 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_1804860059826635300_1001 received exception org.apache.hadoop.hdfs.server.datanode.BlockAlreadyExistsException: Block blk_1804860059826635300_1001 is valid, and cannot be written to.

The solution was to use hostnames every time I referenced a host, either itself or a remote host. I set a host's own name in /etc/hostname and the others in /etc/hosts. I used these hostnames in /hadoop/conf/masters, slaves, and the various conf files.

Every so often I ran into this error in the datanode logs:
... ERROR org.apache.hadoop.dfs.DataNode: java.io.IOException: Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode namespaceID = 308967713; datanode namespaceID = 113030094
        at org.apache.hadoop.dfs.DataStorage.doTransition(DataStorage.java:281)
        at org.apache.hadoop.dfs.DataStorage.recoverTransitionRead(DataStorage.java:121)
        at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:230)
        at org.apache.hadoop.dfs.DataNode.(DataNode.java:199)
        at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:1202)
        at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1146)
        at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:1167)
        at org.apache.hadoop.dfs.DataNode.main(DataNode.java:1326)

I fixed this by deleting tmp/dfs/data on the datanodes where I saw the error. Unfortunately, I had to reformat the HDFS volume after I did this.

I had to raise the ulimit for open files. On Ubuntu nodes I edited /etc/security/limits.conf:
rpark  soft nofile  8192
rpark  hard nofile  8192

For OS X nodes I just edited ~/.profile:
ulimit -n 8192

I ran into this error when copying data into HDFS:
could only be replicated to 0 nodes, instead of 1

The solution was simply to wait for the datanode to start up. I usually saw the error when I immediately copied data into HDFS after starting the cluster.

Port 50070 on the namenode gave me a Web UI to tell me how many nodes were in the cluster. This was very useful.

837 Comments:

«Oldest ‹Older 801 – 837 of 837
Blogger sandhiya said...

Nice..You have clearly explained about it ...Its very useful for me to know about new things..Keep on blogging..
Best JAVA and J2EE Course Training Institute in kanchipuram|

September 11, 2019 at 12:08 AM  
Blogger mani said...

Hi, am a big follower of your blog. I am really happy to found such a helpful and fascinating post that is written in well manner. Thanks for sharing such an informative post. keep update your blog.
Best CCNA Course Training Institute in kanchipuram|

September 11, 2019 at 11:27 PM  
Blogger meenaraj said...

Excellent post!!!. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it.….No:1 AWS Training Academy in Kanchipuram

September 12, 2019 at 5:12 AM  
Blogger meenaraj said...

Excellent post!!!. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it.….No:1 AWS Training Academy in Kanchipuram

September 12, 2019 at 5:30 AM  
Blogger karthik said...

Great post.Thanks for one marvelous posting! I enjoyed reading it;The information was very useful.
Best Best Web Designing and Development Course Training Institute in kanchipuram|

September 12, 2019 at 11:55 PM  
Blogger deepa prabu said...

very nice blogs!!! I have to learning for lot of information for this sites...Sharing for wonderful information. Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing.No:1 LINUX Training Academy in Kanchipuram

September 13, 2019 at 4:40 AM  
Blogger deepa prabu said...

very nice blogs!!! I have to learning for lot of information for this sites...Sharing for wonderful information. Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing.No:1 LINUX Training Academy in Kanchipuram

September 13, 2019 at 5:16 AM  
Blogger bharathidasan said...

great doing. and i happy to visit your blog to know about this wonderful information. thanks for sharing. keep updating.
Best Best Web Designing and Development Course Training Institute in kanchipuram|

September 13, 2019 at 11:25 PM  
Blogger Blessy john said...

Great blog.you put Good stuff.All the topics were explained briefly.so quickly understand for me.I am waiting for your next fantastic blog.Thanks for sharing..No:1 CCNA Training Academy in Kanchipuram

September 14, 2019 at 4:30 AM  
Blogger Blessy john said...

Great blog.you put Good stuff.All the topics were explained briefly.so quickly understand for me.I am waiting for your next fantastic blog.Thanks for sharing..No:1 CCNA Training Academy in Kanchipuram

September 14, 2019 at 5:15 AM  
Blogger uma said...

This blog is really useful and it is very interesting thanks for sharing, it is really good and exclusive.
Best Paper Plates Shop in Chennai | Best Paper Plates Shop in Kovilambakkam

September 14, 2019 at 6:53 AM  
Blogger srihariparu said...

I have read your blog its very attractive and impressive. I like it your blog.Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge.

VMware Certification Training in Chennai | VMware Certification Exam Center in Chennai | VMware Exams Center in Taramani | VMware Certification Exams in Chennai

September 15, 2019 at 11:30 PM  
Blogger mani said...

Your info is really amazing with impressive content..Excellent blog with informative concept. Really I feel happy to see this useful blog, Thanks for sharing such a nice blog..
Best Python Course Training Institute in kanchipuram|

September 15, 2019 at 11:30 PM  
Blogger ramya raj said...

very nice blogs!!! i have to learning for lot of information for this sites...Sharing for wonderful information. Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing..No:1 Blue Prism Training Academy in Kanchipuram

September 16, 2019 at 4:46 AM  
Blogger ramya raj said...

very nice blogs!!! i have to learning for lot of information for this sites...Sharing for wonderful information. Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing..No:1 Blue Prism Training Academy in Kanchipuram

September 16, 2019 at 5:25 AM  
Blogger srihariparu said...

Wow...What an excellent informative blog, really helpful. Thank you so much for sharing such a wonderful post with us.keep updating..
AWS Certifications in Chennai | AWS Exam Centers in Chennai | AWS Certification Exams in Velachery | AWS Exams in Velachery | AWS Online Exam Center in Velachery

September 16, 2019 at 8:50 PM  
Blogger mano karan said...

Very impressive and informative blog.. thanks for sharing your valuable information.. it is very useful and easy to learn as well... keep rocking and updating... looking further..
Best Tally Erp9.0 Course Training Institute in kanchipuram|

September 16, 2019 at 11:14 PM  
Blogger kumaran meena said...

Thanks for sharing such a great information..Its really nice and informative.Great article, thanks for posting....No:1 JAVA Training Academy in Kanchipuram

September 17, 2019 at 3:54 AM  
Blogger kumaran meena said...

Thanks for sharing such a great information..Its really nice and informative.Great article, thanks for posting....No:1 JAVA Training Academy in Kanchipuram

September 17, 2019 at 5:24 AM  
Blogger Venkatesh CS said...

Thanks for sharing valuable information.
Digital Marketing training Course in chennai
digital marketing training institute in chennai
digital marketing training in Chennai
digital marketing course in Chennai
digital marketing course training in omr
digital marketing certification in omr
digital marketing course training in velachery
digital marketing training center in chennai
digital marketing courses with placement in chennai
digital marketing certification in chennai
digital marketing institute in Chennai
digital marketing certification course in Chennai
digital marketing course training in Chennai
Digital Marketing course in Chennai with placement
digital marketing courses in chennai

September 18, 2019 at 2:27 AM  
Blogger rajinikumar said...

Quite Interesting post!!! Thanks for posting such a useful post. I wish to read your upcoming post to enhance my skill set, keep blogging.
Best Web Technologies Course Training Institute in kanchipuram|

September 18, 2019 at 2:30 AM  
Blogger meenaraj said...

Useful information and kindly keep update such a nice blog. Thanks for sharing......No:1 CCNA Training Academy in Kanchipuram

September 18, 2019 at 4:29 AM  
Blogger meenaraj said...

Useful information and kindly keep update such a nice blog. Thanks for sharing......No:1 CCNA Training Academy in Kanchipuram

September 18, 2019 at 4:46 AM  
Blogger creationmaker said...

Amazing blog...
ORACLE TRAINING IN CHENNAI

September 18, 2019 at 11:40 PM  
Blogger srihariparu said...

Very informative and interesting blog, it was so good to read and useful to improve my knowledge as updated one,keep updating..This Concepts is very nice Thanks for sharing.
ISTQB Certification in Chennai | ISTQB Exam Centers in Chennai | Best ISTQB Exams in Velachery

September 18, 2019 at 11:48 PM  
Blogger sandhiya said...

This is excellent information. It is amazing and wonderful to visit your site.Thanks for sharing this information, this is useful to me…
Best Linux Certification Course Training Institute in kanchipuram|

September 19, 2019 at 2:19 AM  
Blogger ramya raj said...

Wow!! What a interesting blog..Thanks The information which you provided is very much useful and easy to learn as well...keep rocking and updating... looking further......No:1 AWS Training Academy in Kanchipuram

September 19, 2019 at 3:52 AM  
Blogger ramya raj said...

Wow!! What a interesting blog..Thanks The information which you provided is very much useful and easy to learn as well...keep rocking and updating... looking further......No:1 AWS Training Academy in Kanchipuram

September 19, 2019 at 4:49 AM  
Blogger ajay said...

It is really very awesome and wonderful to visit your site.Thanks for sharing your informative blog with us.keep updating such a wonderful post..
MicroSoft Azure Certification in Chennai | Azure Exam Centers in Velachery | Azure Exam Centers in Madipakkam

September 19, 2019 at 10:13 PM  
Blogger karthik said...

Quite Interesting post!!! Thanks for posting such a useful post. I wish to read your upcoming post to enhance my skill set, keep blogging.
Best Web Technologies Course Training Institute in kanchipuram|

September 20, 2019 at 2:25 AM  
Blogger Blessy john said...

Nice post... Really you are done a wonderful job. Thanks for sharing such wonderful information with us. Please keep on updating......No:1 CCNP Training Academy in Kanchipuram

September 20, 2019 at 4:37 AM  
Blogger Blessy john said...

Nice post... Really you are done a wonderful job. Thanks for sharing such wonderful information with us. Please keep on updating......No:1 CCNP Training Academy in Kanchipuram

September 20, 2019 at 4:55 AM  
Blogger Unknown said...

Great post. Wonderful information and really very much useful. Thanks for sharing and keep updating.
Best Paper Plates Shop in Chennai | Best Paper Plates Shop in Kovilambakkam

September 20, 2019 at 10:57 PM  
Blogger bharathidasan said...

Nice and informative content..keep sharing such an wonderful blog..
Best Advanced Amazon Web Services Course Training Institute in kanchipuram|

September 21, 2019 at 2:18 AM  
Blogger Unknown said...

Thanks for your Blogs Appreciating the persistence you put into your blog and detailed information you provide.
Aws training chennai | AWS course in chennai
Rpa training in chennai | RPA training course chennai
oracle training chennai | oracle training in chennai
Hadoop Training in chennai | Hadoop training course in chennai


September 21, 2019 at 4:33 AM  
Blogger meenaraj said...

your Blog is really useful for me,its really very interesting and informative information post.keep updating such an amazing post......<a href="https://www.alltechzsolutions.in/selenium-training-in-chennai>No:1 Selenium Training Academy in Kanchipuram</a>

September 21, 2019 at 4:45 AM  
Blogger srihariparu said...

Your Blog is really amazing with useful and helpful content for us.Thanks for sharing.keep updating more information.
Embedded System Training Institute in Chennai | Embedded Training in Velachery | Embedded System Training in Guindy

September 21, 2019 at 5:33 AM  
«Oldest ‹Older 801 – 837 of 837

Post a Comment

Subscribe to Post Comments [Atom]

<< Home