Monday, May 16, 2011

Building a Hadoop cluster

I've recently had to build a Hadoop cluster for a class in information retrieval. My final project involved building a Hadoop cluster.

Here are some of my notes on configuring the nodes in the cluster.

These links on configuring a single node cluster and multi node cluster were the most helpful.

I downloaded the latest Hadoop distribution then moved it into /hadoop. I had problems with this latest distribution (v.21) so I used v.20 instead.

Here are the configuration files I changed:

core-site.xml:
  
    fs.default.name
    hdfs://master:9000
  
  
    hadoop.tmp.dir
    /hadoop/tmp
    A base for other temporary directories.
  

hadoop-env.sh:
# Variables required by Mahout
export HADOOP_HOME=/hadoop
export HADOOP_CONF_DIR=/hadoop/conf
export MAHOUT_HOME=/Users/rpark/mahout
PATH=/hadoop/bin:/Users/rpark/mahout/bin:$PATH

# The java implementation to use.  Required.
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home

hdfs-site.xml:
  
    dfs.replication
    3
  

mapred-site.xml:
  
    mapred.job.tracker
    master:9001
  

masters:
master

slaves:
master
slave1
slave2
slave3
slave4

Be sure to enable password-less ssh between master and slaves. Use this command to create an SSH key with an empty password:
ssh-keygen -t rsa -P ""

Enable password-less ssh login for the master to itself:
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

Then copy id_rsa.pub to each slave and do the same with each slave's authorized_keys file.

I ran into a few errors along the way. Here is an error that gave me a lot of trouble in the datanode logs:
2011-05-08 01:04:30,032 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_1804860059826635300_1001 received exception org.apache.hadoop.hdfs.server.datanode.BlockAlreadyExistsException: Block blk_1804860059826635300_1001 is valid, and cannot be written to.

The solution was to use hostnames every time I referenced a host, either itself or a remote host. I set a host's own name in /etc/hostname and the others in /etc/hosts. I used these hostnames in /hadoop/conf/masters, slaves, and the various conf files.

Every so often I ran into this error in the datanode logs:
... ERROR org.apache.hadoop.dfs.DataNode: java.io.IOException: Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode namespaceID = 308967713; datanode namespaceID = 113030094
        at org.apache.hadoop.dfs.DataStorage.doTransition(DataStorage.java:281)
        at org.apache.hadoop.dfs.DataStorage.recoverTransitionRead(DataStorage.java:121)
        at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:230)
        at org.apache.hadoop.dfs.DataNode.(DataNode.java:199)
        at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:1202)
        at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1146)
        at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:1167)
        at org.apache.hadoop.dfs.DataNode.main(DataNode.java:1326)

I fixed this by deleting tmp/dfs/data on the datanodes where I saw the error. Unfortunately, I had to reformat the HDFS volume after I did this.

I had to raise the ulimit for open files. On Ubuntu nodes I edited /etc/security/limits.conf:
rpark  soft nofile  8192
rpark  hard nofile  8192

For OS X nodes I just edited ~/.profile:
ulimit -n 8192

I ran into this error when copying data into HDFS:
could only be replicated to 0 nodes, instead of 1

The solution was simply to wait for the datanode to start up. I usually saw the error when I immediately copied data into HDFS after starting the cluster.

Port 50070 on the namenode gave me a Web UI to tell me how many nodes were in the cluster. This was very useful.

1651 Comments:

«Oldest ‹Older 1601 – 1651 of 1651
Blogger priyash said...

Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
Correlation vs Covariance

May 31, 2020 at 10:39 PM  
Blogger datasciencecourse said...

cool stuff you have and you keep overhaul every one of us

Correlation vs Covariance

June 2, 2020 at 8:32 PM  
Blogger Saravanan said...

Thank you for taking the time and sharing this information with us. It was indeed very helpful and insightful while being straight forward and to the point.
CCNA Training Institute in Chennai | CCNA Training Institute in Velachery

June 4, 2020 at 4:04 AM  
Blogger Faizal said...

Fantastic Resource to sharing the knowledge's...It helped me with great information's.Looking towards More.
Java training in chennai | Java training in annanagar | Java training in omr | Java training in porur | Java training in tambaram | Java training in velachery

June 4, 2020 at 7:38 AM  
Blogger deiva said...

This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.
Digital Marketing Training Course in Chennai | Digital Marketing Training Course in Anna Nagar | Digital Marketing Training Course in OMR | Digital Marketing Training Course in Porur | Digital Marketing Training Course in Tambaram | Digital Marketing Training Course in Velachery

June 6, 2020 at 2:19 AM  
Blogger Rajesh said...

i am glad to read it.
valuable one ...waiting for next update.
Selenium Training in chennai | Selenium Training in anna nagar | Selenium Training in omr | Selenium Training in porur | Selenium Training in tambaram | Selenium Training in velachery

June 9, 2020 at 12:09 AM  
Blogger subha said...

Awesome blog, very informative content... Thanks for sharing waiting for next update... good work
Ai & Artificial Intelligence Course in Chennai
PHP Training in Chennai
Ethical Hacking Course in Chennai Blue Prism Training in Chennai
UiPath Training in Chennai

June 11, 2020 at 1:50 AM  
Blogger Priyanka said...

Attend The Artificial Intelligence course From ExcelR. Practical Artificial Intelligence course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Artificial Intelligence course.
Artificial Intelligence Course

June 11, 2020 at 8:02 PM  
Blogger karthik said...

Nice Blog...looking amazing...thank you so much for sharing such a wonderful blog...keep sharing... Best CCNA Certification In Chennai|

June 11, 2020 at 11:10 PM  
Blogger datasciencecourse said...

Cool stuff you have and you keep overhaul every one of us

Simple Linear Regression

Correlation vs Covariance

June 14, 2020 at 8:56 PM  
Blogger priyash said...

Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
Correlation vs Covariance
Simple linear regression

June 15, 2020 at 1:02 AM  
Blogger dataexpert said...

This was really one of my favorite website. Please keep on posting. ExcelR Data Science Course In Pune

June 15, 2020 at 3:58 AM  
Blogger karthik said...

very useful really good information thanks for posting such a good information it will hepls the people a lot keep it up , Regards, Best PMP Certification In Chennai|

June 17, 2020 at 12:24 AM  
Blogger Village Talkies said...

Best Corporate Video Production Company in Bangalore and top Explainer Video Company in Bangalore , 3d, 2d Animation Video Makers in Chennai.
Awesome article. good read blog. Thanks for sharing

June 18, 2020 at 1:44 AM  
Blogger datasciencecourse said...

I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!

Simple Linear Regression

Correlation vs Covariance

June 19, 2020 at 7:18 PM  
Blogger Jayalakshmi said...

keep up the good work. this is an Assam post. this to helpful, i have reading here all post. i am impressed. thank you. thi

Dot Net Training in Chennai | Dot Net Training in anna nagar | Dot Net Training in omr | Dot Net Training in porur | Dot Net Training in tambaram | Dot Net Training in velachery





June 21, 2020 at 3:33 AM  
Blogger mehar said...

very informative and good post to follow
Best college for commerce and management

June 22, 2020 at 3:06 AM  
Blogger haritha said...

thank you for the useful post.
Best commerce college in hyderabad

June 22, 2020 at 3:06 AM  
Blogger Premkumar said...

much needed information Best commerce college in hyderabad

June 22, 2020 at 3:07 AM  
Blogger Priyanka said...

Attend The Data Analytics Courses From ExcelR. Practical Data Analytics Courses Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analytics Courses.
Data Analytics Courses

June 22, 2020 at 4:23 AM  
Blogger mehar said...

This blog is very informative and amazing ..
<a href="www.anishcollegehyd.org'> Best BBA college in Hyderabad </a>

June 23, 2020 at 1:15 AM  
Blogger Malar said...

Excellant post!!!. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it. Best Tally Training Center in chennai |

June 25, 2020 at 2:11 AM  
Blogger Natasha said...

Nice Post! It is really interesting to read from the beginning & I would like to share your blog to my circles, keep your blog as updated. Best CCNP Training institute in chennai |

June 26, 2020 at 4:10 AM  
Blogger uma said...

Awesome blog. Your articles really impressed for me, because of all information so nice and unique...
Best Paper Plates Shop in Chennai | Best Paper Plates Shop in thirukkalukundram

June 26, 2020 at 11:07 PM  
Blogger karthik said...

Really it was an awesome article...very interesting to read..You have provided an nice article....Thanks for sharing.. Best Software Testing Training Center in chennai
|

June 27, 2020 at 6:20 AM  
Blogger Anu said...

Nice and good article. It is very useful for me to learn and understand easily. Thanks for sharing your valuable information and time. Please keep updating DevOps Training in Chennai | DevOps Training in anna nagar | DevOps Training in omr | DevOps Training in porur | DevOps Training in tambaram | DevOps Training in velachery

June 28, 2020 at 9:27 PM  
Blogger priyash said...

Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
Correlation vs Covariance
Simple linear regression
data science interview questions

June 29, 2020 at 12:56 AM  
Blogger karthik said...

Good one, it is very useful for me to learn and understand, thanks for sharing your information and ideas.. keep rocks.. Best Agile Scrum Master Exam center in chennai|

June 29, 2020 at 4:29 AM  
Blogger bharathidasan said...

very nice post what a fantastic information you sharing. Keep sharing your post regularly.. Best Embedded System Exam center in chennai|

June 30, 2020 at 2:14 AM  
Blogger ek said...

PMP Certification
You completely match our expectation and the variety of our information.

June 30, 2020 at 2:30 AM  
Blogger jayalatha said...

The article is informative thanks for sharing

Best Commerce college for MEC & CEC in hyderabad

July 2, 2020 at 3:03 AM  
Blogger Malar said...

Great tips and its so easy to understand. really You explain everything in detailed manner and it was very interesting to read. Thank you. Best Data Science Training in chennai|

July 2, 2020 at 4:05 AM  
Blogger haritha said...

useful information . thank you
Best commerce college

July 4, 2020 at 2:13 AM  
Blogger mehar said...

Good post for information
Best management with practical approach

July 4, 2020 at 2:13 AM  
Blogger priyash said...

Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
Correlation vs Covariance
Simple linear regression
data science interview questions

July 5, 2020 at 10:52 PM  
Blogger Sowmiya R said...

Really you have done great job,There are may person searching about that now they will find enough resources by your post

Oracle Training | Online Course | Certification in chennai | Oracle Training | Online Course | Certification in bangalore | Oracle Training | Online Course | Certification in hyderabad | Oracle Training | Online Course | Certification in pune | Oracle Training | Online Course | Certification in coimbatore

July 6, 2020 at 4:40 AM  
Blogger jeffery said...

Nice blog, here I had an opportunity to learn something new in my field. I have an expectation about your future post so please keep updates.. Best PCB design Training center in chennai|

July 6, 2020 at 9:27 PM  
Blogger Natasha said...

We are really grateful for your blog post. You will find a lot of approaches after visiting your post. Great work. Best Matlab Training Center in Chennai |

July 7, 2020 at 9:36 PM  
Blogger dataexpert said...

This is exactly the information I'm looking for, I couldn't have asked for a simpler read with great tips like this... Thanks! ExcelR Data Science Courses

July 7, 2020 at 11:04 PM  
Blogger datasciencecourse said...

This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.

Simple Linear Regression

Correlation vs Covariance

July 9, 2020 at 11:23 PM  
Blogger hrithiksai67 said...

This Was An Amazing ! I Haven't Seen This Type of Blog Ever ! Thankyou For Sharing, data science course in hyderabad with placements

July 10, 2020 at 5:45 AM  
Blogger EXCELR said...

This post is really helpful for us. I certainly love this website, keep on it.
Data Science Course in Hyderabad

July 10, 2020 at 8:23 AM  
Blogger datasciencecourse said...

I am looking for and I love to post a comment that "The content of your post is awesome" Great work!

Simple Linear Regression

Correlation vs Covariance

July 10, 2020 at 8:13 PM  
Blogger Unknown said...

Banquets in Indore
Thanks for sharing such information. This is really helpful for me. you can also visit our blog
https://palmindore.in/blog/banquets-in-indore/

July 10, 2020 at 11:52 PM  
Blogger Premkumar said...

Best blog
Best college for B.com

July 11, 2020 at 2:14 AM  
Blogger mehar said...

Good article and very useful information
Best commerce college

July 11, 2020 at 2:25 AM  
Blogger bill.wood said...

Programming Languages and Methodology: This area addresses the methodology used to formulate problem solving software code and the programming languages that are used to write viable code machine learning course in hyderabad

July 12, 2020 at 1:09 PM  
Blogger srihariparu said...

Thanks for sharing this information, it helped me a lot in finding valuable resources for my career
Java Training Institute in Chennai | Java Training Center in Velachery | java Training in Chennai

July 13, 2020 at 2:35 AM  
Blogger Anu said...

This article resolved my all queries.
DevOps Training in Bangalore | Certification | Online Training Course institute | DevOps Training in Hyderabad | Certification | Online Training Course institute | DevOps Training in Coimbatore | Certification | Online Training Course institute | DevOps Online Training | Certification | Devops Training Online

July 13, 2020 at 5:58 AM  
Blogger kenwood said...

Finally, all this approaches can be used in almost any industry. We should take it into account, when predict the future of some markets and of our society in general. machine learning and ai courses in hyderabad

July 13, 2020 at 1:17 PM  
Blogger Anebellyliza said...

I have read your blog and I gathered some new information through your blog. Thanks for sharing the information
Data Science Training in Hyderabad

July 13, 2020 at 10:20 PM  
«Oldest ‹Older 1601 – 1651 of 1651

Post a Comment

Subscribe to Post Comments [Atom]

<< Home