Monday, May 16, 2011

Building a Hadoop cluster

I've recently had to build a Hadoop cluster for a class in information retrieval. My final project involved building a Hadoop cluster.

Here are some of my notes on configuring the nodes in the cluster.

These links on configuring a single node cluster and multi node cluster were the most helpful.

I downloaded the latest Hadoop distribution then moved it into /hadoop. I had problems with this latest distribution (v.21) so I used v.20 instead.

Here are the configuration files I changed:

core-site.xml:
  
    fs.default.name
    hdfs://master:9000
  
  
    hadoop.tmp.dir
    /hadoop/tmp
    A base for other temporary directories.
  

hadoop-env.sh:
# Variables required by Mahout
export HADOOP_HOME=/hadoop
export HADOOP_CONF_DIR=/hadoop/conf
export MAHOUT_HOME=/Users/rpark/mahout
PATH=/hadoop/bin:/Users/rpark/mahout/bin:$PATH

# The java implementation to use.  Required.
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home

hdfs-site.xml:
  
    dfs.replication
    3
  

mapred-site.xml:
  
    mapred.job.tracker
    master:9001
  

masters:
master

slaves:
master
slave1
slave2
slave3
slave4

Be sure to enable password-less ssh between master and slaves. Use this command to create an SSH key with an empty password:
ssh-keygen -t rsa -P ""

Enable password-less ssh login for the master to itself:
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

Then copy id_rsa.pub to each slave and do the same with each slave's authorized_keys file.

I ran into a few errors along the way. Here is an error that gave me a lot of trouble in the datanode logs:
2011-05-08 01:04:30,032 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_1804860059826635300_1001 received exception org.apache.hadoop.hdfs.server.datanode.BlockAlreadyExistsException: Block blk_1804860059826635300_1001 is valid, and cannot be written to.

The solution was to use hostnames every time I referenced a host, either itself or a remote host. I set a host's own name in /etc/hostname and the others in /etc/hosts. I used these hostnames in /hadoop/conf/masters, slaves, and the various conf files.

Every so often I ran into this error in the datanode logs:
... ERROR org.apache.hadoop.dfs.DataNode: java.io.IOException: Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode namespaceID = 308967713; datanode namespaceID = 113030094
        at org.apache.hadoop.dfs.DataStorage.doTransition(DataStorage.java:281)
        at org.apache.hadoop.dfs.DataStorage.recoverTransitionRead(DataStorage.java:121)
        at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:230)
        at org.apache.hadoop.dfs.DataNode.(DataNode.java:199)
        at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:1202)
        at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1146)
        at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:1167)
        at org.apache.hadoop.dfs.DataNode.main(DataNode.java:1326)

I fixed this by deleting tmp/dfs/data on the datanodes where I saw the error. Unfortunately, I had to reformat the HDFS volume after I did this.

I had to raise the ulimit for open files. On Ubuntu nodes I edited /etc/security/limits.conf:
rpark  soft nofile  8192
rpark  hard nofile  8192

For OS X nodes I just edited ~/.profile:
ulimit -n 8192

I ran into this error when copying data into HDFS:
could only be replicated to 0 nodes, instead of 1

The solution was simply to wait for the datanode to start up. I usually saw the error when I immediately copied data into HDFS after starting the cluster.

Port 50070 on the namenode gave me a Web UI to tell me how many nodes were in the cluster. This was very useful.

264 Comments:

«Oldest ‹Older 201 – 264 of 264
Blogger victoriya said...

Really I enjoy this blog….. Very nice post… Thanks for sharing and keep updating
AWS Training Institute in Chennai | AWS Training Institute in Velachery

October 26, 2018 at 9:36 PM  
Blogger sharma said...

Pretty article! I found some useful information in your blog, it was awesome to read, thanks for sharing this great content to my vision, keep sharing.
Best UIPath Training Institute in Chennai | Best UIPath Training Institute in Velachery

October 27, 2018 at 3:16 AM  
Blogger ancy said...

Thank you so much for sharing such an amazing post with useful information with us. Keep updating such a wonderful blog….
Best CCNA Training Institute in Chennai| CCNA Training Center in Velachery

October 28, 2018 at 10:34 PM  
Blogger ancy said...

Nice post. This post is very helpful. Thank you so much for sharing this post…
Ethical Hacking Training Course in Chennai | Ethical Hacking Training Course in Nanganallur

October 29, 2018 at 2:36 AM  
Blogger Vishal said...

Pretty article! I found some useful information in your blog, it was awesome to read, thanks for sharing this great content to my vision, keep sharing.
Best Selenium Training Institute in Chennai|Best Selenium Training Institute in Saidapet

October 29, 2018 at 2:50 AM  
Blogger kalai hai said...

Nice blog…. with lovely information. Really very useful article for us thanks for sharing such a wonderful blog. Keep updating…..
Best Linux Training Institute in Pallikaranai| Best Linux Training Institute in Velachery

October 29, 2018 at 10:10 PM  
Blogger Babyma said...

Awesome article…. Really very helpful information for us. Keep it up. Keep blogging. Looking to reading your next post,. Thank You for Sharing This Information
Best PCB Training Institute in Taramani| Best PCB Training Institute in Velachery

October 30, 2018 at 9:56 PM  
Blogger Saranya Karthik said...

Nice Blog… Wonderful Information and really very much useful. Thanks for sharing and keep updating…
RPA Training in Chennai | RPA Training in Pallikaranai

October 31, 2018 at 2:18 AM  
Blogger ram ramky said...

Great website and content of your website is really awesome.
Selenium training in Chennai
Selenium Courses in Chennai
best ios training in chennai
Digital Marketing Training in Chennai
JAVA J2EE Training Institutes in Chennai
Best DOT NET Training institute in Chennai
asp.net training in chennai
Selenium Interview Questions and Answers

October 31, 2018 at 4:01 AM  
Blogger Mary Adlena said...

Amazing Article, thank you!. I am very glad to read your informative & practical blog. Kindly keep updating your blog.
Automation Anywhere Training in Chennai | Automation Anywhere Training in Pallavaram

November 1, 2018 at 12:04 AM  
Blogger fawaz shaik said...

Good Post…. Thanks for the useful information it’s very useful to read your blog. Keep blogging article like this.
Best JAVA/J2EE Training Institute in Perungudi| Best JAVA/J2EE Training Institute in Velachery

November 1, 2018 at 2:10 AM  
Blogger rajendran said...

Nice blog, really I feel happy to see this useful blog… Thanks for sharing this valuable information to our vision....
Best AWS Training Institute in Medavakkam| Best AWS Training Institute in Velachery

November 1, 2018 at 9:57 PM  
Blogger rajendran said...

http://www.bashguru.com/2009/09/bash-script-to-check-after-every-one.html?showComment=1541134618580#c8499889638532685776

November 1, 2018 at 10:00 PM  
Blogger teja said...

I simply wanted to thank you so much again. I am not sure the things that I might have gone through without the type of hints revealed by you regarding that situation.
Blue prism Training in Chennai | Blue prism Training in Besant Nagar

November 2, 2018 at 4:21 AM  
Blogger deepti said...

Thank you so much for sharing this worth able content with us. The concept taken here will be useful for my future programs and I will surely implement them in my study.
Best Selenium Training Institute in Chennai|Best Selenium Training Institute in Pallikaranai

November 2, 2018 at 10:14 PM  
Blogger Vishal said...

Awesome post…. This is really helpful for me. I like it. Thanks for sharing……
Best Azure Training Institute in Chennai| Best Azure Training Institute in Velachery

November 3, 2018 at 4:45 AM  
Blogger Deshma Teja said...

Really wonderful post.... My sincere thanks for sharing very useful information to the users... Please continue to share this kind of post
Best Python Exams Center in OMR| Best Python Certification Exams in OMR

November 4, 2018 at 8:15 PM  
Blogger Deshma Teja said...

Really wonderful post.... My sincere thanks for sharing very useful information to the users... Please continue to share this kind of post
Best Python Exams Center in OMR| Best Python Certification Exams in OMR

November 4, 2018 at 8:16 PM  
Blogger muthu said...

Hi Thanks for the nice information its very useful to read your blog.
Automation Anywhere Training in Chennai | Automation Anywhere Training in Ashok Nagar

November 8, 2018 at 11:15 PM  
Blogger vamsi vasi said...

Thank you so much for sharing this worth able content with us. Keep blogging article like this.
Perfect and Outstanding Dotnet Training Institute in Chennai| Perfect and Outstanding Dotnet Training Institute in Adyar

November 8, 2018 at 11:43 PM  
Blogger vamsi vasi said...

Nice and informative article. Thanks for sharing such nice article, keep on updating.
Perfect and Outstanding Dotnet Training Institute in Chennai| Perfect and Outstanding Dotnet Training Institute in Adyar

November 8, 2018 at 11:55 PM  
Blogger Raja said...

Really very nice blog information for this one and more technical skills are improve,i like that kind of post.
Graphic Designing Training in Chennai | Graphic Designing Training in Keelkattalai

November 9, 2018 at 2:34 AM  
Blogger Vishal said...

Wonderful blog… You provided very interesting information here. I feel happy to read this post. I hope that you will write many posts like this… Thanks for sharing and Keep updating…..
Python Training Institute in Chennai | Python Training Institute in Velachery

November 9, 2018 at 8:19 PM  
Blogger kalai hai said...

Useful information and Please keep updating us..... Thanks for sharing
Best Selenium Training Institute in Chennai |Best Selenium Training Institute in Chennai

November 9, 2018 at 10:54 PM  
Blogger fawaz shaik said...

I simply wanted to thank you so much again. I am not sure the things that I might have gone through without the type of hints revealed by you regarding that situation.
UIPath Training in Chennai | UIPath Training in Kanchipuram

November 10, 2018 at 2:10 AM  
Blogger Vimal said...

Really very nice blog information for this one and more technical skills are improve,i like that kind of post.
Ethical Hacking Training in Chennai | Ethical Hacking Training in Thiruvanmiyur

November 10, 2018 at 4:38 AM  
Blogger jerina j said...

I have read your blog. It’s very informative and useful blog. You have done really great job. Keep update your blog.
Android Training in Chennai | Android Training in Porur

November 11, 2018 at 4:20 AM  
Blogger Nithya said...

I have read your blog its very attractive and impressive. I like it your blog.
Java Training in Chennai | Java Training in St.Thomas Mount

November 12, 2018 at 4:33 AM  
Blogger karthik said...

The provided information’s are very useful to me.Thanks for sharing.Keep updating your blog.
RPA Training in Chennai | RPA Training in Pallavaram

November 12, 2018 at 4:45 AM  
Blogger for IT the said...

Great Article

Project Centers in Chennai
Final Year Projects for CSE

November 12, 2018 at 11:07 PM  
Blogger Magesh said...

Very interesting content which helps me to get the in depth knowledge about the technology.
Best Software Testing Training Institute in Chennai | Best Software Testing Training Institute in T.Nagar

November 16, 2018 at 4:29 AM  
Blogger Saravanan said...

Thanks for posting this useful content, Good to know about new things here,Keep updating your blog...
Ethical Hacking Training in Chennai | Ethical Hacking Training in Kanchipuram

November 17, 2018 at 11:19 PM  
Blogger Mounika said...

The provided information’s are very useful to me.Thanks for sharing.Keep updating your blog.
Blue prism Training in Chennai | Blue prism Training in Medavakkam

November 18, 2018 at 11:41 PM  
Blogger Valli said...

Very interesting content which helps me to get the in depth knowledge about the technology.
Software Testing Course in Chennai | Software Testing Course in Perungudi

November 19, 2018 at 2:31 AM  
Blogger vicky m said...

Your Blog is really awesome with useful and helpful content for us.Thanks for sharing ..keep updating more information.
Best Ethical Hacking Training Institute in Chennai | Best Ethical Hacking Training Institute in Velachery

November 20, 2018 at 2:17 AM  
Blogger Petter Marry said...

Great Article

Project Centers in Chennai
Final Year Projects for CSE

November 21, 2018 at 4:21 AM  
Blogger Thiru K said...

Nice blog.. Thanks for sharing informative blog.. its very useful to me..
JAVA and J2EE Training Institute in Chennai | JAVA and J2EE Training Institute in Besant Nagar

November 21, 2018 at 4:54 AM  
Blogger Seetha C said...

I have read your blog its very attractive and impressive. I like it your blog.
Software Testing Course in Chennai | Software Testing Course in Keelkattalai

November 25, 2018 at 4:03 AM  
Blogger Bala V said...

I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.
IOS Training Institute in Chennai | IOS Training Institute in Pallavaram

November 25, 2018 at 11:30 PM  
Blogger Mary Adlena said...

Thanks for sharing this valuable information.. I saw your website and get more details..Nice work...
No.1 Blue Prism Training Institute in Chennai | No.1 Blue Prism Training Institute in Velachery

November 26, 2018 at 3:47 AM  
Blogger Vijay said...

very interesting topic.Helps to gain knowledge about lot of information. Thanks for posting information in this blog.
Selenium Automation Course in Chennai | Selenium Automation Course in Pallikaranai

November 27, 2018 at 2:29 AM  
Blogger Dhanish said...

Thanks for posting this useful content, Good to know about new things here,Keep updating your blog...
Automation Anywhere Certification in Chennai | Automation Anywhere Training in Pallikaranai

November 28, 2018 at 3:56 AM  
Blogger Saravanan said...

Thanks for sharing this valuable information.. I saw your website and get more details..Nice work...
RPA Training in Chennai | RPA Training in Madipakkam

November 30, 2018 at 3:35 AM  
Blogger Ananya Krishnan said...

Good job in presenting the correct content with the clear explanation. The content looks real with valid information. Good Work

DevOps is currently a popular model currently organizations all over the world moving towards to it. Your post gave a clear idea about knowing the DevOps model and its importance.

Good to learn about DevOps at this time.


devops training in chennai | devops training in chennai with placement | devops training in chennai omr | devops training in velachery | devops training in chennai tambaram | devops institutes in chennai | devops certification in chennai | trending technologies list 2018 | devops interview questions and answers

December 3, 2018 at 3:43 AM  
Blogger Dharani M said...

Good information
sas training in Marathahalli

sas training institutes in Marathahalli

December 5, 2018 at 10:47 PM  
Blogger sunshineprofe said...

I’ve bookmarked your site, and I’m adding your RSS feeds to my Google account.
iosh safety course in chennai

December 7, 2018 at 10:09 PM  
Blogger Dhanish said...

Post is very informative… It helped me with great information so I really believe you will do much better in the future.
Blue prism Exam Centers in Chennai | Blue prism Certifications in Kanchipuram

December 10, 2018 at 4:12 AM  
Blogger Aruna Ram said...

Good work!!! Very creativity post and Thank you for your brief explanation. Keep it up.
Hacking Course in Bangalore
Ethical Hacking Training in Bangalore
Ethical Hacking Classes near me
Ethical Hacking Training in Mogappair
Ethical Hacking Course in Chennai
Ethical Hacking Course in Saidapet

December 19, 2018 at 11:16 PM  
Blogger Nandhini said...

I have read your blog its very attractive and impressive. I like it your blog...
Best Python Training institute in Chennai | Best Python Training institute in Thiruvanmiyur

December 24, 2018 at 4:44 AM  
Blogger Suba said...

Generally, I don't make comments on sites, however, I need to say that this post really pushed me to do as such things.

Software testing training in chennai
Salesforce Training in Chennai
Big Data Training in Chennai
Hadoop Training in Chennai
Android Training in Chennai
Selenium Training in Chennai
Digital Marketing Training in Chennai
JAVA Training in Chennai
big data training and placement in chennai

December 29, 2018 at 2:16 AM  
Blogger Unknown said...

Valuable idea shared with program for Hadoop Training in Hyderabad
is the best.

January 2, 2019 at 12:56 AM  
Blogger ruhi rai said...

Learned a lot from your blog. Good creation and hats off to the creativity of your mind. Share more like this.
Loadrunner Training in Chennai
French Classes in Chennai
iOS Training in Chennai

January 3, 2019 at 2:42 AM  
Blogger Nisha Premrahul said...

Very interesting content which helps me to get the in depth knowledge about the technology.
No.1 Ethical Hacking Training institute in Chennai | No.1 Ethical Hacking Training institute in Velachery

January 3, 2019 at 4:42 AM  
Blogger Kayal m said...

Very interesting post, it was too good and good job. Thanks for your sharing with us. I am waiting for your great post.
IELTS coaching in Chennai
IELTS coaching centre in Chennai
IELTS Training in Chennai
Best IELTS coaching in Chennai
Best IELTS coaching centres in Chennai

January 3, 2019 at 10:08 PM  
Blogger alex said...

Play and earn from online casinos at BGAOC we have new online casino games The best online resource for making money.

January 8, 2019 at 5:48 AM  
Blogger Anbarasan14 said...

This blog enabled me to achieve my learning objectives on this topic. Thanks for the efforts taken in sharing the thoughts with us.

TOEFL Coaching in Chennai
Best TOEFL Coaching Institute in Chennai
TOEFL Classes in Chennai
TOEFL Training in Chennai
IELTS Classes in Mumbai
Spoken English Classes in JP Nagar
Spoken English Class in Thiruvanmiyur
Spoken English Classes in Chennai
English Speaking Classes in Mumbai

January 10, 2019 at 11:28 PM  
Blogger akshaya raja said...

I am really enjoying reading your well written articles.
It looks like you spend a lot of effort and time on your blog.
I have bookmarked it and I am looking forward to reading new articles.Keep up the good work..
RPA Training in Chennai
RPA Classes in Chennai
Blue Prism Training in Chennai
Ethical Hacking Training in Chennai
Cloud Computing Training in Chennai
RPA Training in T Nagar
RPA Training in Porur

January 11, 2019 at 9:22 PM  
Blogger Saravanan said...

Thanks its Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us.
Ethical Hacking Training in Chennai | Ethical Hacking Training in Taramani

January 13, 2019 at 1:31 AM  
Blogger zara said...

All information is reliable for new readers. Keep it up Hadoop Training in Chennai

January 17, 2019 at 1:57 AM  
Blogger Mani said...

Your blog is really amazing with smart and cute content.keep updating such an excellent article..
Blueprism Exam Center in Chennai | Blueprism Exam Center in Velachery

January 17, 2019 at 3:33 AM  
Blogger Aman CSE said...

Such a wonderful blog on Machine learning . Your blog almost full information about Machine learning .Your content covered full topics of Machine learning that it cover from basic to higher level content of Machine learning . Requesting you to please keep updating the data about Machine learning in upcoming time if there is some addition.
Thanks and Regards,
Machine learning tuition in chennai
Machine learning workshops in chennai
Machine learning training with certification in chennai

January 19, 2019 at 4:02 AM  
Blogger Dhanish said...

You truly did more than visitors’ expectations. Thank you for rendering these helpful, trusted, edifying and also cool thoughts on the topic.
Best Dot Net Training center in Chennai|Best Dot Net Training center in Velachery

January 20, 2019 at 11:39 PM  
Blogger jefrin adams said...

Informative post thanks for sharing
blue prism training class chennai

January 21, 2019 at 11:11 PM  
Blogger deepti said...

I have read your blog its very attractive and impressive. I like it your blog.
Best MCSE Training Institute in Chennai | Best MCSE Training Institute in Velachery

January 23, 2019 at 10:41 PM  
«Oldest ‹Older 201 – 264 of 264

Post a Comment

Subscribe to Post Comments [Atom]

<< Home