Hadoop部署练习

任务一：

1.1）修改core-site.xml

vi core-site.xml

<property>
    <name>hadoop.tmp.dir</name>
    <value>/usr/local/hadoop/tmp</value>
</property>

1.2）配置 hdfs-site.xml

vi hdfs-site.xml

<property>
        <name>dfs.namenode.name.dir</name>
        <value>/usr/local/hadoop/tmp/dfs/name</value>
</property>

1.3 ）dfs.datanode.data.dir

<property>
        <name>dfs.datanode.data.dir</name>
        <value>/usr/local/hadoop/tmp/dfs/data</value>
</property>

1.4 ）格式化NameNode

格式化HDFS
hadoop namenode -format

查看hdfs情况
hadoop dfsadmin -report

1.5 ）开启NameNode 和DataNode

start-dfs.sh

输入jps查看进程：
5161 SecondaryNameNode
4989 NameNode

start-yarn.sh 

再输入jps查看进程：
5161 SecondaryNameNode
5320 ResourceManager
4989 NameNode

2.1）在hdfs创建用户目录

hdfs dfs -mkdir -p /user/hadoop

2.2）创建input目录，上传数据文件

hdfs dfs -mkdir /input

//删除output目录
hadoop fs -rm -r /output
//本地创建测试文件
cd /usr/local/hadoop/data/
touch test.txt
vi test.txt

this is a test log

cat test.txt

上传：
hadoop fs -put /usr/local/hadoop/data/test.txt /input

2.3）运行hadoop-mapreduce-examples-2.7.3.jar

所在目录：$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar

hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /input /output

2.4）查看输出结果

hadoop fs -ls /output
hadoop fs -cat /output/part-r-00000

3）配置yarn-site.xml

<property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>1024</value>
</property>

<property>
    <name>mapreduce.map.memory.mb</name>
    <value>1024</value>
</property>

<property>
    <name>mapreduce.map.java.opts</name>
    <value>-Xmx768m</value>
</property>

<property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>2048</value>
</property>

<property>
    <name>mapreduce.reduce.java.opts</name>
    <value>-Xmx1536m</value>
</property>

重启Hadoop：

stop-all.sh
start-all.sh