经验总结

  1. 首页 > 生活 > >

Hadoop集群简单入门

生活百科

Hadoop集群搭建自己配置Hadoop的话太过复杂了，因为自己着急学习，就使用了黑马的快照 。如果小伙伴们也想的话可以直接看黑马的课程，快照的话关注黑马程序员公众号，输入Hadoop就能获取资料，到时候直接看课程P9就可以了 。
Hadoop集群启停命令和Web UI

  1. 【Hadoop集群简单入门】手动逐个启停
    优点：准确的启动或关闭进程，避免群起群停 。缺点：多个进程同时操作麻烦
  2. shell脚本一键启停
    前提：配置好SSH免密登录和workers文件 。HDFS集群：start-dfs.sh/stop-dfs.shYARN集群：start-yarn.sh/stop-yarn.shHadoop集群:start-all.sh/start-all.sh
  3. 启停结果查看
    1. jps命令查看进程2. 或者在下载路径下logs文件查看
启动结果：node1：
Hadoop集群简单入门

文章插图
node2：
Hadoop集群简单入门

文章插图
4. Web UI
HDFS Web界面：NameNode所在机器，端口是9870YARN Web界面：ResourceManager所在机器，端口号是8088结果如下：HDFS:
Hadoop集群简单入门

文章插图
其中在这个界面，我们主要是浏览文件系统，即下图：
Hadoop集群简单入门

文章插图
YARN:
Hadoop集群简单入门

文章插图
HDFS初体验[root@node1 ~]# hadoop fs -ls /Found 3 itemsdrwxr-xr-x- root supergroup0 2021-10-26 15:04 /itcastdrwx------- root supergroup0 2021-10-26 15:20 /tmpdrwxr-xr-x- root supergroup0 2021-10-26 15:23 /user[root@node1 ~]# hadoop fs -mkdir itcast[root@node1 ~]# hadoop fs -ls /Found 3 itemsdrwxr-xr-x- root supergroup0 2021-10-26 15:04 /itcastdrwx------- root supergroup0 2021-10-26 15:20 /tmpdrwxr-xr-x- root supergroup0 2021-10-26 15:23 /user[root@node1 ~]# hadoop fs -put anaconda-ks.cfg /itcast[root@node1 ~]# echo 1 > 1.txt[root@node1 ~]# ll总用量 32-rw-r--r--1 root root2 10月 24 21:15 1.txt-rw-------. 1 root root1340 9月11 2020 anaconda-ks.cfgdrwxr-xr-x2 root root55 10月5 00:08 hivedata-rw-------1 root root 23341 10月5 00:11 nohup.out[root@node1 ~]# hadoop fs -put 1.txt /运行结果：
Hadoop集群简单入门

文章插图
MapReduce+YARN初体验[root@node1 ~]# cd /export/server/hadoop-3.3.0/[root@node1 hadoop-3.3.0]# ll总用量 88drwxr-xr-x 2 root root203 7月15 2021 bindrwxr-xr-x 3 root root20 7月15 2021 etcdrwxr-xr-x 2 root root106 7月15 2021 includedrwxr-xr-x 3 root root20 7月15 2021 libdrwxr-xr-x 4 root root288 7月15 2021 libexec-rw-rw-r-- 1 root root22976 7月5 2020 LICENSE-binarydrwxr-xr-x 2 root root4096 7月15 2021 licenses-binary-rw-rw-r-- 1 root users 15697 3月25 2020 LICENSE.txtdrwxr-xr-x 3 root root4096 10月 24 20:33 logs-rw-rw-r-- 1 root users 27570 3月25 2020 NOTICE-binary-rw-rw-r-- 1 root users1541 3月25 2020 NOTICE.txt-rw-rw-r-- 1 root users175 3月25 2020 README.txtdrwxr-xr-x 3 root root4096 7月15 2021 sbindrwxr-xr-x 3 root root20 7月15 2021 share[root@node1 hadoop-3.3.0]# cd share/hadoop/[root@node1 hadoop]# ll总用量 12drwxr-xr-x 2 root root123 7月15 2021 clientdrwxr-xr-x 6 root root217 7月15 2021 commondrwxr-xr-x 6 root root 4096 7月15 2021 hdfsdrwxr-xr-x 5 root root 4096 7月15 2021 mapreducedrwxr-xr-x 7 root root87 7月15 2021 toolsdrwxr-xr-x 8 root root 4096 7月15 2021 yarn[root@node1 hadoop]# cd mapreduce/[root@node1 mapreduce]# ll总用量 5276-rw-r--r-- 1 root root589704 7月15 2021 hadoop-mapreduce-client-app-3.3.0.jar-rw-r--r-- 1 root root803842 7月15 2021 hadoop-mapreduce-client-common-3.3.0.jar-rw-r--r-- 1 root root 1623803 7月15 2021 hadoop-mapreduce-client-core-3.3.0.jar-rw-r--r-- 1 root root181995 7月15 2021 hadoop-mapreduce-client-hs-3.3.0.jar-rw-r--r-- 1 root root10323 7月15 2021 hadoop-mapreduce-client-hs-plugins-3.3.0.jar-rw-r--r-- 1 root root50701 7月15 2021 hadoop-mapreduce-client-jobclient-3.3.0.jar-rw-r--r-- 1 root root 1651503 7月15 2021 hadoop-mapreduce-client-jobclient-3.3.0-tests.jar-rw-r--r-- 1 root root91017 7月15 2021 hadoop-mapreduce-client-nativetask-3.3.0.jar-rw-r--r-- 1 root root62310 7月15 2021 hadoop-mapreduce-client-shuffle-3.3.0.jar-rw-r--r-- 1 root root22637 7月15 2021 hadoop-mapreduce-client-uploader-3.3.0.jar-rw-r--r-- 1 root root281197 7月15 2021 hadoop-mapreduce-examples-3.3.0.jardrwxr-xr-x 2 root root4096 7月15 2021 jdiffdrwxr-xr-x 2 root root30 7月15 2021 lib-examplesdrwxr-xr-x 2 root root4096 7月15 2021 sources[root@node1 mapreduce]# hadoop jar hadoop-mapreduce-examples-3.3.0.jar pi 2 2Number of Maps= 2Samples per Map = 2Wrote input for Map #0Wrote input for Map #1Starting Job2022-10-24 21:26:25,174 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at node1/192.168.88.151:80322022-10-24 21:26:25,982 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1666614796978_00012022-10-24 21:26:26,237 INFO input.FileInputFormat: Total input files to process : 22022-10-24 21:26:26,308 INFO mapreduce.JobSubmitter: number of splits:22022-10-24 21:26:26,492 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1666614796978_00012022-10-24 21:26:26,492 INFO mapreduce.JobSubmitter: Executing with tokens: []2022-10-24 21:26:26,687 INFO conf.Configuration: resource-types.xml not found2022-10-24 21:26:26,688 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.2022-10-24 21:26:27,169 INFO impl.YarnClientImpl: Submitted application application_1666614796978_00012022-10-24 21:26:27,218 INFO mapreduce.Job: The url to track the job: http://node1:8088/proxy/application_1666614796978_0001/2022-10-24 21:26:27,219 INFO mapreduce.Job: Running job: job_1666614796978_00012022-10-24 21:26:38,491 INFO mapreduce.Job: Job job_1666614796978_0001 running in uber mode : false2022-10-24 21:26:38,492 INFO mapreduce.Job:map 0% reduce 0%2022-10-24 21:26:48,699 INFO mapreduce.Job:map 100% reduce 0%2022-10-24 21:26:56,768 INFO mapreduce.Job:map 100% reduce 100%2022-10-24 21:26:56,777 INFO mapreduce.Job: Job job_1666614796978_0001 completed successfully2022-10-24 21:26:56,877 INFO mapreduce.Job: Counters: 54File System CountersFILE: Number of bytes read=50FILE: Number of bytes written=795057FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number of write operations=0HDFS: Number of bytes read=520HDFS: Number of bytes written=215HDFS: Number of read operations=13HDFS: Number of large read operations=0HDFS: Number of write operations=3HDFS: Number of bytes read erasure-coded=0Job CountersLaunched map tasks=2Launched reduce tasks=1Data-local map tasks=2Total time spent by all maps in occupied slots (ms)=13640Total time spent by all reduces in occupied slots (ms)=5089Total time spent by all map tasks (ms)=13640Total time spent by all reduce tasks (ms)=5089Total vcore-milliseconds taken by all map tasks=13640Total vcore-milliseconds taken by all reduce tasks=5089Total megabyte-milliseconds taken by all map tasks=13967360Total megabyte-milliseconds taken by all reduce tasks=5211136Map-Reduce FrameworkMap input records=2Map output records=4Map output bytes=36Map output materialized bytes=56Input split bytes=284Combine input records=0Combine output records=0Reduce input groups=2Reduce shuffle bytes=56Reduce input records=4Reduce output records=0Spilled Records=8Shuffled Maps =2Failed Shuffles=0Merged Map outputs=2GC time elapsed (ms)=906CPU time spent (ms)=5000Physical memory (bytes) snapshot=793993216Virtual memory (bytes) snapshot=8363589632Total committed heap usage (bytes)=677380096Peak Map Physical memory (bytes)=299003904Peak Map Virtual memory (bytes)=2789269504Peak Reduce Physical memory (bytes)=200515584Peak Reduce Virtual memory (bytes)=2788036608Shuffle ErrorsBAD_ID=0CONNECTION=0IO_ERROR=0WRONG_LENGTH=0WRONG_MAP=0WRONG_REDUCE=0File Input Format CountersBytes Read=236File Output Format CountersBytes Written=97Job Finished in 31.796 secondsEstimated value of Pi is 4.00000000000000000000

经验总结扩展阅读


上一篇：立冬节气有什么讲究吃什么 立冬节气风俗忌讳有哪些

下一篇：木床发霉怎么处理妙招