Usage: hdfs dfsadmin Note: Administrative commands can only be run as the HDFS superuser. [-report [-live] [-dead] [-decommissioning] [-enteringmaintenance] [-inmaintenance]]
<!-- 指定YARN的ResourceManager的地址 --> <property> <name>yarn.resourcemanager.hostname</name> <value>node1</value> </property> <!-- yarn的web访问地址 --> <property> <description> The http address of the RM web application. If only a host is provided as the value, the webapp will be served on a random port. </description> <name>yarn.resourcemanager.webapp.address</name> <value>${yarn.resourcemanager.hostname}:8088</value> </property>
Generic options supported are: -conf <configuration file> specify an application configuration file -D <property=value> define a value for a given property -fs <file:///|hdfs://namenode:port> specify default filesystem URL to use, overrides 'fs.defaultFS' property from configurations. -jt <local|resourcemanager:port> specify a ResourceManager -files <file1,...> specify a comma-separated list of files to be copied to the map reduce cluster -libjars <jar1,...> specify a comma-separated list of jar files to be included in the classpath -archives <archive1,...> specify a comma-separated list of archives to be unarchived on the compute machines
The general command line syntax is: command [genericOptions] [commandOptions]
[hadoop@node1 ~]$ hadoop jar /app/hadoop-2.10.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.10.1.jar wordcount /tmp/test.txt /tmp/output 22/05/11 23:19:17 INFO client.RMProxy: Connecting to ResourceManager at node1/199.188.166.111:8032 22/05/11 23:19:19 INFO input.FileInputFormat: Total input files to process : 1 22/05/11 23:19:19 INFO mapreduce.JobSubmitter: number of splits:1 22/05/11 23:19:20 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1652322858586_0001 22/05/11 23:19:20 INFO conf.Configuration: resource-types.xml not found 22/05/11 23:19:20 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 22/05/11 23:19:20 INFO resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE 22/05/11 23:19:20 INFO resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE 22/05/11 23:19:21 INFO impl.YarnClientImpl: Submitted application application_1652322858586_0001 22/05/11 23:19:21 INFO mapreduce.Job: The url to track the job: http://node1:8088/proxy/application_1652322858586_0001/ 22/05/11 23:19:21 INFO mapreduce.Job: Running job: job_1652322858586_0001 22/05/11 23:19:38 INFO mapreduce.Job: Job job_1652322858586_0001 running in uber mode : false 22/05/11 23:19:38 INFO mapreduce.Job: map 0% reduce 0% 22/05/11 23:19:50 INFO mapreduce.Job: map 100% reduce 0% 22/05/11 23:20:02 INFO mapreduce.Job: map 100% reduce 100% 22/05/11 23:20:03 INFO mapreduce.Job: Job job_1652322858586_0001 completed successfully 22/05/11 23:20:03 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=2274 FILE: Number of bytes written=421473 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=3213 HDFS: Number of bytes written=1928 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=8053 Total time spent by all reduces in occupied slots (ms)=8758 Total time spent by all map tasks (ms)=8053 Total time spent by all reduce tasks (ms)=8758 Total vcore-milliseconds taken by all map tasks=8053 Total vcore-milliseconds taken by all reduce tasks=8758 Total megabyte-milliseconds taken by all map tasks=8246272 Total megabyte-milliseconds taken by all reduce tasks=8968192 Map-Reduce Framework Map input records=38 Map output records=335 Map output bytes=4379 Map output materialized bytes=2274 Input split bytes=95 Combine input records=335 Combine output records=87 Reduce input groups=87 Reduce shuffle bytes=2274 Reduce input records=87 Reduce output records=87 Spilled Records=174 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=222 CPU time spent (ms)=2630 Physical memory (bytes) snapshot=396931072 Virtual memory (bytes) snapshot=3804737536 Total committed heap usage (bytes)=194383872 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=3118 File Output Format Counters Bytes Written=1928
[hadoop@node1 ~]$ hadoop jar /app/hadoop-2.10.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.10.1.jar pi 10 100 Number of Maps = 10 Samples per Map = 100 Wrote input for Map #0 Wrote input for Map #1 Wrote input for Map #2 Wrote input for Map #3 Wrote input for Map #4 Wrote input for Map #5 Wrote input for Map #6 Wrote input for Map #7 Wrote input for Map #8 Wrote input for Map #9 Starting Job 22/05/11 23:35:11 INFO client.RMProxy: Connecting to ResourceManager at node1/199.188.166.111:8032 22/05/11 23:35:12 INFO input.FileInputFormat: Total input files to process : 10 22/05/11 23:35:12 INFO mapreduce.JobSubmitter: number of splits:10 22/05/11 23:35:13 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1652322858586_0002 22/05/11 23:35:13 INFO conf.Configuration: resource-types.xml not found 22/05/11 23:35:13 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 22/05/11 23:35:13 INFO resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE 22/05/11 23:35:13 INFO resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE 22/05/11 23:35:13 INFO impl.YarnClientImpl: Submitted application application_1652322858586_0002 22/05/11 23:35:13 INFO mapreduce.Job: The url to track the job: http://node1:8088/proxy/application_1652322858586_0002/ 22/05/11 23:35:13 INFO mapreduce.Job: Running job: job_1652322858586_0002 22/05/11 23:35:24 INFO mapreduce.Job: Job job_1652322858586_0002 running in uber mode : false 22/05/11 23:35:24 INFO mapreduce.Job: map 0% reduce 0% 22/05/11 23:35:42 INFO mapreduce.Job: map 20% reduce 0% 22/05/11 23:36:00 INFO mapreduce.Job: map 20% reduce 7% 22/05/11 23:36:28 INFO mapreduce.Job: map 30% reduce 7% 22/05/11 23:36:29 INFO mapreduce.Job: map 50% reduce 7% 22/05/11 23:36:30 INFO mapreduce.Job: map 70% reduce 7% 22/05/11 23:36:31 INFO mapreduce.Job: map 100% reduce 7% 22/05/11 23:36:32 INFO mapreduce.Job: map 100% reduce 100% 22/05/11 23:36:33 INFO mapreduce.Job: Job job_1652322858586_0002 completed successfully 22/05/11 23:36:33 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=226 FILE: Number of bytes written=2297625 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=2620 HDFS: Number of bytes written=215 HDFS: Number of read operations=43 HDFS: Number of large read operations=0 HDFS: Number of write operations=3 Job Counters Launched map tasks=10 Launched reduce tasks=1 Data-local map tasks=10 Total time spent by all maps in occupied slots (ms)=528176 Total time spent by all reduces in occupied slots (ms)=48013 Total time spent by all map tasks (ms)=528176 Total time spent by all reduce tasks (ms)=48013 Total vcore-milliseconds taken by all map tasks=528176 Total vcore-milliseconds taken by all reduce tasks=48013 Total megabyte-milliseconds taken by all map tasks=540852224 Total megabyte-milliseconds taken by all reduce tasks=49165312 Map-Reduce Framework Map input records=10 Map output records=20 Map output bytes=180 Map output materialized bytes=280 Input split bytes=1440 Combine input records=0 Combine output records=0 Reduce input groups=2 Reduce shuffle bytes=280 Reduce input records=20 Reduce output records=0 Spilled Records=40 Shuffled Maps =10 Failed Shuffles=0 Merged Map outputs=10 GC time elapsed (ms)=9894 CPU time spent (ms)=13350 Physical memory (bytes) snapshot=2232963072 Virtual memory (bytes) snapshot=20908384256 Total committed heap usage (bytes)=1540988928 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=1180 File Output Format Counters Bytes Written=97 Job Finished in 82.602 seconds Estimated value of Pi is 3.14800000000000000000
