java Hive 错误：失败：执行错误，从 org.apache.hadoop.hive.ql.exec.mr.MapRedTask 返回代码 2

Hive Error : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

提问by Ankit Gupta

I have got twitter data using flume on HDFS. Have 3 node cluster and MySQL Metastore for hive.

我在 HDFS 上使用水槽获得了 twitter 数据。有 3 节点集群和用于配置单元的 MySQL Metastore。

When i execute below query

当我执行以下查询时

select user_name.screen_name, user_name.followers_count from BBMP_election_tweets limit 10;

It is giving proper results.

它给出了正确的结果。

But if i run below query

但是如果我在查询下面运行

select user_name.screen_name, user_name.followers_count c from BBMP_election_tweets order by c desc;

It throws below error

它抛出以下错误

Query ID = hduser_20150827102628_15a405ae-59a4-4c90-aa47-045d58c62f02 Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer= In order to limit the maximum number of reducers: set hive.exec.reducers.max= In order to set a constant number of reducers: set mapreduce.job.reduces= Starting Job = job_1439458788229_0016, Tracking URL = http://hdmaster:8088/proxy/application_1439458788229_0016/Kill Command = /usr/local/hadoop/bin/hadoop job -kill job_1439458788229_0016 Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 1 2015-08-27 10:27:33,567 Stage-1 map = 0%, reduce = 0% 2015-08-27 10:27:43,908 Stage-1 map = 50%, reduce = 0% 2015-08-27 10:27:45,015 Stage-1 map = 0%, reduce = 0% 2015-08-27 10:28:31,011 Stage-1 map = 100%, reduce = 100% Ended Job = job_1439458788229_0016 with errors Error during job, obtaining debugging information... Examining task ID: task_1439458788229_0016_m_000001 (and more) from job job_1439458788229_0016

Task with the most failures(4): ----- Task ID: task_1439458788229_0016_m_000001

URL:
http://hdmaster:8088/taskdetails.jsp?jobid=job_1439458788229_0016&tipid=task_1439458788229_0016_m_000001----- Diagnostic Messages for this Task: Error: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 17 more Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140) ... 22 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe not found at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328) at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109) ... 22 more Caused by: java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980) at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141) at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302) ... 24 more

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask MapReduce Jobs Launched: Stage-Stage-1: Map: 2 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL Total MapReduce CPU Time Spent: 0 msec

查询 ID = hduser_20150827102628_15a405ae-59a4-4c90-aa47-045d58c62f02 Total jobs = 1 Launching Job 1 out of 1 编译时确定的reduce 任务数：1 为了更改reducer 的平均负载：seter（。 exec.reducers.bytes.per.reducer=为了限制最大reducer数量：set hive.exec.reducers.max=为了设置恒定数量的reducer：set mapreduce.job.reduces=Starting Job = job_1439458788229_0016 , 跟踪网址 = http://hdmaster:8088/proxy/application_1439458788229_0016/Kill Command = /usr/local/hadoop/bin/hadoop job -kill job_1439458788229_0016 Stage-1的Hadoop作业信息：映射器数量：2；减速器数量：1 2015-08-27 10:27:33,567 Stage-1 map = 0%，reduce = 0% 2015-08-27 10:27:43,908 Stage-1 map = 50%，reduce = 0% 2015 -08-27 10:27:45,015 Stage-1 map = 0%，reduce = 0% 2015-08-27 10:28:31,011 Stage-1 map = 100%，reduce = 100% Ended Job = job_143945878816 有错误在作业期间，获取调试信息...正在检查任务 ID：task_1439458788229_0016_m_000001（以及更多）来自作业 job_1439458788229_0016

失败次数最多的任务（4）： ----- 任务ID：task_1439458788229_0016_m_000001

网址：
http://hdmaster:8088/taskdetails.jsp?jobid=job_1439458788229_0016&tipid=task_1439458788229_0016_m_000001----- 此任务的诊断消息：错误：java.lang.RuntimeException：在 org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util 配置对象时出错.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) 在 org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) 在 java.security.AccessController.doPrivileged(Native Method)在 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) 在 org.apache.hadoop 的 javax.security.auth.Subject.doAs(Subject.java:415)。mapred.YarnChild.main(YarnChild.java:158) 引起： java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 9 导致：java.lang.RuntimeException：在 org.apache.hadoop.util.ReflectionUtils.setConf( ReflectionUtils.java:75) 在 org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14 更多 引起：java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun .reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache。 hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 17 more 引起：java.lang.RuntimeException：地图操作符初始化在org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure失败(ExecMapper.java:140) ... 22 导致：org.apache.hadoop.hive.ql.metadata.HiveException：java.lang。ClassNotFoundException：在 org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328) 处找不到 com.cloudera.hive.serde.JSONSerDe 类。 MapOperator.setChildren(MapOperator.java:338) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109) ... 22 更多 引起：java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe 在 org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980) at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java :141) 在 org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302) ... 24 更多hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328) at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338) at org.apache.hadoop.hive。 ql.exec.mr.ExecMapper.configure(ExecMapper.java:109) ... 22 导致：java.lang.ClassNotFoundException：在 org.apache.hadoop.conf 中找不到类 com.cloudera.hive.serde.JSONSerDe .Configuration.getClassByName(Configuration.java:1980) at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141) at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI (MapOperator.java:302) ... 24 更多hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328) at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338) at org.apache.hadoop.hive。 ql.exec.mr.ExecMapper.configure(ExecMapper.java:109) ... 22 导致：java.lang.ClassNotFoundException：在 org.apache.hadoop.conf 中找不到类 com.cloudera.hive.serde.JSONSerDe .Configuration.getClassByName(Configuration.java:1980) at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141) at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI (MapOperator.java:302) ... 24 更多hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109) ... 22 导致：java.lang.ClassNotFoundException：在 org.apache.hadoop 找不到类 com.cloudera.hive.serde.JSONSerDe .conf.Configuration.getClassByName(Configuration.java:1980) at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141) at org.apache.hadoop.hive.ql.exec.MapOperator .getConvertedOI(MapOperator.java:302) ... 24 更多hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109) ... 22 导致：java.lang.ClassNotFoundException：在 org.apache.hadoop 找不到类 com.cloudera.hive.serde.JSONSerDe .conf.Configuration.getClassByName(Configuration.java:1980) at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141) at org.apache.hadoop.hive.ql.exec.MapOperator .getConvertedOI(MapOperator.java:302) ... 24 更多apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302) ... 24 更多apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302) ... 24 更多

失败：执行错误，从 org.apache.hadoop.hive.ql.exec.mr.MapRedTask MapReduce Jobs Launched: Stage-Stage-1: Map: 2 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL Total 返回代码 2 MapReduce CPU 花费时间：0 毫秒

When checked the logs created. It gives below message

检查时创建的日志。它给出了以下消息

2015-08-27 10:28:48,852 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1439458788229_0016, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@988145f) 2015-08-27 10:28:49,023 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now. 2015-08-27 10:28:49,627 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1439458788229_0016 2015-08-27 10:28:50,363 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2015-08-27 10:28:51,186 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2015-08-27 10:28:51,534 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: Paths:/user/flume/tweets/BBMP_elections/2015/08/25/11/FlumeData.1440484157685:0+52974,/user/flume/tweets/BBMP_elections/2015/08/25/11/FlumeData.1440484188899:0+20119InputFormatClass: org.apache.hadoop.mapred.TextInputFormat

2015-08-27 10:28:51,701 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: PLAN PATH = hdfs://hdmaster:8020/tmp/hive/hduser/55daf291-b44e-414b-b3a5-e326134f9f24/hive_2015-08-27_10-26-28_312_5076847637673296334-1/-mr-10004/6e310852-ea63-4789-93f1-754147923e7f/map.xml 2015-08-27 10:28:51,701 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: ***************non-local mode*************** 2015-08-27 10:28:51,701 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: local path = hdfs://hdmaster:8020/tmp/hive/hduser/55daf291-b44e-414b-b3a5-e326134f9f24/hive_2015-08-27_10-26-28_312_5076847637673296334-1/-mr-10004/6e310852-ea63-4789-93f1-754147923e7f/map.xml 2015-08-27 10:28:51,701 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: Open file to read in plan: hdfs://hdmaster:8020/tmp/hive/hduser/55daf291-b44e-414b-b3a5-e326134f9f24/hive_2015-08-27_10-26-28_312_5076847637673296334-1/-mr-10004/6e310852-ea63-4789-93f1-754147923e7f/map.xml 2015-08-27 10:28:51,719 INFO [main] org.apache.hadoop.hive.ql.log.PerfLogger: 2015-08-27 10:28:51,720 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: Deserializing MapWork via kryo 2015-08-27 10:28:52,043 INFO [main] org.apache.hadoop.hive.ql.log.PerfLogger: 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader: Processing file hdfs://hdmaster:8020/user/flume/tweets/BBMP_elections/2015/08/25/11/FlumeData.1440484157685 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.file is deprecated. Instead, use mapreduce.map.input.file 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.mapred.MapTask: numReduceTasks: 1 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600

2015-08-27 10:28:52,243 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: Initializing operator MAP[3] 2015-08-27 10:28:52,243 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: Initialization Done 3 MAP 2015-08-27 10:28:52,243 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: Operator 3 MAP initialized 2015-08-27 10:28:52,262 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 17 more Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140) ... 22 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe not found at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328) at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109) ... 22 more Caused by: java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980) at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141) at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302) ... 24 more

2015-08-27 10:28:52,269 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the task 2015-08-27 10:28:52,278 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... 2015-08-27 10:28:52,278 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped. 2015-08-27 10:28:52,279 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.

2015-08-27 10:28:48,852 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1439458788229_0016, Ident: (org.apache.hadoop.mapreduce.JobTokenIdentifier.token @988145f) 2015-08-27 10:28:49,023 INFO [main] org.apache.hadoop.mapred.YarnChild：在重试之前休眠 0 毫秒。现在为空。2015-08-27 10:28:49,627 INFO [main] org.apache.hadoop.mapred.YarnChild：mapreduce.cluster.local.dir for child：/tmp/hadoop-hduser/nm-local-dir/usercache/hduser /appcache/application_1439458788229_0016 2015-08-27 10:28:50,363 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id 已弃用。相反，使用 dfs.metrics.session-id 2015-08-27 10:28:51,186 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2015-08-27 10:28:51,534 INFO [主要] 组织。

8020 / TMP /蜂巢/ hduser / 55daf291-b44e-414B-b3a5-e326134f9f24 / hive_2015-08-27_10-26-28_312_5076847637673296334-1 / -mr-10004 / 6e310852-ea63-4789-93f1-754147923e7f / map.xml 2015 - 08-27 10:28:51,719 信息 [主要] org.apache.hadoop.hive.ql.log.PerfLogger: 2015-08-27 10:28:51,720 信息 [主要] org.apache.hadoop.hive.ql。 exec.Utilities: Deserializing MapWork via kryo 2015-08-27 10:28:52,043 INFO [main] org.apache.hadoop.hive.ql.log.PerfLogger: 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader：处理文件 hdfs://hdmaster:8020/user/flume/tweets/BBMP_elections/2015/08/25/11/FlumeData.1440484157685 2015-08-2 28:52,080 信息 [主要] org.apache.hadoop.conf.Configuration.deprecation：不推荐使用 map.input.file。相反，使用 mapreduce.map.input.file 2015-08-27 10:28:52,080 INFO [main] org. apache.hadoop.conf.Configuration.deprecation：不推荐使用 map.input.start。相反，使用 mapreduce.map.input.start 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.length 已弃用。相反，使用 mapreduce.map.input.length 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.mapred.MapTask: numReduceTasks: 1 2015-08-27 10:28:52,228 INFO [main] ] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io. sort.mb: 100 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: 软限制在 83886080 2015-08-27 10:28:52,228 INFO [main] org.apache .hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2015-08-27 10:28:52,228 信息 [主要] org.apache.hadoop.mapred。地图任务：kvstart = 26214396；长度 = 6553600

2015-08-27 10:28:52,243 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator：初始化操作符 MAP[3] 2015-08-27 10:28:52,243 INFO [main] org. apache.hadoop.hive.ql.exec.MapOperator：初始化完成 3 MAP 2015-08-27 10:28:52,243 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator：Operator 3 MAP 初始化 2015- 08-27 10:28:52,262 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: 在 org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils) 配置对象时出错.java:109) 在 org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) 在 org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) 在 org.apache.hadoop。 mapred.MapTask.runOldMapper(MapTask.java:446) 在 org.apache.hadoop.mapred.MapTask。run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs (Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 引起：java。 lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 9 more 引起：java.lang。RuntimeException: 在 org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop 配置对象时出错.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14 更多 引起：java.lang.reflect.InvocationTargetException at sun。反射.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke( Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 17 更多 引起：java。lang.RuntimeException: Map operator 初始化失败在 org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140) ... 22 more 引起：org.apache.hadoop.hive.ql .metadata.HiveException: java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe 在 org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328) 中找不到.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109) ... 22 更多作者：java.lang.ClassNotFoundException：在 org.apache.hadoop.hive.ql 的 org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980) 中找不到类 com.cloudera.hive.serde.JSONSerDe。 plan.PartitionDesc.getDeserializer(PartitionDesc.java:141) 在 org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302) ... 24 更多

2015-08-27 10:28:52,269 INFO [main] org.apache.hadoop.mapred.Task：运行清理任务 2015-08-27 10:28:52,278 INFO [main] org.apache.hadoop.metrics2 .impl.MetricsSystemImpl：停止 MapTask 指标系统... 2015-08-27 10:28:52,278 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl：MapTask 指标系统停止。2015-08-27 10:28:52,279 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl：MapTask 指标系统关闭完成。

Please suggest. How can this issue be resolved. and why is it occuring.

请建议。如何解决这个问题。以及为什么会发生。

Thanks

谢谢

回答by seneque

It is a classpath problem Root cause is java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe

这是一个类路径问题根本原因是 java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe

you can have a look here to add hive-serdes: https://github.com/cloudera/cdh-twitter-example

您可以在这里查看添加 hive-serdes：https: //github.com/cloudera/cdh-twitter-example