Hadoop: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class is expected

Hadoop: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected

提问by dokondr

My MapReduce jobs runs ok when assembled in Eclipse with all possible Hadoop and Hive jars included in Eclipse project as dependencies. (These are the jars that come with single node, local Hadoop installation).

我的 MapReduce 作业在 Eclipse 中组装时运行正常，所有可能的 Hadoop 和 Hive jar 都包含在 Eclipse 项目中作为依赖项。（这些是单节点本地 Hadoop 安装附带的 jar）。

Yet when trying to run the same program assembled using Maven project (see below) I get:

然而，当尝试运行使用 Maven 项目（见下文）组装的相同程序时，我得到：

 Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected

This exception happens when program is assembled using the following Maven project:

使用以下 Maven 项目组装程序时会发生此异常：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.bigdata.hadoop</groupId>
  <artifactId>FieldCounts</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>FieldCounts</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
     <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-hdfs</artifactId>
        <version>2.2.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>2.2.0</version>
    </dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
    <version>2.2.0</version>
</dependency>
<dependency>
    <groupId>org.apache.hive.hcatalog</groupId>
    <artifactId>hcatalog-core</artifactId>
    <version>0.12.0</version>
</dependency>
<dependency>
    <groupId>com.google.guava</groupId>
    <artifactId>guava</artifactId>
    <version>16.0.1</version>
</dependency>
  </dependencies>     
    <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>2.3.2</version>
        <configuration>
            <source>${jdk.version}</source>
            <target>${jdk.version}</target>
        </configuration>
      </plugin>       
  <plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-assembly-plugin</artifactId>
    <executions>
      <execution>
         <goals>
           <goal>attached</goal>
         </goals>
         <phase>package</phase>
         <configuration>
           <descriptorRefs>
             <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
          <archive>
            <manifest>
              <mainClass>com.bigdata.hadoop.FieldCounts</mainClass>
            </manifest>
          </archive>
        </configuration>
     </execution>
  </executions>
</plugin>
 </plugins>
 </build>       
</project>

* Please advise where and how to find compatible Hadoop jars? *

* 请告知在哪里以及如何找到兼容的 Hadoop jars？*

[update_1]I am running Hadoop 2.2.0.2.0.6.0-101

[update_1]我正在运行 Hadoop 2.2.0.2.0.6.0-101

As I have found here: https://github.com/kevinweil/elephant-bird/issues/247

正如我在这里找到的：https: //github.com/kevinweil/elephant-bird/issues/247

Hadoop 1.0.3: JobContext is a class

Hadoop 1.0.3：JobContext 是一个类

Hadoop 2.0.0: JobContext is an interface

Hadoop 2.0.0：JobContext 是一个接口

In my pom.xml I have three jars with version 2.2.0

在我的 pom.xml 中，我有三个版本为 2.2.0 的罐子

hadoop-hdfs 2.2.0
hadoop-common 2.2.0
hadoop-mapreduce-client-jobclient 2.2.0
hcatalog-core 0.12.0

The only exception is hcatalog-corewhich version is 0.12.0, I could not find any more recent version of this jarand I need it!

唯一的例外是hcatalog-core哪个版本是 0.12.0，我找不到这个 jar 的任何更新版本，我需要它！

How can I find which of these 4 jars produces java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected?

我怎样才能找到这 4 个罐子中的java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected哪一个？

Please, give me an idea how to solve this. (The only solution I see is to compile everything from source!)

请给我一个想法如何解决这个问题。（我看到的唯一解决方案是从源代码编译所有内容！）

[/update_1]

[/更新_1]

Full text of my MarReduce Job:

我的 MarReduce 工作全文：

package com.bigdata.hadoop;

import java.io.IOException;
import java.util.*;

import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.util.*;
import org.apache.hcatalog.mapreduce.*;
import org.apache.hcatalog.data.*;
import org.apache.hcatalog.data.schema.*;
import org.apache.log4j.Logger;

public class FieldCounts extends Configured implements Tool {

    public static class Map extends Mapper<WritableComparable, HCatRecord, TableFieldValueKey, IntWritable> {

        static Logger logger = Logger.getLogger("com.foo.Bar");

        static boolean firstMapRun = true;
        static List<String> fieldNameList = new LinkedList<String>();
        /**
         * Return a list of field names not containing `id` field name
         * @param schema
         * @return
         */
        static List<String> getFieldNames(HCatSchema schema) {
            // Filter out `id` name just once
            if (firstMapRun) {
                firstMapRun = false;
                List<String> fieldNames = schema.getFieldNames();
                for (String fieldName : fieldNames) {
                    if (!fieldName.equals("id")) {
                        fieldNameList.add(fieldName);
                    }
                }
            } // if (firstMapRun)
            return fieldNameList;
        }

        @Override
      protected void map( WritableComparable key,
                          HCatRecord hcatRecord,
                          //org.apache.hadoop.mapreduce.Mapper
                          //<WritableComparable, HCatRecord, Text, IntWritable>.Context context)
                          Context context)
            throws IOException, InterruptedException {

            HCatSchema schema = HCatBaseInputFormat.getTableSchema(context.getConfiguration());

           //String schemaTypeStr = schema.getSchemaAsTypeString();
           //logger.info("******** schemaTypeStr ********** : "+schemaTypeStr);

           //List<String> fieldNames = schema.getFieldNames();
            List<String> fieldNames = getFieldNames(schema);
            for (String fieldName : fieldNames) {
                Object value = hcatRecord.get(fieldName, schema);
                String fieldValue = null;
                if (null == value) {
                    fieldValue = "<NULL>";
                } else {
                    fieldValue = value.toString();
                }
                //String fieldNameValue = fieldName+"."+fieldValue;
                //context.write(new Text(fieldNameValue), new IntWritable(1));
                TableFieldValueKey fieldKey = new TableFieldValueKey();
                fieldKey.fieldName = fieldName;
                fieldKey.fieldValue = fieldValue;
                context.write(fieldKey, new IntWritable(1));
            }

        }       
    }

    public static class Reduce extends Reducer<TableFieldValueKey, IntWritable,
                                       WritableComparable, HCatRecord> {

        protected void reduce( TableFieldValueKey key,
                               java.lang.Iterable<IntWritable> values,
                               Context context)
                               //org.apache.hadoop.mapreduce.Reducer<Text, IntWritable,
                               //WritableComparable, HCatRecord>.Context context)
            throws IOException, InterruptedException {
            Iterator<IntWritable> iter = values.iterator();
            int sum = 0;
            // Sum up occurrences of the given key 
            while (iter.hasNext()) {
                IntWritable iw = iter.next();
                sum = sum + iw.get();
            }

            HCatRecord record = new DefaultHCatRecord(3);
            record.set(0, key.fieldName);
            record.set(1, key.fieldValue);
            record.set(2, sum);

            context.write(null, record);
        }
    }

    public int run(String[] args) throws Exception {
        Configuration conf = getConf();
        args = new GenericOptionsParser(conf, args).getRemainingArgs();

        // To fix Hadoop "META-INFO" (http://stackoverflow.com/questions/17265002/hadoop-no-filesystem-for-scheme-file)
        conf.set("fs.hdfs.impl",
                org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
        conf.set("fs.file.impl",
                org.apache.hadoop.fs.LocalFileSystem.class.getName());

        // Get the input and output table names as arguments
        String inputTableName = args[0];
        String outputTableName = args[1];
        // Assume the default database
        String dbName = null;

        Job job = new Job(conf, "FieldCounts");

        HCatInputFormat.setInput(job,
                InputJobInfo.create(dbName, inputTableName, null));
        job.setJarByClass(FieldCounts.class);
        job.setMapperClass(Map.class);
        job.setReducerClass(Reduce.class);

        // An HCatalog record as input
        job.setInputFormatClass(HCatInputFormat.class);

        // Mapper emits TableFieldValueKey as key and an integer as value
        job.setMapOutputKeyClass(TableFieldValueKey.class);
        job.setMapOutputValueClass(IntWritable.class);

        // Ignore the key for the reducer output; emitting an HCatalog record as
        // value
        job.setOutputKeyClass(WritableComparable.class);
        job.setOutputValueClass(DefaultHCatRecord.class);
        job.setOutputFormatClass(HCatOutputFormat.class);

        HCatOutputFormat.setOutput(job,
                OutputJobInfo.create(dbName, outputTableName, null));
        HCatSchema s = HCatOutputFormat.getTableSchema(job);
        System.err.println("INFO: output schema explicitly set for writing:"
                + s);
        HCatOutputFormat.setSchema(job, s);
        return (job.waitForCompletion(true) ? 0 : 1);
    }

    public static void main(String[] args) throws Exception {
        String classpath = System.getProperty("java.class.path");
        //System.out.println("*** CLASSPATH: "+classpath);       
        int exitCode = ToolRunner.run(new FieldCounts(), args);
        System.exit(exitCode);
    }
}

And class for complex key:

和复杂键的类：

package com.bigdata.hadoop;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;

import org.apache.hadoop.io.WritableComparable;

import com.google.common.collect.ComparisonChain;

public class TableFieldValueKey  implements WritableComparable<TableFieldValueKey> {

      public String fieldName;
      public String fieldValue;

      public TableFieldValueKey() {} //must have a default constructor
      //

      public void readFields(DataInput in) throws IOException {
        fieldName = in.readUTF();
        fieldValue = in.readUTF();
      }

      public void write(DataOutput out) throws IOException {
        out.writeUTF(fieldName);
        out.writeUTF(fieldValue);
      }

      public int compareTo(TableFieldValueKey o) {
        return ComparisonChain.start().compare(fieldName, o.fieldName)
            .compare(fieldValue, o.fieldValue).result();
      }

    }

回答by Chiron

Obviously, you have versions incompatibility between you Hadoop and Hive versions. You need to upgrade (or downgrade) your Hadoop version or Hive version.

显然，您的 Hadoop 和 Hive 版本之间存在版本不兼容。您需要升级（或降级）您的 Hadoop 版本或 Hive 版本。

This is due the incompatibility between Hadoop 1 and Hadoop 2.

这是因为 Hadoop 1 和 Hadoop 2 之间不兼容。

回答by SachinJ

Hadoop has gone through a huge code refactoring from Hadoop 1.0to Hadoop 2.0. One side effect is that code compiled against Hadoop 1.0 is not compatible with Hadoop 2.0 and vice-versa. However source code is mostly compatible and thus one just need to recompile code with target Hadoop distribution.

Hadoop 经历了巨大的代码重构，从Hadoop 1.0到Hadoop 2.0. 一个副作用是针对 Hadoop 1.0 编译的代码与 Hadoop 2.0 不兼容，反之亦然。然而，源代码大多是兼容的，因此只需要使用目标 Hadoop 发行版重新编译代码即可。

The exception "Found interface X, but class was expected" is very common when you're running code that is compiled for Hadoop 1.0 on Hadoop 2.0 or vice-versa.

Found interface X, but class was expected当您在 Hadoop 2.0 上运行为 Hadoop 1.0 编译的代码时，异常“ ”非常常见，反之亦然。

You can find the correct hadoop version used in the cluster, then specify that hadoop version in the pom.xml file Build your project with the same version of hadoop used in the cluster and deploy it.

您可以找到集群中使用的正确 hadoop 版本，然后在 pom.xml 文件中指定该 hadoop 版本 使用集群中使用的相同版本的 hadoop 构建您的项目并部署它。

回答by akshat thakar

You need to recompile "hcatalog-core" to support Hadoop 2.0.0. Currently "hcatalog-core" only supports Hadoop 1.0

您需要重新编译“hcatalog-core”以支持 Hadoop 2.0.0。目前“hcatalog-core”仅支持Hadoop 1.0

回答by nikoo28

Look for entries like this

寻找这样的条目

<dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>1.2.1</version>
</dependency>

in your pom.xml. These define the hadoop version to use. Change them or remove them as per your requirements.

在你的 pom.xml 中。这些定义了要使用的 hadoop 版本。根据您的要求更改或删除它们。

回答by Abhiram

Even I ran through this problem. Was trying use HCatMultipleInputs with hive-hcatalog-core-0.13.0.jar. We are using hadoop 2.5.1.

甚至我也遇到了这个问题。正在尝试将 HCatMultipleInputs 与 hive-hcatalog-core-0.13.0.jar 一起使用。我们正在使用 hadoop 2.5.1。

The following code change helped me fix the issue:

以下代码更改帮助我解决了这个问题：

//JobContext ctx = new JobContext(conf,jobContext.getJobID());
JobContext ctx = new Job(conf);