java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found

I am trying to use the Hadoop HDFS Java API to list all files in HDFS. I am able to list the files on remote HDFS by running the code in my local eclipse.

But i get the exception

java.lang.ClassNotFoundException: Class  org.apache.hadoop.hdfs.DistributedFileSystem 
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2290)
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2303)
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:87)
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2342)
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2324)
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351)
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:163)

when I execute the code from a web server.

I have added the below maven dependencies.

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-hdfs</artifactId>
    <version>2.0.0-cdh4.5.0</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-auth</artifactId>
    <version>2.0.0-cdh4.5.0</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>2.0.0-cdh4.5.0</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-core</artifactId>
    <version>2.0.0-mr1-cdh4.5.0</version>
</dependency>

Also I have embedded the required jars into the exported jar and maven has added the same in the buildpath.

If any one has encountered this issue earlier request you to please share the solution.

Answers


I am facing a similar issue with Apache hadoop 2.2.0 realease, I did a workaround by running it as a separate process, by

final Process p = Runtime.getRuntime ().exec ("java -jar {jarfile} {classfile}";
final Scanner output = new Scanner (p.getErrorStream ());
while (output.hasNext ()) {
            try {
                System.err.println (output.nextLine ());
            } catch (final Exception e) {

            }
 }

The jar file contains the implementation using the apache hadoop 2.2.0 jars.

Though, I am still searching for an exact solution.


For me, hadoop-hdfs-2.6.0.jar was missing in zeppelin server's lib dir. I copied in zeppelin lib forder and my problem was resolved. :) and add dependency for hadoop-hdfs-2.6.0.jar in pom.xml also.


Need Your Help

Sorting an array into least to greatest C

c sorting

So right now my program is simply storing the lowest values I input SIZE number of times, can someone help me out with this? I thought that testing the values against 'final' would fix this.

HTML5 Canvas Font Size Based on Canvas Size

javascript html5 canvas

I have the canvas size set to 1/3 the window.innerHeight and window.innerWidth of the browser.