Problem with dependency maven jar , i am getting below error ,

0
votes

2020-11-26 17:47:55 ERROR OctetStreamMetadataHandler:503 - No FileSystem for scheme: file java.io.IOException: No FileSystem for scheme: file at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:166) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287) at org.apache.orc.impl.ReaderImpl.(ReaderImpl.java:342) at org.apache.orc.OrcFile.createReader(OrcFile.java:342) at com.alation.azure.data.lake.metadata.OctetStreamMetadataHandler.scanORCMetadata(OctetStreamMetadataHandler.java:484) at com.alation.azure.data.lake.metadata.OctetStreamMetadataHandler.getSchema(OctetStreamMetadataHandler.java:748) at com.alation.azure.data.lake.main.AzureFileIndexer.lambda$processAzureFiles$0(AzureFileIndexer.java:199) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1494) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:747) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:283) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1603) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

1
javaorc
Looks like there's an asynch file indexer you'd like to run in Azure. Here's a possible explanation: cloud based systems don't have a file system. You're running in a container. You don't have access to the server file system. You'll have to provide access to that data set from an S3 bucket or a database. - duffymo
I am using azure SDK to extract file system and post metadata to some other web, I have a shellcode version, that's working fine, in this case, I use this jar and included this jar inside the war, after that only its not working , - jack
There's no file system in the WAR, either. You only have the CLASSPATH to deal with. Wrong all around. - duffymo
In classpath where I need to add eclipse or my system - jack
Eclipse won't be there when you deploy your system. It's an IDE. I would suggest that you start thinking about this problem from first principles and stop worrying about answering your questions. You want to index data. Where will that data live when you deploy? How will your indexer access it? - duffymo

1 Answers

0
votes

Hadoop will usually run on a server, with full access to a file system.

If you intend to POST data from a browser UI to be indexed, perhaps your Spring Boot web app could write the data to a Rabbit MQ using Spring Stream. You could write a listener to the queue that would write a file to the Hadoop server file system and index the data that way.

The problem is that your web app doesn't have access to a file system. It probably runs in a container, without access to a disk.

Why are you deploying your Spring Boot app as a WAR file? You don't need a Java EE app server. It would be much better to deploy it as a fat, executable JAR file. You only need a Java JVM that way.

Don't bundle a UI in with your Spring Boot controller. Let it be an independent micro service. Keep the UI separate from the service.