Avro hadoop random access to files

Question

I am wondering if Avro supports random access or queries. For example if I create a Avro file call it B.avro which contains 2 binary files X.png and Y.png, is it possible to directly access Y.png? without the need of iterating through the entire file, it would be great if there was a way to access the file contents directly using the the file key.

If not is there any other data structure that would allow me to do this in the hadoop environment sequenceFiles, HAR ? I am basically using Avro as a way to deal with a large number of small files in hadoop, but I would also like to query these files which makes it tough when storing them in larger collections.

Thanks.

Answer 1

I don't know if there is any OOTB feature that allows us to access value through its key. But random access to AVro data files is supported by public void seek(long position), provided by DataFileReader.

You might find MapFile useful. The class MapFile.Reader allows us to get the value for a named key.

Don't mind please if this is exactly not what you need.