I need to get the data from REST JSON API (every 24h) which returns a lot of data in array (lets say 500MB). Now, in order to do that I was thinking of using Spring Batch. I would process JSON in chunks (say 1000 records per chunk) and then index it in bulk in elastic search. The problem is that all those spring batch support classes (JobLauncher, JobExplorer, JobRepository, ...) look too much to me. I suppose I would need just some retryable runnable that would use JsonBufferedReaderFactory and FlatFileItemReader like so:
@Bean
public FlatFileItemReader<SomeObj> ffir(){
FlatFileItemReader<SomeObj> ffir = new FlatFileItemReader<>();
ffir.setBufferedReaderFactory(new JsonBufferedReaderFactory());
ffir.setLineMapper(new JsonLineMapper<>(SomeObj.class));
try {
ffir.setResource(new UrlResource("http://localhost:8080/cars/759"));
} catch (MalformedURLException e) {
e.printStackTrace();
}
return ffir;
}
where reader factory is:
public class JsonBufferedReaderFactory implements BufferedReaderFactory {
@Override
public BufferedReader create(Resource resource, String encoding) throws IOException {
return new JsonBufferedReader(new InputStreamReader(resource.getInputStream(), encoding));
}
private final class JsonBufferedReader extends BufferedReader {
private final ObjectMapper mapper = new ObjectMapper();
private final JsonFactory factory = mapper.getFactory();
private final JsonParser parser;
private ObjectNode node;
JsonBufferedReader(Reader in) throws IOException {
super(in);
parser = factory.createParser(in);
if (parser.nextToken() != JsonToken.START_ARRAY) {
throw new IllegalStateException("Expected an array");
}
}
@Override
public String readLine() throws IOException {
JsonToken nextToken = parser.nextToken();
if (nextToken == JsonToken.START_OBJECT) {
node = mapper.readTree(parser);
return node.toString();
}
if (nextToken == JsonToken.END_ARRAY) {
return null;
}
throw new IllegalStateException("Expected start of object or end of array of objects");
}
@Override
public void close() throws IOException {
super.close();
parser.close();
}
}
}
Is there possibility to run this use-case without all this additional classes that seems like overkill?