Making pagination API calls simulaneously using Spring Webclient

Question

I have to make paginated API calls, using Webclient and combine all the result at last. For ex : Person latest 1000 transaction details. In one call, i am gonna get max 100 Objects in json response (List). I can only get maximum 1000 records for this Person.

In pseudo code - java, it could look something like this

    int Total = 1000;
    int maxEachCall = 100;
    int noOfCalls = maxTotal/maxEachCall;
    int startIndex = 0;

    List<MyDto> mailList = new ArrayList();

    for(int i =startIndex; i<noOfCalls ;i+100){
        List<MyDto> list =  webClient.get().uri(baseUri+"?startIndex"+startIndex)
                        .retrieve()
                        .bodyToMono(MyDto.class)                            
                        .retry(3)
                        .block();
        mainList.addAll(list);
        if(startIndex>900 || list.size()<100){
            break;
        }
    }

How can i write the same thing in reactive way WITHOUT blocking in spring mvc ?

Something like this??? I dont know. Help me

    int Total = 1000;
    int maxEachCall = 100;
    int noOfCalls = maxTotal/maxEachCall;
      int startIndex = 0;
    Flux.range(1,noOfCalls)
        .flatMap(
            // tried to call api using startIndex but can't use startIndex here or manipulate it
        )
        .doOnError(e->LOGGER.info("Exception occured ",e))
        .blockLast(); ???

Michael Berry Michael Berry · Accepted Answer · 2019-11-26T21:38:36

You could do something similar to:

Flux.range(0, 10)
        .map(start -> start * 100)
        .flatMap(
                start -> webClient.get()
                        .uri(baseUri + "?startIndex=" + start)
                        .retrieve()
                        .bodyToMono(MyDtoCollection.class)
                        .map(c -> splitCollectionIntoIndividualMyDto(c)) //or something to that effect, since you'll get 100 `MyDto` at a time and presumably need to split them.
        )
        .collectList();

...which will provide you with a Mono<List<MyDto>> containing all your paginated results. If your stack is fully reactive you would then continue to use that Mono in the reactive chain elsewhere. If not (yet) fully reactive, and you're just using reactor for this bit, you can then choose to block() at the most sensible location.

Making pagination API calls simulaneously using Spring Webclient

1 Answers