How to fetch multiple pages with an Observable in Angular?

Question

So I try it in many ways but didn't find out. All help welcome; I discover Angular and RxJs.

I have a service that fetch ressources on many URL of the swapi API. I don't know in advance how many pages will be fetched. So I use concat for every http.get(url) to create an observable.
Currently, only the first page of data is added to the component (i.e. firstPage); all the requests are sent.

export class PeoplesService {
  urlSource = "https://swapi.co/api/people/";
  pageResponse: GeneralResponse<People>;
  fullResponse: Observable<GeneralResponse<People>>;

  constructor(private _http:HttpClient) {
  }

  getPaged(n: number): string {
    return this.urlSource + "?page=" + n;
  }

  fetch(): Observable<GeneralResponse<People>> {
    let firstPage = this._http
                      .get<GeneralResponse<People>>(this.urlSource);
    firstPage.subscribe(page => {
      this.fullResponse = firstPage; // first page fetched
      let pageToDownload = Math.ceil(page.count / page.results.length);
      for(let i=2; i<=pageToDownload; i++) {
        // Merge all observable (so all request) into one
        concat(this.fullResponse,
               this._http.get<GeneralResponse<People>>(this.getPaged(i)));
      }
    });
    return this.fullResponse;
  }
}

Then the basic code for my component is the following :

  ngOnInit() {
    this.peoplesService.fetch().subscribe(r => this.movies = r.results);
    // a sort of fetch().onNextFetch(this.movies.push(...r.results)) seems better in this case
    // because every data on each pages need to be merged into this.movies
    // or sort of fetch().subscribeUntilCompleted(r => this.peoples = r.results) needed I guess
  }

I haven't found what could be used instead of subscribe (such as a subscribe when Observable has returned everything and collected it all...).

I guess subscribe does not wait for the status "onCompleted" of the Observable and is not called every time to get all returned values. So how do you fetch all datas ?

Is there something to make the Observable act like a stream and pipe it to this.peoples.push(...r.results) ? I don't know if I am on the right track.

Lyokolux Lyokolux · Accepted Answer · 2020-01-11T18:58:12

OK I have found the answer after some hours of work. I use mergeMap() to get the datas from the first Oservable and merge() for unite all Observable into one.

import { HttpClient } from '@angular/common/http';
import { Observable, merge } from 'rxjs';
import { take, map} from 'rxjs/operators';

@Injectable({
  providedIn: 'root'
})
export class PeoplesService {

  private getPaged(n: number): string {
    return this.urlSource + "?page=" + n;
  }

  /**
   * Return an array with the natural number in it.
   * @param size : size
   * @param start : first number
   * Examples : 
   *    naturalNumberArrayFactory(3) returns [1, 2, 3]
   *    naturalNumberArrayFactory(3, 2) returns [3, 4, 5]
   */
  protected static naturalNumberArrayFactory(n: number, start = 1): number[]{
    return [ ...Array(n).keys() ].map(x => x + start);
  }

  fetch(): Observable<GeneralResponse<People>> {
    return this._http
      .get<GeneralResponse<T>>(this.urlSource)
      .pipe(take(1), map(firstPage => {
        // Amount of page to get ( -1 because got firstPage)
        let pageToDownload = Math.ceil(firstPage.count / firstPage.results.length) - 1;
        // Generate array ids and replace them with GET requests
        let observables = GenericService.naturalNumberArrayFactory(pageToDownload, 2)
          .map(id => this._http.get<GeneralResponse<T>>(this.getPaged(id)));
        // Add the first page to avoid to repeat the request
        observables.unshift(of(firstPage));
        return merge(...observables);
      }));
  }
}

How to fetch multiple pages with an Observable in Angular?

2 Answers