Subscribe to long-processing request feedback on NodeJS server from client

Question

I have created a Node JS server which does the following:

Uploads media files (videos and images) to the server using multer
If the media is an image, then resize it using sharp
It the media is a video , then resize and compress it using fluent-ffmpeg
Upload files to Firebase storage for backup

All this is working know fluently. The problem is that, when the size of an uploaded file is big, the request processing takes long time. So I want to show some progress on the client side as below:

State 1. The media is uploading -> n%
State 2. The media is compessing
State 3. The media is uploading to cloud -> n%
State 4. Result -> JSON = {status: "ok", uri: .., cloudURI: .., ..}

Firebase storage API has a functionality like this when we creating an upload task as shown below:

let uploadTask = imageRef.put(blob, { contentType: mime });
uploadTask.on('state_changed', (snapshot) => {
  if (typeof snapshot.bytesTransferred == "number") {
    let progress = (snapshot.bytesTransferred / snapshot.totalBytes) * 100;
    console.log('Upload is ' + progress + '% done');
  }
});

I have found that, it is possible to realize this using websockets, I am interested if there is other methods to do that.

The problem is described also here: http://www.tugberkugurlu.com/archive/long-running-asynchronous-operations-displaying-their-events-and-progress-on-clients

And there is one of the methods Accessing partial response using AJAX or WebSockets? but I am looking for a more flexible and professional solution.

M.Aliyev M.Aliyev · Accepted Answer · 2019-11-05T11:54:51

I have solved this problem using GraphQL Subscriptions. The same approach can be realized using WebSockets. The steps to solve this problem are as below:

Post files to upload server
Generate operation unique ID and send it as response to the client

Ex: response = {op: "A78HNDGS89NSNBDV7826HDJ"}

Create a subscription by opID

Ex: subscription { uploadStatus(op: "A78HNDGS89NSNBDV7826HDJ") { status }}

Every time on status change send request to the GraphQL endpoint, which which publishes the data to the pubsub. To send GraphQL request from nodejs server you can use https://github.com/prisma-labs/graphql-request

Ex:

const { request } = require('graphql-request');
const GQL_URL = "YOUR_GQL_ENDPOINT";
const query = `query {
    notify ("Status text goes here")
}`

request(GQL_URL, query).then(data =>
  console.log(data)
)

notify resolver function publishes the data to the pubsub

 context.pubsub.publish('uploadStatus', {
     status: "Status text"
 });

If you have more complicated architecture, you can use message brokers like RabbitMQ, Kafka etc.

If someone knows other solutions, please let us know )

Subscribe to long-processing request feedback on NodeJS server from client

1 Answers