Google Cloud Platform - Optimise Cloud Function using puppeteer (Node.js)

Question

I have written a function in node.js that works well when I run it locally (~10s to run).

As I want to run it every hour, I have deployed it on Google Cloud Platform. But, there, I always have a TimeOut error.

Therefore, do you have any advice on:

what I should change in my function to make it more efficient?
a alternate way to automate my function so it runs every hour?

FYI my cloud function has the following characteristics:

Node js 8
Memory: 2Go
Timeout: 540 seconds

and the following form:

exports.launchSearch = (req, res) => {

const puppeteer = require('puppeteer');
const url = require('./pageInformation').url;
const pageLocation = require('./pageInformation').location;
const userInformation = require('./userInformation').information;

(async () => {
    const browser = await puppeteer.launch({args: ['--no-sandbox']});
    const page = await browser.newPage();
    await page.goto(url);

    // Part 1
    await page.click(pageLocation['...']);
    await page.type(pageLocation['...'], userInformation['...']); 
    await page.waitFor(pageLocation['...']);
    await page.click(pageLocation['...']);
    ... ~20 other "page.click" or "page.select"

    // Part 2
    var continueLoop = true;

    while (continueLoop) {

        var list = await page.$x(pageLocation['...']);

        if (list.length > 0) {
            await list[0].click();
            var found = true;
            var continueLoop = false;
        } else {
            var afficher = await page.$x(pageLocation['...']);
            if (afficher.length > 0) {
                await afficher[0].click();
            } else {
                var continueLoop = false;
                var found = false;
            };
        };
    };

    // Part 3
    if (found) {
        await page.waitForXPath(pageLocation['...']);
        const xxx = await page.$x(pageLocation['...']);
        await xxx[0].click();
        ... 5 other blocks with exact same 3 lines, but with other elements to click
    };

    await browser.close();
})();

};

I have tried to run it part by part; sometimes it times out at the end of Part 1, sometimes at the end of Part 2. But the whole script never entirely completed.

does it run fine except the timeout? meaning if you do only a few operations, will it execute on time and end the function gracefully? — Horatiu Jeflea
Along the same lines as what @HoratiuJeflea is asking, have you tried adding any logging statements to the function, to see where the time is going, and that you are not getting stuck on, say, initializing the browser? — robsiemb
@HoratiuJeflea thanks for your answer! I have indeed tried executing the beginning of the function first, then added some more lines of the code and executed again, etc. I have noticed that most of the time Parts 1 and 2 work, but Part 3 never completes. — ludmilaex
so there are a lot of lines in part 3 which are not even visible, may I suggest eliminating line by line and seeing exactly which once causes the timeout? that waitForXpath seems a good candidate — Horatiu Jeflea

Marcos Ramirez Marcos Ramirez · Accepted Answer · 2019-10-30T00:43:53

Without having too much context of what your code does, it is hard to point out the root cause, but that I tell is continue debugging your code as Horatio suggested, or you can use a more sophisticated tool like StackDriver to monitoring the performance of your Cloud Functions. Evaluate its pricing if you are interested in.

If Stackdriver is an overkill, simply make use of inline function wrapping to find out the exact place of your routine that consuming all that time. Here is an example:

var start = process.hrtime();
yourfunction();
var elapsed = process.hrtime(start)[1] / 1000000;

console.log("Elapsed:" + elapsed.toFixed(3));

Once you have the exact piece of code that is affecting the execution, then you probably may have to optimize it. Additionally, as I understand that locally it worked perfectly, consider that sometimes processes running in Cloud environment are affected by latency due the 'proximity' of the other resources they consume.

Regarding your second question, about automating your function to be executed every hour. You can take advantage of Cloud Scheduler. It has the capability to make scheduled calls to HTTP/HTTPS endpoints, which Cloud Functions classify as one of those. Make sure to check its pricing also.

Google Cloud Platform - Optimise Cloud Function using puppeteer (Node.js)

1 Answers