Gatling: execute a check on some JSON hidden inside an HTML response

Question

In Gatling, I'd like to perform a check on some JSON included in an HTML response like below:

<!doctype html>
<html lang="fr">
  <head>
    <script>
      var documentLoaded = performance.now();
    </script>
    <link rel="stylesheet" href="/styles/main.f14d8fab5a7e.css">
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">

    <link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png">
    <link rel="manifest" href="/manifest.json">
    <link rel="preconnect" href="https://www.gstatic.com">

    <title data-react-helmet="true">Asus Discount</title>
    <meta data-react-helmet="true" name="description" content="Asus discount”/><meta data-react-helmet="true" name="keywords" content="Asus"/>

  </head>
  <body>
  <div>Some content</div>

  <script>
      var parseStart = performance.now();
  </script>

  <script>
    window.__INITIAL_STATE__ = {some JSON}; <!-- This is what I need -->
    window.__ENV_VARIABLES__ = {some other JSON};
    window.renderTime = '76';
    window.fetchTime = '349';
  </script>
  <script type="text/javascript" charset="utf-8" src="/vendor.e33d9940372.js"></script>
  <script type="application/ld+json" src="/schema.fr.json"></script>
  </body>
</html>

My actual solution (which is working) looks as follow:

def loadPageJsonInHTML(requestName: String, link: String): ChainBuilder ={
  exec(
    http(requestName)
      .get(link)
      .check(regex("""window[.]__INITIAL_STATE__ = ([^;]+)""").find.transform(s => parseSToProdList(s)).saveAs("prod_list")
      )
  )
  doIf("${prod_list.size()}" == 0){
    exec{session => session.markAsFailed}
  }
}

def parseSToProdList(jsonString: String): Seq[String] ={
  val jsonMap = jsonStrToMap(jsonString)
  val buffer = mutable.Buffer.empty[String]
  jsonMap("products").asInstanceOf[Map[String, Any]].foreach{f =>
    if(f._2.asInstanceOf[Map[String, Any]].keySet.exists(_ == "code"))
      buffer.append(f._2.asInstanceOf[Map[String, Any]]("code").asInstanceOf[String])
  }
  buffer.toSeq
}

def jsonStrToMap(jsonStr: String): Map[String, Any] = {
  implicit val formats = org.json4s.DefaultFormats
  parse(jsonStr).extract[Map[String, Any]]
}

However, this solution has several drawbacks:

The check will always succeed as long as the regex is found and doesn't care if there's any product at all in the JSON -> I have to manually check for it later;
Having a function extracting the required data is more difficult to maintain than if I could use a Json Path expression like "$.products.*.code" which could be stored on a centralised paths file for ease of maintenance;
This is the only place where I have to use a transform to check the JSON of a request, making it more difficult to read and understand.

What I'd like to achieve is something that would look a bit like this:

def loadPageJsonInHTML(requestName: String, link: String): ChainBuilder ={
  exec(
    http(requestName)
      .get(link)
      .check(jsonPath("""$.products.*.code""").findAll.saveAs("prod_list")
  )

Or

def loadPageJsonInHTML(requestName: String, link: String): ChainBuilder ={
  exec(
    http(requestName)
      .get(link)
      .check(jsonpJsonPath("""$.products.*.code""").findAll.saveAs("prod_list")
  )

Of course, jsonPath doesn't work since most of the answer is HTML. jsonpJsonPath doesn't work either as there's several Json strings in the response.

Any good input as to how could I do this more effectively (and nicely) while avoiding a regex on some HTML? Thanks in advance

Arnianor Arnianor · Accepted Answer · 2019-05-24T14:53:26

So, after some digging I found a workaround using ".transformResponse", in order to extract the string before it's actually checked, giving it a default value that is parsable in Json. Then, to make sure we actually did find the regex, we make sure that it's not our default value :

  def loadPageJsonInHTML(requestName: String, link: String): ChainBuilder = {
    exec(
      http(requestName)
        .get(link)
        .transformResponse{(session, response) =>
          response.copy(body = new StringResponseBody(
              (for(m <- """window[.]__INITIAL_STATE__ = ([^;]+)""".r
                           .findFirstMatchIn(response.body.string)
                  ) yield m.group(1)
              ).getOrElse("""{"error":"chain not found"}"""),
              UTF_8
            )
          )
        }
        .check(bodyString.not("""{"error":"chain not found"}"""))
        .check(jsonPath("""$.products.*.code""").findAll.saveAs("prod_list")
        )
    )
  }

Gatling: execute a check on some JSON hidden inside an HTML response

1 Answers