xpages load json from documents

Question

I am trying to load JSON from documents in a view and ultimately showing a dojo enhanced data-grid. there are around 1000 documents and I need to check for multiple rows of data. This is legacy application and documents can have details of 80 different users. So my code will generate 80000 Json rows in worst case. Right now it is able to load 70k+ records from 980 documents. My current approach is to create whole JSON and write in a JS variable in browser which is working but it is slow predictably. It takes around 45-80 seconds for Java to generate the JSON.

I changed the approach to load NotesDocument as JSON and parse it on client. This way my java code will loop through only 980 times. I am using document.generateXML() to generate XML and then org.json.XML.toJSONObject() method (this is from json.org jar file) to convert it to JSON. This too is working but it seems to be slower than first approach.

I am not sure how else I can load this much data to browser.

BIGGEST question is : Java code starts executing very long after I open the Xpage. It is called on beforePageLoad event. This I am really worried about. First console message (date-time to check start of code) appears ages after I open the xpage link.

Now for the datagrid, I am displaying 30 rows only but problem is because of multiple rows of data per document I cannot allow users to sort/filter grid until all records are fetched.

Here is my backend Java code for current approach. If you'd like, I can post working code (first approach).

import java.io.IOException;
import java.util.*;

import javax.faces.context.ExternalContext;
import javax.faces.context.FacesContext;
import javax.faces.context.ResponseWriter;
import javax.servlet.http.HttpServletResponse;

import lotus.domino.Database;
import lotus.domino.Document;
import lotus.domino.View;

import com.ibm.xsp.extlib.util.ExtLibUtil;
import com.ibm.commons.util.io.json.*;
import org.json.*;

/**
 * @author agnihotri.a
 * 
 */

 public class AccessRequests {
    public static void mainly() {
        Date dt1 = new Date();
        System.out.println(dt1.toString());
        System.out.println("here");
        Database database = null;
        View accReqView = null;
        Document accReqDoc = null;
        JSONArray jarr = new JSONArray();
        //try out JSONObject here
        //List <JSONObject> ljo = new ArrayList<JSONObject>();

        try {

            /**
             * we need handle current user session to compute common name in
             * user names field
             */
            // Session s = ExtLibUtil.getCurrentSession();
            database = ExtLibUtil.getCurrentDatabase();
            System.out.println("generating grid in : " + database.getFilePath());
            accReqView = database.getView("AccessRequestsGrid");

            // get access request document
            accReqDoc = accReqView.getFirstDocument();
            //int counter = 0;
            while (accReqDoc != null) {
                //counter++;
                jarr.put(org.json.XML.toJSONObject(accReqDoc.generateXML()));
                //ljo.add(org.json.XML.toJSONObject(accReqDoc.generateXML()));
                accReqDoc = accReqView.getNextDocument(accReqDoc);
            }

            ExtLibUtil.getSessionScope().put("allAccReq", jarr);
            //System.out.println(ljo.size());
            //System.out.println(counter);
            //ExtLibUtil.getSessionScope().put("totDocs", ljo.size());

        }

        catch (final Exception ex) {
            // tbd: handle exception
            ex.printStackTrace();
            System.out.println(ex.getStackTrace().toString());
            // return "An Error occured. Check with IT team.";
        } finally {
            // recycle domino objects
            KillDomObjects.incinerate(accReqDoc, accReqView, database);
            final Date dt2 = new Date();
            System.out.println(dt2.toString());
        }
    }

}

Answer 1

I recommend using the xe:viewItemFileService in an XPages REST service. This will return JSON and not have to go through the XML to JSON conversion you mentioned. Also, only return 50 to 100 entries at a time, I've found this to be the best compromise between speed and the amount of data returned. Yes you'll have to configure either endless scrolling or paging, but it's worth it as people will not be happy waiting 45 - 80 seconds to get something on the screen. You want to shoot for < 3 second returns.

If you're determined to write your own JSON delivery, then ensure you only return what is absolutely necessary to make the current page usable. Also take a look at the OpenNTF Domino API, this project has a Document.toJSON() method which is a huge time saver and code size reducer. Not to mention proper implementation of the Java Collections API for looping through collections.

As for storing this data in sessionScope, you're introducing a huge scalability issue here. You must consider that this data will possibly be stored in memory for every user of your application. This is a great way to crash a server or at the very least make performance degrade as more and more users use the system.

You might also want to take a look at these community resources:

• NotesIn9 - http://www.notesin9.com/2012/11/12/notesin9-087-getting-started-with-xpages-rest-custom-control/
• Generating JSON in XPages - https://frostillic.us/blog/posts/EF0B875453B3CFC285257D570072F78F
• Dojo Grids in XPages - http://xcellerant.net/dojo-grids-in-xpages/

Regarding the delay, I see a couple of things in your code:

• The View index might not be built. If your view has @Now or @Today in any of the selection or column formulas, the index will be rebuilt each time you access the view. Depending on the size of the view this could be a real performance bottleneck
• You probably want to be using a ViewNavigator instead of a document collection
• Use ViewEntry and column values instead of opening the document. Opening a document is a very expensive operation when looping through 1000s of documents