Xpages how to sort preselected large amount data from view

0
votes

I have a domino database with the following view:

Project_no    Realization_date    Author
1/2005        2015-01-02          Alex/Acme
3/2015        2015-02-20          John/Acme
33/2015       2016-06-20          Henry/Acme
44/2015       2015-02-13          John/Acme
...

Now I want to get all projects from this view that starts i.e with "3" (partial match), sort them by Realization_date descending and display first 1000 of them on Xpage.

View is large - some selection can give me 500.000 documents.

The FT search view option is not acceptable because it returns 5.000 docs only.

Creation of ArrayList or ListMap resulted with java out of memory exception (java Domino objects are recycled). Exceeding the memory may help of course but we have 30k users so it may be insufficient.

Do you have any ideas how can I achive this?

4
xpageslotus-domino
Out of memory issues or the dreaded PANIC: LookupHandles error. Your screenshot of the view suggests it contains dates. Using getColumnValues() creates a DateTime object (unless you use preferJavaDates). That DateTime object is a child of the Session not the ViewEntry. So recycling the ViewEntry still leaves the handles to the DateTime object, which can cause a crash, unless you're loading the column values into a Vector and calling recycle(Vector). There are various blog posts on views, datetimes and vectors. – Paul Stephen Withers
Paul, thanks for your comment. I had no idea about recycling vector with column data before. I will use it in our projects. But in this case it is not solving my problem that is connected with data and users amount. With 30k organization we can have some 800 users at one time and with 30MB data structure for one sorting i will achive the 24GB java heap. We have tried intersecting view entry collection (all entries sorted) with another collection (filtered) but it loses sorting. – dj_universe
do you use Reader names? – Frantisek Kossuth
> The FT search view option is not acceptable because it returns 5.000 docs only. - what do users want to do with 5k+ documents? I doubt they want to read them all. – Frantisek Kossuth

4 Answers

1
votes

I would optimize data structure for your view. For example make a ArrayList<view entry>, that will represent the minimum information from your view. It mimics the index. The "view entry" is NOT Notes object (Document, ViewEntry), but a simplified POJO that will hold just enough information to sort it (via comparator) and show or lookup real data - for example Subject column to be shown and UNID to make a link to open that document. This structure should fit into few hundred bytes per document. Troublesome part is to populate that structure - even with ViewNavigator it may take minutes to build such list.

2
votes

I think the key is goiong to be what the users want to do with the output, as Frantisek says.

If it's for an export, I'd export the data without sorting, then sort in the spreadsheet.

If it's for display, I would hope there's some paging involved, otherwise it will take a very long time to push the HTML from the server to the browser, so I'd recommend doing an FT Search on Project_no and Realization_date between certain ranges and "chunking" your requests. You'll need a manual pager to load the next set of results, but if you're expecting that many, you won't get a pager that calculates the total number of pages anyway.

Also, if it's an XAgent or displaying everything in one go, set viewState="nostate" on the relevant XPage. Otherwise, every request will get serialized to disk. So the results of your search get serialized to disk / memory, which is probably what's causing the Java memory issues you're seeing.

Remember FT_MAX_SEARCH_RESULTS notes.ini variable can be amended on the server to increase the (default) maximum from 5000.

500,000 is a very high set of results and is probably not going to make it very user-friendly for any subsequent actions on them. I'd probably also recommend restricting the search, e.g. forcing a separate entry of the "2015" portion or preventing entry of just one number, so it has to be e.g. "30" instead of just "3". That may also mean amending your view so the Project_no format displays as @Right("0000" + @Left(Project_no,"/"), 4), so users don't get 3, 30, 31, 32....300, 301, 302...., but can search for "003" and find just 30, 31, 32..., 39. It really depends on what the users are wanting to do and may require a bit of thinking outside the box, to quickly give them access to the targeted set of documents they want to action.

0
votes

Proper recycling should be ok but...

You could also "revert" to classic Domino URLS for ex ?yourviewname?ReadViewEntries&startkey=3&outputformat=JSON and render that JSON via Javascript UI component of some kind

0
votes

If the filtering is based on partial match for the first sorted column, there's a pure Domino based solution. It requires that the Domino server is 8.5.3 or newer (view.resortView was introduced in 8.5.3), and that the realization_date column has click to sort.

  1. Create a filtered collection with getAllEntriesByKey( key, false ) <-- partial match

  2. Call view.resortView( "name_of_realization_date_column" )

  3. Create a collection of all entries, now sorted by realization_date

  4. Intersect the sorted collection with the filtered collection. This gives you the entries you want sorted by realization_date. E.g. sortedCollection.intersect( filteredCollection )

Pseudocode:

..
View view = currentDb.getView( "projectsView" );
view.setAutoUpdate( false );

ViewEntryCollection filteredCollection = view.getAllEntriesByKey( userFilter, False );
// Use index where view is sorted by realization_date
view.resortView( "realization_date" );

// All entries sorted by realization_date
ViewEntryCollection resortedCollection = view.getAllEntries();

resortedCollection.intersect( filteredCollection );
// resortedCollection now contains only the entries in filteredCollection, sorted by realization_date
..

I'm not certain if this would be faster than creating a custom data structure, but I would think it's worth to test :)