Efficient way to get data from lotus notes view

1
votes

I am trying to get all data from view(Lotus Notes) with lotusscript and Python(noteslib module) and export it to csv, but problem is that this takes too much time. I have tried two ways with loop through all documents:

import noteslib
db = noteslib.Database('database','file.nsf')
view = db.GetView('My View')
doc = view.GetFirstDocument()
data = list()
while doc:
    data.append(doc.ColumnValues)
    doc = view.GetNextDocument(doc)

To get about 1000 lines of data it took me 70 seconds, but view has about 85000 lines so get all data will be too much time, because manually when I use File->Export in Lotus Notes it is about 2 minutes to export all data to csv.

And I tried second way with AllEntries, but it was even slower:

database = []
ec = view.AllEntries
ent = ec.Getfirstentry()
while ent:
    row = []
    for v in ent.Columnvalues:
        row.append(v)
    database.append(row)
    ent = ec.GetNextEntry(ent)

Everything that I found on the Internet is based on "NextDocument" or "AllEntries". Is there any way to do it faster?

2
pythonlotus-dominolotusscript
If you were using LotusScript, I would recommend using NotesViewNavigator. That's very efficient, but I don't know if the Python module has something similar. Fwiw, here's the documentation on NotesViewNavigator: ibm.com/support/knowledgecenter/en/SSVRGU_9.0.1/basic/… - Dave Delay
Thanks for you response. In Python I can use every COM classes from LotusNotes. I tried to use NotesViewNavigator, the problem is that time is similar to other methods. Maybe this is normal and I expect too much? This is my code: db = noteslib.CurrentDatabase view = db.GetView('My view') nav = view.CreateViewNav() doc = nav.GetFirst() view.AutoUpdate = False list_of_data = [] for i in range(1000): list_of_data.append(doc.ColumnValues) doc = nav.Getnext(doc) It took about 146 seconds to get 1000 rows of data. - Ala Cris
Where does your code run ? directly on the Domino server or on a client? - umeli
I'm normal user of Domino client, my IT team told us, that they won't create agent which will take data for us. So I wanted to make it by myself, I don't have permission to Domino server. - Ala Cris
Then I suggest to replicate the database locally.. - umeli

2 Answers

2
votes

It is (or at least used to be) very expensive from a time standpoint to open a Notes document, like you are doing in your code. Since you are saying that you want to export the data that is being displayed in the view, you could use the NotesViewEntry class instead. It should be much faster.

Set col = view.AllEntries
Set entry = col.GetFirstEntry()
Do Until entry Is Nothing
    values = entry.ColumnValues   '*** Array of column values
    '*** Do stuff here
    Set entry = col.GetNextEntry(entry)
Loop

I wrote a blog about this back in 2013:
http://blog.texasswede.com/which-is-faster-columnvalues-or-getitemvalue/

1
votes

Something is going on with your code "outside" the view navigation: You already chose the most performant way to navigate a view using "GetFirstDocument" and "GetNextDocument". Using the NotesViewNavigator as mentioned in the comments will be slightly better, but not significant.

You might get a little bit of performance out of your code by setting view.AutoUpdate = False to prohibit the view object to refresh when something in the backend changes. But as you only read data and not change view data that will not give you much of a performance boost.

My suggestion: Identify the REAL bottleneck of your code by commenting out single sections to find out when it starts to get slower:

First attempt:

while doc:
    doc = view.GetNextDocument(doc)

Slow?

If not then next attempt:

while doc:
    arr = doc.ColumnValues
    doc = view.GetNextDocument(doc)

Slow?

If yes: ColumnValues is your enemy... If not then next attempt: 

while doc:
    arr = doc.ColumnValues
    data.append(arr)
    doc = view.GetNextDocument(doc)

I would be very interested to get your results of where it starts to become slow.