how to limit the number of instances of a Lotuscript Agent consuming WS (high loaded Domino server, detect crash/stalling conditions)

Question

I need to limit the number of instances of an Agent. Is the following code the best technique?

If counter > 50 then exit sub
Counter++
Call WS    '//this could time out after 110 seconds
Counter--

This meta code will be written in LOTUSSCRIPT (the agent is already in LS). Concerning the programmatic aspect of the counter, what is the best approach (in heavily concurrent calls):

1. Lock document counter doc then save (loop on doc.lock)
2. save doc and test save conflict (loop on whole get doc if doc.save false, false)
3. LockID=CreateLock(LockName as String) then update (loop on createLock)
4. any better idea?

I have already read: Are web services processed sequentially or in parallel? http://www-10.lotus.com/ldd/ddwiki.nsf/dx/sequential-numbering.htm a ten years old still very accurate document http://www.eview.com/eview/volr6.nsf/0/62B3A667117B484385256F3300576ECF/$File/Guide%20to%20Document%20Locking%20SO%20604.pdf

More about the Context: An agent in a cluster Web Domino server is heavily called (300+ called per minutes during peaks). This agent calls a Domino consumer to another system WS.

Sometime the external system has problems and don't return response "in a reasonable amount of time" (I get Time out after about 110 sec).

The problem: when this appends, all the workers (threads of http, determined by Number active threads: in the server document configuration) are waiting for a response. As the Domino server stops to responds! After the stalled agents timed out, the next call to the agent starts to wait for time out… causing a hugh queue of requests until crashing the server.

To prevent the agent to exhaust the resources (the threads/workers in this case) I'm planning to increment a counter at the beginning of the agent and decrement it when the agent finished. At any time I should have the amount of running instance of the agent. A simple test at the beginning of the agent will make it.

N.B. Using wsConsumer.Settimeout( ms) could lower the time the agent waits but can solve the problem.

Answer 1

As you do not need a "counter" in classic sense, I would do it like that:

1. Create as much "worker"- documents as you want to have threads.
2. Every agent performs a database- search for "unused"- worker- documents like Form = "Worker" & CurrentAgent = ""
3. The agent gets the first document, checks, if "CurrentAgent" is still "" (concurrent agents) and sets a value in "CurrentAgent" (best would be a timestamp) and saves the document. If document has been updated from another Instance: Loop through the collection, until you find one that is "really" free...
4. after finishing the agent sets "CurrentAgent" back to "" and saves the document again.

If an agent does not find a document with the search- formula, then all threads are "busy". You could then call a "fallback" method for this agent to

1. Select ALL Worker- documents with selection Form = "Worker"
2. Run through the document and check, if the CurrentAgent is older than a specific amount of time. That would mean -> The agent died before it was able to release the document. Then just "reuse" this document

In addition you could run a regular agent, that "resets" the worker- threads (for the same reason of dying agents).

If your database contains lots of documents, then I would create the worker- documents in a database of its own because of performance issues.

Why not use a view to get the documents? - as this thing changes documents all over again, the view index would probably not be up to date...

Answer 2

As mentioned I would not recommend using profile documents. They are cached and unless you do some excessive testing (on different ID's) I'd stick with a normal document.

Alternatively have a file on disk which you write the value to. This would allow you to have some control outside of the agent manager process. You can also control the value with locking.

---

The other aspect you need to worry about in LotusScript and web services (which appears to be what you described).

Serializing to/from a SOAP object has an overhead in LotusScript. This overhead is exponential the larger the SOAP message, or the more SOAP request/responses going on in LotusScript.

There is no set value, but you can see this happening with large SOAP requests (eg. Send binary attachment >64MB). The quickest way to check if this is what is happening you can add a print statement at the end of the agent. Something like Print "Agent Code finished". If you have the issue described, then the "Agent completed Execution" will not appear straight after the print statement.

Answer 3

Thank for your help. The solution combining your suggestion: Build a DB that will store the profile doc. This DB should not replicate between the cluster nodes for 2 reasons avoid useless replication, simplify the profile handling that would correspond to the current server.

Below the code

'we need to check If the nb of concurrent Call To the WS is accepable.
'In some same the WS is very slow before Timing out, the problem is that we Use ALL our workers (threads)
'WAITING for timeout cause the WHOLE site To lag! when all the threads are waiting
Const MAXCONCURRENT = 60%
REM Create note collection
Dim dbProfile As New NotesDatabase(db.Server, "profileDB.nsf")
Dim nc As NotesNoteCollection 'NotesDocumentCollection
Dim Concurrent As Integer

    Set nc = dbProfile.CreateNoteCollection(False)
    nc.SelectProfiles = True
    Call nc.BuildCollection
    Concurrent = nc.Count
    If Concurrent > MAXCONCURRENT Then exit sub ' or what ever need to be done

    Dim profileDoc As NotesDocument
    Dim uniqKey As String
    Dim eval As Variant
    eval = Evaluate ("@Unique")
    uniqKey =eval  (0)

    Set profileDoc = dbProfile.getProfileDocument( Uniqkey )    
    profileDoc.save True, True  

'---------- call the WS ----------

    If Not profileDoc Is Nothing Then 
        Call profileDoc.Removepermanently(True)
    End if

in the profile DB schedule an agent:

Use "OpenLogFunctions"
Sub Initialize
On Error GoTo errortrap
Dim session As New NotesSession
Dim db As NotesDatabase
Dim docNote As NotesDocument
Set db = session.CurrentDatabase
Dim nc As NotesNoteCollection
Dim i As long, intcount As Long, delP As long
Dim strNoteID As String
Dim stStat As String
Dim cutoffDate As Variant

Set nc = db.CreateNoteCollection(False)
nc.SelectProfiles = True
Call nc.BuildCollection
intCount = nc.Count
cutoffDate = Now - 0.007' (0.007 is not James Bond but 10 minutes ;-)

stStat = StrRight(StrLeft(db.Server, "/O"),"CN=") & intCount & " concurrent agents running (counting profile docs)"
logEvent stStat , "" , Nothing , ""
strNoteID$ = nc.GetFirstNoteId()
For i = 1 To intCount
    Set docNote = db.GetDocumentByID(strNoteID$)
    If Not(docNote Is Nothing) Then
        stStat = stStat + Chr$(13) + "profile: " + docNote.NameOfProfile +" created: " & docNote.Created 
        If docNote.Created < cutoffDate Then
            docNote.Removepermanently(true)
            stStat = stStat + " REMOVED !"
            delP = delP + 1 
        End If 
    End If
    strNoteID$ = nc.GetNextNoteId(strNoteID$)
Next i
If delP >= 10 Or intcount >= 50 Then

    Dim doc As NotesDocument
    Set doc = New NotesDocument( db )
    doc.Form = "Memo"
    doc.SendTo = "[email protected]"
    doc.Subject = "Checking itemtrace profiles on " + StrRight(StrLeft(db.Server, "/O"),"CN=")+": " & delP & " deleted in a total of " & intCount & " profiles docs"
    doc.body = stStat
    Call doc.Send( False )
End If  
LogEvent "finish checking profiles of "+StrRight(StrLeft(db.Server, "/O"),"CN=")+": " & delP & " deleted in a total of " & intCount & " profiles docs" , "" , docNote, stStat
Exit Sub
errortrap:
logError stStat
Exit Sub
End Sub

the agent performances (profile result from the agent list)

17/06/2014 12:17:47 ZE2
Elapsed time: 281 msec
Methods profiled: 23
Total measured time: 265 msec
Class       Method          Calls   Time
 WebServiceEngi  Invoke         1   218
 NoteCollection  BuildCollectio 1   31
 Database    GetProfileDocument 1   16
 Document    [expandedname] Get 3   0
 Database    GetView            2   0
 Document    GetItemValue       2   0
 Session     CurrentDatabase    2   0
 View    GetFirstDocument       2   0
 Document    Save               1   0
 Session     UserName Get       1   0
 Session     EffectiveUserName  1   0
 Session     NotesVersion Get   1   0
 Session     GetEnvironment     1   0
 Session     DocumentContext    1   0
 Database    Server Get         1   0
 Database    CurrentAccessLevel 1   0
 Session     NotesBuildVersion  1   0
 Document    RemovePermanently  1   0
 NoteCollection  SelectProfiles 1   0   set
 Database    CreateNoteCollect  1   0
 NoteCollection  Count Get      1   0
 WebServiceEngi  Initialize     1   0
 WebServiceEngi  SetTimeout     1   0

Answer 4

Rather than get involved in locking a single document containing a counter, perhaps you could use profile documents. I.e.,

• Choose a NotesDatabase that contains no profile documents for any other purpose.
• At startup, do set profileDoc = NotesDatabase.getProfileDocument("someProfileName",NotesSession.userName) (If username is not necessarily unique per instance, then use something else).
• Before calling the WS, do NotesDatabase.CreateNoteCollection(false), and NotesNoteCollection.setProfile(true), and NotesNoteCollection.BuildCollection() and then check NotesNoteCollection.Count.
• If the count is 1 or more greater than your threshold, then exit with an error indicating "too busy", else go ahead with the WS call
• And in either case, at termination call profileDoc.removePermanently()

My thought is that using profile documents would be faster than using ordinary documents due to the caching that is done. The downside might be that the caching might make the count unreliable for this purpose, but I don't know if that's the case.