OmniThreadLibrary - Code: 1816. Not enough quota is available to process this command

Question

Update 1: I included the stack traces of all threads instead of just the main thread's - I thought it was enough already.

Update 2: I reopened this question, since even after applied the changes illustrated in my own question, I still get the very same error report today...

Update 3: It seems that the error happened when the thread is terminating and the thread message that's being sent when the error happened was COmniTaskMsg_Terminated. Now it's very strange - I've replaced almost all calls to Task.Comm.Send() in my program with a thread-safe queue, so I'm quite sure the number of thread messages go through Task.Comm is a very small, far less than that might fill up the Windows message queue...

---

I use OmniThreadLibrary-3.03a (just upgraded to the latest svn, I'll see...) with Delphi XE4, and get an error reported from the end user, the error message and stack trace of the thread that had this problem are included at the end.

Actually I've reduced the possibility of this error, by replacing the calls to Task.Comm.Send() with a thread-safe queue which is used by the background threads to send logs to the main thread. Now I don't know where to look at, the most possible place is here (but within this code block no Task.Comm.Send() calls...):

  //at this point, we are already in a thread other than the main thread.
  //I've two instances of the very same thread running when the program runs.
  myTasks := Parallel.ParallelTask.NumTasks(aTaskCount);// aTaskCount = 5

  myTasks.Execute(
    //the following anonymous method will be executed in sub-threads and will have
    //multiple instances determined by the aTaskCount param.
    procedure (const aSubTask: IOmniTask)
    begin
      //note: this code block runs in multiple sub-threads in parallel.
      //aUidList can have tens of thousands (or even more) of item.
      while aUidList.Take(uid) and (not Self.task.CancellationToken.IsSignalled) do
      begin
        ThreadedDoSomething();
      end;
    end
  );//END sub-thread

Stacktrace:

process id         : $2f9c
allocated memory   : 181.30 MB
largest free block : 1019.73 MB
executable         : MyProgram.exe
exec. date/time    : 2014-08-02 21:12
version            : 1.0.7.256
compiled with      : Delphi XE4
madExcept version  : 4.0.9
callstack crc      : $8b5fc164, $b1225a03, $7caf0d48
exception number   : 1
exception class    : EOSError
exception message  : System Error. Code: 1816. Not enough quota is available to process this command.

thread $1160 (TOmniThread): <priority:-15>
0045c3de MyProgram.exe System.SysUtils           RaiseLastOSError
0045c35b MyProgram.exe System.SysUtils           RaiseLastOSError
0045c40f MyProgram.exe System.SysUtils           Win32Check
0075b387 MyProgram.exe OtlContainerObserver  252 TOmniContainerWindowsMessageObserverImpl.Send
0076debb MyProgram.exe OtlTaskControl       1378 TOmniTask.InternalExecute
0076db6a MyProgram.exe OtlTaskControl       1274 TOmniTask.Execute
0077423c MyProgram.exe OtlTaskControl       3091 TOmniThread.Execute
004ab243 MyProgram.exe madExcept                 HookedTThreadExecute
0053c596 MyProgram.exe System.Classes            ThreadProc
0040a5b4 MyProgram.exe System                150 ThreadWrapper
004ab129 MyProgram.exe madExcept                 CallThreadProcSafe
004ab18e MyProgram.exe madExcept                 ThreadExceptFrame
75223368 kernel32.dll                            BaseThreadInitThunk
>> created by main thread ($1408) at:
007741a1 MyProgram.exe OtlTaskControl       3080 TOmniThread.Create

main thread ($1408):
0090f52d MyProgram.exe VirtualTrees       33601 TBaseVirtualTree.SortTree
00906fd1 MyProgram.exe VirtualTrees       28348 TBaseVirtualTree.EndUpdate
008f3563 MyProgram.exe VirtualTrees       17031 TBaseVirtualTree.SetRootNodeCount
00b66d99 MyProgram.exe BackupControlFrame   219 TfraBackupControl.AddLog
00b66be2 MyProgram.exe BackupControlFrame   179 TfraBackupControl.AddLog
00b69595 MyProgram.exe BackupControlFrame   844 TfraBackupControl.TimerReadLogTimer
005f430b MyProgram.exe Vcl.ExtCtrls             TTimer.Timer
005f41ef MyProgram.exe Vcl.ExtCtrls             TTimer.WndProc
0053fca4 MyProgram.exe System.Classes           StdWndProc
76777885 USER32.dll                             DispatchMessageW
00656b87 MyProgram.exe Vcl.Forms                TApplication.ProcessMessage
00656bca MyProgram.exe Vcl.Forms                TApplication.HandleMessage
00656f05 MyProgram.exe Vcl.Forms                TApplication.Run
00b83f34 MyProgram.exe MyProgram            130 initialization
75223368 kernel32.dll                           BaseThreadInitThunk

thread $22a4:
771d0156 ntdll.dll     NtWaitForMultipleObjects
75223368 kernel32.dll  BaseThreadInitThunk

thread $1604:
771d0156 ntdll.dll                NtWaitForMultipleObjects
761215e3 KERNELBASE.dll           WaitForMultipleObjectsEx
752219f7 kernel32.dll             WaitForMultipleObjectsEx
76780864 USER32.dll               MsgWaitForMultipleObjectsEx
76780b64 USER32.dll               MsgWaitForMultipleObjects
004ab129 MyProgram.exe  madExcept CallThreadProcSafe
004ab18e MyProgram.exe  madExcept ThreadExceptFrame
75223368 kernel32.dll             BaseThreadInitThunk
>> created by main thread ($1408) at:
738778e1 gdiplus.dll

thread $3060 (TWorkerThread):
771cf8ca ntdll.dll                          NtWaitForSingleObject
76121497 KERNELBASE.dll                     WaitForSingleObjectEx
7522118f kernel32.dll                       WaitForSingleObjectEx
75221143 kernel32.dll                       WaitForSingleObject
008e2dbe MyProgram.exe  VirtualTrees   6364 TWorkerThread.Execute
004ab243 MyProgram.exe  madExcept           HookedTThreadExecute
0053c596 MyProgram.exe  System.Classes      ThreadProc
0040a5b4 MyProgram.exe  System          150 ThreadWrapper
004ab129 MyProgram.exe  madExcept           CallThreadProcSafe
004ab18e MyProgram.exe  madExcept           ThreadExceptFrame
75223368 kernel32.dll                       BaseThreadInitThunk
>> created by main thread ($1408) at:
008e2cda MyProgram.exe  VirtualTrees   6312 TWorkerThread.Create

thread $22c0:
771d1f3f ntdll.dll     NtWaitForWorkViaWorkerFactory
75223368 kernel32.dll  BaseThreadInitThunk

thread $33d0:
771cf8ca ntdll.dll                NtWaitForSingleObject
76121497 KERNELBASE.dll           WaitForSingleObjectEx
7522118f kernel32.dll             WaitForSingleObjectEx
004ab129 MyProgram.exe  madExcept CallThreadProcSafe
004ab18e MyProgram.exe  madExcept ThreadExceptFrame
75223368 kernel32.dll             BaseThreadInitThunk
>> created by thread $31fc at:
7327325b rasman.dll

thread $2dc4:
771cf8ca ntdll.dll               NtWaitForSingleObject
764b2f7b WS2_32.dll              WahReferenceContextByHandle
764b6a25 WS2_32.dll              select
004ab129 MyProgram.exe madExcept CallThreadProcSafe
004ab18e MyProgram.exe madExcept ThreadExceptFrame
75223368 kernel32.dll            BaseThreadInitThunk
>> created by main thread ($1408) at:
75349791 WININET.dll

thread $2148:
771d1f3f ntdll.dll     NtWaitForWorkViaWorkerFactory
75223368 kernel32.dll  BaseThreadInitThunk

thread $2258 (TOmniThread): <priority:-15>
771d0156 ntdll.dll                          NtWaitForMultipleObjects
761215e3 KERNELBASE.dll                     WaitForMultipleObjectsEx
752219f7 kernel32.dll                       WaitForMultipleObjectsEx
76780864 USER32.dll                         MsgWaitForMultipleObjectsEx
00771942 MyProgram.exe  OtlTaskControl 2379 TOmniTaskExecutor.WaitForEvent
00770ddc MyProgram.exe  OtlTaskControl 2148 TOmniTaskExecutor.MainMessageLoop
0076fb2e MyProgram.exe  OtlTaskControl 1849 TOmniTaskExecutor.DispatchMessages
0076e7d9 MyProgram.exe  OtlTaskControl 1636 TOmniTaskExecutor.Asy_Execute
0076dd59 MyProgram.exe  OtlTaskControl 1354 TOmniTask.InternalExecute
0076db6a MyProgram.exe  OtlTaskControl 1274 TOmniTask.Execute
0077423c MyProgram.exe  OtlTaskControl 3091 TOmniThread.Execute
0040a5b4 MyProgram.exe  System          150 ThreadWrapper
004ab129 MyProgram.exe  madExcept           CallThreadProcSafe
004ab18e MyProgram.exe  madExcept           ThreadExceptFrame
75223368 kernel32.dll                       BaseThreadInitThunk
>> created by thread $1160 (TOmniThread) at:
007741a1 MyProgram.exe  OtlTaskControl 3080 TOmniThread.Create

thread $618 (TOTPWorkerThread): <priority:-15>
771d0156 ntdll.dll                          NtWaitForMultipleObjects
761215e3 KERNELBASE.dll                     WaitForMultipleObjectsEx
752219f7 kernel32.dll                       WaitForMultipleObjectsEx
752241d3 kernel32.dll                       WaitForMultipleObjects
0074b749 MyProgram.exe  DSiWin32       1949 DSiWaitForTwoObjects
007782ca MyProgram.exe  OtlComm         478 TOmniCommunicationEndpoint.ReceiveWait
007639cc MyProgram.exe  OtlThreadPool   625 TOTPWorkerThread.Execute
004ab243 MyProgram.exe  madExcept           HookedTThreadExecute
0053c596 MyProgram.exe  System.Classes      ThreadProc
0040a5b4 MyProgram.exe  System          150 ThreadWrapper
004ab129 MyProgram.exe  madExcept           CallThreadProcSafe
004ab18e MyProgram.exe  madExcept           ThreadExceptFrame
75223368 kernel32.dll                       BaseThreadInitThunk
>> created by thread $2258 (TOmniThread) at:
0076348c MyProgram.exe  OtlThreadPool   537 TOTPWorkerThread.Create

thread $5f0 (TOTPWorkerThread): <priority:-15>
771d0156 ntdll.dll                          NtWaitForMultipleObjects
761215e3 KERNELBASE.dll                     WaitForMultipleObjectsEx
752219f7 kernel32.dll                       WaitForMultipleObjectsEx
752241d3 kernel32.dll                       WaitForMultipleObjects
0074b749 MyProgram.exe  DSiWin32       1949 DSiWaitForTwoObjects
007782ca MyProgram.exe  OtlComm         478 TOmniCommunicationEndpoint.ReceiveWait
007639cc MyProgram.exe  OtlThreadPool   625 TOTPWorkerThread.Execute
004ab243 MyProgram.exe  madExcept           HookedTThreadExecute
0053c596 MyProgram.exe  System.Classes      ThreadProc
0040a5b4 MyProgram.exe  System          150 ThreadWrapper
004ab129 MyProgram.exe  madExcept           CallThreadProcSafe
004ab18e MyProgram.exe  madExcept           ThreadExceptFrame
75223368 kernel32.dll                       BaseThreadInitThunk
>> created by thread $2258 (TOmniThread) at:
0076348c MyProgram.exe  OtlThreadPool   537 TOTPWorkerThread.Create

thread $2a84 (TOTPWorkerThread): <priority:-15>
771d0156 ntdll.dll                          NtWaitForMultipleObjects
761215e3 KERNELBASE.dll                     WaitForMultipleObjectsEx
752219f7 kernel32.dll                       WaitForMultipleObjectsEx
752241d3 kernel32.dll                       WaitForMultipleObjects
0074b749 MyProgram.exe  DSiWin32       1949 DSiWaitForTwoObjects
007782ca MyProgram.exe  OtlComm         478 TOmniCommunicationEndpoint.ReceiveWait
007639cc MyProgram.exe  OtlThreadPool   625 TOTPWorkerThread.Execute
004ab243 MyProgram.exe  madExcept           HookedTThreadExecute
0053c596 MyProgram.exe  System.Classes      ThreadProc
0040a5b4 MyProgram.exe  System          150 ThreadWrapper
004ab129 MyProgram.exe  madExcept           CallThreadProcSafe
004ab18e MyProgram.exe  madExcept           ThreadExceptFrame
75223368 kernel32.dll                       BaseThreadInitThunk
>> created by thread $2258 (TOmniThread) at:
0076348c MyProgram.exe  OtlThreadPool   537 TOTPWorkerThread.Create

thread $1ca0 (TOTPWorkerThread): <priority:-15>
771d0156 ntdll.dll                          NtWaitForMultipleObjects
761215e3 KERNELBASE.dll                     WaitForMultipleObjectsEx
752219f7 kernel32.dll                       WaitForMultipleObjectsEx
752241d3 kernel32.dll                       WaitForMultipleObjects
0074b749 MyProgram.exe  DSiWin32       1949 DSiWaitForTwoObjects
007782ca MyProgram.exe  OtlComm         478 TOmniCommunicationEndpoint.ReceiveWait
007639cc MyProgram.exe  OtlThreadPool   625 TOTPWorkerThread.Execute
004ab243 MyProgram.exe  madExcept           HookedTThreadExecute
0053c596 MyProgram.exe  System.Classes      ThreadProc
0040a5b4 MyProgram.exe  System          150 ThreadWrapper
004ab129 MyProgram.exe  madExcept           CallThreadProcSafe
004ab18e MyProgram.exe  madExcept           ThreadExceptFrame
75223368 kernel32.dll                       BaseThreadInitThunk
>> created by thread $2258 (TOmniThread) at:
0076348c MyProgram.exe  OtlThreadPool   537 TOTPWorkerThread.Create

thread $3248 (TOTPWorkerThread): <priority:-15>
771d0156 ntdll.dll                          NtWaitForMultipleObjects
761215e3 KERNELBASE.dll                     WaitForMultipleObjectsEx
752219f7 kernel32.dll                       WaitForMultipleObjectsEx
752241d3 kernel32.dll                       WaitForMultipleObjects
0074b749 MyProgram.exe  DSiWin32       1949 DSiWaitForTwoObjects
007782ca MyProgram.exe  OtlComm         478 TOmniCommunicationEndpoint.ReceiveWait
007639cc MyProgram.exe  OtlThreadPool   625 TOTPWorkerThread.Execute
004ab243 MyProgram.exe  madExcept           HookedTThreadExecute
0053c596 MyProgram.exe  System.Classes      ThreadProc
0040a5b4 MyProgram.exe  System          150 ThreadWrapper
004ab129 MyProgram.exe  madExcept           CallThreadProcSafe
004ab18e MyProgram.exe  madExcept           ThreadExceptFrame
75223368 kernel32.dll                       BaseThreadInitThunk
>> created by thread $2258 (TOmniThread) at:
0076348c MyProgram.exe  OtlThreadPool   537 TOTPWorkerThread.Create

thread $11c8:
771d1f3f ntdll.dll     NtWaitForWorkViaWorkerFactory
75223368 kernel32.dll  BaseThreadInitThunk

With reference to this answer of another question, it seems that the Windows message queue used as the communication channel by OTL is full.

Answer 1

The exception is raised in OTL here following an attempt to call PostMessage.

procedure TOmniContainerWindowsMessageObserverImpl.Send(aMessage: cardinal;
  wParam, lParam: integer);
begin
  Win32Check(PostMessage(cwmoHandle, aMessage, wParam, lParam));
end;

The exception indicates that the message queue for the window with handle cwmoHandle is full and is not being serviced in a timely manner. The handle for this window is created by TOmniMessageQueue when you assign to its OnMessage property. Most likely the thread that makes this assignment is blocked (or blocking for too long) and is not processing messages in a timely manner.

procedure TOmniMessageQueue.SetOnMessage(const value: TOmniMessageQueueMessageEvent);
begin
  if (not assigned(mqWinMsgObserver.OnMessage)) and assigned(value) then begin // set up observer
    mqWinMsgObserver.Window := DSiAllocateHWnd(WndProc);  // CREATED HERE**
    mqWinMsgObserver.Observer := CreateContainerWindowsMessageObserver(
      mqWinMsgObserver.Window, MSG_CLIENT_MESSAGE, 0, 0);
    ContainerSubject.Attach(mqWinMsgObserver.Observer, coiNotifyOnAllInserts);
    mqWinMsgObserver.Observer.Activate;
  end
  else if assigned(mqWinMsgObserver.OnMessage) and (not assigned(value)) then begin // tear down observer
    mqWinMsgObserver.Observer.Deactivate;
    ContainerSubject.Detach(mqWinMsgObserver.Observer, coiNotifyOnAllInserts);
    FreeAndNil(mqWinMsgObserver.Observer);
    DSiDeallocateHWnd(mqWinMsgObserver.Window);
  end;
  mqWinMsgObserver.OnMessage := value;
end;

To answer any more specifically, we would need to see more code showing where and how you are implementing your OTL objects. The only other clue is your post is this ;

thread $1160 (TOmniThread): <priority:-15>

This shows that the thread that raised the exception is running at a lower priority level. This in itself doesn't explain anything other than that your application is actively changing thread priority levels. Without code to analyze there is no way to say, but the structure of your program may be causing Priority Inversion issues that are frustrating the owning thread's ability to process messages.

Answer 2

Here is my solution - for those who met the same problem. But I accepted @J.. is answer, because he helped me to find out this solution.

var
  waitEvent : IOmniCancellationToken;
begin
  // create the 'end of all sub-threads' signal
  waitEvent := CreateOmniCancellationToken;

  myTasks.NoWait.Execute(//Note the NoWait call
  procedure (const aSubTask: IOmniTask)
  begin
    doSomeDownloadTasks()
  end
  ).OnStop(procedure
  begin
    waitEvent.Signal;//once all subthreads are stopped, signal the parent thread.
  end
  );

  // wait for the sub-threads to stop, and while waiting, clear the Windows messages periodically
  while not waitEvent.IsSignalled do
  begin
    doSomeDataSavingTasks();

    //self.Task has a hidden Window handle, here we constantly empty its message queue
    //, to aovid the 'Not enough quota is available to process this command' error.
    Task.Comm.Receive(myMsg);
  end;

Answer 3

I also had that problem and in my case I forgot to remove some task.comm.send() after I removed the IOmniTaskConfig interface from the pipeline stages which where for debugging.

On the other hand you may call MsgData._ReleaseAndClear; to free the message as it is done in TOmniCommunicationEndpoint.SendWait() in case of errors.

procedure TPmOTLTaskConfigLogger.OnMessage(const ATask: IOmniTaskControl;
  const AMsg: TOmniMessage);
begin
  try
    //your code
  finally
    AMsg.MsgData._ReleaseAndClear;
  end;
end;

The following SSCCE raises the "Quota exceeded" exception (and sometimes some AVs and "Queue is full" exception) using a pipeline. The quote exceeded is triggered by otSharedInfo_ref.Monitor.Send() in TOmniTask.InternalExecute().

program sscce_otl_quota_exceeded_2;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  System.SysUtils,
  OtlCommon,
  OtlCollections,
  OtlTask,
  OtlParallel;

procedure Stage1(const input, output: IOmniBlockingCollection; const task: IOmniTask);
var
  LInputValue: TOmniValue;
begin
  for LInputValue in input do
    task.Comm.Send(1, LInputValue.AsInteger);//fill the queue
end;

var
  FPipeline: IOmniPipeline;
  LOutputValue: TOmniValue;
  i: Integer;
begin
  try
    FPipeline := Parallel.Pipeline
      //using multiple thread provokes the "Quota exceeded" exception after some
      //"Queue is full" exceptions and AVs. The "Quota exceeded" is triggered by
      //otSharedInfo_ref.Monitor.Send() in TOmniTask.InternalExecute().
      //Note: Sometimes the debugger hangs (XE3)
      .Stage(Stage1).NumTasks(16)
      .Run;

    for i := 1 to 9999 do
      FPipeline.Input.Add(i);

    FPipeline.Input.CompleteAdding;

    while not FPipeline.WaitFor(10) do
      Sleep(1);

    FPipeline := nil;
    Writeln('End');
    ReadLn;
  except
    on E: Exception do begin
      Writeln(E.ClassName, ': ', E.Message);
      ReadLn;
    end;
  end;
end.

Answer 4

We came across the "Quota exceeded" and "Queue is full" exceptions as well. We resolved the issue through instantiating our own monitor and calling process messages on it.

we were creating our own task and holding a reference to the task controller. On this we would use the communication channels directly to send and receive messages.

//Setup 
FTaskCtrl := CreateTask(taskObj, name)
                .SetTimer(TASK_MSG_PUMP_TIMER_ID, TASK_MSG_PUMP_INTERVAL, DEBUG_MSG_PUMP)
                .OnTaskTerminated(OnTaskTerminated)
                .OnTaskMessage(OnTaskMessage);

//Sending messages to the task
FTaskCtrl.Comm.Send(CONST_MSG_NUM, TOmniValue.CastFrom<TOurMsgObj>(msgObjToSend));

//Getting messages from the task, done once every 100ms
FTaskCtrl.Comm.Receive(msgObj);

Note that the timer message is a heartbeat message to know the worker task is still alive.

In the main loop of our application thread, or a business engine thread we would "pump" the comm to make sure all messages were processed. The pump call would be called approximately every 100ms.

procedure MainClass.Pump;
var
  msg : TOmniMessage;
begin
  if FTaskCtrl.Comm.Receive(msg) then
    OnTaskMessage(FTaskCtrl,msg);
end;

We found that this was leaving too many messages in the queue, so we changed this to.

procedure MainClass.Pump;
var
  msg : TOmniMessage;
begin
  while FTaskCtrl.Comm.Receive(msg) do
    OnTaskMessage(FTaskCtrl,msg);
end;

This would produce burst behaviour. Also we would still get the errors on the task being terminated. The specific termination message sent by the task in closure was not being passed on.

What we found was that even though we were not creating a monitor, one was being created internally. Also its wndproc was not being called, or at least being given time to process message that had been sent to it.

Therefore in the end we reverted to the simpler implemented actually seen in a number of the samples for Omni thread.

//Setup
FMonitor  := TOmniEventMonitor.Create(nil);
FMonitor.OnTaskTerminated := Self.OnTaskTerminated;
FMonitor.OnTaskMessage := Self.OnTaskMessage; 

FTaskCtrl := CreateTask(taskObj, name).SetTimer(DEBUG_TASK_MSG_PUMP_TIMER_ID, DEBUG_TASK_MSG_PUMP_INTERVAL, DEBUG_MSG_PUMP);

FTaskCtrl := FMonitor.Monitor(FTaskCtrl);


//Sending messages to the task
FTaskCtrl.Comm.Send(CONST_MSG_NUM, TOmniValue.CastFrom<TOurMsgObj>(msgObjToSend));


//Once per internal loop of the application/main logic thread. Once every 50ms.
FMonitor.ProcessMessages;

Doing the above meant that the monitors wndproc was called and all its logic was performed. We saw for example that termination messages are correctly handled now.

Thanks to J... for the initial post that tipped us in the right direction.