bokeh multiple live streaming graphs in different objects / register update routine located in other class

Question

I use python and bokeh to implement streamed live graphing. I want to include several live graphs into a gridplot and run into a kind of "deathlock".

The graphs (there are a lot of them) are created by different classes and the figure objects are returned and then used as input to the gridplot() function.

For live graphing curdoc().add_periodic_callback(update1, 300) references the update routine. I call the update routines of the other graphs directly from update1(). This works but gives me following error continuously:

`raise RuntimeError("_pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes")

This is expected behavior since data of the other graphs is altered from the 'outside' of their object and from an 'unregistered update routine'. I want to get rid of this error.

In my main object (where the layout is pieced together and curdoc().add_root() is called) I intend to register the other graphs update routines (which have to be regular object routines, so that they can be referenced.) via curdoc().add_periodic_callback(). the problem with this approach is, that the objects update functions take the self parameter and bokeh does not accept that.

Yet I can not do it without self, cause update() needs to reference the source.stream object.

I have no clue how to solve this or do it the 'correct' way. Suggestions are appreciated.

thanks

for clarification:

main object:

def graph(self):
   .... bokeh code
   @count()
def update(t):
   .... update code

curdoc().add_root(gridplot([[l[0]], [l[1]]], toolbar_location="left", plot_width=1000)) curdoc().add_periodic_callback(update, 300)

this works

generic other object

def graph(self):
     .... bokeh code

def update(self,t): ....

main object: 
curdoc().add_periodic_callback(other_object.update, 300)

this does NOT work.

Rikki Rikki · Accepted Answer · 2021-02-24T03:51:05

"_pending_writes should be non-None when we have a document lock, and we should have the lock when the document changes"

Disclaimer: I've been dealing with this error in my own work for two weeks now, and finally resolved the issue today. (: It's easy when every sample you see comes with a csv file that's read and pushed to the doc, all in the same thread, but when things get real and you have a streaming client doing the same thing, suddenly everything stops working.

The general issue is, Bokeh server needs to keep its version of the document model in sync with what Bokeh client has. This happens through a series of events and communication happening between the client (Javascript running in the browser) and the server (inside an event loop that we will get to later on).

So every time you need to change the document, which essentially affects the model, the document needs to be locked (the simplest reason I could think of is concurrency). The simplest way to get around this issue, is to tell the Bokeh document instance you hold, that you are in need of making a change - and request a callback, so Bokeh manages when to call your delegate and allow you to update the document.

Now, with that said, there are few methods in bokeh.plotting.Document that help you request a callback.

The method you would want to probably use based on your use case, for example, if you need an ASAP callback, is add_next_tick_callback.
One thing to remember is that, the reference/pointer to your doc must be correct.

In order to make sure of that, I did wrap all my charting application into a class, and kept an instance of doc internally to access its add_next_tick_callback when new data is received. The way I could point at the right instance, was to initialize my Bokeh app using bokeh.server.server.Server - when it initializes the app, you will receive a doc variable that it's created before starting the server - that would be the right reference to the doc you present in the app. One benefit for having this "chart initializer" in a class, is that you can instantiate it as many times as you may need to construct more charts/documents.

Now, if you are a fan of data pipelines, and streaming, and use something like StreamZ to stream the data to the Pipe or Buffer instance you may have, you must remember one thing:

Be aware of what happens asynchronously, in a thread, or outside of it. Bokeh relies on tornado.ioloop.IOLoop for the most part, and if you are anywhere near running things asynchronously, you must have come across asyncio.
The event loops on these two modules can conflict, and will affect how/when you can change the document.

If you running your streaming in a thread (as the streaming client I wrote did..), make sure that thread has a current loop, otherwise you will face other similar issues. Threads can cause conflicts with internally created loops, and affect how they interact with each other.

With something like the following:

asyncio.set_event_loop(asyncio.new_event_loop())

Finally, be aware of what @gen.coroutine does in tornado. Your callbacks for the streaming, the way I understood, must be decorated with @gen.coroutine if you are doing things asynchronously.

bokeh multiple live streaming graphs in different objects / register update routine located in other class

1 Answers