ColdFusion Client Variable Showing Outdated Value Intermittently

Question

We have a legacy ColdFusion application using 150 client variables to manage session state. The client variables are centrally stored in a SQL Server database within a 6 application server clustered environment using a round-robin load balancer.

The problem is when the code updates a client variable with a new value, the old value is still being used and displayed even though the new value is updated appropriately in the CData table. This only happens intermittently in an average of 1 out of 1000 updates being made to the client variable using the cfset tag.

Race conditions and caching issues are possible explanations. We “suspect” that the old value is still being cached on one of the 6 application servers. Adobe's documentation clearly states that client variables are cached into memory but does not go into further details.

1) Has anyone also experienced this issue and found a good resolution?

2) What are the implications of moving to a sticky session while we keep using the client variables?

Answer 1

I think I would first take a look closely at your database commits. Client vars are retrieved at the "beginning" of a request and then updated at the "end" of the request with any changes. Consider the case where a cflocation happens at the end of a request. Is it possible that the "next" request draws pages from the DB before the update is committed? In a complex system with replication etc situations like this can happen. 6 clustered web servers is a significant number.

As for caching in memory - CF caches the client var in memory and only "reads" from the store (the database) if the client var is changed (if an update has taken place). So you could be onto something there as well. Still, theoretically the client var cached in memory should be identical to the vars in the data store. Since it stores changes - which are also written to the datastore. This is done in real time (as I understand it) i.e. at the end of the request. All the Changes to the client vars are flushed to the DB. So theoretically even with round robin, if your browser showed up on a new server that did NOT have your client vars in memory it would just go get them out of the DB. That's why I think the DB might be the key here. NOTE: the update behavior might changed based on whether global vars are enabled or disabled. Take a look at each server to see if there is any differences in how this setting is used.

As for sticky sessions: if you are using a hardware based load balancer make sure and explore it's balancing options. What you want is to use sticky sessions and for the LB to divide load among the servers. You want it to be smart enough to know about the actual load (CPU usage usually) not just the gross number of requests that have been divided among the servers. good luck. I love problems like this :)

Answer 2

Thank you very much for your helpful reply!

It is possible that one of the application servers is still attempting to process an update of the client variable while the next application server has already received the "next" request and then uses a cached value instead of the updated value. This would indicate a load issue on our application servers. The serialization and deserialization of the 150 client variables may be a factor. However as you pointed out, the database could be committing much later (or responding much slower) than expected under certain conditions while the application servers are running normally. The client variables are stored in it's own database but is on the same database server as our main database.

That's interesting of you to mention that all the changed client variables are "flushed" to the database at the end of the request. But just to clarify, are they "flushed" out of memory (and to the database) and on all the application servers at the same time? Thereby removing any outdated cached values on the other servers and hence, requiring the servers to retrieve the updated value from the database. Alternately once the database commits, only then would ColdFusion update client variables on the rest of the application servers regardless if whether or not there was a cached value already in memory?

The global variable updates are disable on all our 6 (CF9) production servers for performance reasons. We have this enabled on our testing site which has 3 (CF9) servers but intermittent occurrences are still happening and even at a much higher rate.

Additionally ....

The setDomainCookies attribute in the CFApplication tag is "no". This setting has always been the same since the legacy application was written over 10 years ago. However, Adobe's documentation does indicate that "yes" is require for clustered environments. We are in the middle of creating a simple test page to further analyzed how the Cookie.CFID and Cookie.CFTOKEN values are shared between the application servers as well as looking further into a caching problem by using the test page.