Multi-tenancy: Individual database per tenant

8
votes

We are developing a multi-tenant application. With respect to architecture, we have designed shared middle tier for business logic and one database per tenant for data persistence. Saying that, business tier will establish set of connections (connection pool) with the database server per tenant. That means application maintain separate connection-pool for each tenant. If we expect around 5000 tenants, then this solution needs high resource utilization (connections between app server and database server per tenant), that leads to performance issue.

We have resolved that by keeping common connection pool. In order to maintain single connection pool across different databases, we have created a new database called ‘App-master’. Now, we always connect to the ‘App-master’ database first and then change the database to tenant specific database. That solved our connection-pool issue.

This solution works perfectly fine with on-premise database server. But it does not work with Azure Sql as it does not support change database.

Appreciate in advance to suggest how to maintain connection pool or better approach / best practice to deal with such multi-tenant scenario.

2
c#azureazure-sql-databaseconnection-poolingmulti-tenant
Pools are per different connection strings. That means if you have 5000 tenants, the pool size is 5000*100 connections (100 connections in a pool per connection string). I wonder what issues you talk about then. We maintain several multitenant applications with few thousands of tenants and haven't observed any issues around pooling that would lead us to a custom pool implementation. - Wiktor Zychla
We have a similar system in Azure and the initial connection time to a DB is noticeably slower when compared to an existing pooled connection. - knightpfhor
Quick Q: are you connecting to the DB directly from outside the cloud or (big hint) are you using webservices keeping the connectivity issues firmly at the Azure end? - SeanCocteau
@WiktorZychla, Thanks for your response. Actually, we believe it should become bottleneck for us, so want be cautious about that. So, you mean you have already gone through such scenario and you didn't find any issue at any end (SQL Server / all other app server)? - Hitesh
@SeanCocteau, at present we are trying from on-premise dev box, but final production will be on Azure. - Hitesh

2 Answers

2
votes

I have seen this problem before with multiple tenancy schemes with separate databases. There are two overlapping problems; the number of web servers per tenant, and the total number of tenants. The first is the bigger issue - if you are caching database connections via ADO.net connection pooling then the likelihood of any specific customer connection coming into a web server that has an open connection to their database is inversely proportional to the number of web servers you have. The more you scale out, the more any given customer will notice a per-call (not initial login) delay as the web server makes the initial connection to the database on their behalf. Each call made to a non-sticky, highly scaled, web server tier will be decreasingly likely to find an existing open database connection that can be reused.

The second problem is just one of having so many connections in your pool, and the likelihood of this creating memory pressure or poor performance.

You can "solve" the first problem by establishing a limited number of database application servers (simple WCF endpoints) which carry out database communications on behalf of your web server. Each WCF database application server serves a known pool of customer connections (Eastern Region go to Server A, Western Region go to Server B) which means a very high likelihood of a connection pool hit for any given request. This also allows you to scale access to the database separately to access to HTML rendering web servers (the database is your most critical performance bottleneck so this might not be a bad thing).

A second solution is to use content specific routing via a NLB router. These route traffic based on content and allow you to segment your web server tier by customer grouping (Western Region, Eastern Region etc) and each set of web servers therefore has a much smaller number of active connections with a corresponding increase in the likelihood of getting an open and unused connection.

Both these problems are issues with caching generally, the more you scale out as a completely "unsticky" architecture, the less likelihood that any call will hit cached data - whether that is a cached database connection, or read-cached data. Managing user connections to allow for maximum likelihood of a cache hit would be useful to maintain high performance.

0
votes

Another method of restricting the number of connection pools per app server is to use Application Request Routing (ARR) to divide up your tenants and assign them to subsets of the web tier. This lends itself to a more scalable "pod" architecture where a "pod" is a small collection of web/app servers coupled to a subset of the databases. A good article on this approach is here: http://azure.microsoft.com/blog/2013/10/31/application-request-routing-in-csf/

If you are building a multi-tenant DB application Azure you should also check-out the new Elastic Scale client libraries that simplify data-dependent routing and facilitate cross-shard queries and management operations. http://azure.microsoft.com/en-us/documentation/articles/sql-database-elastic-scale-documentation-map/