Before I start with my questions let me explain a bit about the architecture we need to use:
We will have one Central application instance.
- this application instance is deployed with business administration webapp - it is used to change "internal" data (I'll talk about this in a moment)
- this application instance is actually a cluster of servers
We will have n (n > 0 && n <= 3000) Local application instances - 1 for each "location"
- this instances serve as data-processors for their locations
- this instance does not use full set of Central data - only subset limited to what is needed for data-processing
- each location may (and we should expect it) go offline for extended periods of time (let's say 2 weeks top)
Each location will have several hundred "clients" (let's say up to 300) - outside applications that will provide data for processing
Now putting everything together: If location is online, then the clients may talk to either Local instance or Central. However, if location is offline, then the Central is not available and the clients can only talk with Local instance. In such cases Local instances should process the data as if they were Central (so according to the rules defined in the Central - the "internal" data I've talked before), cache the result and when the location becomes online, synchronize it to Central (just the result, no recalculation on Central). At the same time Local instances should always keep their "internal" data synchronized with Central (when they are online, if they are offline we assume the data is "fresh" as long as the offline time did not exceed the 2 weeks threshold). Or, looking at it from the other side - whenever something changes on Central it needs to be pushed to all available Local instances.
So to summarize (and finally ask my question) we need a way to synchronize data from Central instance to several thousands of Local instances, we also need a way to send Local changes to Central. Considering the amount of Local instances, the possible high traffic (each local can have up to 300 clients, each client can generate several requests per minute, the calculation of each request can take a lot of time, and the result may be big)) and all of the other constraints (for example Central instance will be cluster of Weblogic servers, but each Local will be single WildFly, the databases will also be different for Central and Local - including different schema, what would be the best approach for this communication and synchronization problem?
