Play framework - loading data from external database

Question

I've been working on a Play application (1.2.4) which will do some crunching on logs. It's not really the typical thing the framework would be used for (basically I'll probably be the only user), but it's so fast to test an idea that I decided to use this.

I'll be loading a lot of different kinds of events and related accounts stored in the log (database) of a separate application, store them in the local database and then do correlation between the events. Until now I have just fetched the logs to file and imported them to the database, but it's clumsy, slow, error prone and the files grow pretty large when the timeframe is extended.

I'd like to figure out a way to get read access to two external databases, run queries and then create locally stored objects in the local database. Both external databases have different schemas, and of course the local one is different two. The connection is only needed during the import, and since the tables I'm importing data from contain a lot of columns out of which I just need a few (and the servers are far away) I would prefer to query for just the columns I need. I'm fine with directly creating models stored locally directly out of the result-set, so I have no need to even define models based on the external database. I might also need to fetch data out of external database A based on data I get from database B (for example when a log row contains a user I have not stored locally, fetch his data from the other server before storing the event). The connections are only needed during import which in theory is a one time event. Also a big bonus would be the possibility to configure the datasources settings without having to restart (for example storing the host / port in the local database and reading the values before starting the import).

Most threads I find about using multiple DB's seem to be related to reducing load by spreading data with the same schema over several instances. I did find some threads about having separate databases with a different schema, but could not quite figure out how well the solutions would adapt to my needs. Any feedback would be appreciated.

Answer 1

You can connect to your external databases in the same way you usually would - JDBC, Hibernate, or whatever. Play adds a bit of framework magic into internally managed models, but there's nothing stopping you adding n different access methods for n frameworks. You will have to manage transactional state, etc, yourself but that's no different to usual.

Note that you can also use Spring to configure these external databases.

If this is a one-hit import, you may want to consider writing a migration tool to do this, and have your main application focus on its day job. You can share the DB model between your main app and your migration app, and so write the migration app using Play. We did this on a previous project, and it worked very well.