How to get snapshot consistency when extracting data from MS Dynamics CRM?

1
votes

I've been researching how to extract data from an MS Dynamics CRM 2011/3 Online instance so that I can replicate entire CRM entities in a target database.

I've looked at the Retrieve and RetrieveAll operations of the Organisation web service. These are able to extract data from a single CRM entity (entity type).

There's also the FetchXML interface, that can retrieve data using a complex query, from multiple entities.

It's possible that will be be no quiet time, when there are no data changes being made by users, or via web services, that I could use to extract data from the system in order to get a consistent snapshot of the data.

If I was able to access the SQL Server database directly I would be able to set an isolation level for a transaction and extract all data within that transaction, and get a consistent view of data.

I think FetchXML would give me a consistent snapshot, but only of the data queried by each call to it.

I could use FetchXML to query all the entities I'd like to replicate, in a single call, and then renormalise the data, with some ETL code, on my target database. That query wouldn't be nice though (complex and possibly non performant, and impacting the system performance).

So, basically my problem is this: if I extract from each entity in turn, and the database is changing whilst I'm extracting, I'm highly like to get an inconsistent data set in my target database.

How can I get a consistent snapshot of data to access?

1
dynamics-crm-2011

1 Answers

1
votes

You can contact support through the Support Portal and request a database backup. Then you can just restore that database to your On-Premise installation through Deployment Manager.

EDIT

After your comments below, I suggest a "push" model instead of a "pull" model. You'll need to create plugins for Create/Update/Delete on all entities in which you are interested in CRM Online. These plugins will push those updates to your database (probably through your own web-service). Since these plugins happen inside the transaction, if your web service throws an error you can cancel the source action in CRM, thus guaranteeing transactional consistency.

Once you get these plugins up and running, you can do a one-time export and your plugins will keep it up to date from there.