Migrating existing Cosmosdb collections to Autopilot

Question

I have a number of existing collection with manual RU provisioning which I would like to migrate to be Autopilot managed to better automatically deal with varying levels of demand.

The collections contain many GB of historical timeseries data, and I cannot have any downtime where new or historical data is not available to customers. I must also ensure no data is lost during the migration

Once a day, a new day of data is bulk uploaded to the cosmosdb collection, and the collections can be queried at any time by the customer-facing service in front of them.

For migration, I was considering the following: 1. Create new autopilot collection 2. Modify service to query both old and new collection and deduplicate any data present in both 3. Redirect data upload to new collection 4. Use ADF (Azure data factory) to copy the contents of the old collection to the new Autopilot one 5. Update service to only query the new collection 6. Drop old collection.

Is this the best migration strategy, or is there an alternative approach which would provide a better customer experience, or be less work?

Answer 1

While in Preview you will need to manually migrate data to AutoPilot containers. Once we GA we are planning to allow customers to seamlessly migrate containers from regular to AutoPilot throughput.

For the scenario you describe I find it easier to use ChangeFeed when I need to do a near zero downtime migration. Create a new AutoPilot configured container, then create an Azure Function using the Cosmos DB bindings to read from the source container and write to the new AutoPilot container to allow data to stay in sync.

Rewrite your consuming apps to use the new container and your bulk load scripts to write to the new container. Once that is done, deploy the changes. I like using slots for Web Apps (or whatever you choose) for zero or near zero downtime.

One thing to keep an eye on is since you are bulk loading this data, Azure Functions will likely fall far behind keeping the data in sync. You'll want to monitor to see how long that takes just so you know when you can flip the switch on the migration.

Hope that helps.