Core Data Concurrency Import Performance

Question

I’m working on an app connected to a web service which retrieves lot of data during app launch. I use concurrency to avoid UI blocking. I choosed the following Core Data Stack pattern : background private moc —> main moc —> writer private moc —> coordinator —> file.

The problem occures when operations are being imported. The CPU is 100% used and the app gets slow along the process. I work with batches of 300 objects for a total import of about 10,000 objects.

For each batch, an NSOperation is created with an associated temporary moc, child of the background one. Operation is enqueue in an NSOperationQueue. When the importing jobs are done, the app get even slower, depending on the number of jobs running. I also note that when the app is killed, and relaunched, it’s really way more usable and fast.

My memory footprint changes between 40Mo and 60Mo when importing. Do you think it’s too much?

Do you think my stack pattern is appropriate for my needs? Should I migrate to a stack with 2 coordinators?

Moreover, when fetching data to display in tableView, should I use performBlockAndWait to get data immediately before displaying the view ?

Thanks for your help

Answer 1

Your stack as described is fine.

CPU usage can be misleading. You want to make sure you are not on the main thread as that will cause most of your slowness and/or stuttering in the app.

When you watch your app in Instruments, what is taking the most time? How much time is spent on the main queue?

In general, imports shouldn't be causing the CPU to sit at 100%. If you are doing that from a background thread there is most likely some performance gains to be made.

If should share your import code and or Instruments trace so that I can see what is going on.

Answer 2

I think your setup is problematic. You state that the child of the background managed object context is main thread and that you create such children to import. This is bound to cause UI glitches.

Also, I believe that relying on NSOperation is unnecessary over-engineering. You should use the NSManagedObjectContext block APIs instead.

My recommended setup would be:

RootContext (background, writing to persistent store) -> parent of
MainContext (foreground, UI) -> parent of
WorkerContext(s) (background, created and discarded ad hoc)

You can create worker contexts in the callbacks of your web calls to do the heavy lifting for the import. Make sure you are using the block APIs and confine all objects to the local context. You save the context to push the changes up to the main thread (which can already start displaying data before it is saved to the store), and periodically, you save the main context and the writer context, always using the bock APIs.

A typical such saveContext function that can be called thread safe (here self refers to the data manager singleton or app delegate):

func saveContext () {
    if self.managedObjectContext.hasChanges  {
       self.managedObjectContext.performBlocAndWait {
            do { try self.managedObjectContext.save() }
            catch let error as NSError {
                print("Unresolved error while saving main context \(error), \(error.userInfo)")
            }
        }
        self.rootContext.performBlockAndWait {
            do { try self.rootContext.save() }
            catch let error as NSError {
                print("Unresolved error while saving to persistent store \(error), \(error.userInfo)")
            }
        }
    }
}

Answer 3

After few days of test and instruments trace, I can give you more details. The following snippet shows how I save my context (based on parent/child pattern) from a shared instance :

- (void)save {
    [self.backgroundManagedObjectContext performBlockAndWait:^{
        [self saveContext:self.backgroundManagedObjectContext];

        [self.mainManagedObjectContext performBlock:^{
            [self saveContext:self.mainManagedObjectContext];

            [self.writerManagedObjectContext performBlock:^{
                [self saveContext:self.writerManagedObjectContext];
            }];
        }];
     }];
}

- (void)saveContext:(NSManagedObjectContext*)context {
     NSError *error = nil;

     if ([context hasChanges] && ![context save:&error]) {
         NSLog(@"Unresolved error %@, %@", error, [error userInfo]);
     }
}

Then each import job is perform on the background context thank to a synchronous operation. The following method is fired in operation main.

- (void)operationDidStart
{
    NSManagedObjectContext *moc = self.context;
    NSMutableArray *insertedOrUpdatedObjects = [NSMutableArray array];
    NSMutableArray *subJSONs = [NSMutableArray array];
    NSUInteger numberOfJobs = ceil((double)self.JSONToImport.count/self.batchSize);

    for (int i = 0; i < numberOfJobs; i++) {
        NSUInteger startIndex = i * self.batchSize;
        NSUInteger count = MIN(self.JSONToImport.count - startIndex, self.batchSize);
        NSArray *arrayRange = [self.JSONToImport subarrayWithRange:NSMakeRange(startIndex, count)];
        [subJSONs addObject:arrayRange];
    }

    __block NSUInteger  numberOfEndedJobs = 0;

    for (NSArray *subJSON in subJSONs) {
        [moc performBlock:^{
            [self startJobWithJSON:subJSON context:moc completion:^(NSArray *importedObjects, NSError *error) {

                numberOfEndedJobs++;

                if (!error && importedObjects && importedObjects.count > 0) {
                    [insertedOrUpdatedObjects addObjectsFromArray:importedObjects];
                }

                if (numberOfEndedJobs == numberOfJobs) {
                    [[CoreDataManager manager] save];

                    if (self.operationCompletion) {
                        self.operationCompletion(self, insertedOrUpdatedObjects, error);
                    }
                }
            }];
        }];
    }
}

As you can see, I segment my import in batches (of 500). The operation perform each batch on the background context queue and I save my stack when all batches are ended.

It seems the save method take 23% of CPU usage for each thread thanks to Time Profiler.

Hope to be as clear as possible.