PouchDB replication speed up?

Question

I am using CouchDB/PouchDB as an off-line tile server for a Cordova mapping app. Initial replication of the CouchDB 215mb db on phone takes over two hours on local network. CouchDB to CouchDB replication takes 6 minutes on local network. Db includes 65000 documents with png attachments of 1-2KB.

function managePouchDB(networkState) {
        // replicate tiles from local workstation
        var remoteCouchdb_osm_bc = "http://192.168.12.5:5984/osm_bc_simple_z6-13_tiles";
        var tiledb_rep_opts = {
            // http://pouchdb.com/api.html#replication
            live: true,
            retry: true,
            // batch size: tried 10 - 100; 10 works best for osm tiles replication w/ nginx
            batch_size: 10, // default = 100
            attachments: true
        };

        // replicate osm bc tiles on check
        var pouchTilesDB_osm_bc_baselayer_replicate = pouchTilesDB_osm_bc_baselayer.replicate.from(remoteCouchdb_osm_bc, tiledb_rep_opts);
        var pouchTilesDB_osm_bc_storedState = localStorage.getItem("pouchTilesDB_osm_bc_storedState");
        if (pouchTilesDB_osm_bc_storedState === "true") {
            pouchTilesDB_osm_bc_baselayer.replicate.from(remoteCouchdb_osm_bc, tiledb_rep_opts);

I have reduced batch size, which yields a reduction in replication time, but still only see ~ 2mb/min replication.

Online CouchDB tiles called directly display well. Using CouchDB V1.6.1, PouchDB V4.0.3

Where would the probable bottlenecks be? Solutions or ideas?

Answer 1

The replication protocoll es extremely chatty. That cost's a lot of time in initial replication.

Check out this tool: https://github.com/nolanlawson/pouchdb-load

Answer 2

As Alex said, you can use pouchdb-load for faster replications.

However, the slowness will be fixed automatically in CouchDB 1.7 and CouchDB 2.0 when those get released, so you can just wait for that. I believe Cloudant has already shipped the fix as well. See this JIRA for details on the speedup.

Answer 3

My added code:

var remoteCouchdb_osm_bc_dump ="http://192.x.x.x:8092/data/osm_tiles_dump.txt";

if ((pouchTilesDB_osm_bc_storedState === "true") && (networkState === "WiFi_connection" || "wifi")) {
   pouchTilesDB_osm_bc_baselayer.load(remoteCouchdb_osm_bc_dump , {
        proxy: remoteCouchDB_osm_bc
   }).then(function () {
        // done loading! handoff to regular replication
        console.log("Done loading osm dump to PouchDB");
        pouchTilesDB_osm_bc_baselayer.info().then(console.log.bind(console));
        return pouchTilesDB_osm_bc_baselayer.replicate.from(remoteCouchdb_osm_bc, tiledb_rep_opts);        
   }).catch(function (err) {
         // HTTP error or something like that
         console.log("error on osm_bc dump to PouchDB");
});

Doesn't throw the above errors when run from phone, but causes app to crash & restart. Doesn't happen without the db.load. Dump file downloads in less than 1 minute to phone, all docs seem to be in db per

pouchTilesDB_osm_bc_baselayer.info().then(console.log.bind(console));

Then throws error and crashes.

CORS errors were thrown when debugging in Chromium and got sorted via Access-Control-Allow-Origin: for Nginx (http://enable-cors.org/server_nginx.html). Had short=circuit, CORS error was from accessing dump.txt file, not CouchDB. Doesn't show for Cordova app in phone.

While a replication is "chatty" that does not fully explain why CouchDB to PouchDB is so much slower than CouchDB to CouchDB replication. Will test CouchDB 2.0.

I can wget dump file from laptop. Took 41s.