CouchDB filtered replication: Amend doc_id after first complete replication

4
votes

NPM's (node package manager) registry uses CouchDB to store meta information and tarballs of packages at a CouchDB instance at http://registry.npmjs.org/registry .. I use the following replication document (CouchDB 1.1.0) to replicate a subset of the registry to my corporate CouchDB:

{ "_id": "fetch-npm-registry", "doc_ids": [ "coffee-script", "nodeunit" ], "source": "http://couchdb.mycompany.com:5984/registry", "target": "registry", }

[BTW, the CouchApp handling this is at https://github.com/isaacs/npmjs.org (also with complete installation instructions)].

If I want to add another dependency to one of my packages, my naive thought was I just amend the doc_ids list (say, to ["coffee-script", "nodeunit", "npm"]) and start the replication again.

This however doesn't work: The replication is finished immediately and the package I wanted to add to the replication (in this case "npm") is missing.

[The workaround known to me is to delete the target database, replicate again and - because I also use this local registry to publish my proprietary packages - re-publish my local packages. sigh]

Amendment 18.11.2011

Here's what I think what happens (not at all a CouchDB internals expert, but maybe there is some truth to it):

After the first successful replication, CouchDB stores the last (highest?) Sequence ID of the latest document it replicated in a hidden document in the database (I once knew how to access those, pointers welcome). Then, when I change the doc_ids this cached information about the last successful replication (the sequence ID) is not invalidated (or cleared). Then, when it's told to replicate again with the same database it compares the Sequence IDs and decides everything is fine.

2
node.jscouchdbreplicationnpm
Have you checked for conflicts? - Marcello Nuccio
No, but there shouldn't be any. I didn't push (as in npm publish) my own version of npm in the meantime. I have some packages that are original content (my proprietary ones) and the rest should be replicated from upstream. These two sets of packages are disjoint. - peritus
Then check for permissions. Error reporting during replication is weak... check the logs too. Some more information would be useful: CouchDB version; compiled or packaged; what OS? - Marcello Nuccio
CouchDB 1.1.0, compiled from source either on OSX (but also happens on Ubuntu). Will make another clean test run with checking logs. - peritus
I wrote what I think happens as an amendment to the original question. - peritus

2 Answers

1
votes

1) http://registry.npmjs.org/registry is not registry database, but http://registry.npmjs.org is.

2)

{ "_id": "fetch-npm-registry", "doc_ids": [ "coffee-script", "nodeunit" ], "source": "http://couchdb.mycompany.com:5984/registry", "target": "registry", }

Are you sure that you to replicate from http://couchdb.mycompany.com:5984/registry and not http://registry.npmjs.org ?

0
votes

I tried it with Couchbase Single Server 2.0 Developer Preview 5. There it works. My curl command was (but takes quite a few minutes):

curl -X POST 'http://localhost:5984/_replicate' -H 'Content-Type: application/json' -d '{"doc_ids": [ "coffee-script", "nodeunit", "npm" ], "source": "http://registry.npmjs.org:5984/registry", "target": "registry", "create_target": true}

It is based on some Apache COuchDB trunk version.

Could you please try it with Apache CouchDB 1.1.1. I remember that there was a replicator bug with empty IDs (and the npm repository has such a Document), which was fixed.

Cheers, Volker