RavenDB Replication Issue - database cannot be found

Question

Have followed the documentation, but am unable to make replication work for RavenDB over the WAN.

Scenario:

Using Raven build #2261
Master DB: has a local name of "it23"
Slave DB: has a remote name of "http://184.169.xxx.xxx" (xxx's are for privacy)

On both servers I have created a database called "TonyTest".

On the Master db, I have set up replication using the following document:

{
  "Destinations": [
    {
      "Url": "http://184.169.xxx.xxx:8080",
      "Username": null,
      "Password": null,
      "Domain": null,
      "ApiKey": null,
      "Database": "TonyTest",
      "TransitiveReplicationBehavior": "None",
      "IgnoredClient": false,
      "Disabled": false,
      "ClientVisibleUrl": null
    }
  ]
}

When browsing to the remote server using the same URL of: http://184.169.xxx.xxx:8080, the RavenDB studio launches correctly, and I can see the TestTony database. This seems to confirm that the URL is formatted correctly.

However, the master database immediately generates a document showing failures:

{   
  "Destination": "http://184.169.xxx.xxx:8080/databases/TonyTest",  
  "FailureCount": 142 
}

When we look at the logs for the REMOTE db, we see that there IS communication with the master, but the replication doesn't complete.

Debug   3/9/2013 12:19:44 AM    Document with key 'Raven/Replication/Sources/http://it23:8080/databases/TonyTest' was not found     Raven.Storage.Esent.StorageActions.DocumentStorageActions

It looks like the remote server is saying that the db "TonyTest' can't be found, but it IS created.

Can anyone spot my mistake?

Per Ayende's request, here are some log samples from LOCAL server after attempting to setup replication (again I replaced IPs with xxx for privacy). We do not see any errors in the LOCAL db's log. And we do see errors popup in the REMOTE db log. This seems to imply that the LOCAL db is connecting to the REMOTE db, but the replication does not happen. Here are the LOCAL logs:

Debug   3/11/2013 3:17:00 PM    No work was found, workerWorkCounter: 17626, for: ReducingExecuter, will wait for additional work       Raven.Database.Indexing.WorkContext
Debug   3/11/2013 3:17:00 PM    Going to index 1 documents in IndexName: Raven/DocumentsByEntityName, LastIndexedEtag: 00000001-0000-0100-0000-000000002265: (Raven/Replication/Destinations/184.169.xxx.xxx8080databasesTonyTest)      Raven.Database.Indexing.AbstractIndexingExecuter
Debug   3/11/2013 3:17:00 PM    Document with key 'Raven/Studio/PriorityColumns' was not found      Raven.Storage.Esent.StorageActions.DocumentStorageActions
Debug   3/11/2013 3:16:56 PM    Going to index 1 documents in IndexName: Raven/DocumentsByEntityName, LastIndexedEtag: 00000001-0000-0100-0000-000000002256: (Raven/Replication/Destinations/184.169.xxx.xxx8080databasesTonyTest)      Raven.Database.Indexing.AbstractIndexingExecuter

Update 3/11 8:24p Pacific time I am now seeing the following errors in the MASTER/Local raven logs:

Failed to close response

System.AggregateException: One or more errors occurred. ---> System.Net.HttpListenerException: An operation was attempted on a nonexistent network connection
   at System.Net.HttpResponseStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at Raven.Database.Util.Streams.BufferPoolStream.Dispose(Boolean disposing) in c:\Builds\RavenDB-Stable\Raven.Database\Util\Streams\BufferPoolStream.cs:line 144
   at System.IO.Stream.Close()
   at Raven.Database.Impl.ExceptionAggregator.Execute(Action action) in c:\Builds\RavenDB-Stable\Raven.Database\Impl\ExceptionAggregator.cs:line 23
   --- End of inner exception stack trace ---
   at Raven.Database.Impl.ExceptionAggregator.ThrowIfNeeded() in c:\Builds\RavenDB-Stable\Raven.Database\Impl\ExceptionAggregator.cs:line 38
   at Raven.Database.Server.Abstractions.HttpListenerResponseAdapter.Close() in c:\Builds\RavenDB-Stable\Raven.Database\Server\Abstractions\HttpListenerResponseAdapter.cs:line 94
   at Raven.Database.Server.Abstractions.HttpListenerContextAdpater.FinalizeResponse() in c:\Builds\RavenDB-Stable\Raven.Database\Server\Abstractions\HttpListenerContextAdpater.cs:line 92
---> (Inner Exception #0) System.Net.HttpListenerException (0x80004005): An operation was attempted on a nonexistent network connection
   at System.Net.HttpResponseStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at Raven.Database.Util.Streams.BufferPoolStream.Dispose(Boolean disposing) in c:\Builds\RavenDB-Stable\Raven.Database\Util\Streams\BufferPoolStream.cs:line 144
   at System.IO.Stream.Close()
   at Raven.Database.Impl.ExceptionAggregator.Execute(Action action) in c:\Builds\RavenDB-Stable\Raven.Database\Impl\ExceptionAggregator.cs:line 23<---

The destination log isn't relevant. What is the error you see in the source? — Ayende Rahien
Hello Ayende. I have copied some of the Local/Source logs above. We do not seem to see any errors that I recognize. Any ideas? — Pete Lunenfeld
Another update, I created a 2nd RavenDB instance on my PC, and was able to replicate perfectly. What special configurations are needed for replication over the WAN? — Pete Lunenfeld
Another update. I have opened all the ports on my firewall that would allow my remote/slave db to 'see' the Master db. Still no luck. The only logs I get that help are on the remote/slave: Document with key 'Raven/Replication/Sources/it23:8080/databases/Pete' was not found. it23 is the network name of my master/local db. Do I need to use the "ClientVisibleUrl" property to repoint back to the master. I can't find the documentation for how to use that property. — Pete Lunenfeld
When I replicate on the LAN, the "Raven/Replication/Sources/" is instantly created on the slave server. But over the WAN that file is NOT created. Could this be permission thing? — Pete Lunenfeld

Pete Lunenfeld Pete Lunenfeld · Accepted Answer · 2013-03-13T02:45:34

Solved this. Although I don't understand the full reason why.

On the SLAVE server, you must set the raven.server.exe config file to have the following key:

<add key="Raven/AnonymousAccess" value="All"/>

The default was

<add key="Raven/AnonymousAccess" value="Get"/>.

The default worked fine when the master and slave were on the same machine. But when the master and slave were on separate machines (either on the LAN, or across the WAN) replication failed.

I could never find a log entry on the master that pointed toward his problem. The only log entry I could see was on the slave which said that Raven/Replication/Sources/ was not found. I realized that the master was connecting to the slave, but the slave was unable to create the "Raven/Replication/Sources/" document remotely.

RavenDB Replication Issue - database cannot be found

1 Answers