unable to run mongo-connector

Question

I have installed mongo-connector in the mongodb server.

I am executing by giving the command

mongo-connector -m [remote mongo server IP]:[remote mongo server port] -t [elastic search server IP]:[elastic search server Port]  -d elastic_doc_manager.py

I also tried with this since mongo is running in the same server with the default port.

mongo-connector -t [elastic search server IP]:[elastic search server Port]  -d elastic_doc_manager.py

I am getting error

Traceback (most recent call last):
  File "/usr/local/bin/mongo-connector", line 9, in <module>
    load_entry_point('mongo-connector==2.3.dev0', 'console_scripts', 'mongo-connector')()

  File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/util.py", line 85, in wrapped
    func(*args, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/connector.py", line 1037, in main
    conf.parse_args()

  File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/config.py", line 118, in parse_args
    option, dict((k, values.get(k)) for k in option.cli_names))

  File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/connector.py", line 820, in apply_doc_managers
    module = import_dm_by_name(dm['docManager'])

  File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/connector.py", line 810, in import_dm_by_name
    "Could not import %s." % full_name)

**mongo_connector.errors.InvalidConfiguration: Could not import mongo_connector.doc_managers.elastic_doc_manager.py.**

NOTE: I am using python2.7 and mongo-connector 2.3

Elastic search server is 2.2

Any suggestions ?

[edit] After applying Val's suggestion:

2016-02-29 19:56:59,519 [CRITICAL] mongo_connector.oplog_manager:549 - Exception during collection dump

Traceback (most recent call last):

File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/oplog_manager.py", line 501, in do_dump

upsert_all(dm)

File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/oplog_manager.py", line 485, in upsert_all dm.bulk_upsert(docs_to_dump(namespace), mapped_ns, long_ts)

File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/util.py", line 32, in wrapped

return f(*args, **kwargs)

File "/usr/local/lib/python2.7/dist-packages/mongo_connector-2.3.dev0-py2.7.egg/mongo_connector/doc_managers/elastic_doc_manager.py", line 190, in bulk_upsert

for ok, resp in responses:

File "/usr/local/lib/python2.7/dist-packages/elasticsearch-1.9.0-py2.7.egg/elasticsearch/helpers/init.py", line 160, in streaming_bulk

for result in _process_bulk_chunk(client, bulk_actions, raise_on_exception, raise_on_error, **kwargs):

File "/usr/local/lib/python2.7/dist-packages/elasticsearch-1.9.0-py2.7.egg/elasticsearch/helpers/init.py", line 132, in _process_bulk_chunk

raise BulkIndexError('%i document(s) failed to index.' % len(errors), errors)

BulkIndexError: (u'2 document(s) failed to index.',..document_class=dict, tz_aware=False, connect=True, replicaset=u'mss'), u'local'), u'oplog.rs')

2016-02-29 19:56:59,835 [ERROR] mongo_connector.connector:302 - MongoConnector: OplogThread unexpectedly stopped! Shutting down

---

Hi Val,

I connected with another mongodb instance, which had only one database, having one collection with 30,000+ records and I was able to execute it succesfully. The previous mongodb collection has multiple databases (around 7), which internally had multiple collections (around 5 to 15 per databases) and all were having good amount of documents (ranging from 500 to 50,000) in the collections.

Was Mongo-connector failing because of huge data residing in the mongo database ?

I have further queries

a. Is is possible to get indexing done of only specific collections in the mongodb, residing in different databases? I wan to index only specific collections (not the entire database). How can I achieve this ?

b. In elasticsearch i can see duplicate indexes for one collection. First one is with the database name (as expected), other one with the name mongodb_meta, both of them having same data, if I am changing the collection, the update is happening in both the collections.

c. Is it possible to configure the output index name or any other parameters any how?

Answer 1

I think the only issue is that you have the .py extension on the doc manager (it was needed before mongo-connector 2.0), you simply need to remove it:

mongo-connector -m [remote mongo server IP]:[remote mongo server port] -t [elastic search server IP]:[elastic search server Port]  -d elastic_doc_manager

Answer 2

I found this option to run specific collection only.

$ mongo-connector -m mongodbserver:27017 -t elasticserver:9200 -d elastic_doc_manager --oplog-ts oplogstatus.txt --namespace-set database.collection

Answer 3

It started working after giving below command with --oplog-ts option.

mongo-connector -m localhost:27017 -t localhost:37017 -d mongo_doc_manager --oplog-ts oplogstatus.txt

But its failing if i use a config file. Kindly advise how to resolve this issue.

C:\Dev\mongodb\mongo-connector>mongo-connector -c myconfig.json --oplog-ts oplogstatus.txt
Fatal Exception
Traceback (most recent call last):
  File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\config.py", line 110, in parse_args
    self.load_json(f.read())
  File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\config.py", line 132, in load_json
    parsed_config = json.loads(text)
  File "C:\Program Files\Python\lib\json\__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "C:\Program Files\Python\lib\json\decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Program Files\Python\lib\json\decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid \escape: line 6 column 21 (char 201)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\util.py", line 90, in wrapped
    func(*args, **kwargs)
  File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\connector.py", line 1059, in main
    conf.parse_args()
  File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\config.py", line 112, in parse_args
    reraise(errors.InvalidConfiguration, *sys.exc_info()[1:])
  File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\compat.py", line 9, in reraise
    raise exctype(str(value)).with_traceback(trace)
  File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\config.py", line 110, in parse_args
    self.load_json(f.read())
  File "C:\Program Files\Python\lib\site-packages\mongo_connector-2.5.0.dev0-py3.6.egg\mongo_connector\config.py", line 132, in load_json
    parsed_config = json.loads(text)
  File "C:\Program Files\Python\lib\json\__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "C:\Program Files\Python\lib\json\decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Program Files\Python\lib\json\decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
mongo_connector.errors.InvalidConfiguration: Invalid \escape: line 6 column 21 (char 201)

Answer 4

Try this. pip install 'elastic2-doc-manager[elastic5]'

mongo-connector -m localhost:27017 -t localhost:9200 -d elastic2_doc_manager

Answer 5

answer on github

Your strategy seems sound to me. Here's how to do this:

Generate a mongo-connector timestamp file:

Run mongo-connector --no-dump.

Stop mongo-connector right after it starts up. Now you have an oplog.timestamp file pointing to the latest entry on the oplog.

Run mongodump on the primary. The dump already reflects all the changes that mongo-connector saw in the oplog.

Run mongorestore with the dump from (2) on the target MongoDB.

Restart mongo-connector. Pass in the file generated in (1) to the --oplog-ts option.

I'll add this to the wiki.