Elastic Apm python agent connection problem

1
votes

I have a very basic Django APM agent setup:

ELASTIC_APM = {
# Set required service name. Allowed characters:
# a-z, A-Z, 0-9, -, _, and space
'SERVICE_NAME': 'bourse',

# Set custom APM Server URL (default: http://localhost:8200)
'SERVER_URL': '127.0.0.1:8200',
'LOG_LEVEL': "debug",
'LOG_FILE': "/home/hamed/Desktop/log.txt",}

My APM server is up and running on localhost:8200. But it seems my APM agent can't make a connection to the APM server. Here is a part of my log file that I think cause the problem:

Skipping instrumentation of urllib3. Module botocore.vendored.requests.packages.urllib3.connectionpool not found
HTTP error while fetching remote config: HTTPConnectionPool(host='config', port=80): Max retries exceeded with url: /v1/agents (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe2625ed610>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

Here are my APM logs:

Inserting elasticapm log_record_factory into logging
Location of cgroup files failed, skipping cgroup metrics
Detected PID change from None to 13595, starting threads
Starting transport thread
Starting metrics thread
Starting metrics collect timer
Starting config thread
Instrumented jinja2, jinja2.Template.render
Skipping instrumentation of pymssql. Module pymssql not found
Instrumented psycopg2, psycopg2.extensions.register_type, psycopg2._json.register_json, psycopg2.extensions.quote_ident, psycopg2.extensions.encrypt_password
Skipping instrumentation of aiohttp_client. Module aiohttp.client not found
Instrumented psycopg2, psycopg2.connect
requests.packages.urllib3.connectionpool.HTTPConnectionPool.urlopen already instrumented, skipping
Skipping instrumentation of urllib3. Module botocore.vendored.requests.packages.urllib3.connectionpool not found
Instrumented urllib3, urllib3.connectionpool.HTTPConnectionPool.urlopen
Skipping instrumentation of mysql_connector. Module mysql.connector not found
Skipping instrumentation of httpx. Module httpx not found
Skipping instrumentation of aiopg. Module aiopg.cursor not found
Instrumented tornado_handle_request_exception, tornado.web.RequestHandler._handle_request_exception
Instrumented tornado_render, tornado.web.RequestHandler.render
Instrumented redis, redis.connection.Connection.send_packed_command
Instrumented redis, redis.client.Pipeline.execute
Skipping instrumentation of python_memcached. Module memcache not found
Skipping instrumentation of pylibmc. Module pylibmc not found
Skipping instrumentation of graphql. Module graphql.backend.core not found
Skipping instrumentation of graphql. Module graphql.backend.cache not found
Skipping instrumentation of httpcore. Module httpcore._async.connection not found
Skipping instrumentation of pymongo. Module pymongo.bulk not found
Instrumented django_template_source, django.template.base.Parser.extend_nodelist
Instrumented urllib, urllib.request.AbstractHTTPHandler.do_open
Skipping instrumentation of pymongo. Module pymongo.collection not found
Skipping instrumentation of pymongo. Module pymongo.cursor not found
Skipping instrumentation of pyodbc. Module pyodbc not found
Skipping instrumentation of elasticsearch_connection. Module elasticsearch_async.connection not found
Skipping instrumentation of elasticsearch_connection. Module elasticsearch._async.http_aiohttp not found
Skipping instrumentation of httpcore. Module httpcore._sync.connection not found
Instrumented tornado_request_execute, tornado.web.RequestHandler._execute
Skipping instrumentation of httpx. Module httpx not found
Instrumented django_template, django.template.Template.render
Skipping instrumentation of asyncpg. Module asyncpg.connection not found
Skipping instrumentation of sqlite. Module pysqlite2.dbapi2 not found
Instrumented sqlite, sqlite3.connect, sqlite3.dbapi2.connect
Skipping instrumentation of pymysql. Module pymysql not found
Skipping instrumentation of cassandra. Module cassandra.cluster not found
Instrumented elasticsearch_connection, elasticsearch.connection.http_urllib3.Urllib3HttpConnection.perform_request, elasticsearch.connection.http_requests.RequestsHttpConnection.perform_request
Instrumented redis, redis.client.Redis.execute_command
Instrumented asyncio_sleep, asyncio.tasks.sleep
Skipping instrumentation of mysql. Module MySQLdb not found
Instrumented requests, requests.sessions.Session.send
Skipping instrumentation of graphql. Module graphql.execution.executors.sync not found
Skipping instrumentation of graphql. Module graphql.execution.executors.gevent not found
Skipping instrumentation of graphql. Module graphql.execution.executors.asyncio not found
Skipping instrumentation of graphql. Module graphql.execution.executors.process not found
Skipping instrumentation of graphql. Module graphql.execution.executors.thread not found
Skipping instrumentation of pymemcache. Module pymemcache.client.base not found
Skipping instrumentation of pymemcache. Module pymemcache.client.hash not found
Skipping instrumentation of botocore. Module botocore.client not found
Inserting elasticapm log_record_factory into logging
Location of cgroup files failed, skipping cgroup metrics
Detected PID change from None to 13607, starting threads
Starting transport thread
Starting metrics thread
Starting metrics collect timer
Starting config thread
Instrumented elasticsearch_connection, elasticsearch.connection.http_urllib3.Urllib3HttpConnection.perform_request, elasticsearch.connection.http_requests.RequestsHttpConnection.perform_request
Instrumented django_template, django.template.Template.render
Skipping instrumentation of pylibmc. Module pylibmc not found
Instrumented django_template_source, django.template.base.Parser.extend_nodelist
Instrumented redis, redis.connection.Connection.send_packed_command
Skipping instrumentation of graphql. Module graphql.backend.core not found
Skipping instrumentation of graphql. Module graphql.backend.cache not found
Instrumented asyncio_sleep, asyncio.tasks.sleep
Skipping instrumentation of pymongo. Module pymongo.cursor not found
Skipping instrumentation of elasticsearch_connection. Module elasticsearch_async.connection not found
Skipping instrumentation of elasticsearch_connection. Module elasticsearch._async.http_aiohttp not found
Instrumented redis, redis.client.Redis.execute_command
Skipping instrumentation of mysql_connector. Module mysql.connector not found
Skipping instrumentation of sqlite. Module pysqlite2.dbapi2 not found
Instrumented sqlite, sqlite3.connect, sqlite3.dbapi2.connect
Instrumented requests, requests.sessions.Session.send
Skipping instrumentation of aiohttp_client. Module aiohttp.client not found
Skipping instrumentation of pymemcache. Module pymemcache.client.base not found
Skipping instrumentation of pymemcache. Module pymemcache.client.hash not found
Instrumented tornado_handle_request_exception, tornado.web.RequestHandler._handle_request_exception
requests.packages.urllib3.connectionpool.HTTPConnectionPool.urlopen already instrumented, skipping
Skipping instrumentation of urllib3. Module botocore.vendored.requests.packages.urllib3.connectionpool not found
Instrumented urllib3, urllib3.connectionpool.HTTPConnectionPool.urlopen
Skipping instrumentation of pymssql. Module pymssql not found
Skipping instrumentation of pymysql. Module pymysql not found
Skipping instrumentation of pymongo. Module pymongo.collection not found
Skipping instrumentation of httpcore. Module httpcore._sync.connection not found
Instrumented psycopg2, psycopg2.connect
Instrumented urllib, urllib.request.AbstractHTTPHandler.do_open
Skipping instrumentation of aiopg. Module aiopg.cursor not found
Skipping instrumentation of httpx. Module httpx not found
Skipping instrumentation of botocore. Module botocore.client not found
Skipping instrumentation of pymongo. Module pymongo.bulk not found
Skipping instrumentation of cassandra. Module cassandra.cluster not found
Instrumented tornado_render, tornado.web.RequestHandler.render
Skipping instrumentation of graphql. Module graphql.execution.executors.sync not found
Skipping instrumentation of graphql. Module graphql.execution.executors.gevent not found
Skipping instrumentation of graphql. Module graphql.execution.executors.asyncio not found
Skipping instrumentation of graphql. Module graphql.execution.executors.process not found
Skipping instrumentation of graphql. Module graphql.execution.executors.thread not found
Instrumented psycopg2, psycopg2.extensions.register_type, psycopg2._json.register_json, psycopg2.extensions.quote_ident, psycopg2.extensions.encrypt_password
Skipping instrumentation of httpx. Module httpx not found
Skipping instrumentation of asyncpg. Module asyncpg.connection not found
Instrumented tornado_request_execute, tornado.web.RequestHandler._execute
Skipping instrumentation of httpcore. Module httpcore._async.connection not found
Instrumented jinja2, jinja2.Template.render
Skipping instrumentation of pyodbc. Module pyodbc not found
Skipping instrumentation of mysql. Module MySQLdb not found
Instrumented redis, redis.client.Pipeline.execute
Skipping instrumentation of python_memcached. Module memcache not found
Checking for new config...
HTTP error while fetching remote config: HTTPConnectionPool(host='config', port=80): Max retries exceeded with url: /v1/agents (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe2625ed610>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
Remote config unchanged
Checking for new config...
HTTP error while fetching remote config: HTTPConnectionPool(host='config', port=80): Max retries exceeded with url: /v1/agents (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f44c17c2a30>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
Remote config unchanged
flushing due to time since last flush 9.503s > max_flush_time 9.503s
flushing due to time since last flush 10.827s > max_flush_time 10.827s
flushing due to time since last flush 9.971s > max_flush_time 9.971s
flushing due to time since last flush 9.114s > max_flush_time 9.114s
flushing due to time since last flush 9.763s > max_flush_time 9.763s
Collecting metrics
Collecting metrics
flushing due to time since last flush 10.874s > max_flush_time 10.874s
Failed to submit message: 'Connection to APM Server timed out (url: intake/v2/events, timeout: 5 seconds)'
flushing due to time since last flush 10.796s > max_flush_time 10.796s
Failed to submit message: 'Connection to APM Server timed out (url: intake/v2/events, timeout: 5 seconds)'
Cancelling collect timer
Cancelling collect timer

On my APM server, I'm not receiving any requests from my agent. I checked the APM server log files.

1
djangoelasticsearchapmelastic-apm
This doesn't look right: HTTPConnectionPool(host='config', port=80) – Val
what should it be instead ?host and port? – HamedTaherkhani
I'm just not sure where those values come from... – Val
I added urllib3 to my django requirements. Is there anything else I should do with urllib3? – HamedTaherkhani
Could you try setting 'SERVER_URL': 'http://127.0.0.1:8200'? The Temporary failure in name resolution seems odd to me and I wonder if it's failing to parse that URL – Colton Myers

1 Answers

1
votes

Commenting out 'SERVER_URL': '127.0.0.1:8200' solved the problem.