Docker Filebeat Nginx Logs

Question

I m using filebeat as docker and when ı point my nginx logs in filebeat.yml ı m not able to see nginx logs in kibana here is my filebeat.yml. I have elastichsearch and kibana containers ready to go. When I start filebeat container in the logs it says that given log paths are configured. but ı can not visualize any nginx logs on kibana

  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

# filebeat.autodiscover:
#   providers:
#     - type: docker
#       hints.enabled: true

filebeat.autodiscover:
    providers:
      - type: docker
        hints.enabled: true
        templates:
          - condition:
              contains:
                docker.container.image: nginx
            config:
              - type: docker
                containers.ids:
                    - "${data.docker.container.id}"
              - module: nginx
                access:
                  enabled: true
                  var.paths: ["/var/log/nginx/user_service_access.log"]
                  containers:
                      stream: "stdout"
                error:
                  enabled: true
                  var.paths: ["/var/log/nginx/user_service_access.log"]
                  containers:
                      stream: "stderr"
                      
processors:
- add_cloud_metadata: ~

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:ip_address:9200}'

and an example of my nginx site conf

    listen 80;
    listen [::]:80;

    # For https
    # listen 443 ssl;
    # listen [::]:443 ssl ipv6only=on;
    # ssl_certificate /etc/nginx/ssl/default.crt;
    # ssl_certificate_key /etc/nginx/ssl/default.key;

    server_name user_service.test;
    root /var/www/user_service/public;
    index index.php index.html index.htm;

    location / {
        try_files $uri $uri/ /index.php$is_args$args;
        proxy_pass http://ip_address:5601;
    }

    location ~ \.php$ {
        try_files $uri /index.php =404;
        fastcgi_pass php-upstream;
        fastcgi_index index.php;
        fastcgi_buffers 16 16k;
        fastcgi_buffer_size 32k;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        #fixes timeouts
        fastcgi_read_timeout 600;
        include fastcgi_params;
    }

    location ~ /\.ht {
        deny all;
    }

    location /.well-known/acme-challenge/ {
        root /var/www/letsencrypt/;
        log_not_found off;
    }

    error_log /var/log/nginx/user_service_error.log;
    access_log /var/log/nginx/user_service_access.log;
}```

Answer 1

Try running the filebeat in debug mode to check for any failures in filebeat configurations. From the filebeat home, try running:

filebeat -e -c filebeat.yml -d "*"

Answer 2

autodiscovery config for filebeat 7.9.x looks like this it allows for nginx log parsing from container's streams

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      templates:
        - condition.contains:
            docker.container.image: nginx
          config:
            - module: nginx
              access.input:
                type: docker
                containers:
                  ids: "${data.docker.container.id}"
                  stream: "stdout"
              error.input:
                type: docker
                containers:
                  ids: "${data.docker.container.id}"
                  stream: "stderr"