Google cloud trace + Gcloud logging in logs viewer

Question

I've stuck with problem with google cloud logging and google cloud trace using google cloud kubernetes

I've the application which consumes gcloud pubsub topic and I want to unify logs in trace of every pubsub message handle func call

My Gcloud Logging handler code

class GCLHandler(CloudLoggingHandler):
    def emit(self, record):
        message = super(GCLHandler, self).format(record)

        resource = Resource(
            type='k8s_container',
            labels={
                'cluster_name': os.environ['CLUSTER_NAME'],
                'container_name': os.environ['POD_APP_NAME'],
                'location': os.environ['CLUSTER_LOCATION'],
                'namespace_name': os.environ['POD_NAMESPACE'],
                'pod_name': os.environ['POD_NAME'],
                'project_id': _settings.PROJECT_NAME
            }
        )

        labels: Dict[str, Any] = {
            'k8s-pod/app': os.environ['POD_APP_NAME'],
            'k8s-pod/app_kubernetes_io/managed-by': os.environ['POD_MANAGED_BY'],
            'k8s-pod/pod-template-hash': os.environ['POD_TEMPLATE_HASH']
        }
        trace = getattr(record, 'traceId', None)
        if trace is not None:
            trace = f'projects/{_settings.PROJECT_NAME}/traces/{trace}'
        self.transport.send(
            record,
            message,
            resource=resource,
            labels=labels,
            trace=trace,
            span_id=getattr(record, 'spanId', None)
        )

I use opensensus integration with gcloud trace and logging, so I can get traceId and spanId and pass it into gcloud logging transport, it work fine and LogEntry in logs viewer contains proper traceId and spanId

My code of using gcloud trace looks like

config_integration.trace_integrations(['logging'])
logger = logging.getLogger(__name__)

exporter = stackdriver_exporter.StackdriverExporter(
    project_id=settings.PROJECT_NAME
)


async def handle_message(message: Message) -> None:
    tracer = Tracer(exporter=exporter, sampler=AlwaysOnSampler())
    with tracer.span(name=f'Message#{message.message_id}'):
        logger.debug(f'debug')
        logger.info(f'info')
        logger.warning(f'warning')

So, I can these logs in Logs Viewer, but they aren't gropped in one trace, but if I use gcloud trace viewer and search by traceId, I will find this trace with connected logs. Q: There is any way to display trace in logs viewer as it displayed in any appengine service as appengine.googleapis.com/Frequest_log?

As I can see, you want to be able to display trace in logs viewer as it displayed in any GAE service as appengine.googleapis.com/Frequest_log, but you found no way to configure it, am I right? — Serhii Rohoza
@SerhiiRohoza Yes, you're right. At the moment, I recognized that I should create parent log with different from childlogs logName. I will try to do that and if it works I will create answer — Nikita Davydov
Okay. I'm looking forward to hear from you. If no luck, I have a workaround but it's time consuming. — Serhii Rohoza
@SerhiiRohoza No, it doesn't work, I created simple TextEntry with logName pubsub_message and after that I sent some logs with logName stdout, and they're not groupped. Is it true that I can do groupped logs only with httpRequest log? — Nikita Davydov
Have no time for it, sorry :( I created fake http_request payload and groupped my logs — Nikita Davydov

Unknown Unknown · Accepted Answer · 2020-11-10T07:44:09

As it was confirmed by @Nikita Davydov in the comment section there's a workaround: you can create a fake http_request payload to group logs.

If it doesn't work for you, you can file a feature request at Google Public Issue Tracker in order to change current behavior.

Google cloud trace + Gcloud logging in logs viewer

1 Answers