GCP stackdriver logging logs format changed in bucket from folder per container to stdout\stderr

1
votes

i have a question, similar as describe here: GKE kubernetes container stdout logs format changed

in old version of stackdriver i had 1 sink with filter like this:

resource.type=container,
resource.namespace_id=[NAMESPACE_NAME] 
resource.pod_id=[POD_NAME]

and logs was stored in bucket pretty well, like this:

logName=projects/[PROJECT-NAME]/logs/[CONTAINER-NAME]

...so i had folders whith logs for each container.

But now i updated my stackdriver logging+monitoring to last version and now i have 2 folders stdout\stderr which contains all logs for all containers!

logName=projects/[PROJECT-NAME]/logs/stdout
logName=projects/[PROJECT-NAME]/logs/stderr

All logs from many containers stored in this single folders! This is pretty uncomfortable =(

I'v read about this in docs: https://cloud.google.com/monitoring/kubernetes-engine/migration#changes_in_log_entry_contents

The logName field might change. Stackdriver Kubernetes Engine Monitoring log entries use stdout or stderr in their log names whereas Legacy Stackdriver used a wider variety of names, including the container name. The container name is still available as a resource label.

...but i can't find solution! Please, help me, how to make container per folder logging, like it was in old version of stackdriver?

1
kubernetesgoogle-kubernetes-enginegoogle-cloud-stackdrivergoogle-cloud-logging
It looks like expected behavior described in the documentation. If you need some functions that aren't available at the moment you can try to create Feature Request at Google Issue Tracker issuetracker.google.com/issues/… - Serhii Rohoza
Not sure, that this function not exist, maybe i'm just not find proper solution yet. - kikis
Maybe need to use "Sink Service/Custom destination" instead of "Sink Service/Cloud Storage", and make somehow folders, which can be created depends on Labels, for instance storage.googleapis.com/BUCKET-NAME/%resource.labels.container_name% - kikis
I've attach link to Google Issue Tracker with a possible workaround issuetracker.google.com/issues/149300373 and already opened Feature Request. - Serhii Rohoza

1 Answers

1
votes

Here is a workaround that has been suggested:

  1. Create a different sink for each of your containers filtered by resource.labels.container_name
  2. Export each sink to a different bucket

Note: If you configure each separate sink to the same bucket the logs will be combined.

More details at Google Issue Tracker