cockroachdb/kubernetes logging lots of errors on stackdriver

1
votes

I've just noticed from a very nasty gcp bill that cockroachDB has logged 1.5tb of errors on stackdriver, costing me several hundred dollars in just a few days. Sadly I had left it on 100% logging. The errors look like this and are piling up multiple times per second.

E  I180712 11:18:41.963205 106 server/status/runtime.go:223  [n2]
runtime stats: 1.5 GiB RSS, 283 goroutines, 254 MiB/54 MiB/441 MiB GO alloc/idle/total, 
918 MiB/1.1 GiB CGO alloc/total, 
2175.51cgo/sec, 
0.16/0.02 %(u/s)time, 0.00 %gc (1x)

Does anybody know what they mean, and how to stop them?

2
kubernetesgoogle-cloud-stackdrivercockroachdb

2 Answers

2
votes

These are all CockroachDB logs, not just errors. This is indicated by the I prefix (meaning Info) in the CockroachDB log line. The listed log lines show basic memory information for the cockroach process. This is logged every 10 seconds.

If you wish to persist logs I would recommend filtering by severity.

This can be done by CockroachDB itself when redirecting logs to stderr by using: --logtostderr=Level where Level is one of Info, Warning, Error, or Fatal.

If you are saving the raw logs, you could do a quick pass to discard anything not starting with a desired prefix. This will however not be as accurate as the --logtostderr method as you would need to handle multi-line log entries.

1
votes

Marc's answer is the "right" one, but as a workaround you could exclude specific logs from Stackdriver: https://cloud.google.com/logging/docs/exclusions

Log exclusions give fine grain control, and wouldn't require re-configuring or restarting the job.