Log rotation on logs consuming disk space in Google Cloud Kubernetes pod

拥有回忆 提交于 2020-06-25 18:08:41

问题


We have a pod in a Google Cloud Platform Kubernetes cluster writing JsonFormatted to StdOut. This is picked up by Stackdriver out of box. However, we see the disk usage of the pod just growing and growing, and we can't understand how to set a max size on the Deployment for log rotation.

Documentation on Google Cloud and Kubernetes is unclear on this.

This is just the last hour:


回答1:


Are you sure that disk usage of the pod is high because of the logs? If the application writes logs to stdout, it doesn't use any disk space inside the pod. All logs are usually stored in a log file on the node’s filesystem and can be managed by the node logrotate process.

Perhaps application uses pod's disk space for something else, like temp files or debug information?

Here is the part of documentation related to log rotation:

Logging at the node level:

Everything a containerized application writes to stdout and stderr is handled and redirected somewhere by a container engine. For example, the Docker container engine redirects those two streams to a logging driver, which is configured in Kubernetes to write to a file in json format.

An important consideration in node-level logging is implementing log rotation, so that logs don’t consume all available storage on the node.

Kubernetes currently is not responsible for rotating logs, but rather a deployment tool should set up a solution to address that. For example, in Kubernetes clusters, deployed by the kube-up.sh script, there is a logrotate tool configured to run each hour.

You can also set up a container runtime to rotate application’s logs automatically, e.g. by using Docker’s log-opt.

In the kube-up.sh script, the latter approach is used for COS image on GCP, and the former approach is used in any other environment. In both cases, by default rotation is configured to take place when log file exceeds 10MB.

As an example, you can find detailed information about how kube-up.sh sets up logging for COS image on GCP in the corresponding script.

Here is a part of the script related to logrotate:

# Installs logrotate configuration files
function setup-logrotate() {
  mkdir -p /etc/logrotate.d/
  # Configure log rotation for all logs in /var/log, which is where k8s services
  # are configured to write their log files. Whenever logrotate is ran, this
  # config will:
  # * rotate the log file if its size is > 100Mb OR if one day has elapsed
  # * save rotated logs into a gzipped timestamped backup
  # * log file timestamp (controlled by 'dateformat') includes seconds too. This
  #   ensures that logrotate can generate unique logfiles during each rotation
  #   (otherwise it skips rotation if 'maxsize' is reached multiple times in a
  #   day).
  # * keep only 5 old (rotated) logs, and will discard older logs.
  cat > /etc/logrotate.d/allvarlogs <<EOF
/var/log/*.log {
    rotate ${LOGROTATE_FILES_MAX_COUNT:-5}
    copytruncate
    missingok
    notifempty
    compress
    maxsize ${LOGROTATE_MAX_SIZE:-100M}
    daily
    dateext
    dateformat -%Y%m%d-%s
    create 0644 root root
}
EOF

}


来源:https://stackoverflow.com/questions/50718608/log-rotation-on-logs-consuming-disk-space-in-google-cloud-kubernetes-pod

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!