Logging of Pgpool-II on Kubernetes

logging

Logging is an important topic and particularly useful for troubleshooting, debugging and monitoring. Many applications have their own built-in logging mechanism. Pgpool-II logging mechanism is similar to PostgreSQL logging mechanism. Pgpool-II log management system supports two ways for logging messages (stderr and syslog) and the logging collector process collects the logs sent to stderr and redirects them into log files. However, how can we manage the logging system on Kubernetes? In this blog, I will describe how to manage Pgpool-II container logs on Kubernetes.

Logging on Kubernetes 

Kubernetes supports the functionality to view the container logs by using “kubectl logs” command. 

kubectl logs <pod name>

However, logging on Kubernetes is complicated because the logs will be deleted when containers are terminated or recreated.

Fluentd helps you to collect container logs and send the logs to desired destinations such as Amazon S3, MySQL, MongoDB, etc.

How to use Fluentd on Kubernetes?

Kubernetes logs the stdout and stderr streams of each container in a pod to a file in JSON format. The log file is located in /var/log/containers directory of the node where the pod is running, and the filename contains the pod name, the namespace and the container name.

To read the containerized application log files and expose to other destinations, we need a node-level logging agent that runs on each node.

Fluentd is an open-source log collector that allows you to collect and analyze various application logs and send them to your desired data storage destinations. In this article we use Fluentd as the logging agent. 

Because Fluentd must run on each node, we deploy it as a DaemonSet on Kubernetes. In the next sections, I will describe how to deploy and configure Fluentd on Kubernetes to manage Pgpool-II container logs.

Managing Pgpool-II logs using Fluentd

Pgpool-II container writes its logs to stderr streams.
The pgpool container log files will be stored on each node like:

$ ll /var/log/containers/ | grep pgpool
lrwxrwxrwx 1 root root  94  3月 31 10:27 pgpool-565c85fbd-9sddg_default_pgpool-8e4f9154eac929f8b2fdfa4e59faa934da41374efc28b5c047b42158079b4661.log -> /var/log/pods/default_pgpool-565c85fbd-9sddg_e23dc5a4-7677-4fd8-972c-7df5016f780b/pgpool/0.log
lrwxrwxrwx 1 root root  94  3月 31 10:27 pgpool-565c85fbd-kbpzq_default_pgpool-b6e86e2dcfc6753bf57b306daa559467b89b201e55f4ff18964bed1113dc8f00.log -> /var/log/pods/default_pgpool-565c85fbd-kbpzq_b2af5056-fd0f-4168-bbbc-206c0b27e8ae/pgpool/0.log

Step 1 : Grant permissions

First, we need to grant Fluentd permissions so that Fluentd is able to collect logs from pgpool container. Here we assume that pgpool pod is deployed in the default namespace.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluentd
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - list
  - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: default

Let’s save the manifests above in fluentd_rbac.yaml and create it.

$ kubectl create -f fluentd_rbac.yaml
serviceaccount/fluentd created
clusterrole.rbac.authorization.k8s.io/fluentd created
clusterrolebinding.rbac.authorization.k8s.io/fluentd created

Step 2 : Configure Fluentd to collect Pgpool-II logs

We can use the manifest provided by fluentd-kubernetes-daemonset to deploy fluentd as a DaemonSet. By default the configuration file fluent.conf under /fluentd/etc will be used to start Fluentd. In this article we use this default Fluentd configuration file and mount Pgpool-II's dedicated configuration file defined in ConfigMap below to pgpool container.

The ConfigMap is:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-pgpool-configmap
  labels:
    app: fluentd-pgpool-config
data:
  pgpool.conf: |-
    <source>
      @type tail
      path /var/log/containers/*_pgpool-*.log
      pos_file /var/log/pgpool.log.pos
      tag pgpool
      read_from_head true
      <parse>
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </parse>
    </source>

    <filter pgpool>
      @type record_transformer
      enable_ruby
      <record>
        log ${record["log"].strip}
      </record>
    </filter>

    <filter pgpool>
      @type parser
      key_name log
      <parse>
         @type none
      </parse>
    </filter>

    <match pgpool>
      @type s3
      s3_bucket "#{ENV['S3_BUCKET_NAME']}"
      s3_region "#{ENV['S3_BUCKET_REGION']}"
      path ${tag[0]}
      s3_object_key_format "pgpool-log/%Y-%m-%d/pgpool-log-%{index}.%{file_extension}"
      <inject>
        time_key time
        tag_key tag
        localtime false
      </inject>
      <buffer>
        @type file
        path /var/log/fluentd-buffers/s3-pgpool.buffer
        timekey "#{ENV['S3_TIMEKEY'] || '3600'}"
        timekey_use_utc true
        chunk_limit_size "#{ENV['S3_CHUNK_LIMIT_SIZE'] || '256m'}"
      </buffer>
    </match>

Let’s save the ConfigMap above in fluentd-pgpool-config.yaml and create it.

kubectl create -f fluentd-pgpool-config.yaml

Step 3 : Deploy Fluentd as DaemonSet

Here we use the latest Docker image v1.12 under https://github.com/fluent/fluentd-kubernetes-daemonset/tree/master/docker-image/. Because I will save the logs to Amazon S3, I use debian-s3.

In the step 2 we have defined the Pgpool-II's dedicated ConfigMap fluentd-pgpool-configmap. Next, Let's deploy Fluentd DaemonSet and mount the ConfigMap defined in the step 2 to /fluentd/etc/conf.d.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-daemonset
spec:
  selector:
    matchLabels:
      name: fluentd-pod
  template:
    metadata:
      labels:
        name: fluentd-pod
    spec:
      serviceAccountName: fluentd
      containers:
        - name: fluentd-container
          image: fluent/fluentd-kubernetes-daemonset:v1.12-debian-s3-1
          env:
            - name: S3_BUCKET_NAME
              value: "your s3 bucket name"
            - name: S3_BUCKET_REGION
              value: "your s3 bucket region"
            - name: FLUENTD_SYSTEMD_CONF
              value: "disable"
            - name: FLUENT_UID
              value: "0"
          resources:
            requests:
              cpu: 200m
              memory: 500Mi
            limits:
              memory: 1Gi
          volumeMounts:
            - name: config-volume
              mountPath: /fluentd/etc/conf.d
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: config-volume
          configMap:
            name: fluentd-pgpool-configmap
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

Let’s save the DaemonSet manifest above in fluentd.yaml and create it.

kubectl create -f fluentd.yaml

Now, you can find the logs stored in Amazon S3.


 

 


Comments

Popular posts from this blog

Connection Pooling in Pgpool-II

Monitoring PostgreSQL Cluster via Pgpool-II with Prometheus

Authentication in Pgpool-II