Logging of Pgpool-II on Kubernetes

-

logging

Logging is an important topic and particularly useful for troubleshooting, debugging and monitoring. Many applications have their own built-in logging mechanism. Pgpool-II logging mechanism is similar to PostgreSQL logging mechanism. Pgpool-II log management system supports two ways for logging messages (stderr and syslog) and the logging collector process collects the logs sent to stderr and redirects them into log files. However, how can we manage the logging system on Kubernetes? In this blog, I will describe how to manage Pgpool-II container logs on Kubernetes.

Logging on Kubernetes 

Kubernetes supports the functionality to view the container logs by using “kubectl logs” command. 

kubectl logs <pod name>

However, logging on Kubernetes is complicated because the logs will be deleted when containers are terminated or recreated.

Fluentd helps you to collect container logs and send the logs to desired destinations such as Amazon S3, MySQL, MongoDB, etc.

How to use Fluentd on Kubernetes?

Kubernetes logs the stdout and stderr streams of each container in a pod to a file in JSON format. The log file is located in /var/log/containers directory of the node where the pod is running, and the filename contains the pod name, the namespace and the container name.

To read the containerized application log files and expose to other destinations, we need a node-level logging agent that runs on each node.

Fluentd is an open-source log collector that allows you to collect and analyze various application logs and send them to your desired data storage destinations. In this article we use Fluentd as the logging agent. 

Because Fluentd must run on each node, we deploy it as a DaemonSet on Kubernetes. In the next sections, I will describe how to deploy and configure Fluentd on Kubernetes to manage Pgpool-II container logs.

Managing Pgpool-II logs using Fluentd

Pgpool-II container writes its logs to stderr streams.
The pgpool container log files will be stored on each node like:

$ ll /var/log/containers/ | grep pgpool
lrwxrwxrwx 1 root root  94  3月 31 10:27 pgpool-565c85fbd-9sddg_default_pgpool-8e4f9154eac929f8b2fdfa4e59faa934da41374efc28b5c047b42158079b4661.log -> /var/log/pods/default_pgpool-565c85fbd-9sddg_e23dc5a4-7677-4fd8-972c-7df5016f780b/pgpool/0.log
lrwxrwxrwx 1 root root  94  3月 31 10:27 pgpool-565c85fbd-kbpzq_default_pgpool-b6e86e2dcfc6753bf57b306daa559467b89b201e55f4ff18964bed1113dc8f00.log -> /var/log/pods/default_pgpool-565c85fbd-kbpzq_b2af5056-fd0f-4168-bbbc-206c0b27e8ae/pgpool/0.log

Step 1 : Grant permissions

First, we need to grant Fluentd permissions so that Fluentd is able to collect logs from pgpool container. Here we assume that pgpool pod is deployed in the default namespace.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluentd
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - list
  - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: default

Let’s save the manifests above in fluentd_rbac.yaml and create it.

$ kubectl create -f fluentd_rbac.yaml
serviceaccount/fluentd created
clusterrole.rbac.authorization.k8s.io/fluentd created
clusterrolebinding.rbac.authorization.k8s.io/fluentd created

Step 2 : Configure Fluentd to collect Pgpool-II logs

We can use the manifest provided by fluentd-kubernetes-daemonset to deploy fluentd as a DaemonSet. By default the configuration file fluent.conf under /fluentd/etc will be used to start Fluentd. In this article we use this default Fluentd configuration file and mount Pgpool-II's dedicated configuration file defined in ConfigMap below to pgpool container.

The ConfigMap is:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-pgpool-configmap
  labels:
    app: fluentd-pgpool-config
data:
  pgpool.conf: |-
    <source>
      @type tail
      path /var/log/containers/*_pgpool-*.log
      pos_file /var/log/pgpool.log.pos
      tag pgpool
      read_from_head true
      <parse>
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </parse>
    </source>

    <filter pgpool>
      @type record_transformer
      enable_ruby
      <record>
        log ${record["log"].strip}
      </record>
    </filter>

    <filter pgpool>
      @type parser
      key_name log
      <parse>
         @type none
      </parse>
    </filter>

    <match pgpool>
      @type s3
      s3_bucket "#{ENV['S3_BUCKET_NAME']}"
      s3_region "#{ENV['S3_BUCKET_REGION']}"
      path ${tag[0]}
      s3_object_key_format "pgpool-log/%Y-%m-%d/pgpool-log-%{index}.%{file_extension}"
      <inject>
        time_key time
        tag_key tag
        localtime false
      </inject>
      <buffer>
        @type file
        path /var/log/fluentd-buffers/s3-pgpool.buffer
        timekey "#{ENV['S3_TIMEKEY'] || '3600'}"
        timekey_use_utc true
        chunk_limit_size "#{ENV['S3_CHUNK_LIMIT_SIZE'] || '256m'}"
      </buffer>
    </match>

Let’s save the ConfigMap above in fluentd-pgpool-config.yaml and create it.

kubectl create -f fluentd-pgpool-config.yaml

Step 3 : Deploy Fluentd as DaemonSet

Here we use the latest Docker image v1.12 under https://github.com/fluent/fluentd-kubernetes-daemonset/tree/master/docker-image/. Because I will save the logs to Amazon S3, I use debian-s3.

In the step 2 we have defined the Pgpool-II's dedicated ConfigMap fluentd-pgpool-configmap. Next, Let's deploy Fluentd DaemonSet and mount the ConfigMap defined in the step 2 to /fluentd/etc/conf.d.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-daemonset
spec:
  selector:
    matchLabels:
      name: fluentd-pod
  template:
    metadata:
      labels:
        name: fluentd-pod
    spec:
      serviceAccountName: fluentd
      containers:
        - name: fluentd-container
          image: fluent/fluentd-kubernetes-daemonset:v1.12-debian-s3-1
          env:
            - name: S3_BUCKET_NAME
              value: "your s3 bucket name"
            - name: S3_BUCKET_REGION
              value: "your s3 bucket region"
            - name: FLUENTD_SYSTEMD_CONF
              value: "disable"
            - name: FLUENT_UID
              value: "0"
          resources:
            requests:
              cpu: 200m
              memory: 500Mi
            limits:
              memory: 1Gi
          volumeMounts:
            - name: config-volume
              mountPath: /fluentd/etc/conf.d
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: config-volume
          configMap:
            name: fluentd-pgpool-configmap
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

Let’s save the DaemonSet manifest above in fluentd.yaml and create it.

kubectl create -f fluentd.yaml

Now, you can find the logs stored in Amazon S3.


 

 


Comments

Post a Comment

Popular posts from this blog

Installing Pgpool-II on Debian/Ubuntu

-
First of all, many thanks to the PostgreSQL Global Development Group (PGDG) for creating Pgpool-II packages for Debian and Ubuntu. This post shows you step by step how to install Pgpool-II using official APT repository provided by PGDG. Prerequisites This blog assumes you have already installed two PostgreSQL 14 servers and setup streaming replication between the PostgreSQL servers.   For testing purpose,we set trust authentication in pg_hba.conf . In a production environment, Please follow the PostgreSQL and Pgpool-II documentation to configure proper authentication settings.  Install Pgpool-II To use the PostgreSQL apt repository, follow the steps below. Create the repository configuration file: # echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list Import the repository signing key: # wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add - Update the package lists: # ...
Read more

Authentication in Pgpool-II

-
Image
    Pgpool-II is a proxy software for PostgreSQL cluster and it supports several authentication methods. How does Pgpool-II perform user authentication? In this post, I will introduce the authentication methods which Pgpool-II supports and how Pgpool-II authentication mechanism works. Authentication methods in Pgpool-II Pgpool-II supports several authentication methods: Trust Authentication MD5 Password Authentication   scram-sha-256 Authentication (Pgpool-II 4.0 or later) Certificate Authentication   PAM Authentication LDAP Authentication (from the next major release 4.2)  Starting with Pgpool-II 4.0, Pgpool-II supports scram-sha-256 authentication. scram-sha-256 authentication method is strongly recommended because it is the most secure password-based authentication method. How does Pgpool-II authentication mechanism work? Since Pgpool-II is a PostgreSQL proxy that works between clients and PostgreSQL servers, the authentication comprises two steps: Authen...
Read more

Query Load Balancing in Pgpool-II

-
Image
Pgpool-II is a PostgreSQL cluster management tool. The major features of Pgpool-II are: Connection pooling Query load balancing Automated failover Watchdog (High availability of Pgpool-II) Replication In memory query cache In this post, I will describe the query load balancing mechanism in Pgpool-II and the relevant configuration parameters. Why use query load balancing? Query load balancing can distribute database server workloads across multiple PostgreSQL servers. Nowadays, most database systems use multiple replicated database servers to achieve high availability for PostgreSQL. Pgpool-II takes the advantage of the replication feature in order to distribute the workloads across multiple PostgreSQL servers. If there is a PostgreSQL cluster with multiple PostgreSQL servers, Pgpool-II is able to distribute READ queries across those PostgreSQL servers. The benefit of READ queries load balancing is, improve the system's throughput reduce the load on each PostgreSQL server optimize...
Read more