Centralized Logging of Application Pod Logs

Overview

SnappyFlow can collect & parse application logs from pods in 2 ways:

• Collect logs locally by running sfKubeAgent as a sidecar container inside application pod
• Collect logs centrally through sfPod, which is explained in this page

Procedure for Centralized Logging

• User runs a busybox sidecar container in the application pod with log files mounted to the container. Busybox tails & streams the application logs to stdout

• Add SnappyFlow labels:

  • snappyflow/projectName and snappyflow/appName: These are mandatory labels for SnappyFlow monitoring. sfPod collects logs only from pods that have these labels and collected logs are organized under projectName/appName hierarchy in SnappyFlow
  • snappyflow/component: This label is used to signal to sfPod on which parser to apply to parse the logs. List of standard parsers packaged with sfPod. If no label is present, sfPod will apply SnappyFlow’s generic parser which collects the whole log line as a message.

• sfPod runs as daemon-set in all the Kubernetes data nodes. It picks up logs from stdout of tagged pods, parses the logs based on component tag and ships parsed logs to SnappyFlow under projectName/appName hierarchy.

How to tag application Pods with project and application name labels

Running Pods

Use the following kubectl commands to tag your application pods with the appropriate tags:

kubectl label pods <pod_name> snappyflow/projectname=<project_name> --namespace <appnamespace>
kubectl label pods <pod_name> snappyflow/appname=<app_name> --namespace <appnamespace>

Automatically apply labels to new Pods

To automatically apply right labels for new pods which get created due to various reasons such as upgrade, restarts etc. apply labels to pod templates. If you are using helm chart, best practice is to define labels in values.yaml and use these labels parameter in pod template section of Deployment, StatefulSet, Daemonset or other Kubernetes controller.

Below is one example values.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sfapm-ui
  labels:
    app: sfapm
    role: ui
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sfapm
      role: ui
  template:
    metadata:
      labels:
        app: sfapm
        role: ui
        snappyflow/appname: demo-application
        snappyflow/projectname: demo-project

Example

Centralized logging for nginx-access logs

• Configure Nginx to drop logs in the required format in /var/log/nginx folder using config map

• Add busy box container to tail logs from access logs and stream to stdout

• Signal to sfPod to use “nginx” parser using label “component”

Nginx Pod YAML

kind: Pod 
apiVersion: v1 
metadata: 
 name: my-first-pod 
 labels: 
  snappyflow/appname: <app_name> 
  snappyflow/projectname: <project_name> 
  snappyflow/component: nginx 
spec: 
 containers: 
 - name: nginx-container 
   image: nginx:latest 
   imagePullPolicy: IfNotPresent 
   ports: 
   - name: http 
     containerPort: 80 
     protocol: TCP 
   volumeMounts: 
   - name: varlog 
     mountPath: /var/log/nginx 
   - name: nginx-config 
     mountPath: /etc/nginx/nginx.conf 
     subPath: nginx.conf 
 - name: busybox-container 
   image: busybox 
   command: ["/bin/sh", "-c"] 
   args: ["tail -n+1 -f /var/log/nginx/access1.log"] 
   volumeMounts: 
   - name: varlog 
     mountPath: /var/log/nginx 
 volumes: 
 - name: nginx-config 
   configMap: 
     name: nginx-config 
 - name: varlog 
   emptyDir: {} 

Config map for Nginx configuration

apiVersion: v1 
kind: ConfigMap 
metadata: 
  name: nginx-configmap 
  labels: 
    snappyflow/appname: <app_name> 
    snappyflow/projectname: <project_name> 
data: 
  nginx.conf: | 
    worker_processes  5; 
    events { 
        worker_connections  4096; 
    } 
    http { 
        default_type application/octet-stream; 
        log_format upstream_time  '$remote_addr:$remote_port $remote_user [$time_local] ' 
                       '"$request" $status $body_bytes_sent ' 
                       '"$http_referer" "$http_user_agent" "$http_referer" ' 
                       'rt=$request_time uct=$upstream_connect_time uht=$upstream_header_time urt=$upstream_response_time'; 
        server { 
            listen          80; 
            error_log /var/log/nginx/error1.log; 
            access_log /var/log/nginx/access1.log upstream_time; 
            location /nginx_status { 
                stub_status; 
            } 
        } 
    }