How to implement centralized logging on Kubernetes using EFK/ELK

KUBERNATES COURSE

In a Kubernetes cluster, logs are the heartbeat of your applications: they tell you what’s happening, where errors occur, and how services are performing. However, the distributed nature of microservices and the volatility of pods make it difficult to collect, correlate, and analyze logs without a centralized solution. The EFK (Elasticsearch, Fluentd/Fluent Bit, Kibana) and ELK (Elasticsearch, Logstash, Kibana) stacks are the two most common approaches to building a reliable, scalable, and observable logging pipeline.

In this guide you’ll learn how to design, install, and optimize a centralized logging system on Kubernetes with EFK/ELK, adopting best practices for security, cost, and performance.

Architecture and log flow

The typical log flow in Kubernetes follows these steps:

– Containers write to stdout/stderr; the container runtime saves logs as files in /var/log/containers with CRI format.

– A DaemonSet agent (Fluent Bit, Fluentd, or Filebeat) runs on each node, tails the files, enriches with Kubernetes metadata (namespace, pod, container, labels), and sends to a backend (Elasticsearch directly or via Logstash).

– Elasticsearch indexes and makes logs searchable; Kibana provides visualizations, dashboards, and advanced searches.

EFK or ELK: which to choose?

– EFK with Fluent Bit/Fluentd: Fluent Bit is lightweight and performant for forwarding; Fluentd offers a rich ecosystem of plugins and transformations. Great when you want to minimize overhead on nodes and keep elasticity.

– ELK with Logstash: Logstash excels at complex transformations and multiple pipelines. However, it is more resource-heavy. Often Filebeat/Fluent Bit is used for collection and Logstash as an intermediate hop only where needed.

Practical tip: start with EFK (Fluent Bit → Elasticsearch → Kibana); introduce Logstash only if you have complex pipelines or special integrations.

Planning: prerequisites and sizing

Before deploying the stack:

– Storage: prefer SSDs for Elasticsearch data nodes; define a StorageClass with persistent volumes (PVC). Plan retention and index strategy to control costs.

– Resources: assign requests/limits for agents and for Elasticsearch. Size the JVM heap at 50

– Security: enable TLS and authentication for Elasticsearch/Kibana; apply RBAC and network policies (NetworkPolicy). Avoid sending PII or secrets in logs.

– Versions: use compatible releases between Elasticsearch/Kibana and your agents.

Quick install with Helm (EFK)

Below is a reference installation in a dedicated logging namespace. Note: the commands shown are suitable for a test environment; in production enable TLS, secure credentials, replication, and appropriate resources.

kubectl create namespace logging

# Add the Helm repositories
helm repo add elastic https://helm.elastic.co
helm repo add fluent https://fluent.github.io/helm-charts
helm repo update

# Elasticsearch (single node for testing)
helm install es elastic/elasticsearch 
  --namespace logging 
  --set replicas=1 
  --set minimumMasterNodes=1 
  --set resources.requests.cpu=500m 
  --set resources.requests.memory=1Gi 
  --set resources.limits.memory=2Gi 
  --set volumeClaimTemplate.resources.requests.storage=20Gi

# Kibana
helm install kibana elastic/kibana 
  --namespace logging 
  --set service.type=ClusterIP

# Fluent Bit as DaemonSet, output to Elasticsearch
helm upgrade --install fluent-bit fluent/fluent-bit 
  --namespace logging 
  --set backend.type=es 
  --set backend.es.host=es-master.logging.svc 
  --set backend.es.port=9200 
  --set backend.es.index=logs-kubernetes 
  --set input.tail.parser=cri 
  --set tolerations[0].operator=Exists 
  --set resources.requests.cpu=100m 
  --set resources.requests.memory=128Mi

# Local access to Kibana
kubectl -n logging port-forward svc/kibana-kibana 5601:5601

If you prefer ELK, you can replace Fluent Bit with Filebeat and introduce Logstash as a processor:

# Filebeat as DaemonSet
helm install filebeat elastic/filebeat 
  --namespace logging 
  --set daemonset.enabled=true 
  --set elasticsearch.hosts[0]=http://es-master.logging.svc:9200

# (Optional) Logstash
helm install logstash elastic/logstash 
  --namespace logging

Enrichment and parsing configuration

Agents must enrich logs with Kubernetes metadata and handle multiline formats (e.g., stack traces). With Fluent Bit you can easily customize inputs, filters, and outputs. Minimal ConfigMap example with enrichment:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-custom
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        1
        Parsers_File parsers.conf
    [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        Tag               kube.*
        Parser            cri
        Mem_Buf_Limit     10MB
        Skip_Long_Lines   On
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Keep_Log            Off
        Annotations         Off
    [OUTPUT]
        Name            es
        Match           kube.*
        Host            es-master.logging.svc
        Port            9200
        Logstash_Format On
        Index           logs-kubernetes
  parsers.conf: |
    [PARSER]
        Name        cri
        Format      regex
        Regex       ^(?<time>.+?) (?<stream>stdout|stderr) (?<flags>[FP]) (?<log>.*)$

Apply the ConfigMap and mount the volume into the Fluent Bit DaemonSet via the chart values or a patch. For Java stack traces or multiline logs, use your agents’ multiline plugin or preprocess in code to emit structured JSON: this choice simplifies parsing and queries.

Index management, retention, and costs

Define clear retention rules: how many days to keep logs and with what index granularity. Best practices:

– Use indices per environment and/or namespace (e.g., logs-k8s-prod-yyyy.mm.dd) to limit cardinality and simplify permissions.

– Apply Index Lifecycle Management (ILM) for automatic rollover, shrink, and delete. Keep few shards per index (1–3) depending on volume; avoid tiny shards.

– Compress and limit high-cardinality fields (for example, avoid indexing redundant full-text messages if not necessary; use keyword fields on labels).

Effective visualizations and queries in Kibana

After creating the Index pattern (for example, logs-kubernetes*), use Discover to filter by:

– kubernetes.namespace_name, kubernetes.pod_name, kubernetes.container_name

– level, service, trace.id if you send structured logs

Build dashboards that combine errors per namespace, log rates, and correlations with releases (labels such as app, version). Set alerts on error patterns or on the absence of logs (an often underrated signal).

Scalability and performance

– Collectors: prefer Fluent Bit for high loads; enable filesystem buffering to handle backpressure. Set limits/requests to prevent eviction.

– Elasticsearch: separate roles (data, master, ingest) for larger clusters; allocate heap and SSD storage, enable replica >=1 in production. Avoid over-sharding. Monitor heap, file descriptors, merges, and queues.

– Reduce noise: increase application log level only when needed; adopt sampling for trace logs and filter repetitive health checks.

Security and compliance

– Enable TLS and authentication on Elasticsearch and Kibana; restrict access with NetworkPolicy and ingress with mTLS/Basic Auth.

– Apply RBAC in Kibana for teams/namespaces using separate indices or document-level filters.

– Avoid logging secrets or PII; mask sensitive data upstream with filters or in applications.

Quick troubleshooting

– Check pod status:

kubectl -n logging get pods -o wide

– Check the DaemonSet logs:

kubectl -n logging logs ds/fluent-bit -c fluent-bit --tail=200

– Check indices and cluster health:

kubectl -n logging port-forward svc/es-master 9200:9200
curl http://localhost:9200/_cat/indices?v
curl http://localhost:9200/_cluster/health?pretty

Alternatives and integrations

– OpenSearch + OpenSearch Dashboards as an open-source alternative compatible with Elasticsearch APIs.

– OpenTelemetry Collector to unify logs, metrics, and traces, with export to Elastic or other backends.

– Managed solutions: Elastic Cloud, cloud providers, or SaaS offerings to reduce operational burden.

Conclusions

Well-designed centralized logging in Kubernetes accelerates observability and debugging, as well as cost governance and security. Start with a lean EFK pipeline, introduce Logstash only when advanced transformations are needed, apply ILM for retention, and secure the cluster with TLS, RBAC, and network policies.

Ready to level up your observability? Put the steps from this guide into practice on your staging cluster, measure the results, and optimize. If you want hands-on support or a customized production checklist, contact us: we’ll build a solid, secure, and scalable log pipeline together.

Se vuoi farmi qualche richiesta o contattarmi per un aiuto riempi il seguente form

    Comments