How to implement centralized logging on Kubernetes using EFK/ELK
In a Kubernetes cluster, logs are the heartbeat of your applications: they tell you what’s happening, where errors occur, and how services are performing. However, the distributed nature of microservices and the volatility of pods make it difficult to collect, correlate, and analyze logs without a centralized solution. The EFK (Elasticsearch, Fluentd/Fluent Bit, Kibana) and ELK (Elasticsearch, Logstash, Kibana) stacks are the two most common approaches to building a reliable, scalable, and observable logging pipeline.
In this guide you’ll learn how to design, install, and optimize a centralized logging system on Kubernetes with EFK/ELK, adopting best practices for security, cost, and performance.
Architecture and log flow
The typical log flow in Kubernetes follows these steps:
– Containers write to stdout/stderr; the container runtime saves logs as files in /var/log/containers with CRI format.
– A DaemonSet agent (Fluent Bit, Fluentd, or Filebeat) runs on each node, tails the files, enriches with Kubernetes metadata (namespace, pod, container, labels), and sends to a backend (Elasticsearch directly or via Logstash).
– Elasticsearch indexes and makes logs searchable; Kibana provides visualizations, dashboards, and advanced searches.
EFK or ELK: which to choose?
– EFK with Fluent Bit/Fluentd: Fluent Bit is lightweight and performant for forwarding; Fluentd offers a rich ecosystem of plugins and transformations. Great when you want to minimize overhead on nodes and keep elasticity.
– ELK with Logstash: Logstash excels at complex transformations and multiple pipelines. However, it is more resource-heavy. Often Filebeat/Fluent Bit is used for collection and Logstash as an intermediate hop only where needed.
Practical tip: start with EFK (Fluent Bit → Elasticsearch → Kibana); introduce Logstash only if you have complex pipelines or special integrations.
Planning: prerequisites and sizing
Before deploying the stack:
– Storage: prefer SSDs for Elasticsearch data nodes; define a StorageClass with persistent volumes (PVC). Plan retention and index strategy to control costs.
– Resources: assign requests/limits for agents and for Elasticsearch. Size the JVM heap at 50
– Security: enable TLS and authentication for Elasticsearch/Kibana; apply RBAC and network policies (NetworkPolicy). Avoid sending PII or secrets in logs.
– Versions: use compatible releases between Elasticsearch/Kibana and your agents.
Quick install with Helm (EFK)
Below is a reference installation in a dedicated logging namespace. Note: the commands shown are suitable for a test environment; in production enable TLS, secure credentials, replication, and appropriate resources.
kubectl create namespace logging # Add the Helm repositories helm repo add elastic https://helm.elastic.co helm repo add fluent https://fluent.github.io/helm-charts helm repo update # Elasticsearch (single node for testing) helm install es elastic/elasticsearch --namespace logging --set replicas=1 --set minimumMasterNodes=1 --set resources.requests.cpu=500m --set resources.requests.memory=1Gi --set resources.limits.memory=2Gi --set volumeClaimTemplate.resources.requests.storage=20Gi # Kibana helm install kibana elastic/kibana --namespace logging --set service.type=ClusterIP # Fluent Bit as DaemonSet, output to Elasticsearch helm upgrade --install fluent-bit fluent/fluent-bit --namespace logging --set backend.type=es --set backend.es.host=es-master.logging.svc --set backend.es.port=9200 --set backend.es.index=logs-kubernetes --set input.tail.parser=cri --set tolerations[0].operator=Exists --set resources.requests.cpu=100m --set resources.requests.memory=128Mi # Local access to Kibana kubectl -n logging port-forward svc/kibana-kibana 5601:5601
If you prefer ELK, you can replace Fluent Bit with Filebeat and introduce Logstash as a processor:
# Filebeat as DaemonSet helm install filebeat elastic/filebeat --namespace logging --set daemonset.enabled=true --set elasticsearch.hosts[0]=http://es-master.logging.svc:9200 # (Optional) Logstash helm install logstash elastic/logstash --namespace logging
Enrichment and parsing configuration
Agents must enrich logs with Kubernetes metadata and handle multiline formats (e.g., stack traces). With Fluent Bit you can easily customize inputs, filters, and outputs. Minimal ConfigMap example with enrichment:
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-custom
namespace: logging
data:
fluent-bit.conf: |
[SERVICE]
Flush 1
Parsers_File parsers.conf
[INPUT]
Name tail
Path /var/log/containers/*.log
Tag kube.*
Parser cri
Mem_Buf_Limit 10MB
Skip_Long_Lines On
[FILTER]
Name kubernetes
Match kube.*
Kube_Tag_Prefix kube.var.log.containers.
Merge_Log On
Keep_Log Off
Annotations Off
[OUTPUT]
Name es
Match kube.*
Host es-master.logging.svc
Port 9200
Logstash_Format On
Index logs-kubernetes
parsers.conf: |
[PARSER]
Name cri
Format regex
Regex ^(?<time>.+?) (?<stream>stdout|stderr) (?<flags>[FP]) (?<log>.*)$
Apply the ConfigMap and mount the volume into the Fluent Bit DaemonSet via the chart values or a patch. For Java stack traces or multiline logs, use your agents’ multiline plugin or preprocess in code to emit structured JSON: this choice simplifies parsing and queries.
Index management, retention, and costs
Define clear retention rules: how many days to keep logs and with what index granularity. Best practices:
– Use indices per environment and/or namespace (e.g., logs-k8s-prod-yyyy.mm.dd) to limit cardinality and simplify permissions.
– Apply Index Lifecycle Management (ILM) for automatic rollover, shrink, and delete. Keep few shards per index (1–3) depending on volume; avoid tiny shards.
– Compress and limit high-cardinality fields (for example, avoid indexing redundant full-text messages if not necessary; use keyword fields on labels).
Effective visualizations and queries in Kibana
After creating the Index pattern (for example, logs-kubernetes*), use Discover to filter by:
– kubernetes.namespace_name, kubernetes.pod_name, kubernetes.container_name
– level, service, trace.id if you send structured logs
Build dashboards that combine errors per namespace, log rates, and correlations with releases (labels such as app, version). Set alerts on error patterns or on the absence of logs (an often underrated signal).
Scalability and performance
– Collectors: prefer Fluent Bit for high loads; enable filesystem buffering to handle backpressure. Set limits/requests to prevent eviction.
– Elasticsearch: separate roles (data, master, ingest) for larger clusters; allocate heap and SSD storage, enable replica >=1 in production. Avoid over-sharding. Monitor heap, file descriptors, merges, and queues.
– Reduce noise: increase application log level only when needed; adopt sampling for trace logs and filter repetitive health checks.
Security and compliance
– Enable TLS and authentication on Elasticsearch and Kibana; restrict access with NetworkPolicy and ingress with mTLS/Basic Auth.
– Apply RBAC in Kibana for teams/namespaces using separate indices or document-level filters.
– Avoid logging secrets or PII; mask sensitive data upstream with filters or in applications.
Quick troubleshooting
– Check pod status:
kubectl -n logging get pods -o wide
– Check the DaemonSet logs:
kubectl -n logging logs ds/fluent-bit -c fluent-bit --tail=200
– Check indices and cluster health:
kubectl -n logging port-forward svc/es-master 9200:9200 curl http://localhost:9200/_cat/indices?v curl http://localhost:9200/_cluster/health?pretty
Alternatives and integrations
– OpenSearch + OpenSearch Dashboards as an open-source alternative compatible with Elasticsearch APIs.
– OpenTelemetry Collector to unify logs, metrics, and traces, with export to Elastic or other backends.
– Managed solutions: Elastic Cloud, cloud providers, or SaaS offerings to reduce operational burden.
Conclusions
Well-designed centralized logging in Kubernetes accelerates observability and debugging, as well as cost governance and security. Start with a lean EFK pipeline, introduce Logstash only when advanced transformations are needed, apply ILM for retention, and secure the cluster with TLS, RBAC, and network policies.
Ready to level up your observability? Put the steps from this guide into practice on your staging cluster, measure the results, and optimize. If you want hands-on support or a customized production checklist, contact us: we’ll build a solid, secure, and scalable log pipeline together.

Comments