Logo

9. Loki

like Prometheus, but for logs. Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream. Loki differs from Prometheus by focusing on logs instead of metrics, and delivering logs via push, instead of pull.

architecture

9.1. Install docker-compose

sudo curl -L "https://github.com/docker/compose/releases/download/v2.14.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

9.2. Start Loki, Grafana, Promtail，AppMesh

cd app-mesh/script
docker-compose -f docker-compose-all-in-one.yaml up -d

9.3. Configure Grafana

  1. Open Grafana on 3000 port (http://grafana_node:3000/)

  2. Add DataSource: Loki

  3. Input Loki address: http://script_loki_1:3100 (this address is Grafana access Loki docker container name)

  4. Select Explore -> Log labels -> job

9.4. Stop and clean

cd app-mesh/script
docker-compose -f docker-compose-all-in-one.yaml stop
docker-compose -f docker-compose-all-in-one.yaml rm -f

9.5. Design

Loki is a log aggregation system, component loki run as Server and promtail run as log connect and push agent. promtail is always run as daemon-set on each log collection node. For appmesh node, bellow logs need to be collected together:

  1. Default system log in /var/log

  2. appmesh service log in /opt/appmesh/log/appsvc.log

  3. Managed application output log in /opt/appmesh/work/*.log So override promtail configuration (/etc/promtail/config.yml) to combine the 3 log target in one promtail configuration, each node only start one promtail docker container.

9.6. Reference