Loki

App Mesh can manage all app applications running on Linux and provide build-in Prometheus exporter the metrics can be used to monitor hosted application detail running behavior. Loki can be used to collect all applications and App Mesh logs.

Grafana can be used as Prometheus and Loki web portal, with those components, we can build a runtime metrics and log aggregate system with a flexible GUI.

1.1. Deploy all component by docker compose

1.1.1. install docker-compose

sudo curl -L "https://github.com/docker/compose/releases/download/v2.14.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

1.1.2. Clone App Mesh source files

git clone --depth=1 https://github.com/laoshanxi/app-mesh.git
cd app-mesh

1.1.3. Deploy all in one YAML

cd app-mesh/script
$ docker-compose -f docker-compose-all-in-one.yaml up -d
Creating script_prometheus_1    ... done
Creating script_loki_1          ... done
Creating script_consul_1        ... done
Creating script_grafana_1       ... done
Creating script_node_exporter_1 ... done
Creating script_alertmanager_1  ... done
Creating script_promtail_1      ... done
Creating script_appmesh_1       ... done
Creating script_appmesh-ui_1    ... done

Then you can access App Mesh UI (https://192.168.3.24/) with initial user (admin/admin123).

Monitor system: Grafana, Prometheus, Node Exporter, promtail, Alert Manager

1.2. Login Grafana

Open target host http://192.168.3.24:3000/

Input initial password admin/admin and change a new password for Grafana

Add Prometheus Data Source with default address(http://localhost:9090/)

In Grafana Explorer page, you can query App Mesh metrics

Add Loki Data Source with default address(http://localhost:3100/)

In Grafana Exploere page, you can query all app stdout of App Mesh managed applications and App Mesh logs

1.3. Prometheus UI (alertmanager)

Open Prometheus WEB portal at http://192.168.3.24:9090/

Query metrics: appmesh_prom_process_memory_gauge

Stop node exporter to trigger alertmanager, you will get bellow email:

docker stop script_node_exporter_1

1.3.1. Clean docker compose started containers

cd app-mesh/script
docker-compose -f docker-compose-all-in-one.yaml stop
docker-compose -f docker-compose-all-in-one.yaml rm -f