There are many logging solutions available for dealing with log data. Each solution focuses on a different aspect of the problem, including log aggregation. These tools and software are both open-source and proprietary and can be integrated into cloud providers platforms. They also offer a range of capabilities that will meet your needs. Grafana Loki, a new industry solution.
Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. A Loki-based logging stack consists of 3 components: promtail is the agent, responsible for gathering logs and sending them to Loki, loki is the main server and Grafana for querying and displaying the logs. In this article, I will talk about the 1st component, that is Promtail.
Promtail is an agent that ships local logs to a Grafana Loki instance, or Grafana Cloud. It is typically deployed to any machine that requires monitoring. Below are the primary functions of Promtail:
- Discovers targets
- Log streams can be attached using labels
- Logs are pushed to the Loki instance
Promtail currently can tail logs from two sources. These are the local log files and the systemd journal (on AMD64 machines). Promtail must first find information about its environment before it can send any data from log files directly to Loki. This includes locating applications that emit log lines to files that require monitoring. Prometheus’ service discovery mechanism is borrowed by Promtail, but it only currently supports static and Kubernetes service discovery. Promtail is deployed to each local machine as a daemon and does not learn label from other machines. While kubernetes service Discovery fetches the Kubernetes API Server required labels, static covers all other uses.
Configuration for Promtail Web Server
Prometheus’s promtail configuration is done using a scrape_configs section. relabel_configs allows you to control what you ingest and what you drop and the final metadata to attach to the log line. You can configure the web server that Promtail exposes in the Promtail.yaml configuration file:
server: http_listen_address: 127.0.0.1 http_listen_port: 9080
Loki Push API
Promtail can be configured to receive logs via another Promtail client or any Loki client. This is done by exposing the Loki Push API using the loki_push_api Scrape configuration. This might prove to be useful in a few situations:
- Complex network infrastructures that allow many machines to egress are not ideal.
- You are using Docker Logging Driver to create complex pipelines or extract metrics from logs.
- serverless setups where many ephemeral log sources want to send to Loki, sending to a Promtail instance with use_incoming_timestamp == false can avoid out-of-order errors and avoid having to use high cardinality labels.
Once Promtail has set of targets (i.e. things to read from like files), and all labels have been correctly set, it will begin tailing (continuously reading the logs from targets). After enough data has been read into memory, or after a timeout, it flushes the logs to Loki as one batch.
Promtail will keep track of the offset it last read in a position file as it reads data from sources (files, systemd journal, if configurable). By default, the positions file is stored at /var/log/positions.yaml. Promtail can continue reading from the same location it left in case the Promtail instance is restarted.
- Get Promtail binary zip at the release page. Download Promtail binary zip from the release page
curl -s https://api.github.com/repos/grafana/loki/releases/latest | grep browser_download_url | cut -d '"' -f 4 | grep promtail-linux-amd64.zip | wget -i –
After the file has been downloaded, extract it to /usr/local/bin
unzip promtail-linux-amd64.zip sudo mv promtail-linux-amd64 /usr/local/bin/promtail
$ promtail --version promtail, version 2.6.0 (branch: HEAD, revision: 525040a32) build user: root@5d9e7a4c92e6 build date: 2022-01-12T16:48:53Z go version: go1.16.2 platform: linux/amd64
- In the /usr/local/bin directory, create a YAML configuration for Promtail:
sudo vim /etc/promtail-local-config.yaml
- Add the following content to the file:
server: http_listen_port: 9080 grpc_listen_port: 0 positions: filename: /data/loki/positions.yaml clients: - url: http://localhost:3100/loki/api/v1/push scrape_configs: - job_name: system static_configs: - targets: - localhost labels: job: varlogs __path__: /var/log/*log
- Make a service for Promtail. We will now configure Promtail to be a service, so it can continue running in the background.
sudo vim /etc/systemd/system/promtail.service
Add this script:
[Unit] Description=Promtail service After=network.target [Service] Type=simple User=root ExecStart=/usr/local/bin/promtail -config.file /etc/promtail-local-config.yaml [Install] WantedBy=multi-user.target EOF
- Reload and start Promtail service
sudo systemctl daemon-reload sudo systemctl start promtail.service
Confirm if service is in running state:
$ systemctl status promtail.service
promtail.service – Promtail service
Loaded: loaded (/etc/systemd/system/promtail.service; disabled; vendor preset: enabled)
Active: active (running) since Thu 2022-07-07 10:22:16 UTC; 5s ago
Main PID: 13667 (promtail)
Tasks: 6 (limit: 1267)
└─15381 /usr/local/bin/promtail -config.file /etc/promtail-local-config.yaml
Jul 07 10:22:16 ubuntu systemd: Started Promtail service.
Jul 07 10:22:16 ubuntu promtail: level=info ts=2022-07-07T10:22:16.812189099Z caller=server.go:225 http=[::]:9080 grpc=[::]:35499 msg=”server listening on>
Jul 07 10:22:16 ubuntu promtail: level=info ts=2020-07-07T11
This example uses Promtail for reading the systemd-journal. The promtail user will not yet have the permissions to access it. Add the user promtail into the systemd-journal group
usermod -a -G systemd-journal promtail
You can stop the Promtail service at any time by typing
sudo service promtail stop sudo service promtail status
Remote access may be possible if your Promtail server has been running. If localhost is not required to connect to your server, type
iptables -A INPUT -p tcp -s localhost — dport 9080 -j ACCEPT iptables -A INPUT -p tcp — dport 9080 -j DROP iptables -L
So that is all the fundamentals of Promtail you needed to know. Promtail is a logs collector built specifically for Loki. It uses the same service discovery as Prometheus and includes analogous features for labelling, transforming, and filtering logs before ingestion into Loki. Go ahead, setup Promtail and ship logs to Loki instance or Grafana Cloud.