I have unique log collection techniques

This article is simultaneously published on Candy·Tribe

As more and more servers are opened, the containers running on each server become increasingly mixed, and collecting logs has become a troublesome task. I usually use Docker for easier management, and since I don't quite understand Kubernetes, I haven't built a cluster, making log collection a rather awkward task. If I use the log command that comes with Docker or Docker Compose, I not only have to connect to the original server and enter the service directory, but also input a long string of commands. After successfully executing, I still have to search for the needed parts in a long output, and if I'm not careful, the content I just found gets overwritten by new logs, which is quite inelegant. So I tried to find a solution that could quickly and conveniently collect running logs and perform on-demand queries and organization. Recently, I was researching Grafana and Loki, and I found the official Docker log plugin, so I decided to write another article to document the configuration files and commands used, making it easier for friends in need to quickly solve similar problems.

Please adjust the relevant configuration files according to your usage to avoid interference caused by some sample configurations.

Preparing Grafana + Loki#

Preparing the Joint Startup File#

Loki can be understood as the server that actually collects logs, while Grafana can be understood as the frontend that queries data from the server. The two are independent yet interconnected and can be deployed within the same local area network or separated and connected via the public network. Since this is a specialized scenario for pure log collection, I directly used a docker-compose to associate the startup.

Moreover, since this server is dedicated solely to log collection and does not need to share port 443 with other services, I also wrote the Caddy configuration together.

The docker-compose.yml file is as follows:

version: '3.4'
services:

  loki:
    image: grafana/loki
    container_name: grafana-loki
    command: -config.file=/etc/loki/local-config.yml
    networks:
      - internal_network
    volumes:
      - ./config/loki.yml:/etc/loki/local-config.yml
      - ./data/loki:/tmp/loki

  grafana:
    depends_on:
      - grafana-db
      - loki
    image: grafana/grafana
    container_name: grafana
    networks:
      - internal_network
      - external_network
    volumes:
      - ./config/grafana.ini:/etc/grafana/grafana.ini
    restart: unless-stopped

  grafana-db:
    image: postgres:15-alpine
    restart: always
    container_name: grafana-db
    volumes:
      - ./data/db:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: grafana
      POSTGRES_PASSWORD: password
      POSTGRES_DB: grafana
      POSTGRES_INITDB_ARGS: "--encoding='UTF8' --lc-collate='C' --lc-ctype='C'"
    networks:
      - internal_network

  caddy:
    image: caddy
    restart: always
    ports:
      - "443:443"
    networks:
      - internal_network
      - external_network
    volumes:
      - ./config/Caddyfile:/etc/caddy/Caddyfile
      - ./ssl:/ssl:ro
      - ./data/caddy/data:/data
      - ./data/caddy/config:/config

networks:
  internal_network:
    internal: true
  external_network:

The only exposed port is Caddy's 443 port, and strong HTTPS is enabled at the CDN to ensure that there are no requests directed to port 80.

Preparing the Configuration for Each Service#

The configuration files are uniformly categorized and placed in the config directory for easier management.

Grafana#

The configuration file for Grafana grafana.ini is as follows:

[database]
type = postgres
host = grafana-db:5432
name = grafana
user = grafana
password = password

Loki#

The configuration file for Loki loki.yml is as follows:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

analytics:
  reporting_enabled: false

# See https://grafana.com/docs/loki/latest/configuration/
# S3 storage can be enabled, but it's not necessary for now

Since I don't know what alertmanager_url should be set to, and as a log collection service, I am not considering alerting needs for now, I kept the sample value unchanged.

It is important to note the auth_enabled configuration item. This field allows multiple organizations to share the same Loki service by setting different region request headers. When enabled, the X-Scope-OrgID must be used in HTTP calls to distinguish the organization making the request, and this must be included when submitting and retrieving data. However, since this is for single-party use and does not involve sharing data, this option is disabled.

Loki itself does not have HTTP authorization checks, so it is recommended to write the authorization request headers in the configuration of the frontend reverse proxy program. For example, here it is placed in the Caddy configuration file, requiring authorization checks only for requests from public traffic, while Grafana does not need to verify when using internal connections.

Caddy#

Here, I used the source server certificate provided by the CDN provider, so there is no need for Caddy to apply for it, and the configuration file is as follows:

example.com {
    tls /ssl/example.com/cert.pem /ssl/example.com/key.pem
    reverse_proxy grafana:3000
}
loki.example.com {
    tls /ssl/example.com/cert.pem /ssl/example.com/key.pem
    reverse_proxy loki:3100
    basicauth * {
        user $2a$14$QZIF1JLM9lFPjAl6PQzBDeANmB4Ssufc0qUnP9nI7QYnv.QevEzyi
    }
}
loki-grpc.example.com {
    tls /ssl/example.com/cert.pem /ssl/example.com/key.pem
    reverse_proxy loki:9095
    basicauth * {
        user $2a$14$QZIF1JLM9lFPjAl6PQzBDeANmB4Ssufc0qUnP9nI7QYnv.QevEzyi
    }
}

The authorization part uses BasicAuth, which is a username + password method; the password is not stored in plain text but is pre-computed using Caddy's hash-password feature to enhance security as much as possible.

/srv # caddy hash-password
Enter password: 
Confirm password: 
$2a$14$QZIF1JLM9lFPjAl6PQzBDeANmB4Ssufc0qUnP9nI7QYnv.QevEzyi
/srv #

After completing the configuration, you also need to place the corresponding SSL certificate in the specified location to avoid errors due to missing certificate files.

Once all configurations are complete, you should be able to start using the docker-compose up -d command.

Managing Permissions for the Loki Data Directory#

Loki in the container does not start as the root user, so the root permissions set by Docker will cause access to be denied. You need to manually execute a command like the following to open the permissions of Loki's data directory for container use:

chown -R 10001:10001 ./data/loki/

After modifying, restart the Loki container.

Associating Loki as a Data Source for Grafana#

Enter the deployed Grafana, log in with the default username and password admin, and after changing the password, you can start adding data sources.

If following the above docker-compose configuration scheme, you only need to select Loki and enter http://loki:3100 in the HTTP URL field, and then save and test. At this point, Loki will report an error indicating that no labels can be found because there are no logs yet, but don't worry; it will be fine once logs appear later.

Preparing the Server#

Since a Docker environment is used, I won't elaborate on the specific commands here, starting directly from installing the Loki log plugin.

This refers to the tutorial provided by Docker Driver Client, and please note that related content may be updated.

Installing the loki-docker-driver Plugin#

To send data to Loki, a plugin is needed to convert Docker logs into data with detailed label information and send it to the deployed server using a Loki-compatible interface. You can install this plugin using the following command:

docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

After installation, you can check the installed Docker plugins using docker plugin ls.

If there are subsequent upgrades needed, you can refer to the commands provided in the original link for upgrading:

docker plugin disable loki --force
docker plugin upgrade loki grafana/loki-docker-driver:latest --grant-all-permissions
docker plugin enable loki
systemctl restart docker

Then you can proceed to configure the plugin. Since I need to collect all Docker logs on the host machine and do not need to distinguish between different organizations, I will configure the /etc/docker/daemon.json file with the following content:

{
    "log-driver": "loki",
    "log-opts": {
        "loki-url": "https://user:[email protected]/loki/api/v1/push"
    }
}

After completing the configuration, you need to restart the Docker process for the changes to take effect.

It is important to note that this will not take effect in real-time for all containers; the logging system of already created containers has been fixed, and it will only be effective for newly created containers. If you urgently need log collection, you can try using docker-compose up -d --force-recreate to recreate all containers.

Querying in Grafana#

Once the configuration is complete and the newly created containers start working and generating logs, you should be able to query them in Grafana. Since this is for single-party use and does not involve detailed visualization needs, I did not configure a Dashboard but directly entered the Explore feature for interactive queries.

Once logs appear, you can filter by labels to identify the source. This plugin will provide six filtering labels: host, compose_project, compose_service, container_name, source, and filename, among which:

host refers to the hostname of the host machine that generated the logs,
compose_project, compose_service, or container_name can be used to locate the container,
source indicates whether the log source is standard output stdout or error log stderr,
filename refers to the original log file (usually not important),

Using these labels along with Grafana's time control, you can quickly query the running logs of a specific container. No more worries about scattered and numerous logs!

Filtered Logs