Grafana cloud monitoring

grafana, kotlin, vertx

Introduction

For any software project, being able to see what is going on is extremely important. Monitoring, visibility, or observability are all terms that are used for this. For my side projects I use grafana cloud and it is by far the easiest way to get metrics, and logs out into a simple hosted service. The free tier is generous enough that it makes it feasible for large deployments as well.

Prometheus

Prometheus lets me send my metrics up to grafana cloud. Within my kotlin vert.x application I use the following piece of code to create the registry

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
fun getPrometheusRegistry(): PrometheusMeterRegistry {
  val registry = PrometheusMeterRegistry(PrometheusConfig.DEFAULT)
  ClassLoaderMetrics().bindTo(registry)
  JvmMemoryMetrics().bindTo(registry)
  JvmGcMetrics().bindTo(registry)
  ProcessorMetrics().bindTo(registry)
  JvmThreadMetrics().bindTo(registry)
  UptimeMetrics().bindTo(registry)
  FileDescriptorMetrics().bindTo(registry)

  return registry
}

fun getVertxOptions(): VertxOptions {
  return VertxOptions().setMetricsOptions(
    MicrometerMetricsOptions()
      .setEnabled(true)
      .setMicrometerRegistry(getPrometheusRegistry())
  )
}

val vertx = Vertx.vertx(getVertxOptions())

val registry = BackendRegistries.getDefaultNow() as PrometheusMeterRegistry

Then at /metrics I run registry.scrape() and return the output as the response.

The actual prometheus instance is also quite simple. Since it mainly sends the metrics up to the grafana cloud instance it makes it quite lightweight to bring into my docker compose stack

1
2
3
4
5
6
## docker-compose.yml
prometheus:
  container_name: prometheus
  image: prom/prometheus:latest
    volumes:
      - ./prometheus/prometheus.yaml:/etc/prometheus/prometheus.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
## Prometheus.yml
global:
  scrape_interval: 30s
  evaluation_interval: 30s
remote_write:
  - url: https://prometheus-blocks-prod-us-central1.grafana.net/api/prom/push
    basic_auth:
      username: <userID>
      password: <APIKey>
scrape_configs:
  - job_name: "prom-metrics"
    metrics_path: "/admin/v1/metrics" # the path at which your application exposes metrics
    static_configs:
      - targets: ["<host>:<port>"]

Few minutes after running docker-compose up I have metrics up in my grafana cloud dashboard.

Promtail

Promtail is grafana’s offering at a log shipper. It is also super easy to get running in my docker compose stack.

1
2
3
4
5
6
7
8
9
## docker-compose.yml
promtail:
  container_name: promtail
  image: grafana/promtail:2.5.0
  volumes:
    - ./promtail/config.yaml:/etc/promtail/config.yaml
    - /var/lib/docker/containers:/var/lib/docker/containers
    - /var/log/:/var/log/
  command: -config.file=/etc/promtail/config.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
## Config.yaml
server:
  http_listen_port: 0
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

client:
  url: https://<User>:<APIKey>@logs-prod-us-central1.grafana.net/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*.log


- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      __path__: /var/lib/docker/containers/*/*log

  pipeline_stages:
  - json:
      expressions:
        output: log
        stream: stream
        attrs:
  - json:
      expressions:
        tag:
      source: attrs
  - regex:
      expression: (?P<image_name>(?:[^|]*[^|])).(?P<container_name>(?:[^|]*[^|])).(?P<image_id>(?:[^|]*[^|])).(?P<container_id>(?:[^|]*[^|]))
      source: tag
  - timestamp:
      format: RFC3339Nano
      source: time
  - labels:
      tag:
      stream:
      image_name:
      container_name:
      image_id:
      container_id:
  - output:
      source: output

Few minutes after running docker-compose up docker logs are now being shipped up to Loki, which is grafana’s indexer of logs that allows rich querying, slicing, and dicing.

Conclusion

Grafana cloud is like the heroku of monitoring for developers. Being able to see the traffic going through the service without having to ssh into the server is such a great dev experience to have. Being able to set all that up and blog about it in an evening is even better.