Grafana cloud monitoring

Introduction

For any software project, being able to see what is going on is extremely important. Monitoring, visibility, or observability are all terms that are used for this. For my side projects I use grafana cloud and it is by far the easiest way to get metrics, and logs out into a simple hosted service. The free tier is generous enough that it makes it feasible for large deployments as well.

Prometheus

Prometheus lets me send my metrics up to grafana cloud. Within my kotlin vert.x application I use the following piece of code to create the registry

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
fun getPrometheusRegistry(): PrometheusMeterRegistry {
val registry = PrometheusMeterRegistry(PrometheusConfig.DEFAULT)
ClassLoaderMetrics().bindTo(registry)
JvmMemoryMetrics().bindTo(registry)
JvmGcMetrics().bindTo(registry)
ProcessorMetrics().bindTo(registry)
JvmThreadMetrics().bindTo(registry)
UptimeMetrics().bindTo(registry)
FileDescriptorMetrics().bindTo(registry)

return registry
}

fun getVertxOptions(): VertxOptions {
return VertxOptions().setMetricsOptions(
MicrometerMetricsOptions()
.setEnabled(true)
.setMicrometerRegistry(getPrometheusRegistry())
)
}

val vertx = Vertx.vertx(getVertxOptions())

val registry = BackendRegistries.getDefaultNow() as PrometheusMeterRegistry

Then at /metrics I run registry.scrape() and return the output as the response.

The actual prometheus instance is also quite simple. Since it mainly sends the metrics up to the grafana cloud instance it makes it quite lightweight to bring into my docker compose stack

1
2
3
4
5
6
## docker-compose.yml
prometheus:
container_name: prometheus
image: prom/prometheus:latest
volumes:
- ./prometheus/prometheus.yaml:/etc/prometheus/prometheus.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
## Prometheus.yml
global:
scrape_interval: 30s
evaluation_interval: 30s
remote_write:
- url: https://prometheus-blocks-prod-us-central1.grafana.net/api/prom/push
basic_auth:
username: <userID>
password: <APIKey>
scrape_configs:
- job_name: "prom-metrics"
metrics_path: "/admin/v1/metrics" # the path at which your application exposes metrics
static_configs:
- targets: ["<host>:<port>"]

Few minutes after running docker-compose up I have metrics up in my grafana cloud dashboard.

Promtail

Promtail is grafana’s offering at a log shipper. It is also super easy to get running in my docker compose stack.

1
2
3
4
5
6
7
8
9
## docker-compose.yml
promtail:
container_name: promtail
image: grafana/promtail:2.5.0
volumes:
- ./promtail/config.yaml:/etc/promtail/config.yaml
- /var/lib/docker/containers:/var/lib/docker/containers
- /var/log/:/var/log/
command: -config.file=/etc/promtail/config.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
## Config.yaml
server:
http_listen_port: 0
grpc_listen_port: 0

positions:
filename: /tmp/positions.yaml

client:
url: https://<User>:<APIKey>@logs-prod-us-central1.grafana.net/loki/api/v1/push

scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*.log


- job_name: containers
static_configs:
- targets:
- localhost
labels:
job: containerlogs
__path__: /var/lib/docker/containers/*/*log

pipeline_stages:
- json:
expressions:
output: log
stream: stream
attrs:
- json:
expressions:
tag:
source: attrs
- regex:
expression: (?P<image_name>(?:[^|]*[^|])).(?P<container_name>(?:[^|]*[^|])).(?P<image_id>(?:[^|]*[^|])).(?P<container_id>(?:[^|]*[^|]))
source: tag
- timestamp:
format: RFC3339Nano
source: time
- labels:
tag:
stream:
image_name:
container_name:
image_id:
container_id:
- output:
source: output

Few minutes after running docker-compose up docker logs are now being shipped up to Loki, which is grafana’s indexer of logs that allows rich querying, slicing, and dicing.

Conclusion

Grafana cloud is like the heroku of monitoring for developers. Being able to see the traffic going through the service without having to ssh into the server is such a great dev experience to have. Being able to set all that up and blog about it in an evening is even better.