conf.d, you sneaky fiend!

Cola Gangsters! Holy shit, friends… I just spent some 3 hours fighting with Grafana on the old homelab dojo, and I wanted to get this ode to dumbassery preserved for posterity.

The belligerents:

Read on... TL;DR at the bottom.

So, ok. I wanted to get Grafana + Prometheus back on my tEcH sTaCk, and I thought it would be a good idea to just get a little Alpine LXC container going on Proxmox to get the job done. No need to dockerize/podman-ize anything. Simple stuff. In and out. 20 minutes. Or was it?

Gangsters, I tell ya... I was not. Some of you (many) who are more adept at this thing than me (low bar) will laugh at my mishaps. Or feel empathy... or some sort of kinship that only people who fight with configuration files for literal hours can share.

Don't get me wrong, I just love Alpine. It's the best. It runs everything you throw at it, keeps it to itself, no fuzz, no frills. Great package collection, too. apk just works! My beef, however, is with one Konstantin Kulikov. Look, KK, I don't know you, and I'm grateful for all your work, but, dude... your approach to the Grafana package just fucking killed me.

What happened?

Simply put, every document on the planet will tell you that the grafana.ini file should be used to configure things like the IP addresses Grafana will listen at. Set http_addr to nothing at all, and Grafana will listen on all interfaces. Set it to 127.0.0.1, and it will only listen at the localhost. Simple enough, right? Wrong. It would not listen on anything but the localhost, no matter what http_addr says.

After many attempts to remedy the situation, to right this wrong, and have pretty graphs with mostly data I don't even care about, I had the presence of spirit to see what Grafana was up to.

ps auxw | grep grafana gave me a clue:

537 grafana   0:01 /usr/bin/grafana server -config /etc/grafana.ini -homepath /usr/share/grafana cfg:paths.data=/var/lib/grafana/data cfg:paths.plugins=/var/lib/grafana/plugins cfg:paths.provisioning=/var/lib/grafana/provisioning cfg:server.http_addr=127.0.0.1 cfg:log.mode=syslog

Clearly, cfg:server.http_addr=127.0.0.1 was harshing my mellow. But, where the fuck did that come from? It's not coming from grafana.ini. It's not coming from /etc/init.d/grafana. Although /etc/init.d/grafana does mention $GRAFANA_OPTS. Nice, but WTF is THAT?

By sheer luck, I stumbled upon this. It's an email between someone who was having the very same issue and one Konstantin Kulikov, maintainer of the Grafana package for Alpine Linux. The key exchange is this:

>Would you be open to removing this line from the conf.d file?

It's not ok for unsecured services in default configuration to listen
on all interfaces. So no.

Which, fair point. But, the maintainer decided to hardcode http_addr to the localhost ON A COMPLETELY DIFFERENT FILE than the config file mentioned by the official documentation. If you don't know that some Grafana configs from the Alpine Linux apk are set in /etc/conf.d/grafana and not /etc/grafana.ini, you're bound to relive my nightmare. Maybe add a couple of lines to the package page saying that? Don't be like me, friends. I hope this gets indexed by Google or whatever. If one unlucky bastard can save some time by reading this, I'll be happy. If you found this while having this issue, reach out to me on Mastodon, and I'll send you a Cola Gang Industries sticker.

TL;DR

http_addr on the Alpine Linux apk version of Grafana is hardcoded to 127.0.0.1. You'll need to change/remove this in /etc/conf.d/grafana. Changing /etc/grafana.ini will not work until you remove the cfg:server.http_addr=127.0.0.1 line from /etc/conf.d/grafana.

God.