Metrics, metrics everywhere

Building an Iceshrimp.NET Grafana dashboard.

What is Grafana?

Grafana open source software enables you to query, visualize, alert on, and explore your metrics, logs, and traces wherever they are stored.

https://grafana.com/docs/grafana/latest/introduction

Iceshrimp.NET Grafana Dashboard Installation

Resources:

https://grafana.com/

https://grafana.com/docs/grafana/latest/setup-grafana/installation/debian/

https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-grafana-on-ubuntu-22-04

Update Everything

The following may be a bit of overkill but I run this command quite frequently:

sudo apt update && sudo apt upgrade && sudo apt remove && sudo apt clean && sudo apt autoremove && sudo apt autoclean

Install part I

sudo apt-get install -y apt-transport-https software-properties-common wget

Add Grafana APT repository and key

wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null

echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Install part II

sudo apt-get update

sudo apt-get install grafana

Enable Grafana Server

sudo /bin/systemctl daemon-reload

sudo /bin/systemctl enable grafana-server

sudo systemctl start grafana-server

sudo systemctl status grafana-server

Change listening port

Then, if iceshrimp.NET is using the default port, change the Grafana listening port.

sudo systemctl stop grafana-server

sudo nano /etc/grafana/grafana.ini

Change:

# The http port  to use
;http_port = 3000

To:

# The http port  to use
# ;http_port = 3000
http_port = 'port of your choice'

Restart

sudo systemctl restart grafana-server

Update firewall

sudo ufw allow 'port of your choice' comment "Grafana traffic"

sudo ufw status verbose

A slight installation detour

Even though Grafana only READs from the database, I prefer to have a dedicated user per process.

First, CREATE a PostgreSQL User with READ-ONLY access to the ‘iceshrimp‘ database:

sudo -u postgres psql

CREATE ROLE grafana_user WITH LOGIN PASSWORD 'super_long_password';

Then, connect as that User:

\connect iceshrimp;

GRANT CONNECT ON DATABASE "iceshrimp" TO grafana_user;

GRANT USAGE ON SCHEMA public TO grafana_user;

GRANT SELECT ON ALL TABLES IN SCHEMA public TO grafana_user;

Then, alter the privileges the new User has:

ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO grafana_user;

Create Account(s)

Then, go to the Grafana homepage and create the first Account.

Remember that by default this will be an Admin (i.e. will have elevated privileges).

http://<server-ip>:port of your choice

Iceshrimp.NET Grafana Dashboard

I am aware that Grafana can hook into all sorts of other stuff; this was a learning exercise for me and as such is limited to using the PostgreSQL database as a data source.

Current Grafana Dashboard

Links go to the SQL used to pull the data into the Grafana panel.

Most of the time this was done with the ‘Builder’ tool but a couple of queries are handwritten.

00. Instance Info:

  • Just a Header really (i.e. the top of the page content)

01. Overview:

  • Federated Instances
    Shows the total number of Federated instances (i.e. how many instances this installation is connected to)
  • Blocked Instances
    Shows the total number of blocked instances (i.e. how many instances this installation is blocks)
  • Accounts
    Shows the total number of Accounts; this includes the instance.actor (i.e. accounts have been created on this instance)
  • Notes
    Shows the total number of Notes (i.e. how many Notes this installation has processed)
  • Blocked Accounts
    Shows the total number of blocked Accounts that fall outside of blocked Instances (i.e. how many Accounts that have been individually blocked)
  • Muted Accounts
    Total number of muted Accounts (i.e. how many Accounts that have been individually muted)

02. Jobs:

  • Job Queue by Status
    Current status of Notes (jobs) being processed (i.e. the current workload in respect of job processing)

03. Notes:

04. Sessions:

05. Misc:

  • Is Cat ?
    Yes, really (i.e. whether or not Accounts identify as a cat)
  • Note Reactions
    A list of posted Reactions, ranked by usage (i.e. the most used emoji)

Setup a Sub Domain for the Dashboard

Resources:

https://grafana.com/tutorials/run-grafana-behind-a-proxy

First, set up a basic nginx grafana.conf to ensure domain resolution works:

sudo nano /etc/nginx/sites-available/grafana.conf

server {
    listen              80;
    listen              [::]:80;
    server_name         grafana.example.net;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }
}

Then, ‘enable’ the new site by linking:

sudo ln -s /etc/nginx/sites-available/grafana.conf /etc/nginx/sites-enabled/

Then, restart nginx:

sudo nginx -t && sudo systemctl restart nginx

Secondly, setup a basic non-SSL nginx conf file for Grafana:

sudo nano /etc/nginx/sites-available/grafana.conf

map $http_upgrade $connection_upgrade {
    default upgrade;
        '' close;
}

upstream grafana {
    server localhost:3000;
}

server {
    server_name         grafana.example.net;
    root                /usr/share/nginx/html;
    index               index.html index.htm;

    location / {
    proxy_set_header Host $host;
        proxy_pass http://grafana;
    }

    # Proxy Grafana Live WebSocket connections.
    location /api/live/ {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_pass http://grafana;
  }
}

Then, ‘Check and Restart’ nginx:

sudo nginx -t && sudo systemctl restart nginx

Then, ‘Restart and Check Grafana’:

sudo systemctl restart grafana-server

sudo systemctl status grafana-server

Finally, Set up a Grafana SSL reverse.

sudo certbot

The result (i.e. successful) should look something like this:

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/viz.consummatetinkerer.net/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/viz.consummatetinkerer.net/privkey.pem

Then, ‘Check and Restart nginx’:

sudo nginx -t && sudo systemctl restart nginx

Then, ‘Restart and Check Grafana’:

sudo systemctl restart grafana-server

sudo systemctl status grafana-server

There are a slew of Grafana posts on Google about updating the grafani.ini to get SSL functioning properly.

However I did none of those things and it seems to be functioning quite well.

A screenshot of the iceshrimp.NET Grafana Dashboard showing various metrics as gauges and large numbers. Iceshrimp.NET Grafana Dashboard Screenshot