Monitoring a Celestia consensus node

Monitoring of a consensus node & validator for the Celestia blockchain, with Prometheus and Grafana. We use a separate server for Prometheus and Grafana. You can rent a VPS with minimal specifications

Prometheus is an open source database management system written in Go. An interesting feature of Prometheus is that it itself pulls metrics from a given set of services. Due to this, Prometheus cannot clog any data queues, which means monitoring will never become a system bottleneck.

---

Before you start, you need to make sure that prometheus is enabled in your validator node by checking the value of the variable in:

.celestia-app/config/config.toml

Specify your listening port (26660) and be sure to open it in your Firewall.

For the configuration to take effect, you need to restart your node.

Install Prometheus

Updating repositories

apt update && apt upgrade -y

Install the necessary utilities

apt install curl iptables build-essential git wget jq make gcc nano tmux htop nvme-cli pkg-config libssl-dev libleveldb-dev tar clang bsdmainutils ncdu unzip libleveldb-dev -y
apt install python3-pip -y
pip install yq

Create a prometheus user

sudo useradd -m -s /bin/bash Prometheus 
sudo groupadd --system Prometheus  
sudo usermod -aG Prometheus Prometheus

Download and install Prometheus

sudo mkdir /var/lib/prometheus
for i in rules rules.d files_sd; do sudo mkdir -p /etc/prometheus/${i}; done
mkdir -p /tmp/prometheus && cd /tmp/prometheus
curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest | grep browser_download_url | grep linux-amd64 | cut -d '"' -f 4 | xargs wget

tar xvf prometheus*.tar.gz
cd prometheus*/
sudo mv prometheus promtool /usr/local/bin/

Check versions

prometheus --version
promtool --version

Move the files

sudo mv prometheus.yml /etc/prometheus/prometheus.yml
sudo mv consoles/ console_libraries/ /etc/prometheus/

Create a service files

sudo tee /etc/systemd/system/prometheus.service<<EOF
[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=Prometheus
Group=Prometheus
ExecReload=/bin/kill -HUP \$MAINPID
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries \
--web.listen-address=0.0.0.0:9090 \
--web.external-url=

SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target
EOF

Port 9090 can be replaced with your own (for example 8080).

for i in rules rules.d files_sd; do sudo chown -R Prometheus:Prometheus /etc/prometheus/${i}; done
for i in rules rules.d files_sd; do sudo chmod -R 775 /etc/prometheus/${i}; done
sudo chown -R Prometheus:Prometheus /var/lib/prometheus/

Start service

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus

Prometheus installed! Press CTL+C to exit.

Now we can access the Prometheus interface by accessing the ip of the server where we have installed it: (ip node Prometheus):9090/status

In the future, you can add additional information about your other servers to prometheus.yml in order to monitor several servers at once. The configuration will depend on your settings and json for Grafana.

Setting up Prometheus

Go back to your Prometheus server and edit the prometheus.yml file.

sudo nano /etc/prometheus/prometheus.yml

Enter the following parameters, the prometheus.yml file should look like this:

- job_name: celestia-consensus
  static_configs:
  - targets: ['localhost:26660']

ATTENTION - here the standard Prometheus port (9090) has been replaced with port (8080)

Restart the Prometheus service.

sudo systemctl restart prometheus
sudo journalctl -u prometheus -f --no-hostname -o cat

Install Grafana

Install some dependencies

sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

Updating repositories

echo "deb https://packages.grafana.com/enterprise/deb estable main "| sudo tee -a /etc/apt/sources.list.d/grafana.list

Create the user to manage

sudo useradd -m -s /bin/bash grafana
sudo groupadd --system Grafana
sudo usermod -aG Grafana grafana

Install Grafana and update your packages

sudo apt-get install -y adduser libfontconfig1
wget https://dl.grafana.com/enterprise/release/grafana-enterprise_9.3.2_amd64.deb
sudo dpkg -i grafana-enterprise_9.3.2_amd64.deb

Start the server

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl status grafana-server

sudo systemctl enable grafana-server.service

Check the logs

journalctl -u grafana-server -f

Now is the time to go to the browser. Navigate to http://your_grafana_ip:3000 and you should see the Grafana home page. To log in your initial username is admin and your password is admin.

The system immediately asks us to come up with a new password.

After changing the password, click "Configuration" and then "Data Sources".

Now click on Add data source and select the Prometheus data source.

Now enter the IP address with port 9090. If you have decided to run Prometheus and Grafana on the same server, type http://localhost:9090 If you have separated Grafana and Prometheus into two servers, then enter the IP address of Prometheus.

Save the settings.

Now you need to download the json files of the dashboard. To do this, you need to know the ID of the json file or download it from someone or create it yoursel. You can find json here.

Now click on Dashboard and then on Manage.

Now click on Import and then on Upload JSON file.

Upload the JSON file you downloaded earlier, then select the Prometheus data source you just configured and click the +Import button.