Proxmox Monitoring with Grafana

In this tutorial, we will learn how to set up monitoring for Proxmox using a metric server (InfluxDB2) and a visualization tool (Grafana).

In my home lab, I am running many self-hosted applications and virtual machines, mostly on the Proxmox server. Recently I came across some great monitoring tools to integrate with my current setup. With this new setup, I am able to monitor some critical metrics like CPU, storage, VMs in a clean interface.

In this tutorial, we will cover:

  • Installation of a metric server for Proxmox with InfluxDB.
  • Securing the communication between Proxmox and InfluxDB with SSL certificates.
  • Visualising the data in Grafana dashboard.

My current monitoring setup collects metrics from the Linux server and docker containers into a Prometheus DB. I am using Grafana for visualization. However, a monitoring solution was missing for Proxmox.

Methods to load Proxmox data into DB

There are numerous ways to get Proxmox data into Prometheus DB.

  1. There is a PVE exporter that can connect to Proxmox interface and publish the data in Prometheus format.
  2. You can also set a Prometheus job that can scrape data.
  3. You can also use the Proxmox’s built-in function to send metrics to an influx DB or Graphite. But of-course, this would require installation of a new metric server.

All of the above options are viable, but I am limiting myself to the last one. I would use InfluxDB instead of Prometheus. For this, I will install InfluxDB2 on the Docker server and use the built-in function of Proxmox to send data to it.

I have also found the link to a great Grafana dashboard for the new data source (InfluxDB). It is also customizable. Here is the link to it.

Setting up the monitoring stack

Installing the InfluxDB2 component

Influx DB is the main DB to store metrics from Proxmox. Later, these metrics would be sent to Grafana for visualization.

Installation of InfluxDB2 is very straightforward. A detailed start-up guide can be found here.

In the install guide, there are details for installation on multiple platforms, as shown below:

Note: There are 2 different versions of InfluxDB, and they are not reverse compatible. However, discussing each is beyond the scope of this post.

 I have installed the InfluxDB on my docker server by using Portainer. You can proceed with any other deployment of your choice.

Note: It’s important to expose port 8086, which is the main port to connect to an InfluxDB. It also needs to be accessible from the source server.

Expose port 8086 to connect to InfluxDB

Storage configuration

Always store data in a persistent volume, otherwise the data gets wiped everything a container is restarted.

Notice that for InfluxDB2, the path also needs to have /influxdb2.

Network settings

It is very important to connect the InfluxDB to a custom docker network. Here, in my case, the front end should be the same as the Grafana dashboard.

This also allows you to use DNS lookups with a container’s hostname.

If you want to use the docker-compose to deploy InfluxDB or if you want to use the compose file as a template to deploy it in Portainer, you can find it here on my personal GitHub page.

Intital user setup

Once the container is running, you can just open a web browser and connect to the port 8086 and a wizard will guide you to set up the initial deployment settings.

Type in the browser to access it. Replace IP with your docker container’s IP.

Securing communication with SSL

Communication over HTTP is not encrypted. Usually, people don’t consider metrics and log files as sensitive data. However, in case auth tokens are stolen, anyone can have access to DB. This puts production environments in danger.

Security can be enhanced with reverse proxy. In my setup, I am using self-signed certificates.

I’m using wildcard self-signed certificates in my home network. I have uploaded these certificates to my home server where the InfluxDB is running and mounted this folder inside the container as a ‘read-only bind’.

I have also given administrative permissions to my user who is running the container.

Permissions details on server:

The InfluxDB documentation provides a command that you can put at the end of the binary. Both files need to be provided:

  • The full chain certificate.
  • A certificate key file.

Run influxdb with TLS flags

Start InfluxDB with TLS command-line flags:

influxd \ --tls-cert="<path-to-crt>" \ --tls-key="<path-to-key>"

When you are running InfluxDB and docker, you can simply put these arguments in the command attribute of your compose file or the Portainer deployment as shown below:

When you set up the SSL certificates correctly, you should see that InfluxDB will use the HTTPS protocol now to expose the server on the same port.

Container Logs

Now, we can securely set up our credentials and do the initial setup. We will set up an organization and create a storage bucket where the data for Proxmox should be stored. 

Send proxmox data to InfluxDB

In the Proxmox UI, you can now enable the external metric server as follows:

Give InfluxDB a name and set other values as per the initial setup.

Note: IP address should be same as the Docker where InfluxDB is installed.

Finding access tokens

You would also need access tokens for authentication. You can find the token in InfluxDB GUI as shown below. Just copy the token and paste it to the Proxmox metrics configuration.

However, you might see the below error as we are using SSL certificates. There seems to be no option to skip this validation for now.

It is important that Proxmox trusts InfluxDB’s SSL certificate, and this requires two things to be set up correctly:

  1. A self-signed certificate that is valid for the ip address.
  2. The DNS name that you used in your metrix server configuration.

Hence, if the certificate isn’t valid for my IP address, Proxmox will never trust this certificate.

Thus, I have created my self-signed certificate with a subject alternative name for the wildcard domain and the IP address.

Explicitly specifying trust between Proxmox and InfluxDB

However, trust should also be set up between Proxmox and the CA which has assigned the certificate. Otherwise, Proxmox doesn’t trust where the certificate is coming from, and you need to explicitly specify that the certificate should be trusted.

To explicitly specify, copy the CA cert into the Proxmox server and place it into the user/local/share/ca-certificates folder.

Also, update the certificates to reload them.

Now the self-signed certificate should be imported into the trusted root CA stores of Proxmox. The connection is trusted as the InfluxDB cert is signed with it.

Check if Proxmox is sending data to InfluxDB

Once everything is in place, it is a good practice to check if data is being correctly sent to InfluxDB.

You can see it when you go into the database and query the storage bucket in InfluxDB GUI.

Select these options in the InfluxDB GUI:
Check these options and you should see some metrics.
Metrics in the InfluxDB

Once data is available in the bucket, there is a bunch of other stuff that can be done such as:

  • Loading Data.
  • Setting up alerts.
  • Building a dashboard.

Visualization with Grafana

Now we have the data loaded into the storage, we can visualize it using different tools. In this post, I am using the Grafana dashboard.

Grafana is a powerful tool to visualize metrics. It is very flexible and can query data from almost every kind of data source.

If you are new to Grafana, you can use my Boilerplates for reference.

Proxmox Flux Grafana Dashboard

  1. Add data source in Grafana.

To set up the Grafana dashboard template as mentioned earlier, add a new data source.

Change query language to “FLUX”

Enter the settings:

2. Import Dashboard in Grafana

The Grafana dashboard we are using here is written n Flux. You can review and download it here.

Import this dashboard by copying the ID:

Add the ID in Grafana GUI.

Select bucket and view data as shown below:

Once data is loaded, you can see it in the dashboard.

This dashboard is customizable and panels can be adjusted.