This pages shows the different components that need to be installed so a monitoring server have all the necessary tools to:
Query language to check time-series data.
Scrap metrics from other servers.
Alerting system.
An app to monitor time-series data, alerts.
| Table of Contents | ||
|---|---|---|
|
This guide does not include configuration for each service. Each service should be configured to work accordingly with nginx and have connectivity between services.
...
Prometheus
Prometheus is an open-source tool that provides a query language for time-series data. It collects and stores metrics in a time-series database. To install prometheus the user must follow this steps:
Download prometheus with the following command:
| Code Block |
|---|
sudo wget https://github.com/prometheus/prometheus/releases/download/v2.47.0/prometheus-2.47.0.linux-amd64.tar.gz |
Extract the files with the following command:
| Code Block |
|---|
sudo tar vxf prometheus*.tar.gz |
Create a System User for Prometheus:
| Code Block |
|---|
sudo groupadd --system prometheus
sudo useradd -s /sbin/nologin --system -g prometheus prometheus |
Create Directories for Prometheus:
| Code Block |
|---|
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus |
Navigate to the Prometheus Directory:
| Code Block |
|---|
cd prometheus*/ |
Move Files:
| Code Block |
|---|
#Move the Binary Files
sudo mv prometheus /usr/local/bin
sudo mv promtool /usr/local/bin
#Move other Files
sudo mv console* /etc/prometheus
sudo mv prometheus.yml /etc/prometheus |
Set Owner:
| Code Block |
|---|
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
sudo chown prometheus:prometheus /etc/prometheus
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
sudo chown -R prometheus:prometheus /var/lib/prometheus |
Change Prometheus configuration file:
| Code Block |
|---|
sudo nano /etc/prometheus/prometheus.yml |
Add the following initial configuration so prometheus server can start:
| Code Block |
|---|
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'node'
static_configs:
- targets: ['localhost:9100'] |
Create prometheus service:
| Code Block |
|---|
sudo nano /etc/systemd/system/prometheus.service |
Add the following configuration to the service file:
| Code Block |
|---|
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target |
Reload the daemon configuration:
| Code Block |
|---|
sudo systemctl daemon-reload |
Enable Prometheus:
| Code Block |
|---|
sudo systemctl enable prometheus |
Start Prometheus:
| Code Block |
|---|
sudo systemctl start prometheus |
Check that prometheus is running without issues:
| Code Block |
|---|
sudo systemctl status prometheus |
Prometheus is now running on port 9090
...
Node Exporter
Node exporter is an agent that extracts system metrics and expose that information so prometheus can read those metrics.
Download node_exporter compressed pack:
| Code Block |
|---|
wget https://github.com/prometheus/node_exporter/releases/download/v*/node_exporter-*.*-amd64.tar.gz |
Extract the binary:
| Code Block |
|---|
tar xvfz node_exporter-*.*-amd64.tar.gz |
Access the directory of the extracted file:
| Code Block |
|---|
cd node_exporter-*.*-amd64 |
Change folder execution permissions:
| Code Block |
|---|
chmod +x node_exporter |
Run node_exporter binary
| Code Block |
|---|
./node_exporter |
Run the following command to check the service status:
| Code Block |
|---|
sudo systemctl status node_exporter |
Now node_exporter is running on port 9100.
...
Grafana
An open-source analytics and interactive visualization web application used for monitoring application performance.
Install the prerequisite packages:
| Code Block |
|---|
sudo apt-get install -y apt-transport-https software-properties-common wget |
Import the GPG key:
| Code Block |
|---|
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
|
To add a repository for stable releases, run the following command:
| Code Block |
|---|
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list |
To add a repository for beta releases, run the following command:
| Code Block |
|---|
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com beta main" | sudo tee -a /etc/apt/sources.list.d/grafana.list |
Run the following command to update the list of available packages:
| Code Block |
|---|
sudo apt-get update |
To install Grafana OSS, run the following command:
| Code Block |
|---|
sudo apt-get install grafana |
Now grafana is running on port 3000.
...
Alertmanager
Handles alerts sent by client applications such as the Prometheus server.
Download prometheus alertmanager:
| Code Block |
|---|
wget https://github.com/prometheus/alertmanager/releases/download/v0.21.0/alertmanager-0.21.0.linux-amd64.tar.gz |
Create prometheus user:
| Code Block |
|---|
sudo groupadd -f alertmanager
sudo useradd -g alertmanager --no-create-home --shell /bin/false alertmanager
sudo mkdir -p /etc/alertmanager/templates
sudo mkdir /var/lib/alertmanager
sudo chown alertmanager:alertmanager /etc/alertmanager
sudo chown alertmanager:alertmanager /var/lib/alertmanager |
Unpack alertmanager binary:
| Code Block |
|---|
tar -xvf alertmanager-0.21.0.linux-amd64.tar.gz
mv alertmanager-0.21.0.linux-amd64 alertmanager-files |
Install prometheus alertmanager:
| Code Block |
|---|
sudo cp alertmanager-files/alertmanager /usr/bin/
sudo cp alertmanager-files/amtool /usr/bin/
sudo chown alertmanager:alertmanager /usr/bin/alertmanager
sudo chown alertmanager:alertmanager /usr/bin/amtool |
Install prometheus alertmanager configuration file:
| Code Block |
|---|
sudo cp alertmanager-files/alertmanager.yml /etc/alertmanager/alertmanager.yml
sudo chown alertmanager:alertmanager /etc/alertmanager/alertmanager.yml |
Setup prometheus alertmanager service:
| Code Block |
|---|
sudo vi /usr/lib/systemd/system/alertmanager.service |
Add the following configuration to the service file:
| Code Block |
|---|
[Unit]
Description=AlertManager
Wants=network-online.target
After=network-online.target
[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/bin/alertmanager \
--config.file /etc/alertmanager/alertmanager.yml \
--storage.path /var/lib/alertmanager/
[Install]
WantedBy=multi-user.target |
Modify permissions to file:
| Code Block |
|---|
sudo chmod 664 /usr/lib/systemd/system/alertmanager.service |
Reload systemd and start alertmanager service:
| Code Block |
|---|
sudo systemctl daemon-reload
sudo systemctl start alertmanager |
Enable alertmanager service to be available on machine restart:
| Code Block |
|---|
sudo systemctl enable alertmanager.service |
Check alertmanager service status:
| Code Block |
|---|
sudo systemctl status alertmanager |
Now alertmanager is running on port 9093.
...
Telegraf
An open source server agent that helps you collect metrics from your stacks, sensors, and systems.
Run the following command to install telegraf from influxDB repository:
| Code Block |
|---|
curl --silent --location -O \
https://repos.influxdata.com/influxdata-archive.key \
&& echo "943666881a1b8d9b849b74caebf02d3465d6beb716510d86a39f6c8e8dac7515 influxdata-archive.key" \
| sha256sum -c - && cat influxdata-archive.key \
| gpg --dearmor \
| sudo tee /etc/apt/trusted.gpg.d/influxdata-archive.gpg > /dev/null \
&& echo 'deb [signed-by=/etc/apt/trusted.gpg.d/influxdata-archive.gpg] https://repos.influxdata.com/debian stable main' \
| sudo tee /etc/apt/sources.list.d/influxdata.list
sudo apt-get update && sudo apt-get install telegraf |
For Telegraf to start pushing metric to influxDB the configuration file must be modified.
...
InfluxDB
A high performance Time Series Database which can store data ranging from hundreds of thousands of points per second.
Update the OS packages:
| Code Block |
|---|
sudo apt-get update |
Setup the OS repositories to add influxdb:
| Code Block |
|---|
wget -q https://repos.influxdata.com/influxdata-archive_compat.key |
| Code Block |
|---|
echo '393e8779c89ac8d958f81f942f9ad7fb82a25e133faddaf92e15b16e6ac9ce4c influxdata-archive_compat.key' | sha256sum -c && cat influxdata-archive_compat.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg > /dev/null |
| Code Block |
|---|
echo 'deb [signed-by=/etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg] https://repos.influxdata.com/debian stable main' | sudo tee /etc/apt/sources.list.d/influxdata.list |
Update OS package list:
| Code Block |
|---|
sudo apt-get update |
Install influxDB:
| Code Block |
|---|
sudo apt-get install influxdb2 |
Check influxDB version:
| Code Block |
|---|
influx version |
Start influxDB service:
| Code Block |
|---|
sudo systemctl start influxdb |
Enable the service so influxDB can be available on machine restart:
| Code Block |
|---|
sudo systemctl enable influxdb |
Check the service status:
| Code Block |
|---|
sudo service influxdb status |
InfluxDB is now available on port 8086