Seitenhierarchie
Zum Ende der Metadaten springen
Zum Anfang der Metadaten

Since openITCOCKPIT 3.6

With openITCOCKPIT 3.6 all performance data gets written to Whisper (Carbon). This helps to reduce disk IO

and solve several issues we had with RRDTool.

Docker Container

The new Graphing Backend of openITCOCKPIT runs inside of different Docker containers. We use Docker Compose to manage the containers.

All containers can be controlled through systemd

Start Graphing Backend
systemctl start openitcockpit-graphing.service
Stop Graphing Backend
systemctl stop openitcockpit-graphing.service
Graphing Backend status
systemctl status openitcockpit-graphing.service


You could also use the Docker CLI to query the status of the containers

Graphing Backend status
docker container ps

Logfile

cd /usr/share/openitcockpit/docker/container/graphing
docker-compose logs -f

docker-compose.yml file

The file docker-compose.yml is located at /usr/share/openitcockpit/docker/container/graphing/docker-compose.yml.

Warning

Do not edit the docker-compose file manually. It will be get automatically generated by openITCOCKPIT.


Carbon-C-Relay

The Carbon-C-Relay is a load balancer infront of multiple Carbon-Cache instances.

The default LINE_RECEIVER_PORT (Plaintext protocol based on TCP) is 2003 and gets exposed to 127.0.0.1:2003.

You can change the port via the openITCOCKPIT web interface.

The configuration file gets automatically generated to: /etc/openitcockpit/carbon/carbon-c-relay.conf


Carbon-Cache instances

Is responsible to write and read data from Whisper files.

Default Whisper path on openITCOCKPIT server: /var/lib/graphite/whisper

The configuration file gets automatically generated to: /etc/openitcockpit/carbon/carbon.conf


Graphite-Web

Provides a HTTP-JSON-API to easy access the data saved in Whisper files.

Graphite-Web will send a query to the Carbon-Cache instances.

The configuration file gets automatically generated to: /etc/openitcockpit/carbon/local_settings.py


Grafana

Grafana is a powerful and easy to use tool, to create graph dashboards. It is used by openITCOCKPIT in the background through the API.


Grafana itself could be reached by https://<openITCOCKPIT-address>/grafana/

The configuration file gets automatically generated to: /etc/openitcockpit/grafana/grafana.ini

The username for the Grafana interface is "admin".

A random password will be generated during the installation process. It is stored at: /etc/openitcockpit/grafana/admin_password

The API-Key used by openITCOCKPIT is located at: /etc/openitcockpit/grafana/api_key

Storage path of Whisper files

By default all Whisper files will be saved to:  /var/lib/graphite/whisper

The used storage schema is: /var/lib/graphite/whisper/<prefix>/<$HOSTNAME$>/<$SERVICEDESC$>/<$METRIC$>.wsp


You can use the openITCOCKPIt debugging tool to replace UUIDs with host and service names.

ls -la /var/lib/graphite/whisper/openitcockpit/ | oitc debug -s

Configuration changes

If required, you can adjust the configuration of the Graphing Backend via the openITCOCKPIT interface.

Navigate to Configuration → Config file editor and select the docker-compose.yml of the Graphing Backend.

In the next step, you can adjust all important settings of the Graphing Backend like storage path, number of Carbon-Cache instances and so on.

General overview


Use shipped Graphite and Grafana Server (recommended and default)

openITCOCKPIT is shipped with multiple Docker containers as explained above.

By default openITCOCKPIT will use the shipped Docker containers to store performance data.


In this section, the documentation describes the default values for this configuration.


Store performance data to shipped Graphite

Navigate to Configuration → Config file editor and select the Statusengine.php configuration file.

Configuration keyDefault valueDescription
graphite_port2003Carbon cache server address (Exposed LINE_RECEIVER_PORT of the Carbon-C-Relay)
graphite_address127.0.0.1

Carbon cache server address (Exposed address of the Carbon-C-Relay)

graphite_prefixopenitcockpitPrefix added to every metric stored in carbon (Metric: <prefix>.<$HOSTNAME$>.<$SERVICEDESC$>.<$METRIC$>)


Read performance data from shipped Graphite (openITCOCKPIT itself)

Navigate to Configuration → Config file editor and select the graphite.php configuration file.

Configuration keyDefault valueDescription
graphite_web_host127.0.0.1Address of the Graphite-Web server openITCOCKPIT use to query data from Whisper files
graphite_prefixopenitcockpitPrefix added to every metric stored in carbon (Metric: <prefix>.<$HOSTNAME$>.<$SERVICEDESC$>.<$METRIC$>)
graphite_web_port8888Port of the Graphite-Web server.
use_httpsNoIf HTTPS should be used to establish connection. Not required for local running Graphite-Web instance
use_proxyNoUse given proxy server from Configuration → Proxy Settings. Not required for local running Graphite-Web instance


In the next step, open the perfdatabackend.php configuration file and make sure, that the selected backend is "Whisper"



Configure shipped Grafana to use shipped Graphite

Navigate to Configuration → Grafana

Configuration valueDefault valueDescription
Grafana URLgrafana.dockerThe address of the Grafana server. grafana.docker will be matched automatically to the local shipped Grafana
Grafana API KeyRandom generatedStored in the file /etc/openitcockpit/grafana/api_key. Please use the value out of this file.
Grafana PrefixopenitcockpitPrefix added to every metric stored in carbon (Metric: <prefix>.<$HOSTNAME$>.<$SERVICEDESC$>.<$METRIC$>)
Use HTTPSYesIf HTTPS should be used to establish connection.
Use ProxyNoUse given proxy server from Configuration → Proxy Settings.
Ignore SSL CertificateYesAlso trust self-signed certificates.
Dashboard StylelightUse the light or dark style of Grafana
HostgroupsemptyIf this value is set Grafana-Dashboards will be generated for all hosts in given hostgroup. (If empty, all hosts will get an generated dashboard)
Hostgroups (excluded)emptyDo not generate Grafana-Dashboards for hosts in given hostgroups.


Use external Graphite and Grafana Server (expert)

To split up system load across multiple nodes, you can also use external systems to store performance data.


Store performance data in external Graphite

Navigate to Configuration → Config file editor and select the Statusengine.php configuration file.

Configuration keyValueDescription
graphite_port20032003 is the default LINE_RECEIVER_PORT of Graphite
graphite_addressxxx.xxx.xxx.xxx

Hostname or IP-Address of the external Graphite server you want to use

graphite_prefixopenitcockpitPrefix added to every metric stored in carbon (Metric: <prefix>.<$HOSTNAME$>.<$SERVICEDESC$>.<$METRIC$>)


Read performance data from external (openITCOCKPIT itself)

Navigate to Configuration → Config file editor and select the graphite.php configuration file.

Configuration keyValueDescription
graphite_web_hostxxx.xxx.xxx.xxxHostname or IP-Address of the external Graphite-Web server
graphite_prefixopenitcockpitPrefix added to every metric stored in carbon (Metric: <prefix>.<$HOSTNAME$>.<$SERVICEDESC$>.<$METRIC$>)
graphite_web_port80Port of the external Graphite-Web server.
use_httpsYes/NoIf HTTPS should be used to establish connection.
use_proxyYes/NoUse given proxy server from Configuration → Proxy Settings.


In the next step, open the perfdatabackend.php configuration file and make sure, that the selected backend is "Whisper"



Configure external Grafana to use external Graphite

Navigate to Configuration → Grafana

Configuration valueValueDescription
Grafana URLxxx.xxx.xxx.xxxThe address of the external Grafana server.
Grafana API Key

You need to create an Grafana API key with "Editor" permissions


Grafana Prefixopenitcockpit

Prefix added to every metric stored in carbon (Metric: <prefix>.<$HOSTNAME$>.<$SERVICEDESC$>.<$METRIC$>)

Use HTTPSYes/NoIf HTTPS should be used to establish connection.
Use ProxyYes/NoUse given proxy server from Configuration → Proxy Settings.
Ignore SSL CertificateYes/NoAlso trust self-signed certificates.
Dashboard Stylelight/darkUse the light or dark style of Grafana
HostgroupsemptyIf this value is set Grafana-Dashboards will be generated for all hosts in given hostgroup. (If empty, all hosts will get an generated dashboard)
Hostgroups (excluded)emptyDo not generate Grafana-Dashboards for hosts in given hostgroups.


Migrate form RRDTool to Whisper

Since openITCOCKPIT 3.6 performance data (used for charts) get stored to Whisper (Carbon).

Notice: There is no automatically migration when updating from an older version of openITCOCKPIT to version 3.6 or newer. For this reason openITCOCKPIT will show empty charts until new data was collected.


To keep the migration from RRDTool to Whisper as simple as possible, we provide an own tool: rrd2whisper.


Download and Installation of rrd2whisper

We provide dedicated packages of rrd2whisper for every supported distribution on the official download page.

Internet connection required

To install rrd2whisper an working internet connection is required. If you need to use an HTTP-Proxy to connect to the outside world, please configure it first.

cd /tmp

wget https://github.com/it-novum/rrd2whisper/releases/download/1.0/rrd2whisper_$(lsb_release -c -s)_1.0.tar.gz
tar xfv rrd2whisper_$(lsb_release -c -s)_1.0.tar.gz

cp -r rrd2whisper /opt/

rrd2whisper is now installed to the path /opt/rrd2whisper.

Running migration

The migration process will read all .rrd files from /opt/openitc/nagios/share/perfdata and save them as .wsp files in /var/lib/graphite/whisper.

Please make sure you have enough free disk space available before you start the migration process. Usually Whisper only needs the half of the disk space which RRDtool used.


Test of the migration

Before you start the migration for all files, you should do a test run where only one rrd file is going to be processed.

Test the migration
/opt/rrd2whisper/bin/console rrdfinder --graphite-host=127.0.0.1 --limit=1 | tee /var/log/rrd2whisper.log

By passing the parameter --limit=1 only one file will be processed. You should see a similar output on your terminal:


In the given example the service with the UUID b123beaf-0575-4ec4-b353-328df2fba0e3 got migrated. In the next step, open the openITCOCKPIT interface and search for the UUID. Make sure the chart shows the migrated performance data.



Running the migration

After a successful test, the migration of all performance data can be started.

Depending of the amount of performance data that needs to be migrated, the migration process could take several hours. The migration process will not impact your monitoring or collection of new performance data.

Migration using default parameters
/opt/rrd2whisper/bin/console rrdfinder --graphite-host=127.0.0.1 | tee /var/log/rrd2whisper.log

The output of the migration will be logged to /var/log/rrd2whisper.log.



To speed up the migration process, you can pass the option --high-mem. This option should only be used, if your system has more than 32GB memory! By passing the option --delete-rrd-files the migration will automatically delete migrated .rrd files.

Danger

Passing the option --delete-rrd-files could cause data loss in case of an error!

Migration using --high-mem and --delete-rrd-files
/opt/rrd2whisper/bin/console rrdfinder --graphite-host=127.0.0.1 --high-mem --delete-rrd-files | tee /var/log/rrd2whisper.log


Use --help to see all options.

/opt/rrd2whisper/bin/console rrdfinder --help

In case of error

In case the migration process crashes or any other unexpected error occurred, the migration process can be restarted. Every successfully processed .rrd file will be marked with an .ok file, so no records will be migrated twice. 

[16:02]root@graphing/tmp# ls /opt/openitc/nagios/share/perfdata/abfb63a8-5187-46bf-a6f1-5120fcac66b6/*
/opt/openitc/nagios/share/perfdata/abfb63a8-5187-46bf-a6f1-5120fcac66b6/279b7b4a-4812-4bc5-a251-cec81e11a705.ok
/opt/openitc/nagios/share/perfdata/abfb63a8-5187-46bf-a6f1-5120fcac66b6/279b7b4a-4812-4bc5-a251-cec81e11a705.rrd
/opt/openitc/nagios/share/perfdata/abfb63a8-5187-46bf-a6f1-5120fcac66b6/279b7b4a-4812-4bc5-a251-cec81e11a705.xml
/opt/openitc/nagios/share/perfdata/abfb63a8-5187-46bf-a6f1-5120fcac66b6/27e67fe9-16fd-4ce2-90d1-442e1e7a1164.ok
/opt/openitc/nagios/share/perfdata/abfb63a8-5187-46bf-a6f1-5120fcac66b6/27e67fe9-16fd-4ce2-90d1-442e1e7a1164.rrd
/opt/openitc/nagios/share/perfdata/abfb63a8-5187-46bf-a6f1-5120fcac66b6/27e67fe9-16fd-4ce2-90d1-442e1e7a1164.xml
/opt/openitc/nagios/share/perfdata/abfb63a8-5187-46bf-a6f1-5120fcac66b6/30a267ae-05a4-432a-9066-15d14eb6a6e8.ok
/opt/openitc/nagios/share/perfdata/abfb63a8-5187-46bf-a6f1-5120fcac66b6/30a267ae-05a4-432a-9066-15d14eb6a6e8.rrd
/opt/openitc/nagios/share/perfdata/abfb63a8-5187-46bf-a6f1-5120fcac66b6/30a267ae-05a4-432a-9066-15d14eb6a6e8.xm



openITCOCKPIT versions <= 3.5.x

In openITCOCKPIT 3.5 and previous versions RRDTool was used to store performance data to.

All .rrd files where located at /opt/openitc/nagios/share/perfdata/$HOSTNAME$/$SERVICEDESC$.rrd.

The performance data get processd by Statusengine 2.x. No additional configuration in nagios.cfg required.

  • Keine Stichwörter