This article will show you how to create basic environment for monitoring your distributed application in one place using Graphite and Grafana. Recently I did somehow similar configuration on CentOS Linux so I will show you how to make first step into world of monitoring in this particular OS (version 7.2 actually ;))

What we want to achieve?

Basically we want to create environment where any number of applications (hosted via NGINX in this example, but it could be anything) are reporting to single Graphite. Then we want Grafana for creating some nice dashboards that will give us just in time information what’s happening within our applications.

 

Of course this could be much more complicated – Nginx can be used only as LoadBalancers for multiple instances for some application modules. Then we want to both – load balancers and apps to tell our graphite about their real load. We will try to cover more complicated solutions in future.

Why Graphite and Grafana?

Basically because it’s somehow standard these days in IT. Grafana is just a platform that will give you nice dashboards.

Of course you can use tool such as Zabbix but this is a bit more different approach. First of all you have to provide more configuration info in zabbix to give you desired charts. Think about this solution that we want to create as a push notification system to Graphite. And everything – no matter how many instances of specific applications you have will show immediately when application start. I’m not familiar if zabbix can show you that many options on one dashboard and adopt to various numbers of data providers. Imagine auto-scalable system where number of instances of particular module can vary.

For Graphite instead of default database (Whisper).

Can I use other linux distro?

Yes you can.

Graphite – installation

Please use all commands with administrative privileges.

Before you start please update your distribution and install EPEL repository:

Now we can install httpd server (Yes Apache – I don’t mind which server will serve grafana and graphite and this is most common solution so you will get more results while searching for any troubleshooting). Also in this step we will install graphite and MariaDB – our MySQL server.

If you don’t want to use MariaDB but different database or just default one you don’t have to install all mysql/mariadb related packages.

Within next steps we will start and enable http server as well as myslq server:

Now we should create database for graphite (please change passwords!):

Graphite – configuration

Lets start to configure our graphite installation 🙂 We will use mostly default configuration so let’s copy default example configuration files:

Now we should configure graphite to use our database. Please edit /etc/graphite-web/local_settings.py  and add our database configuration:

Now we can generate database schema:

Last step is to set some file permissions:

Now we can reconfigure Apache to serve our graphite! Please note that I serve graphite on port 8080.

Last step is to add port 8080 to httpd configuration so Apache will listen on that port. Please add port 8080 so you will have in /etc/httpd/conf/httpd.conf two ports configured:

At the end restart apache and you should have Graphite installed and configured.

Grafana – installation

This one is easy 🙂

And if we want to serve this also via httpd:

Done.

You should now see at http port 80 grafana. Please add graphite as default source. After installation there is only one user: admin with password admin

Security

Also – In my configuration graphite is accessible only from localhost so I don’t add additional security to this part of monitoring environment. All modules have to send data via tcp port 2003 which is accessible internal only. Only one access point for whole monitoring solution is Grafana – it have simple user management so you can use it out of the box. If you need to send data to graphite via internet please read this article before you start.

NGINX Monitoring

Next step is to provide simple but efficient nginx monitoring. First you need to install nginx, collectd and collectd plugin for nginx:

We will use Stub Status Module to give status information from NGINX then we will use collectd to send data over network to installed in earlier steps graphite. On some environments Stub Status Module is already compiled into nginx. If its not please use same steps as in Requirement section on this page.

Now we need to add information about endpoint that will serve our status information. Please add to /etc/nginx/nginx.conf  this code (in server section):

Please note that this endpoint will be accessible only from local machine. If you want to check if its ok. Use wget -q -O - "$@" http://localhost/nginx_status to get information about current nginx status.

Now we have to configure collectd. Everything is in /etc/collectd.conf  file. We want to enable nginx plugin and write_graphite plugin:

Restart nginx:

At this point  you will be able to create and add graphs. In next article in this series I will show you how to configure Grafana to adapt to multiple instances of nginx and show reasonable information just in time.