How to Monitor Your Rails/Passenger App with Munin
Munin is a great tool to monitor resources on your server, showing graphs over time, so that you can analyze resource trends to find what’s killing your server before it causes major problems. It is also very configurable and can be made to profile and graph just about anything via plugins. And with a couple tricks, you can get it to monitor your Phusion Passenger application with ease.
Update: If you have RVM installed on your server and need Munin to work with RVM’s Passenger installation, follow all of the instructions below, and then make the changes described in Monitor Passenger with Munin when using RVM.*
* These changes will still use your server’s non-RVM-installed default Ruby to run the Munin plugin, which will in turn use your RVM-installed Ruby to run the passenger-status and passenger-memory-stats commands.
If you already have Munin installed and working, and just want to know how to get the Passenger Plugins working, you can skip directly to Install Munin Passenger Plugins, Configure Munin for Passenger Stats, and be sure to read the Gotcha.
Install Munin and Munin-node
The first step is to install Munin. If you have your server running Ubuntu, this is pretty easy. One you’ve SSH’d into your server, enter:
If you’re running another flavor of Linux, see the Munin’s Linux install instructions. For Mac OSX, see Mac install instructions.
Install Munin Passenger Plugins
The next step is to install the Passenger plugins. The first is passenger_status:
sudo mv 20319.txt /usr/share/munin/plugins/passenger_status
sudo chmod a+x /usr/share/munin/plugins/passenger_status
sudo ln -s /usr/share/munin/plugins/passenger_status /etc/munin/plugins/passenger_status
The second plugin is passenger_memory_stats:
sudo mv 21391.txt /usr/share/munin/plugins/passenger_memory_stats
sudo chmod a+x /usr/share/munin/plugins/passenger_memory_stats
sudo ln -s /usr/share/munin/plugins/passenger_memory_stats /etc/munin/plugins/passenger_memory_stats
No go ahead and restart Munin-node (this is the process that runs Munin at regular intervals):
Configure Munin
Now here are where those couple of tricks come in to get Munin playing nicely with your Rails application. First we want to tell Munin where to store the html and graph images that you can access through the browser.
And change the following lines:
[yoursite.com]
Note that the htmldir can really be any directory, but make sure it’s persistent (i.e. if you’re using Capistrano to keep revisions of your app on the server, make sure the munin directory is in the shared directory outside of your rails app root directory).
Also note that the [yoursite.com] part is only a descriptive name, so it really doesn’t matter what you call it. If you have a more complex application that runs from multiple directories or multiple servers, you can group Munin stats, and in this case, you actually have to put some thought into this line. But that’s outside the scope of this article, so you can read up on customizing Munin Master on your own time if you’d like.
Now if you haven’t already, you need to actually create that directory you just told Munin about:
mkdir -p munin
sudo chown munin:munin munin
Configure Munin For Passenger Stats
And finally, you need to allow the munin user to run the passenger-status and passenger-memory-stats commands without a password, since they both require sudo powers to run properly.
And at the bottom of the file add:
Gotcha
At this point, Munin is suppose to start doing it’s stuff and all is happy in the ruby-munin marriage. For me, however, this was not the case. After combing the Munin error logs and digging through the Munin documentation and code more than I care to admit, I realized that Munin-node needs to preface the passenger stat commands with ruby. So, here’s how we fix that:
And then add this to the bottom of that file:
user munin
command ruby %c
Final Munin Restart
Now we’ll give Munin-node one more restart and we’re in business.
After waiting a few minutes, you should start to see .html files and graphs and whatnot in the /path/to/your/rails/shared/directory/munin directory. If not, you want to check out the Munin-node error logs to see what’s going on. On Ubuntu, this is found at /var/log/munin/munin-node.log.
View Munin From Your Browser
So now, Munin is running, but what good is it if you can’t view the pretty output? Typically the best way to do this is through a sub-domain. If you have yoursite.com, let’s set up Munin to be viewable from munin.yoursite.com. All you need to do is point to the /path/to/your/rails/shared/directory/munin directory for munin.yoursite.com in your site’s server conf file. For example, if you’re on Apache, you’d just do this in your sites-available/yoursite.conf
ServerName munin.yoursite.com
DocumentRoot /path/to/your/rails/shared/directory/munin
</VirtualHost>
And make sure you have the proper A-name record setup for munin.yoursite.com with your NameServers.
Once domain propogation magic happens, you should now be able to see this when you navigate to munin.yoursite.com
Munin main page
The Passenger stats will be in the “App” section.
Have fun now monitoring your Passenger Rails application with passenger usage trending. Remember to use your newfound power for good.
(irqstats patch)
If you find that the irqstats graph doesn’t work, you may need to patch the irqstats plugin. Basically you just need to sudo nano /etc/munin/plugins/irqstats and change
to
if (($irq =~ /^\d+$/) || ($irq =~/^[A-Z]{3}$/)) {
Subscribe to the feed
Follow @alfajango on Twitter
Like Alfa Jango on Facebook
[...] getting started, we highly recommend employing Munin (instructions for Munin with Passenger here) and NewRelic Performance Monitoring for your Passenger/Rails application, so that you can [...]
[...] Jango Blog – has a couple articles on monitoring and tuning [...]
[...] Happy caching! For more tips for getting the best performance out of your server, check out Performance Tuning for Phusion Passenger (An Introduction), and How to Monitor Your Rails/Passenger App with Munin. [...]
Hi,
Thanks for the tips!
Dmitry
Thanks for this, exactly what I wanted to do.
I did have a slight issue when importing directly onto RedHat…
It seems that the original editor used to create 21391.txt and 20319.txt added the ^M character onto the end of the lines and when I executed the files, it gave an ‘error 256′ inside the munin-node.log and spit out a bad interpreter error to stdout.
After a bit of googling, I found out that if you open the files in vim with a ‘-b’ argument, it’ll show you the ^M on the end of each line, which you can then delete as you wish.
All in all, great plugin, thanks for this!
@Graham: Awesome, I’m glad it helped!
Concerning the ^M characters, that happens because it sounds like your RedHat distribution downloaded the text file with it’s original DOS encoding, which handles newlines in text files differently than Unix encodings. You should check out the dos2unix utility so that you don’t have to manually get rid of all of the ^M characters.
Awesome post, worked like a charm! Thank you.
This is post is great! I modified the passenger_memory_stats gist a bit to report also global memory stats. So we have an integrated view of the memory and how passenger relates to that.
The modified version is also a gist: http://gist.github.com/562483
For people using collectd there is a handy bash script to do a similar thing:
https://gist.github.com/493581
Where do the ‘/usr/bin’ executables come from in the sudo section?
@EH: Those executables are installed when you install Phusion Passenger. They’re command-line utilities that output the current status and memory stats for Passenger. The Munin plugins simply parse and pipe their output to Munin.
For more info, check out the Analysis and System Maintenance section of the Passenger Users Guide.
Hi Steve,
Great post! I have munin all set up bar the passenger stats(no app section). When I tail munin-node.log I get the following error https://gist.github.com/786200. Any thoughts??
Damian
Hey Damian, I’m not sure. It sounds like Munin is trying to execute ruby, in the passenger status script, but that it can’t find ruby in your path. It looks like your munin-node user cannot find ruby on your machine. Maybe it’s not in the munin-node user’s or global path. I dunno, try running the munin passenger script manually from your terminal, like this:
# Should display something like this:
# => max.value 2
# => running.value 0
# => active.value 0
# => sessions.value 0
I had the problem that I got
: No such file or directory
strace revealed that
execve(“/bin/ruby\r”, ["ruby\r", "./passenger_memory_stats"], [/* 19 vars */]) = -1 ENOENT (No such file or directory)
- there are windows newlines in it. Running dos2unix on them made it work.
why didn’t I read comments before losing an effing hour?
windows line terminators sucks.
What fixed it for me was
since previously they were rwxrwx–x (not world readable)
I am unable to download 20319.txt file. I am Getting download error
Please update new download link.
Steve – I followed all the steps here. The one I got stuck on was
in /etc/munin/plugin-conf.d/munin-node
[passenger_*]
user munin
command ruby %c
The ruby command had to be replaced with the rvm wrapper /usr/local/rvm/wrappers/ruby-1.9.2-p290/ruby
however I still dont see the app section on my munin page. I see [ apache disk munin network processes system ] but no app section and hence no passenger stats. No errors in the munin log either. Any tips on how to proceed with debugging ?
Thanks Steve,
I followed your instructions to install the Munin passenger plugin on my VPS, with couple of chagnes to run ruby plugin with Single-User RVM install.
I created a Gist to help others setup with Single-User RVM – https://gist.github.com/1538846
Hi Steve,
Great post! It helped me though I still had troubles to fix.
Thanks!