Rotating Rails Log Files
One of the chores I dislike most is cleaning up and clearing out the log files in my Rails applications. On some of my higher profile sites, I get a lot of spiders probing my applications for security holes. They don’t succeed (knock on wood) but they do fill up my log files with errors.
I finally decided to get smart and get lazy (the two best traits a programmer can have), and I set up automatic log rotation on all of my Rails applications. The idea behind log rotation is simple: make a back up of the current log file, continue logging into a new or cleared log file, and discard log files that are older than a certain date.
Your webserver probably already rotates its own log files. For Apache, they are probably located in /etc/httpd/logs and they are probably rotated weekly. These logs store everything Apache does. Simple webserver stats and traffic analysis tools make use of these log files to show who visits a site when and what pages are viewed.
While it is possible to configure your Rails application to log to your Apache log files, I do not think it is a good practice. It’s much better to give each Rails application its own log file—it will be easier to find important Rails errors, it will keep your Apache logs cleaner and Rails is set up to keep its own logs by default. Fortunately, on a Linux server the built-in logrotate program will make the process super-easy. After the jump, I’ll walk you through the steps to get it set up.
Configuring logrotate
logrotate gets its configuration information from the file /etc/logrotate.conf. If you can’t find it, you can type locate logrotate.conf from the command line to see where it lives. Open up that file in a text editor (nano /etc/logrotate.conf will work) and append the following lines at the end:
1 2 3 4 5 6 7 8 9 10 | # Rotate Rails application logs /path/to/your/rails/applicaton/log/*.log { daily missingok rotate 7 compress delaycompress notifempty copytruncate } |
Here’s what each line does:
/path/to/your/rails/applicaton/log/*.log - Use the full path to the log directory of your Rails application (symbolic links are acceptable too). “*.log” will rotate any file in the log directory with .log at the end. If you only want to rotate certain log files you can be more specific.
daily - Rotates the log files every day. You could specify weekly or monthly instead.
missingok - Don’t issue an error message if log files are missing.
rotate 7 - The maximum number of log files to keep. Once you have more than this number, the oldest file will be deleted. I set it to keep seven days worth but feel free to change this number.
compress - Compress old versions of log files to save space (uses gzip by default).
delaycompress - Delays the compression until the next log rotation. It’s a minor point and probably not strictly necessary, but it makes sure that the log file is truly no longer active before compressing it.
notifempty - If the log file is empty, there’s no need to rotate it. You can remove this option if you want to rotate even blank log files; just keep in mind that you may erase a log file that has lots of information to make room for your blank log file.
copytruncate - Makes a backup copy of the current log and then clears the log file for continued writing. The alternative is to use create which will perform the rotation by renaming the current file and then creating a new log file with the same name as the old file. I strongly recommend that you use copytruncate unless you know that you need create. The reason why is that Rails, FastCGI, Mongrel, etc. may still keep pointing to the old log file even though its name has changed and they may require restarting to locate the new log file. copytruncate avoids this by keeping the same file as the active file.
If you have more than one Rails application, you can repeat this code to rotate them all one-after-another. There other options you can specify, man logrotate will show you them all. I haven’t used them but the options to mail log files on creation or deletion look interesting. It is also possible to have rotate the logs once they get to a certain size instead of at a certain time.
Using logrotate
To use the log rotation you just configured, you have two choices. 1) Wait for the next day (or whatever time period you specified). If you configured it correctly, rotation should occur automatically and without further commands. 2) Run it immediately by typing /usr/sbin/logrotate -f /etc/logrotate.conf on the command line. (The -f is for “force it to run now.)
That’s all there is to it! Now your log files won’t fill up to an unmanagable size yet you’ll still be able to go back and track any recent errors.
November 22nd, 2007 at 10:32 am
Kevin,
Thanks much for this excellent article. I particularly appreciate the detailed explanations for each configuration statement that goes in logrotate.conf.
The advantage of copytruncate over create is clear as crystal now.
November 24th, 2007 at 2:52 pm
Thanks for the great post! Question: is there any way to specify what time you want the rotation to run? If not, is there a standard time it backs up at? Thanks!
November 26th, 2007 at 1:02 pm
@Ian: It runs as a cron job, so it will run at whatever time your daily, weekly or monthly cron jobs run. AFAIK, you can’t give it a more specific time.
November 29th, 2007 at 4:35 pm
Kevin –
Thanks. I did some more research and discovered that the daily/weekly/monthly option is to specify criteria for backup. So if you have it set to daily and, for some reason, the cron job runs twice, the log(s) will not be rotated.
Of course you can override this behavior with the -f (–force) option.
Thanks for the post!
January 24th, 2008 at 5:04 pm
You can actually do log rotation from within rails by adding the following line to the Rails::Initializer.run block in your app’s /config/environment.rb:
config.logger = Logger.new(config.log_path, 50, 1024**2)
The 2nd argument being the number of .log files you’d like to keep, and the 3rd being the size in bytes that the files are allowed to reach before they’ll be rotated. So this line would keep up to a maximum of 50 1MB files. Not quite as configurable as logrotate perhaps (it doesn’t support compression for example), but it’s nice to be able to keep all of the config within the app.
See here for more:
http://blog.caboo.se/articles/2005/12/15/who-said-size-is-not-important
March 12th, 2008 at 10:39 am
[...] established a standard deployment location of /usr/local/sites/sitename for our apps. Following this advice I think I’ll use standard logrotate to move the logs around. I’ve read that log [...]
July 18th, 2008 at 9:39 am
Thanks for the excellent article! I’ve tried (following different advice from various sources) using the method suggested by Matt, but I can’t get it to work. I will definitely try this method. I feel that in most cases that the job of rotating logs is probably not the job of the application or web server. It’s probably debatable as to where this kind of functionality should sit (as is the question of any domain-specific functionality), but it probably comes down to each individual case.
December 24th, 2008 at 4:44 pm
Thanks for the great article! I had just come to the point in building an application where I knew I’d need to solve the log file problem. Definitely a helpful article. I really appreciated the thorough explanations of each configuration option.
February 17th, 2009 at 6:05 pm
Thanks for the article. If you are doing this on a ubuntu or debian based system make sure that logrotate is actually installed. It’s easy to be fooled since apache and mysql install rotate scripts in /etc/logrotate.d
To install:
sudo apt-get install logrotate
add your new config file to: /etc/logrotate.d
test:
sudo logrotate -d /etc/logrotate.c/my_servers_rotate_script
force a rotate:
sudo logrotate -f /etc/logrotate.c/my_servers_rotate_script
March 3rd, 2009 at 6:25 am
[...] if there are more than 3 of them they get deleted. Maximum filesystem usage: 30mb An alternative is using logrotate which is a harder to setup and maintain, but offers more [...]
April 22nd, 2009 at 9:36 am
The problem with using copytruncate instead of create is that there is the possibility of loosing log data if your process writes to the original file after the copy, but before the truncate. Using create instead of copy truncate solves this problem, but you have to be able to signal your process (restart works) so that it starts writing to the new log file. Here is logrotate file I’m using for production mongrels: http://gist.github.com/99826
April 25th, 2010 at 6:54 pm
This configuration works really well for me on a collection of rails apps; it even restarts Passenger for each app after rotating the logs: http://gist.github.com/378839
June 16th, 2010 at 3:30 am
Thanks for you tip,
I use http://marklunds.com/articles/one/410 , it more easy to implement in rails application.
What do you suggest? Should we use logrotate or this one(I mentioned)?