Tuesday 31 July 2012

Remote logging for Apache and Amazon AWS

One of the changes that I have started in my current job has been to move the environment to Amazon Web Services. By and large this suits us well, but one of the issues we do have is that, as instances spin up and down, we lose logs.

My current solution to this is to set up a logging server and route the logs from the web heads to that, if you're not interested in Linux plumbing it's time to go away and read something else.

There are 3 areas that need configuring Apache, the web head's logging system and the logging server. Our systems are running Ubuntu 12.04 LTS (hey, the Rezillos on Last.fm :) ), and that uses rsyslog rather than stock syslog -so YMMV

Apache

This is just an edit of the httpd.conf file (or a file it includes) to repoint ErrorLog and CustomLog. We also set up a new blackbox logging format to help debugging.

ErrorLog "|/usr/bin/logger -p local1.info -t apache2"
LogFormat "%v:%a/0 %X %t \"%r\" %s/%>s %{pid}P/%{tid}P %T/%D %I/%O/%B" blackbox
CustomLog "|/usr/bin/logger -p local1.info -t apache2" blackbox

As you can see this uses a pipe into logger(-p sets the priotity and -t the tag), when you reload your Apache you should find the log messages from it going into the local syslog.

Local Rsyslog


I basically trashed /etc/rsyslog.conf to send everything from local.1 to the logging server, local.1 probably isn't the right channel for this -but it's working.

$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
local1.* @ec2-46-137-82-138.eu-west-1.compute.amazonaws.com


Messages are sent via UDP -as we just want a 'fire and forget' system, if you want to be sure your messages arrive you can tell rsyslog to use TCP

One thing to note is the use of the DNS for the Elastic IP of the server, rather than the IP address itself -this should ensure that traffic is internal to Amazon -and thus free, using the IP address will route it externally, which costs.

Another poi ntis the use of the Elastic IP -rather than just the IP address of the server. This is another AWS funny, servers aren't guranteed to keep their IP if they shutdown and restart, wheras the Elastic IP should always be ther,e using it saves you having to update all your clients if the server IP changes.

Server Rsyslog

Uncomment UDP syslog reception in rsyslog.conf
# provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
 

In /etc/rsyslog.d I created a .conf file containing :


:programname, isequal, "apache2" /var/log/oneGiantHeapOfLogs.log

This tests for the tag name 'apache2' that we set using logger in the apache.conf file, if it finds it the message goes to the big log file.


Job Done.


Debugging

  • you can call logger directly on the commandline with a message and tail the various log files you are looking at to see if the message arrives.
  • On the server 'netstat -nlp' will show you if udp is listening on the logging port (514)
  • On the client you can write directly to the port on the server with netcat -u 10.229.70.230 514
  • Put rsyslog messages into debug mode with *.* /var/log/rsyslog_debug.log;RSYSLOG_DebugFormat early on in rsyslog.conf
Eliza Carthy playing  'Moss Meg' now.

No comments:

Post a Comment

linkedin