How to configure MongoDB to log to PaperTrailApp on Amazon Linux

In this post we will configure MongoDB to log to PaperTrailApp

What is MongoDB?

MongoDB is a document database that provides high performance, high availability, and easy scalability.

What is PaperTrailApp

PaperTrailApp is a hosted log management service for servers, apps, and cloud services

You will need to create an account with PaperTrail

Settings up MongoDB to log to PaperTrailApp

Access the MongoDB instances using SSH and run this command

sudo yum -y install gcc ruby-devel rubygems gcc-c++ openssl-devel
sudo gem install remote_syslog
sudo gem install eventmachine
sudo vi /etc/log_files.yml

Update the file with the below configuration

files:
  - /var/log/mongo/mongod.log
destination:
  host: logs.papertrailapp.com
  port: XXXXX   # Papertrail dest port (default: 514)
exclude_patterns: # exclude noise generated by over zealous C# driver prior to v1.8 connection recycling
  - connection accepted
  - end connection

Where XXXXX is the port number provided to you by PaperTrailApp

Start the logging service

Configure auto start of logging service

wget https://raw.github.com/papertrail/remote_syslog/master/examples/remote_syslog.init.d

The file needs editing to force logging to use the full hostname and change the startup priority.

By default PaperTrailApp will reduce a hostname of

mongodb.eu-west-1a.XXX_XXX_XXX_XXX

to just

mongodb

This means all the logs from all MongoDB instances will appear to come from the same source

Apr 24 01:13:09 mongodb mongod.log:  Wed Apr 24 00:13:09.188 [conn505716] query

So add this argument to the end of the process start command

--hostname $HOSTNAME

to add the hostname to the log

...
start(){
    echo -n $"Starting $prog: "

    unset HOME MAIL USER USERNAME
    $prog -c $config --pid-file $pid_file "$EXTRAOPTIONS" --hostname $HOSTNAME
...

So that a log entry look like this

// example log entry
Apr 24 01:13:09 mongodb.eu-west-1b.xxx_xxx_xxx_xxx mongod.log:  Wed Apr 24 00:13:09.188 [conn505716] query ...

The default start up priority occurs before that of

/etc/init.d/cloud-init-user-scripts 

at time of writing this was

chkconfig: 2345 99 99

In order for hostname changes to have completed, alter the start up priority from 90 to 99 (i.e. start remote_syslog after cloud-init-user-scripts):

chkconfig: 345 99 5

And edit cloud-init-user-scripts to guarantee that it runs before remote_syslog

sudo vi /etc/init.d/cloud-init-user-scripts
# chkconfig: 345 98 98

Save and run the following to realise chkconfig priorty edits.

chkconfig cloud-init-user-scripts resetpriorities
sudo mv remote_syslog.init.d /etc/init.d/remote_syslog
sudo chown root:root /etc/init.d/remote_syslog
sudo chmod u+x /etc/init.d/remote_syslog
sudo chkconfig --add remote_syslog
sudo service remote_syslog start

Configure rsyslog

Configure General syslog Messages

sudo vi /etc/rsyslog.conf 

Add this as the first non-comment line

# Required to force logs to use their full hostname
$PreserveFQDN on

Add this to the very end of the file:

# VoucherCloud Paper Trail config
*.*                                         @logs.papertrailapp.com:XXXXX

Where XXXXX is the port provided by PaperTrailApp

Save and quit vi

Restart rsyslog

sudo service rsyslog restart

Further Reading

Original instructions are below, but the steps that are ultimately run are very different so this link is just for reference



Comments

No comments yet.

Add Yours

  • Author Avatar

    YOU


Comment Arrow




About Author

Robert

Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning hands down.