Beaver ~ A Light Logstash Shipper

The ELK stack has long been a useful tool for sysadmins but apache error logs have the awful habit of requiring either shipped to a local file or to syslog. They don’t support piping to a program like their access_log counterparts. I’d tried the syslog route but trying to discern who shipped the log and search for it was tricky at best. I’d been looking around for sometime for a decent one with many available out there settled on beaver for its simplicity and ease of installation.

System Config Ubuntu 14.04 Ubuntu system python

Install You can install it by hand following the steps below or you can install it using the saltstack formula over at https://github.com/joshboon/beaver-formula

Logstash Config

# Config logstash
# in your logstash config file
# in the input section
 tcp {
 port => 57084
 type => "beaver"
 }

# in the filter section
if [type] == "beaver" {
 json{
 source => message
 }
 }
# restart logstash
service logstash restart

Manual Install

pip install beaver==36.2.0
mkdir -p /etc/beaver/conf.d
echo "[beaver]
format: json
# required for logstash, set to 0 if logstash < 1.2
logstash_version: 1
queue_timeout: 60
# lots of transport options, pick your poison http://python-beaver.readthedocs.io/en/latest/user/usage.html
transport: tcp
tcp_host: $LOGHOST
tcp_port: 57084" > /etc/beaver/beaver.conf
# a sample log monitoring config
echo "[/var/log/mail.log]
type: mail-log
tags: mail" > /etc/beaver/conf.d/mail.conf

# start beaver
beaver -c /etc/beaver/beaver.conf -C /etc/beaver/conf.d
# You will need to write an upstart or copy the init.d script from https://github.com/joshboon/beaver-formula/tree/master/beaver/files and configure it

Salt Install

Add github.com/joshboon/beaver-formula as a gitfs endpoint see https://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html 
# For states that you're looking to add log monitoring to
include:
  - beaver
# Add the log file beaver config
/etc/beaver/conf.d/vhost-{{ vhost['servername'] }}.conf:
  file:
    - managed
    - source: salt://apache-vhosts/beaver-error.conf
    - user: root
    - group: root
    - mode: 0644
    - template: jinja
    - backup: minion
    - require: 
      - file: /etc/beaver/conf.d
    - context:
      servername: {{ vhost['servername'] }}
      apacheenv: {{ pillar['apacheenv'] }}
# add the beaver file template
cat apache-vhosts/beaver-error.conf 
[/var/www/vhosts/{{servername}}/statistics/logs/error_log]
type: apache-error
tags: {{apacheenv}},{{servername}}
# add the pillar data
beaver:
  lookup:
    global:
      transport: tcp
      tcp_port: 57084
      tcp_host: $LOGHOST
      logstash_version: 1
      output: json
# once that is in place you should be able to deploy the config and see beaver installed and configured for the logging expected.