Debian

Logstash recipe: Akamai on ELK

One of the perks of working for the new company is the usage of cool tools and providers. One of the new providers is Akamai – state of the art CDN/EdgeCache provider – and also the first one to exist.
The cool new tool: Kibana4!

Just a quick introduction to Kibana: Kibana belongs to the ELK Stack (Elasticsearch, Logstash and Kibana) – and as you spotted correctly comes in last, as it forms the Web/Userinterface to the underlying Elasticsearch database. Logstash sits somewhere in between and is a powerful tool to parse many log formats and also to inject them into Elasticsearch. Elasticsearch itself holds the data and offers a search engine/index.

Why do you need ELK? In a multi-server environment you will want to have your logs somewhere centralized – that is so you do not need to log into each box. Also you want to maybe have some kind of webinterface so you do simple tasks without some commandline-fu – like filtering all failed cronjobs.
There are some great tools that can achieve this as well, like syslog-ng or Graylog.

Wanna see what you are going to get? Here you go:

BTW, yes this is a demo dashboard only, meaning a lot is most probably redundant to Google Analytics – nevertheless it is more exact as it will also capture bots and file requests where no JavaScript is being loaded. The possibilities are of course fare more extensive.

This recipe will take care of three major points:

  • Actual grok filter to match the logs
  • Fix @timestamp to be parsed directly from log-line (as the logs come in batches and often also not in chronological order)
  • Apply GeoIP (via maxmind) filter to ClientIP so we can create cool looking maps on our dashboard

1) First things first

Currently there are two options to receive Logs from Akamai, via FTP and via email. You will want to receive it via FTP so I would suggest to setup a ftp server on your ELK setup.
Akamai will either send the log file gzipped or GPGP encrypted, both formats that Logstash can not read in-house, so you will need to build a script to translate it into plain/text.
Be smarter than me and chose a ftp-daemon that supports uploadscripts, like pure-ftpd or proftpd. It is much easier than using vsftpd + some funky logfile-analyzer-upload-script.

2) Setup Akamai Log Delivery Service (LDS)

  • Log into your Luna Control Center
  • Configure > Log Delivery
  • Select your Object > “Begin Delivery”
  • Make sure you choose “Combined + Cookie + Host Header” as log format – this will give us the possibility to extinguish between different projects later on Kibana

My settings look approx. like this:

3) Use the following Logstash configuration

$ sudo nano /etc/logstash/conf.d/11-akamai-access.conf
input {
file {
path => "/home/logs/incoming/akamai/access_*"
exclude => "*.gz"
type => "akamai-access"
}
}

filter {
if [type] == "akamai-access" {
grok {
match => { "message" => "%{IP:clientip} - - [%{HTTPDATE:timestamp}] %{HOSTNAME:hostname} "%{WORD:verb} /%{HOSTNAME:origin}%{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} %{NUMBER:bytes:int} "(?:%{URI:referrer}|-)" %{QS:agent} %{QS:cookie}" }
}

date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
}
}

if [clientip] {
geoip {
source => "clientip"
target => "geoip"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
}
}
}

4) Thats it!

  • Restart/reload logstash
  • Wait a few for the log files to come in (might take some hours)
  • Explore the data and create some nice visuals!

Some final notes: there are some major advantages (well and also disadvantages) when analyzing logs directly from the CDN/EdgeCache:

  1. You will get the actual Client-IP (you might be able to redirect it through your ELB until down to your EC2 – but that might be hell of a job)
  2. You will get more accurate Data, as in the best scenario your actual webserver will only get hit once a day 😉

One of the disadvantages: you do not (though there might be products for that) get the data in real time.

Compile Lighttpd with LibreSSL

As LibreSSL is gaining popularity you might want to switch your compiled Lighttpd version with one that uses LibreSSL for your https.

Tested on Debian Squeeze, but should work on Wheezy/Ubuntu in a similar way.

Prerequisites

$ sudo apt-get install make gcc libev-dev libpcre3-dev zlib1g-dev libbz2-dev gamin libgamin-dev liblua5.1-0-dev
$ wget
http://download.lighttpd.net/lighttpd/releases-1.4.x/lighttpd-1.4.35.tar.gz
http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.0.5.tar.gz
$ tar xvfz libressl-2.0.5.tar.gz && tar xvfz lighttpd-1.4.35.tar.gz

Compile & Install LibreSSL

We are installing it in a non-standard path so it won’t interfer with your existing openssl/libssl(-dev)

$ cd libressl-2.0.5
$ ./configure --prefix=/opt/libressl
$ make
$ sudo make install

Verify the LibreSSL Installation

$ /opt/libressl/bin/openssl version
LibreSSL 2.0

Compile Lighttpd with LibreSSL

$ cd ../lighttpd-1.4.35
$ wget https://gist.github.com/lifeofguenter/7ef3fe9e089fcb24baed/raw/316108a350f69d622c17d0801cc429388cf36cef/lighttpd-libressl.patch
$ patch -p1 < lighttpd-libressl.patch
$ ./configure
--prefix=/usr
--sysconfdir=/etc
--localstatedir=/var
--with-libev
--with-pcre
--with-zlib
--with-bzip2
--with-fam
--with-lua
--with-openssl=/opt/libressl
$ make
$ sudo make install

Verify the Lighttpd Installation

$ lighttpd -v
lighttpd/1.4.35 (ssl) - a light and fast webserver
Build-Date: Aug 11 2014 12:54:04

Please have a look at the following URLs for further Documentation on configuring Lighttpd + SSL:

Compile tinyproxy as anonymous proxy

Tinyproxy is a small light-weight proxy daemon for Linux environments. Espescially usefull if you have some spare dedicated/virtual servers running with multiple IPs. In this tutorial I will show you how to compile it from source on a Debian server – as the current official .deb package does not allow running it in complete anonymous mode.

Additionally if you don’t want to install asciidoc (1GB!!!) you can use the modified patch from my compile steps (original). My version of the patch works with the current stable (1.8.3) version of tinyproxy. Not using the patch might give you following error:

checking for a2x... no
configure: error: Test for asciidoc failed. See the file 'INSTALL' for help.

 

Compile


$ wget --no-check-certificate https://banu.com/pub/tinyproxy/1.8/tinyproxy-1.8.3.tar.bz2
$ wget --no-check-certificate https://github.com/lifeofguenter/patches/raw/master/tinyproxy/tinyproxy-1.8.3-no-asciidoc.patch
$ tar xvfj tinyproxy-1.8.3.tar.bz2
$ cd tinyproxy-1.8.3/
$ patch -p1 < ../tinyproxy-1.8.3-no-asciidoc.patch
$ ./configure
--prefix=/usr
--sysconfdir=/etc
--localstatedir=/var
--disable-xtinyproxy
--disable-filter
--disable-upstream
--disable-reverse
--enable-transparent
$ make && sudo make install

Config


$ sudo nano /etc/tinyproxy.conf
User nobody
Group nogroup

Port 8888
BindSame yes
Timeout 600
DefaultErrorFile "/usr/share/tinyproxy/default.html"
StatFile "/usr/share/tinyproxy/stats.html"
Syslog On
LogLevel Error
PidFile "/var/run/tinyproxy/tinyproxy.pid"
MaxClients 100
MinSpareServers 5
MaxSpareServers 20
StartServers 10
MaxRequestsPerChild 10000
Allow 127.0.0.1
DisableViaHeader Yes
ConnectPort 443
ConnectPort 563

Run


$ sudo mkdir /var/run/tinyproxy
$ sudo chown nobody:nogroup /var/run/tinyproxy/
$ sudo tinyproxy