Graylog Syslog Grok Extractor

I have several apps in my k8s cluster which run behind a lighttpd web server where I wanted the access logs to be sent to graylog. The setup of lighttpd is relatively basic:

# /etc/lighttpd/lighttpd.conf
var.basedir  = "/var/www/localhost"
var.logdir   = "/var/log/lighttpd"
var.statedir = "/var/lib/lighttpd"
server.modules = (
    "mod_alias",
    "mod_accesslog",
    "mod_magnet"
)
include "mime-types.conf"
server.username      = "lighttpd"
server.groupname     = "lighttpd"
server.port          = 80
server.document-root = var.basedir + "/htdocs"
server.pid-file      = "/run/lighttpd.pid"
server.indexfiles    = ("index.html")
server.errorlog-use-syslog = "enable"
accesslog.use-syslog       = "enable"

Next we need to tell rsyslog to send logs to graylog:

# /etc/rsyslog.d/dxc-graylog.conf
*.* @graylog-udp.graylog.svc.cluster.local:5410;RSYSLOG_SyslogProtocol23Format

Note that you will need to change the URL and port to match the graylog syslog URL and port you have. Also notice that we are using RSYSLOG_SyslogProtocol23Format format. This results in a log message with a message value similar to the following in graylog: 10.33.1.106 10.42.2.9 - [28/Oct/2020:06:28:06 -0400] "GET / HTTP/1.1" 200 6341 "-" "kube-probe/1.16" This should all work; however, we would like to have the message parsed - enter the grok pattern! In Graylog, go to your Syslog Input and click Manage Extractors. Add an Extractor with the following info:

• select extractor type for the message field with a Grok Pattern
• enter grok pattern:

%{SYSLOGHOST:host} %{SYSLOGHOST:referer} - \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}

• Condition: Always try to extract
• Extraction strategy: Copy
• Extractor title: RFC5424
• click the Create extractor That should be all that is needed, go to your syslog stream and check (make sure you pick a message that came in after you created the extractor) to ensure the new fields are being created!