Jephe Wu - http://linuxtechres.blogspot.com
Objective: Use TLS-encrypted TCP connection for Linux client and normal UDP port 514 for Windows client to send all syslog to centralized rsyslog RHEL5 server
Environment: Windows 2003 server with Snare for Windows(freware) installed. CentOS5 servers for Linux rsyslog server and clients.
Steps:
1. CentOS 5 rsyslog rpm is quite old and without SSL support, download rsyslog latest stable version and compile (version 5.4.0), don't use 5.2.0 which is quite buggy.
rsyslog compliation needs mysql and gnutls devel packages
yum install gnutls*
yum install mysql*
then
compile rsyslog:
./configure --enable-mysql --enable-gnutls --enable-mail
make
make install
Note:
1) when compiling rsyslog 5.4.0 stable version on RHEL5.1, I encountered this issue:
runtime/glbl.c:134: undefined reference to `ATOMIC_STORE_1_TO_INT'
then it is solved this way:
a) In the "configure" file, I simply added the "-march=i686" option in bold
...
elif test $ac_cv_prog_cc_g = yes; then
if test "$GCC" = yes; then
CFLAGS="-g -O2 -march=i686"
else
CFLAGS="-g"
fi
else
...
b) I executed ./configure again, then ran make clean and make, and I passed this error.
2. Configure rsyslogd on the CentOS5 server
2.1) configure TLS encryption
Refer to http://www.rsyslog.com/doc-rsyslog_tls.html and http://www.rsyslog.com/doc/rsyslog_secure_tls.html to generate CA and individual peer certificates, put them under /etc/rsyslog/ (create folder first)
2.2) /etc/rsyslog.conf
# chkconfig syslog off
# service syslog stop
put /usr/local/sbin/rsyslogd into /etc/rc.local
then configure /etc/rsylog.conf as follows:
==============
# for Snare
$EscapeControlCharactersOnReceive off
#load modules
$ModLoad immark # provides --MARK-- message capability
$ModLoad imudp # provides UDP syslog reception
$ModLoad imtcp # provides TCP syslog reception
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
$ModLoad ommail # direct email sending
# discard some messages
:msg, regex, ".*Failure Audit.*Account Logon.*Logon attempt by: MICROSOFT_AUTHEN
TICATION_PACKAGE_V1.*eporeplro" ~
# email sending
$ModLoad ommail
$ActionMailSMTPServer mailrelay.domain.com
$ActionMailFrom rsyslog@domain.com
$ActionMailTo jephe.wu@domain.com
$template mailSubject,"Rsyslog Alert for %hostname%"
$template mailBody,"%msg%"
$ActionMailSubject mailSubject
$ActionExecOnlyOnceEveryInterval 60
:msg, regex, ".*Audit.*Account Management.*" :ommail:;mailBody
$ActionMailSMTPServer mailrelay.domain.com
$ActionMailFrom rsyslog@domain.com
$ActionMailTo jephe.wu@domain.com
$template mailSubject,"Rsyslog Alert for %hostname%"
$template mailBody,"%msg%"
$ActionMailSubject mailSubject
$ActionExecOnlyOnceEveryInterval 60
:msg, regex, ".*Failure Audit.*Account Logon.*Logon attempt by: MICROSOFT_AUTHEN
TICATION_PACKAGE_V1.*" :ommail:;mailBody
# note: fore each :msg, regex, command, you have to use a complete set of mail setting first.
# make gtls driver the default
$DefaultNetstreamDriver gtls
# certificate files
$DefaultNetstreamDriverCAFile /etc/rsyslog/ca.pem
# note: this ca.pem should be distributed to all clients for tls connection, ca.pem is only needed for rsyslog tcp client, not the other cert.pem and key.pem files.
$DefaultNetstreamDriverCertFile /etc/rsyslog/cert.pem
$DefaultNetstreamDriverKeyFile /etc/rsyslog/key.pem
$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
$InputTCPServerStreamDriverAuthMode anon # client is NOT authenticated
$InputTCPServerRun 10514 # start up listener at port 10514
$UDPServerRun 514
$AllowedSender UDP, 10.0.0.1,10.0.0.2 # permit only Snare from Windows server, so normal Linux syslog client which runs at udp port 514 cannot connect to it insecurely.
# create directories automatically and give permission
$CreateDirs on
$DirCreateMode 0755
# templates
$template DailyPerHostLogs,"/var/log/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%.log"
*.* -?DailyPerHostLogs
# Other usages examples
# $template 1,"/var/log/rsyslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%-maillog.log"
# if $source == 'app2' and $syslogfacility-text == 'mail' then -?1
--------
note:
a. optionally, you can also use allowedsender tcp such as this
$AllowedSender TCP, 127.0.0.1, 10.0.0.0/24
to only allow certain or some hosts to connect to rsyslog server through tcp protocol.
Refer to rsyslog.conf configuration directive for $allowedsender at
http://www.rsyslog.com/doc/rsconf1_allowedsender.html
b. If a host is disallowed to connect to rsylog server, on the rsyslog server itself, the log will be showing as follows:
TCP message from disallowed sender jephe.domain.com discarded
====================
3. configure rsyslog on Linux as syslog client (Sequence is important for remote message logging when rsyslog server is shut down)
note: you can refer to http://www.rsyslog.com/doc/rsyslog_secure_tls.html to configure the mutual authentication for real secure.
3.1) stop syslog
# chkconfig syslog off
# service syslog stop
put /usr/local/sbin/rsyslogd into /etc/rc.local
3.2) /etc/rsyslog.conf
# certificate files - just CA for a client
$DefaultNetstreamDriverCAFile /etc/rsyslog/ca.pem
# set up the action (put SSL first before *.* message forwarding)
$DefaultNetstreamDriver gtls # use gtls netstream driver
$ActionSendStreamDriverMode 1 # require TLS for the connection
$ActionSendStreamDriverAuthMode anon # server is NOT authenticated
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
# check website: http://www.rsyslog.com/doc-rsyslog_reliable_forwarding.html
# this reliable log forwarding function is different with RELP protocol which is more reliable
$CreateDirs on
$DirCreateMode 0755
$ModLoad imuxsock # local message reception
$WorkDirectory /var/log/rsyslog # default location for work (spool) files (create this folder first)
$ActionQueueType LinkedList # run asynchronously
$ActionQueueFileName rsyslog # set file name, also enables disk mode
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionResumeRetryCount -1 # infinite retries if host is down
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
*.* @@log1.jephe.com:10514 # send (all) messages
#note:
#The rsyslog queueing subsystem tries to buffer to memory. So even if the remote server goes #offline, no disk file is generated. File on disk are created only if there is need to, for example if rsyslog #runs out of (configured) memory queue space or needs to shutdown (and thus persist yet unsent messages). #Using main memory and going to the disk when needed is a huge performance benefit. You do not need to #care about it, because, all of it is handled automatically and transparently by rsyslog.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
note:
a. If you are not using yum to install rsyslog, you should also add rsyslog to /etc/logrotate.d/syslog . that file should look similar to this, otherwise, once the /var/log/secure etc files are rotated weekly, the log will not go to that file anymore if someone tried to ssh and failed.
/var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron {
sharedscripts
postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
/bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> /dev/null || true
endscript
}
Leaving the syslog line in that file won't hurt anything.b. you can put the following into /etc/rc.local
/usr/local/sbin/rsyslogd -c4 -f /etc/rsyslog.conf
4) Install Snare for Windows on Windows 2003 servers
use snare for windows with rsyslog :
$EscapeControlCharactersOnReceive off
note: see http://wiki.rsyslog.com/index.php/Using_Snare_as_a_client_on_Windows
5 Testing and debug
5.1) run 'logger testing' on Linux command line to test
5.2) check log1 server /var/log/messages or /var/log/rsyslog/YEAR/MONTH/DAY/logfilename to see if there's any error after manually running /usr/local/sbin/rsyslog
6 Other usage examples
6.1)A solution to rotate logs on a daily basis is to use dynamic files and after a while rotate the log files away. Quick sample:
/etc/rsyslog.conf:
#####################################################
# Log everything to a per host daily logfile #
#####################################################
$template DailyPerHostLogs,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%.log"
*.* -?DailyPerHostLogs
/etc/cron.hourly/syslog-bzip2:
# Compress *.log-files not changed in more than 24 hours:
find /var/log/syslog/2008 -type f -mtime +1 -name "*.log" -exec bzip2 '{}' \;
6.2) Other examples
$template tplSiteID,"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%,siteID,%msg%"
*.* @@centralsrv.example.net;tplSiteID
$template 1,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%-maillog.log"
if $source == 'hpay1' and $syslogfacility-text == 'mail' then -?1
:msg, regex, ".*Failure Audit.*Account Logon.*Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1.*" -?1
7. References:
a. Replace sysklogd with rsyslog as the default in Fedora: - http://fedoraproject.org/wiki/Releases/FeatureRsyslog
b. You might want to take a look at Octopussy: Perl/XML Logs Analyzer at
http://sourceforge.net/projects/syslog-analyzer/files/
c. debug rsyslog - rsyslogd -c4 -dn (interactive mode)
d. rsyslogd -v (to show which are the things compiled in, SSL?)
e. http://www.homershut.net/wiki/rsyslogd%20client/server for an working example
g. http://www.rsyslog.com/doc/rsconf1_actionexeconlywhenpreviousissuspended.html for fallback
*.* @@primary-syslog.example.com
$ActionExecOnlyWhenPreviousIsSuspended on
& @@secondary-1-syslog.example.com # & is used to have more than one action for
& @@secondary-2-syslog.example.com # the same selector - the mult-action feature
& /var/log/localbuffer
$ActionExecOnlyWhenPreviousIsSuspended off # to re-set it for the next selector
================
Added by Jephe on 30 Oct 2012
h. example of /etc/rsyslog.conf for rsyslog server:
#If set to on (the default), rsyslog emits message on startup and shutdown
#as well as when it is HUPed. This information might be needed by some log analyzers. If set to off, no such #status messages are logged, what may be useful for other scenarios. [available since 4.7.0 and 5.3.0]
# Log anything (except mail) of level info or higher. # Don't log private authentication messages! # *.info;mail.none;authpriv.none;cron.none /var/log/messages
# info is level 6
LOG LEVELS 
7 debug 6 info 5 notice 4 warning, warn 3 error, err 2 critical, crit 1 alert 0 emergency, emerg
debug is the highest and emergency is the lowest level.debug is the highest level because it basically says to log every peep.$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
*.info;mail.none;authpriv.none;cron.none /var/log/messages
authpriv.* /var/log/secure
mail.* -/var/log/maillog
cron.* /var/log/cron
*.emerg *
uucp,news.crit /var/log/spooler
local7.* /var/log/boot.log
# missingok is required if any of above files might not exist such as /var/log/rsyslog/*/spooler, otherwise, it won't work.
Posts
