How to enable core dump for application users under RHEL5

Jephe Wu - http://linuxtechres.blogspot.com

Objective: enable core dump for application and users
Environment: RHEL5


Steps:
1. enable it for interactive login users globally
By default, it's disabled in Linux, you can change file /etc/profile.
from
    # No core files by default
    ulimit -S -c 0 > /dev/null 2>&1
to
    ulimit -c unlimited >/dev/null 2>&1

or for individual user, edit your $HOME/.bash_profile.
   
2. for those program started by daemon or services, please add the following into /etc/sysconfig/init
#added by Jephe for enabling core dump for application users
DAEMON_COREFILE_LIMIT='unlimited'

This environment variable will be picked up by /etc/init.d/* daemons/services.

To enable core dump for individual daemon /etc/init.d/abc
add the following into that file /etc/init.d/abc after ". /etc/rc.d/init.d/functions" line - RHEL
DAEMON_COREFILE_LIMIT='unlimited'

or for other distribution
ulimit -c unlimited >/dev/null 2>&1
echo /tmp/core > /proc/sys/kernel/core_pattern



3. specify core dump file location
By default, the core dump file will be generated at program directory, you can change it to /tmp as follows:

echo "/tmp/core" > /proc/sys/kernel/core_pattern

or add it to /etc/sysctl.conf
kernel.core_pattern = /tmp/core
then run 'sysctl -p'

note:
a.it was 'core' in /proc/sys/kernel/core_pattern

b.core dump file will be generated as /tmp/core.$PID, provided kernel.core_uses_pid=1 which is default in RHEL

c. you can also set kernel.core_pattern as
kernel.core_pattern = /tmp/core-%e-%s-%u-%g-%p-%t

    %% - A single % character
    %p - PID of dumped process
    %u - real UID of dumped process
    %g - real GID of dumped process
    %s - number of signal causing dump
    %t - time of dump (seconds since 0:00h, 1 Jan 1970)
    %h - hostname (same as ’nodename’ returned by uname(2))
    %e - executable filename

3.1 for suid program
echo 2 > /proc/sys/fs/suid_dumpable  (for RHEL5)

4. testing
a. ssh as normal user, to run command 'sleep 1000 &'
c. run 'kill -s SIGSEGV $$'.
d. assuming 15043 is the PID, check if file /tmp/core.15043 exists.
e. if successful, logout then ssh again to the server, restart your program

4.1 to revoke core dump settings above
edit /etc/profile
edit /etc/sysconfig/init
echo 0 > /proc/sys/kernel/suid_dumpable
echo core > /proc/sys/kernel/core_pattern

5. References:
a. for setuid program, core dumps are not generated to prevent sensitive information to be leaked.
According to Redhat, to enable it
For Red Hat Enterprise Linux 5: "suidsafe" (recommended) - protect privileged information by
having the core dump be owned by and only readable for root:
echo 2 > /proc/sys/fs/suid_dumpable

For Red Hat Enterprise Linux 5: "debug" (may cause privileged information to be leaked):
echo 1 > /proc/sys/fs/suid_dumpable

For Red Hat Enterprise Linux 4:
echo 2 > /proc/sys/kernel/suid_dumpable
For Red Hat Enterprise Linux 3:
echo 1 > /proc/sys/kernel/core_setuid_ok

to enable them persistent over reboot:
fs.suid_dumpable = 2 # RHEL 5 only
kernel.suid_dumpable = 2 # RHEL 4 only
kernel.core_setuid_ok = 1 # RHEL 3 only
kernel.core_pattern = /tmp/core

b. commands:
ulimit -c
ulimit -a
ulimit -c 2000 (limit core dump to 2000 bytes)
you might want to limit the individual user core dump size through /etc/security/limits.conf
add something like this:

jephe soft core unlimited

c. ssh login and /etc/security/limits.conf consideration
When you set some restriction for users in limits.conf and you login through ssh, then you need to make sure /etc/ssh/sshd_config to have 'usePAM yes'
For more information on ssh and ulimit, please refer to setting bash shell limits for oracle user - http://linuxtechres.blogspot.com/2010/09/setting-bash-shell-limits-for-oracle.html

d. how to read core dump file
gdb program_path corefile_path