Linux limits.conf and limits.d

Jephe Wu - http://linuxtechres.blogspot.com

Objective: find out how limits.conf and /etc/security/limits.d/ works

Questions:

1. what order are files under /etc/security/limits.d/ read?

Firstly, read /etc/security/limits.conf file first, then individual files from /etc/security/limits.d directory are read. The order will be
special characters,
numbers in ascending order
uppercase letters
lowercase letters in alphabetical order

If two files have same entry, then the entry read last will take effect.

2.  applicable domain

• The limits.conf and limits.d only applies for the applications that uses PAM service
• And make sure the following line appears in /etc/pam.d/system-auth.

session     required      pam_limits.so

• Make sure there's no sufficient line for session part before pam_limits.so in /etc/pam.d/system-auth

3. ssh and pam_limits
limits.conf and limits.d/* are not working for ssh shell, because by default /etc/ssh/sshd_config doesn't use PAM. Change UsePAM from no to yes to fix the issue.

4. difference between open files setting in limits.conf and the fs.file-max in /proc/sys/fs/file-max
 File descriptor has two types: per session limit (/etc/security/limits.conf or limits.d/*) and system-wide limit (cat /proc/sys/fs/file-max), and nofile setting in limits.conf cannot be set to unlimited.

5.  all configuration files for setting up user nofile
a. # grep nofile /etc/security/limits.conf
b.# grep -r nofile /etc/security/limits.d/
c.# grep ^UsePAM /etc/ssh/sshd_config
d.# grep -r ulimit /etc/bashrc /etc/profile /etc/profile.d/
e.# grep ulimit ~user/.bashrc ~user/.bash_profile

6. soft limit vs hard limit
Normal user or unprivileged process can alter soft limit, range from 0 up to hard limit, also can lower its hard limit irreversibly.

Hard limit can only be changed by root user

How to set soft limit for a application process?

• Use "ulimit -Sn " command to change the soft limits in runtime.
• Directly update the proc files of respective process PID.
• 
  

[root@jephe 1521]# more limits 

Limit                     Soft Limit           Hard Limit           Units     

Max cpu time              unlimited            unlimited            seconds   

Max file size             unlimited            unlimited            bytes     

Max data size             unlimited            unlimited            bytes     

Max stack size            10485760             unlimited            bytes     

Max core file size        0                    unlimited            bytes     

Max resident set          unlimited            unlimited            bytes     

Max processes             14841                14841                processes 

Max open files            1024                 4096                 files     

Max locked memory         65536                65536                bytes     

Max address space         unlimited            unlimited            bytes     

Max file locks            unlimited            unlimited            locks     

Max pending signals       14841                14841                signals   

Max msgqueue size         819200               819200               bytes     

Max nice priority         0                    0                    

Max realtime priority     0                    0                    

Max realtime timeout      unlimited            unlimited            us 

But soft limit 14 can be increased up to hard limit 19 as follows,e.g. we are increasing from 14 to 18

echo -n "Max open files=18:19" > /proc/1521/limits

setting bash shell limits for oracle user

Jephe Wu -  http://linuxtechres.blogspot.com

Objective: understand the concept of ulimit nofile and nproc settings for oracle under ssh environment
Environment: RHEL 5, CentOS 5, Oracle 11g


Concept:
Bash shell can set maximum allowable number of open file descriptors (handles) or the maximum number of processes available to a user.

To see all limits settings for a current user under bash shell, login to CentOS 5, run 'ulimit -a' to check.

1.  Setting file descriptors/handles for entire Linux operating system
The maximum number of file handles denotes the maximum number of open files on a Linux system.
to see the setting , run
cat /proc/sys/fs/file-max

to check the current usage: run
cat /proc/sys/fs/file-nr
1154 133 65536

it shows the total allocated file handles, the number of currently unused file handles, the maximum file handles that can be allocated (also found in /proc/sys/fs/file-max).

To configure it:
# echo 65536 > /proc/sys/fs/file-max
or
# sysctl -w fs.file-max=65536

To make it permanent:
echo "fs.file-max=65536" >> /etc/sysctl.conf

2.  set Maximum Number of Open File Descriptors for the Oracle User:
 There's still per user limit after above file-max. It is not recommend to set hard limit for nofile for the oracle user equal to /proc/sys/fs/file-max, otherwise, once oracle user used up the file handles, the whole system also used up the file handles, so, the system cannot assign any more file handles for login process.

Modify the /etc/security/limits.conf file as root and make it like this: (use 63536 instead of 65536)

oracle soft nofile 63536
oracle hard nofile 63536

In order to make it work, pam_limits should be configured in the /etc/pam.d/
system-auth as follows, or in /etc/pam.d/sshd for ssh, /etc/pam.d/su for su, or /etc/pam.d/
login for local access and telnet

session required pam_limits.so
session required pam_unix.so

Same thing we can use for the number of processes:
oracle soft nproc 16384
oracle hard nproc 16384


3.  How many file descriptors are being used in your Linux system

File Descriptors

File descriptors are allocated dynamically by the kernel for performance reasons. use

sysctl fs.file-nr 

to check all 3 values.

Open Files
lsof | wc -l
8124
This tells you that there are 8124 files by applications on the system. The same file opened by two applications will be counted twice. Normally, this value is bigger than fs.file-max.

lsof lists all open files, including files which are not using file descriptors - such as current working directories, memory mapped library files, and executable text files.

How to check file descriptors and open files for pid 1234:
file descriptor: 

ls -l /proc/1234/fd/

open files:

 lsof | grep 1234

4. FAQ:

a. I tried to setup the soft and hard nofile limits for root. But when I
 tried to ssh as root user, the limits set do not take into effect. Why?

According to redhat knowledge base you can 

specify the nofile ulimit values in /etc/init.d/sshd init script such as ulimit -n 4096 

For other similiar issues, you might need to disable

UsePrivilegeSeparation

Why it happens? some openssh version has problem like below:

Due to the manner in which SSH logins are implemented. 
When a user logs in via SSH, the SSH daemon process forks a separate 
process to handle that specific connection. As such, the context this 
forked process runs in is owned by the user who logged in. Since regular
 users are not allowed to modify their ulimit upwards, the higher value 
specified in /etc/security/limits.conf fails to take effect.

This is because the calling program, i.e. the forked sshd process, 

lacks the permissions to perform the upward modification.