Jephe Wu - http://linuxtechres.blogspot.com
Objective: to understand how BDB (DB4) log file remove works and automatically remove old logfiles to prevent disk space full
background: Ubuntu 8.04.2 , Berkeley DB 4.2.52, OpenLDAP
Steps:
1. BDB version used by openldap
root@ldap1:~/ # ldd /usr/sbin/slapd
linux-gate.so.1 => (0xb7faa000)
libldap_r-2.4.so.2 => /usr/lib/libldap_r-2.4.so.2 (0xb7f62000)
liblber-2.4.so.2 => /usr/lib/liblber-2.4.so.2 (0xb7f55000)
libdb-4.2.so => /usr/lib/libdb-4.2.so (0xb7e7c000)
root@ldap1:~/ # dpkg -l| grep db
ii console-tools 1:0.2.3dbs-65ubuntu7 Linux console and font utilities
ii dbus 1.1.20-1ubuntu3.2 simple interprocess messaging system
ii libconsole 1:0.2.3dbs-65ubuntu7 Shared libraries for Linux console and font
ii libdb4.2 4.2.52+dfsg-4 Berkeley v4.2 Database Libraries [runtime]
2. log file removal
http://sepp.oetiker.ch/db-4.2.52-mo/ref/transapp/logfile.html
Log files may be removed at any time, as long as:
- the log file is not involved in an active transaction.
- a checkpoint has been written subsequent to the log file's creation.
- the log file is not the only log file in the environment.
method 3:
Call the DB_ENV->set_flags method from the application, with the DB_LOG_AUTOREMOVE (http://sepp.oetiker.ch/db-4.2.52-mo/api_c/env_set_flags.html#DB_LOG_AUTOREMOVE) flag, to remove any log files that are no longer needed on an ongoing basis. With this configuration, Berkeley DB will automatically remove log files, and the application will not have an opportunity to copy the log files to backup media.Two ways to implement checkpoint (besides DB_LOG_AUTOREMOVE flag in DB_CONFIG)
a. checkpoint settings in slapd.conf
checkpoint kbyte min
The checkpoint directive defines the time between checkpoint operations in BDB (the database can only be recovered from the last checkpoint).
The frequency of checkpointing determines the time during which data may be unrecoverable by BDB in the event of a system failure. If NOT using the dbnosync this time could be set to a reasonably long period, say, 10 mins or more, if the dbnosync directive is being used 5 - 15 mins or less if practical. kbytes is the number of kilobytes written to the directory and min is the time in minutes. Whichever occurs first determines the period between checkpoints.
OpenLDAP default is NO CHECKPOINTING - you should always supply a checkpoint directive. See the BDB Chapter 12 Section 15 documentation for more information. This directive may be replaced by using DB_CONFIG file with the txn_checkpoint directive. Examples:
checkpoint 128 15
# check point whenever 128k data bytes written or
# 15 minutes has elapsed whichever occurs first
b. txn_checkpoint in DB_CONFIG file
# more DB_CONFIG
# see more detail config parameter at http://sepp.oetiker.ch/db-4.2.52-mo/ref/env/intro.html
set_flags DB_LOG_AUTOREMOVE
txn_checkpoint 1024 5 0
DB_CONFIG example:
http://www.zytrax.com/books/ldap/ch6/bdb.html#db-config
root@ldap1:/etc/ldap/ # grep checkpoint slapd.conf
# first DIT definition
database bdb
...
# DIT will act as a provider
overlay syncprov
checkpoint 1024 5
syncprov-checkpoint 100 10
3. working configuration example:
The following configuration makes logfile auto remove possible in my testing environment.
- DB_CONFIG (sitting at the same directory as BDB database files)
set_cachesize 0 2097152 0
# default cache size is 256KB
# sets a database cache of 0G(first column) + 2M(2*1024*1024=2097152) and
# do not allows fragmentation
# does NOT replace slapd.conf cachesize
# this is a database parameter, for slapd.conf cachesize paramter, refer to
# http://www.zytrax.com/books/ldap/ch6/bdb.html#cachesize
# see http://sepp.oetiker.ch/db-4.2.52-mo/api_c/env_set_cachesize.html
# The database environment's cache size may also be set using the environment's DB_CONFIG file. The syntax of the entry in that file is a single line with the string "set_cachesize", one or more whitespace characters, and the cache size specified in three parts: the gigabytes of cache, the additional bytes of cache, and the number of caches, also separated by whitespace characters. For example, "set_cachesize 2 524288000 3" would create a 2.5GB logical cache, split between three physical caches. Because the DB_CONFIG file is read when the database environment is opened, it will silently overrule configuration done before that time.
set_lk_max_objects 1500
set_lk_max_locks 1500
set_lk_max_lockers 1500
set_flags DB_LOG_AUTOREMOVE
txn_checkpoint 1024 5 0
# replaces checkpoint in slap.conf
# writes checkpoint if 128K written or every 15 mins
# 0 = no writes - no update
- slapd.conf (put the following checkpoint in the every BDB definition)
4. References:
http://www.openldap.org/doc/admin24/maintenance.html
Administrators can change the size limit of a single log file (by default 10MB), and have old log files removed automatically, by setting up DB environment (see below). The reason Berkeley DB never deletes any log files by default is that the administrator may wish to backup the log files before removal to make database recovery possible even after a catastrophic failure, such as file system corruption.
Log file names are log.XXXXXXXXXX (X is a digit). By default the log files are located in the BDB backend directory. The db_archive tool knows what log files are used in current transactions, and what are not. Administrators can move unused log files to a backup media, and delete them. To have them removed automatically, place set_flags DB_LOG_AUTOREMOVE directive in DB_CONFIG.
Note: If the log files are removed automatically, recovery after a catastrophic failure is likely to be impossible.
http://www.zytrax.com/books/ldap/ch6/bdb.html#samples
cachesize in slapd.conf
Format:
cachesize integer
The cachesize directive defines the number of entries that the LDAP backend will maintain in memory. Do not confuse this directive with the BDB set_cachesize directive - they control different behaviours.
For maximum performance this figure should be as high as practical or as close as practical to the number of records maintained in the directory. The default is 1000. Examples:
cachesize 10000
# LDAP maintains 10,000 entries in memory
See also Performance chapter.
http://www.openldap.org/faq/data/cache/1072.html
Posts
