How to compile mod_proxy module additionally while the Apache httpd is running

Jephe Wu - http://linuxtechres.blogspot.com

Environment: CentOS 4.4 32bit web server,  Apache web server is running within chroot environment, it's also Cognos web gateway server.
Objective: we need to enable mod_proxy module to use reverse proxy method  instead of using web gateway method to make httpd to take to Cognos server on application network segmen. (As somehow the cognos web gateway is not able to take to Cognos server anymore unless using reverse proxy method)

Tried to use apxs to compile mod_proxy.c and mod_proxy_http.c, it's not working.


Steps:
1.  find out the apache httpd version first
cd /usr/local/chroot/usr/local/apache2
bin/httpd -V


2. find out the original compiling options
cd /usr/local/chroot/usr/local/apache2/build
cat config.nice

[root@web1 build]# more config.nice
#! /bin/sh
#
# Created by configure

"./configure" \
"--prefix=/usr/local/apache2" \
"--enable-mods-shared=all" \
"--enable-ssl=shared" \
"--enable-rewrite=shared" \
"--enable-headers=shared" \
"--enable-expires=shared" \
"--enable-cern-meta=shared" \
"--enable-unique-id=shared" \
"--enable-mime-magic=shared" \
"--with-ssl=/usr/local/ssl" \
"$@"

3. find where the original Apache httpd installation files are, then compile mod_proxy.

"./configure" "--prefix=/usr/local/apache2_test" "--enable-mods-shared=all" "--enable-ssl=shared" "--enable-rewrite=shared" "--enable-proxy=shared" "--enable-headers=shared" "--enable-expires=shared" "--enable-cern-meta=shared" "--enable-unique-id=shared" "--enable-mime-magic=shared" "--with-ssl=/usr/local/ssl" "$@"

use different installation prefix directory, then run 'make;make install' to install everything into /usr/local/apache2_test.

4. copy necessary mod_proxy modules into /usr/local/chroot/usr/local/apache2/modules/

cd /usr/local/chroot/usr/local/apache2/modules/
cp /usr/local/apache2_test/modules/mod_proxy.so .
cp /usr/local/apache2_test/modules/mod_proxy_http.so .

5. modify /usr/local/chroot/usr/local/apache2/conf/httpd.conf to enable mod_proxy

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
#LoadModule proxy_connect_module modules/mod_proxy_connect.so
#LoadModule proxy_ftp_module modules/mod_proxy_ftp.so


6. configure virutalhost to enable mod_proxy directives

    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
    RewriteRule .* - [F]

rewriterule ^/reporting/(.*) http://10.0.0.1/cognos8/$1 [P,L]
rewriterule ^/cognos8/(.*) http://10.0.0.1/cognos8/$1 [P,L]


7. use USR1 to kill Apache parent process
ps -efH to find out the parent process id
then kill -USR1 thatid
ps -efH to check again.

Troubleshooting Apache/PHP 500 error

Jephe Wu - http://linuxtechres.blogspot.com

Problem: apache access log shows 500 error code whenever accessing a PHP page which should be prompted to download a docx file.
Environment: CentOS 5.6 64bit, Apache httpd 2.2.3, PHP 5.2.10 from centos test repository


Troubleshooting steps:

1. When problem happens, browser doesn't show anything, only apache access log files shows 500 error code which indicates some sort of issues.

2. Since it's php page, we need to enable error reporting in /etc/php.ini(centos) or /etc/php5/apache2/php.ini(ubuntu). The following is for enabling error reporting.

error_reporting  =  E_ALL & ~E_NOTICE
display_errors = On
display_startup_errors = Off
log_errors = On

This way, we can see some kind of error showing on the screen when we access that php page. Here is the error showing on the screen after enabling that:

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 71 bytes) in /var/www/xxxx/xxxxxx.php on line 297.

3. check memory limit in /etc/php.ini

root@reports01:/etc/ # cat php.ini  | grep memory
memory_limit = 256M; Maximum amount of memory a script may consume (256)

Actually, 67108864 bytes is exactly 64M, which is not 256M that indicated in php.ini.

4. find out where 64M is defined in the php code

root@reports01:/var/www/docgen# grep 64M * -R
lib/header.php:ini_set('memory_limit', '64M');


5. remove that line in php code, problem solved.

References:
1. how to use php 5.2 for CentOS 5.6?
http://wiki.centos.org/HowTos/PHP_5.1_To_5.2

2. PHP 5.2 on Ubuntu 10.04 Lucid Lynx
http://randyfay.com/node/63

How to setup compression for high volume zabbix server

Jephe Wu - http://linuxtechres.blogspot.com 

Environment: CentOS 5 64bit, zabbix server with very large items.
Objective: make 'latest data' with group and host set to 'all' page loading faster.


Steps:
1. make sure deflate_module is enabled in apache configuration

LoadModule deflate_module modules/mod_deflate.so

2. filtered by type
for example:

<Location />
    AddOutputFilterByType DEFLATE text/html text/plain text/xml text/x-js text/css
</Location>

Note: you can use Lynx or similar to do a headers dump on the file and it will tell you want the mime type is,

e.g.:

$ lynx -head -dump http://www.example.com/js/jquery-1.2.6.min.js
HTTP/1.1 200 OK
Date: Wed, 25 Jun 2008 03:25:53 GMT
Server: Apache/2.2.3 (CentOS)
Last-Modified: Wed, 25 Jun 2008 03:24:47 GMT
ETag: "38016-d9de-37f45dc0"
Accept-Ranges: bytes
Content-Length: 55774
Connection: close
Content-Type: application/x-javascript

<Location />
    AddOutputFilterByType DEFLATE application/x-javascript
</Location>
--------------------
3. example settings
root@zabbixserver01:/var/log/httpd/ # more /etc/httpd/conf.d/deflate.conf
<IfModule mod_deflate.c>
        AddOutputFilterByType DEFLATE text/plain
        AddOutputFilterByType DEFLATE text/html
        AddOutputFilterByType DEFLATE text/xml
        AddOutputFilterByType DEFLATE text/css
        AddOutputFilterByType DEFLATE application/xml
        AddOutputFilterByType DEFLATE application/xhtml+xml
        AddOutputFilterByType DEFLATE application/rss+xml
        AddOutputFilterByType DEFLATE application/javascript
        AddOutputFilterByType DEFLATE application/x-javascript
        DeflateCompressionLevel 9
        BrowserMatch ^Mozilla/4 gzip-only-text/html
        BrowserMatch ^Mozilla/4\.0[678] no-gzip
        BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

    # Don't compress images
    SetEnvIfNoCase Request_URI \
    \.(?:gif|jpe?g|png)$ no-gzip dont-vary

    # Make sure proxies don't deliver the wrong content
    Header append Vary User-Agent env=!dont-vary
       
    DeflateFilterNote Input instream
        DeflateFilterNote Output outstream
        DeflateFilterNote Ratio ratio
        LogFormat '"%r" %{outstream}n/%{instream}n (%{ratio}n%%)' deflate
    CustomLog logs/deflate_log deflate
</IfModule>

4. References
a. Apache mod_deflate example:
http://httpd.apache.org/docs/2.0/mod/mod_deflate.html

b. http://www.linuxjournal.com/article/6802?page=0,1

c. use php zlib compress by modifying php.ini

PHP with the --with-zlib configure option and then reconfiguring the php.ini file.

Below is what the output buffer method looks like:

output_buffering = On
output_handler = ob_gzhandler
zlib.output_compression = Off

The ZLIB method uses:

output_buffering = Off
output_handler =
zlib.output_compression = On

Upgrading Openssl without touching the existing running Apache httpd

Jephe Wu - http://linuxtechres.blogspot.com

Environment: Linux server, httpd 2.2.15, Openssl 1.0.0
Objective: upgrading Openssl from 1.0.0 to 1.0.0a without touching running Apache configuration


Steps: 
firstly, backup /usr/local/apache2 directory.

1. compile new version of openssl 1.0.0a first
mv /usr/local/ssl /usr/local/ssl.1.0.0
cd ~jephe
cd openssl1.0.0a
./config;make;make test;make install

2. reconfigure apache with the new version of ssl
cd ~jephe
cd httpd-2.2.15
./config.nice
make
 ---------
note:
[root@web1 httpd-2.2.15]# more config.nice
#! /bin/sh
#
# Created by configure

"./configure" \
"--prefix=/usr/local/apache2" \
"--enable-mods-shared=most ssl rewrite deflate headers expires mime-magic unique-id" \
"--with-ssl=/usr/local/ssl" \
"$@"
----------
Now, running the following command to generate new ssl module

/home/jephe/httpd-2.2.15/srclib/apr/libtool --silent --mode=install cp mod_ssl.la /tmp
then copy /tmp/mod_ssl.so to replace the working one /usr/local/apache2/modules/ssl/mod_ssl.so(backup first)

or directly install to /usr/local/apache2/modules/

cd ~jephe/httpd-2.2.15/modules/ssl
../../build/instdso.sh /home/jephe/httpd-2.2.15/srclib/apr/libtool mod_ssl.la /usr/local/apache2/modules/ (backup mod_ssl.so first)

3. restart apache and confirm the version
/usr/local/apache2/bin/apachectl graceful
less /usr/local/apache2/logs/error_log

4. References
http://httpd.apache.org/docs/2.2/programs/apxs.html
http://httpd.apache.org/docs/2.2/dso.html

Install Apache 2.2 with OpenSSL and Tomcat connector

Jephe Wu - http://linuxtechres.blogspot.com

Objective: Install Apache, Openssl and Tomcat connector with the latest version to address vulnerabilities
Environment: CentOS 4.4, use chroot configuration (/chroot)


Steps:
1. download the latest softwares from individual websites:
httpd-2.2.15
openssl 1.0.0a
tomcat connector 1.2.30

2. install openssl 1.0.0a
if you have installed previous version of openssl, rename /usr/local/ssl to /usr/local/ssl.version
cd /usr/local; mv ssl ssl.0.9.8d
cd openssl-1.0.0a
./config;make;make test;make install
note: all the files will be written to /usr/local/ssl directory.

3. install httpd 2.2.15
"./configure" \
"--prefix=/usr/local/apache2" \
"--enable-mods-shared=most ssl rewrite deflate headers expires mime-magic unique-id" \
"--with-ssl=/usr/local/ssl" \
"$@"

make;make install

note: Some basic Apache configuration in httpd.conf
LoadModule authz_host_module modules/mod_authz_host.so 

#note: without above, you will get error below when you run 'bin/apachectl configtest' under /usr/local/apache2 directory
#Syntax error on line 167 of /usr/local/apache2/conf/httpd.conf:
#Invalid command 'Order', perhaps misspelled or defined by a module not included in the server configuration

LoadModule deflate_module modules/mod_deflate.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule env_module modules/mod_env.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule mime_module modules/mod_mime.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
Include conf/extra/httpd-default.conf

4.  tomcat connector installation
http://tomcat.apache.org/connectors-doc/

cd /home/jephe/apache2215/tomcat-connectors-1.2.30-src/native
./configure --with-apxs=/usr/local/apache2/bin/apxs
make
cd /home/jephe/apache2215/tomcat-connectors-1.2.30-src/native/apache-2.0
cp mod_jk.so /usr/local/apache2/modules/
add the following line to /usr/local/apache2/conf/httpd.conf LoadModule directive

LoadModule jk_module modules/mod_jk.so

note:
1. you can find the building instruction at http://tomcat.apache.org/connectors-doc/webserver_howto/apache.html

5. Hardening apache
a. change conf/extra/httpd-default.conf
ServerSignature Off
ServerTokens Prod
MaxKeepAliveRequests 10000
TraceEnable off  => disable trace method

b. disable SSL2 and weak ciphers

SSLCipherSuite ALL:!ADH:!EXPORT56:!RC4:+HIGH:+MEDIUM:!LOW:!SSLv2:!EXP:!eNULL

c. put into chroot environment
new method: use apache 2.2. builtin chroot 
Since Apache 2.2.10,  it supports chroot, all you need to do is just put chrootdir /path in the main configuration file httpd.conf. You don't have to create anything under /path

startup Apache with command /usr/local/apache2/bin/apachectl start

Advantage: simplify chroot configuration, Apache 2.2.10 above only, it works with cronolog.
Disadvantage: it doens't work with cognos web gateway, got 'internal server error'. Also, when I tried to use Redirect index.html https://domain/path/to  in Apache virutal host configuration, it doesn't work, I have to change it to RewriteRule ^/$ https://domain/path/to to make this kind of redirection work.

old method: user manual chroot settings
search google.com for 'chroot apache', some page is at
http://www.faqs.org/docs/securing/chap29sec254.html
http://www.linux.com/archive/articles/36331

change /etc/init.d/apache starting script for $HTTPD
to  /usr/sbin/chroot /chroot $HTTPD
note: we use /chroot as chroot directory

[root@web1 conf]# ls /chroot
dev  etc  lib  opt  usr  var

note: according to my test, when you use manual chroot, if you use cronolog , you will hit  the problem, apache cannot find the path of the cronolog, so cannot start up Apache


Advantage: Apache 2.2 doesn't work with this kind of chroot if you are using cronolog, has not tried cognos web gateway, I don't think it works either, because when I was testing cronolog, it cannot find the program path
apache 2.0 is working with both cronolog and cognos web gateway
Disadvantage:  configuration is a bit difficult

6. testing Apache configuration
a.  syntax test
cd /usr/local/apache2
bin/apachectl configtest

b. Qualys SSL server test
https://www.ssllabs.com/ssldb/index.html

7. solution for Apache 2.0, chroot and cognos web gateway, cronolog
Using tranditional chroot, Apache 2.0.63, openssl 0.9.8o version.
If you using openssl 1.0.0.a version, apache 2.0.63 cannot compile successfully.

References:
http://httpd.apache.org/docs/2.2/programs/apxs.html
http://httpd.apache.org/docs/2.2/dso.html

How to check openssl renegotiation and weak cipher vulnerability

Jephe Wu - http://linuxtechres.blogspot.com
Environment: Apache httpd server with openssl
Objective: check renegotiation and weak cipher vulnerability and patch them


Steps:
1. How to check if a website supports openssl renegotiation and weak cipher?
method 1: openssl s_client command
use openssl command that comes with CentOS 5.5:

openssl s_client -connect jephe.domain.com:443
it will show 'secure renegotiation is NOT supported or supported' message

use openssl ciphers to know all the ciphers on the client Linux PC, then you can use the following commands to check specific cipher support on server:
openssl s_client -connect jephe.domain.com:443 -cipher LOW:EXP  - check if it suports low or exp ciphers, for what are the low or exp ciphers, see  http://www.openssl.org/docs/apps/ciphers.html#
                                                                 
openssl s_client -connect jephe.domain.com:443 -cipher EXP-RC4-MD5 - check specific cipher

openssl s_client -connect jephe.domain.com:443 -cihper MEDIUM


method 2: use public ssl database report
https://www.ssllabs.com/ssldb/index.htm


method3: use downloaded tool
download sslciphercheck from http://www.woany.co.uk/downloads/
then use sslciphercheck -h jephe.domain.com to check all supported ciphpers



2. How to patch it
For renegotiation vulnerability, you can upgrade to openssl 1.0a version.
For weak cipher, you can use the  following ciphersuite configuration in Apache

SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:!LOW:!SSLv2:!EXP:!eNULL

You can disable RC4 also, so it becomes:



SSLCipherSuite ALL:!ADH:!EXPORT56:!RC4:+HIGH:+MEDIUM:!LOW:!SSLv2:!EXP:!eNULL


note:

1. you can list all ciphers the current openssl supports on the server:


/usr/local/ssl/bin/openssl ciphers | sed -e 's#:#\n#g' | sort

3. References
http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2009-3555
redhat knowledge base: https://access.redhat.com/kb/docs/DOC-20491

                                                      

How to disable TRACE method for Apache

Jephe Wu - http://linuxtechres.blogspot.com

Environment: Apache web server
Objective: Disable trace/track method


Steps:
1. How to test to confirm if a website supports trace/track
 a. for port 80

TRACE / HTTP/1.1
 Host: jephe.domain.com
 [CR]

If you see positive response which is 200 code, that means it supports trace. 

If it's 403 Forbidden, it means it doesn't support.

or if you don't have telnet, you can use nc command

nc jephe.domain.com 80
TRACE / HTTP/1.1
Host: jephe.domain.com 

or

telnet server/ipaddr 80

OPTIONS * HTTP/1.0  => capital letter.

if you saw 'TRACE' appear in allowed methods, then trace method is allow, although 

you can disable the echo using rewrite rule:

TRACE and OPTIONS

The OPTIONS method can be used by a client to determine which methods are allowed. Even when TRACE is disabled using the mod_rewrite method above, the OPTIONS response will report that TRACE is enabled. However, TRACE will be rejected with a 403 error before it is processed, and the potentially harmful behavior of TRACE — echoing input data to the response — won't be allowed.

b. for port 443

openssl s_client -connect jephe.domain.com:443

.....

TRACE / HTTP/1.1
 Host: jephe.domain.com
 [CR]

or 

openssl s_client -connect jephe.domain.com:443

....

OPTIONS * HTTP/1.0

[CR]

[CR]

Allow: GET,HEAD,POST,OPTIONS,TRACE 

2. How to disable it?

method 1: TraceEnable off (Available in Apache 1.3.34, 2.0.55 and later), 

put it in httpd.conf main configuration file

simpler, more direct, and requires less overhead than using mod_rewrite.

method 2:  mod_rewrite in every virtual host

# Block TRACE/TRACK XSS vector
 RewriteEngine On
 RewriteCond %{REQUEST_METHOD} ^TRAC(E|K)
 RewriteRule .* - [F]

3. why we should disable trace method

The HTTP TRACE method asks a web server to echo the contents of the request back to the client for debugging purposes.
the complete request, including HTTP headers, is returned in the entity-body of a TRACE response. Using features that provide client-side HTTP protocol support, such as XMLHTTP ActiveX or XMLDOM scripting objects, a web site can cause browsers to issue TRACE requests. The site can read the TRACE response, including sensitive header information such as cookies or authentication data.

4. References

a. Web servers enable HTTP TRACE method by default - http://www.kb.cert.org/vuls/id/867593

b. Secure Apache TRACE Vulnerabilities - http://cobaltfaqs.com/index.php/Secure_Apache_TRACE_Vulnerabilities

c. http://httpd.apache.org/docs/2.2/mod/core.html#traceenable

How to setup Apache httpd 2.2 with openssl and tomcat

Objective: Use the latest httpd 2.2.X and openssl, mod_jk(tomcat connector) to setup a web server with tomcat service.


Steps:
1. download the latest openssl and follow the document to compile and make install

2. use the following configure parameters for httpd
./configure --prefix=/usr/local/apache2 --enable-mods-shared=most ssl rewrite deflate headers expires --with-ssl=/usr/local/ssl
make;make install

note: after that, you can comment out all unnecessary modules from httpd.conf.

3. follow document of mod_jk and use apxs to generate the module file then copy to apache modules directory

FAQ:
1.  when startup apache in chroot envirnment /chroot, got the following error in error.log
caught SIGTERM, shutting down
Digest: generating secret for digest authentication ...
no such file or directory: Digest: error generating secret: no such file or directory

note: you can just simply disable digest module auth_digest_module

2.  When startup httpd, got the following error:
syntax error on line xxx of /usr/local/apache2/conf/httpd.conf
Invalid command 'Order', perhaps misspelled or defined by a module not included in the server configuration

Note: must enable module authz_host_module, in httpd 2.2.X, the module is replacing the old module mod_access which is required by order command in httpd.conf

3. How to know what configuration command I used before for compiling httpd
Find the config.nice file from httpd source directory.

How to tune Apache performance and harden it

• understand How Apache works first

the main process is responsible to accept the new incoming connections and pass it to "worker" process, once the worker finishs its job which is reading user's request and send back response. it tells the main process it has done the job and waiting for the new connection

• Apache and keepalive issue - reduce keepalive value to 2 seconds to allow Apache to terminate the connection soon enough, also
  

Imagine there're 1000 users connecting to your website at the same moment, how do you handle it? by default, Apache has setting for keepalive 15 seconds and timeout 300 seconds(5 minutes).

If there's no keepalive, each connection will only serve one file which needs to be downloaded for browser display, this is not efficient method since normally a website page contains a few files in order to be properly displayed.

You can reduce the KeepAlive timeout to 2 seconds (do not turn if off). 2 seconds is enough to let the client has the enouchg time to request all the files needed for a page display, also without having to open more than one connection, also let Apache to terminate the connection soon enough so that it can handle more clients then normal.

• use a worker threaded MPM

use a work thread MPM will use less memory and increase the connection it will be able to handle.

• Some other tips

Disable ExtendedStatus
HostnameLookups off
Setting Options -Indexes FollowSymLinks (so Apache doesn't have to check if the file is symbolic link or not, it just follow)
reasonable connection timeout value than 300 (use 30 seconds or less to avoid those modem user hog the connection)

• use thttpd for static files

You can farm all your static files like images to thttpd , make sure it supports keep alive.

• Apache is the only service on the web server

You can tune StartServers, MinSpareServers, MaxSpareServers, MaxClients and MaxRequestsPerChild parameters accordingly if you are running 3-tier archtecture such as Apache, Java-enabled middle server and database backend.

You can have a big StartServers value and MinSpareServer(0) and MaxSpareServers value the same as StartServers value.

If you trust your application, you can set MaxRequestPerClient a very big value.

• hardening

user apache 2.2 builtin chroot (chrootdir /path)
use 'TraceEnable Off' to disable trace/track in main httpd.conf
use ServerTokens Prod