NOTE: This document is only relevant for CentOS 4.0-4.5.
Problems with Apache 2.0 and 2.2 mod_ssl on Certain Systems
Certain older versions of OpenSSL contain a bug that causes Apache to fail while starting on systems that have a large number of VirtualHosts, at least one of which is used for SSL access. RedHat and CentOS 4.6+ and 5.x have been patched for this issue. If your system is running one of those distributions and is up to date, you should not encounter this bug. If you are running a different distribution and encounter segmentation faults when starting up Apache 2.0 or 2.2, you may need to contact your Linux distribution vendor and inquire about this problem.
Apache 1.3 is not affected by this issue.
Detailed Analysis
The core problem is that versions of OpenSSL before 0.9.7k and 0.9.8c have a fault that appears when certain functions are used after FD_SETSIZE (generally 1024) file descriptors have been opened:
http://marc.info/?l=openssl-dev&m=114301777619250&w=2
With Apache 1.3, initialization of mod_ssl occurs before httpd.conf is processed. With Apache 2.0 and 2.2, most initialization of mod_ssl occurs after httpd.conf has been processed, meaning that all the VirtualHost logfiles will be open when ssl_init_Module() is executed. The functions in OpenSSL affected by this issue at present are:
RAND_status(), called by ssl_init_Module() -> ssl_rand_seed()
RSA_generate_key(), called by ssl_init_Module() -> ssl_tmp_keys_init() -> ssl_tmp_key_init_rsa()
It does not appear that there are any faults once initialization of mod_ssl is completed on Apache 2.0 and 2.2. Starting Apache with a trimmed down httpd.conf file, copying the complete httpd.conf file over it and then restarting Apache does not cause a segfault because ssl_init_Module() is not executed again.
How Do You Know Whether An Install of Apache 2.x Suffers From This Problem?
A quick way of checking for this specific problem is by capturing the output of httpd startup via the strace program. The OpenSSL bug manifests at the end of the output as:
- Reading from
/dev/urandom
- Closing
/dev/urandom
- Get the current time
- Seg Fault
A more positive identification can be done by disabling all SSL Vhosts in httpd.conf and restarting Apache. If Apache restarts without error, then the problem lies with mod_ssl/OpenSSL.
To pinpoint the exact function call causing the segmentation fault, Apache must be recompiled with debugging symbols so that a usable backtrace can be created with GDB. In order to recompile apache with debugging symbols, you must simply set the appropriate CFLAGS and then run EasyApache like so:
export CFLAGS="-g"
/scripts/easyapache –build
How Can This Bug Be Fixed?
Unfortunately, the fix provided by Red Hat is only in CentOS/RHEL 5 and Fedora at this time. Older versions are left with the following options:
- Update Your Operating System If Possible - Granted this is no trivial task but then you’ll have the latest openssl without this openssl bug.
- Use Only Apache 1.3 - Tests with Apache 1.3 show it is not susceptible to this bug, due to the way it performs some module initialization before processing httpd.conf, while Apache 2.x does the module initialization after httpd.conf processing.
- Patch Apache 2.x - At this time, cPanel patches Apache 2.0 and 2.2
mod_ssl to avoid the call to RAND_status() for older versions of OpenSSL. On some systems, this is sufficient to avoid any problems. Reordering the call to RSA_generate_key() is more involved since that function must not run before other initialization functions have executed. It is possible that other function calls during mod_ssl’s initialization also cause segmentation faults under some circumstances.
After applying a patch, if you choose that option, please be sure to test Apache again to see if it is still suffering from the issue.
NOTE: It was previously recommended to use /opt/openssl to work around this issue however that method does not ensure that you have the latest patched version of OpenSSL as /opt/openssl is not updated by either EasyApache or the system package manager. This can be a security concern and requires you to manually track and update the openssl installation in /opt/openssl. In addition, when using the /opt/openssl workaround, multiple copies of libssl can be/are loaded into memory. When a function is called by a program looking to interact with libssl, there may be two (potentially different) copies of that function loaded into memory. There’s no real way to determine which function is being called and therefore troubleshooting issues with a /opt/openssl implementation becomes increasingly difficult if not impossible. For these reasons, we do not recommend using /opt/openssl to work around this bug.
Tags:
Apache,
centos,
cPanel,
EasyApache,
redhat,
rhel,
ssl
Related posts
EasyApache Apache, centos, cPanel, redhat, rhel, ssl