segmentation fault with httpd 2.4.54 #293
Comments
|
There is a known issue since the dawn of time in restart/reload behaviour and watchdog tasks. The problem is that the order in which watchdog tasks and server shut down is not coordinated well enough and OpenSSL shutdown may stumble over its feet. One of the stack traces you show is a from a child exiting. Just to confirm that we are investigating the right thing: can you confirm crashes without the child exiting? |
On opening this issue I thought there are various stack traces that end up in a segfault. Now that I look at it more closely I recognized that there are only the two cases that I have mentioned in my first post. Segfaults that arise during exiting of childs tend to be in the minority. So yes, there are lots of crashes without the child exiting. |
|
The first stack trace shows and that means the lock pointer is NULL which is, afaict not supposed to happen. Openssl has a RUN_ONCE init that allocates locks for the RAND functions to use. Which means either that this did not properly work or that someone wiped memory he was not supposed to. Can you compile the mod_md from 2.4.53 in the 2.4.54 into your test server to check that the problems disappear? That would lay the blame solely at my door... |
|
Hmm, looking at the differences between 2.4.53 and 2.4.54, there was a change in JSON reference handling. Can you try the following patch? |
I tested this out.... the segfaults occur also with mod_md from 2.4.53.
Unfortunately, the patch does not solve the problem. |
|
Well, that means the latest changes are not causing the issues on your system. Have you switched/updated other components? Maybe from OpenSSL 1.1 to 3.0? Trying to circle things down here... |
|
Ok, scanning the code and the report again. Can you produce a log at Since the bug also happens with the 2.4.43 version, this points to a mistake in my handling of libcurl. |
|
Oh, I made a mistake building httpd 2.4.54 with mod_md from 2.4.53... I did it again and I have to revise my statement: indeed, with mod_md 2.4.53 the problem did not exist yet. I'll send you next the logs file via mail... |
|
I'm using 2.4.54 but only with @arminabf thank you for the regression testing! |
|
@whereisaaron I am investigating the issue with @arminabf and so far, he is the only one seeing it. I proposed a fix in the current master that restores the old behaviour in handling |
|
@arminabf confirmed the issue to be fixed in the last patch which I now release as v2.4.19. |
asfgit
pushed a commit
to apache/httpd
that referenced
this issue
Aug 25, 2022
*) mod_md: a new directive `MDStoreLocks` can be used on cluster
setups with a shared file system for `MDStoreDir` to order
activation of renewed certificates when several cluster nodes are
restarted at the same time. Store locks are not enabled by default.
Restored curl_easy cleanup behaviour from v2.4.14 and refactored
the use of curl_multi for OCSP requests to work with that.
Fixes <icing/mod_md#293>.
git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1903677 13f79535-47bb-0310-9956-ffa450edef68
asfgit
pushed a commit
to apache/httpd
that referenced
this issue
Aug 25, 2022
*) mod_md: a new directive `MDStoreLocks` can be used on cluster
setups with a shared file system for `MDStoreDir` to order
activation of renewed certificates when several cluster nodes are
restarted at the same time. Store locks are not enabled by default.
Restored curl_easy cleanup behaviour from v2.4.14 and refactored
the use of curl_multi for OCSP requests to work with that.
Fixes <icing/mod_md#293>.
git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/branches/2.4.x@1903678 13f79535-47bb-0310-9956-ffa450edef68
netbsd-srcmastr
pushed a commit
to NetBSD/pkgsrc
that referenced
this issue
Jan 20, 2023
Changes with Apache 2.4.55 *) SECURITY: CVE-2022-37436: Apache HTTP Server: mod_proxy prior to 2.4.55 allows a backend to trigger HTTP response splitting (cve.mitre.org) Prior to Apache HTTP Server 2.4.55, a malicious backend can cause the response headers to be truncated early, resulting in some headers being incorporated into the response body. If the later headers have any security purpose, they will not be interpreted by the client. Credits: Dimas Fariski Setyawan Putra (@nyxsorcerer) *) SECURITY: CVE-2022-36760: Apache HTTP Server: mod_proxy_ajp Possible request smuggling (cve.mitre.org) Inconsistent Interpretation of HTTP Requests ('HTTP Request Smuggling') vulnerability in mod_proxy_ajp of Apache HTTP Server allows an attacker to smuggle requests to the AJP server it forwards requests to. This issue affects Apache HTTP Server Apache HTTP Server 2.4 version 2.4.54 and prior versions. Credits: ZeddYu_Lu from Qi'anxin Research Institute of Legendsec at Qi'anxin Group *) SECURITY: CVE-2006-20001: mod_dav out of bounds read, or write of zero byte (cve.mitre.org) A carefully crafted If: request header can cause a memory read, or write of a single zero byte, in a pool (heap) memory location beyond the header value sent. This could cause the process to crash. This issue affects Apache HTTP Server 2.4.54 and earlier. *) mod_dav: Open the lock database read-only when possible. *) mod_proxy_http2: apply the standard httpd content type handling to responses from the backend, as other proxy modules do. *) mod_dav: mod_dav overrides dav_fs response on PUT failure. *) mod_proxy_hcheck: Honor worker timeout settings. [Yann Ylavic] *) mod_http2: version 2.0.10 of the module, synchronizing changes with the gitgub version. This is a partial rewrite of how connections and streams are handled. - an APR pollset and pipes (where supported) are used to monitor the main connection and react to IO for request/response handling. This replaces the stuttered timed waits of earlier versions. - H2SerializeHeaders directive still exists, but has no longer an effect. - Clients that seemingly misbehave still get less resources allocated, but ongoing requests are no longer disrupted. - Fixed an issue since 1.15.24 that "Server" headers in proxied requests were overwritten instead of preserved. - A regression in v1.15.24 was fixed that could lead to httpd child processes not being terminated on a graceful reload or when reaching MaxConnectionsPerChild. When unprocessed h2 requests were queued at the time, these could stall. - Improved information displayed in 'server-status' for H2 connections when Extended Status is enabled. Now one can see the last request that IO operations happened on and transferred IO stats are updated as well. - When reaching server limits, such as MaxRequestsPerChild, the HTTP/2 connection send a GOAWAY frame much too early on new connections, leading to invalid protocol state and a client failing the request. The module now initializes the HTTP/2 protocol correctly and allows the client to submit one request before the shutdown via a GOAWAY frame is being announced. - :scheme pseudo-header values, not matching the connection scheme, are forwarded via absolute uris to the http protocol processing to preserve semantics of the request. Checks on combinations of pseudo-headers values/absence have been added as described in RFC 7540. Fixes #230. - A bug that prevented trailers (e.g. HEADER frame at the end) to be generated in certain cases was fixed. See #233 where it prevented gRPC responses to be properly generated. - Request and response header values are automatically stripped of leading and trialing space/tab characters. This is equivalent behaviour to what Apache httpd's http/1.1 parser does. The checks for this in nghttp2 v1.50.0+ are disabled. - Extensive testing in production done by Alessandro Bianchi (@alexskynet) on the v2.0.x versions for stability. Many thanks! *) mod_proxy_http2: fixed #235 by no longer forwarding 'Host:' header when request ':authority' is known. Improved test case that did not catch that the previous 'fix' was incorrect. *) mod_proxy_hcheck: hcmethod now allows for HTTP/1.1 requests using GET11, HEAD11 and/or OPTIONS11. [Jim Jagielski] *) mod_proxy: The AH03408 warning for a forcibly closed backend connection is now logged at INFO level. [Yann Ylavic] *) mod_ssl: When dumping the configuration, the existence of certificate/key files is no longer tested. [Joe Orton] *) mod_authn_core: Add expression support to AuthName and AuthType. [Graham Leggett] *) mod_ssl: when a proxy connection had handled a request using SSL, an error was logged when "SSLProxyEngine" was only configured in the location/proxy section and not the overall server. The connection continued to work, the error log was in error. *) mod_proxy_hcheck: Re-enable workers in standard ERROR state. *) mod_proxy_hcheck: Detect AJP/CPING support correctly. *) mod_http2: Export mod_http2.h as public header. [Stefan Eissing] *) mod_md: a new directive `MDStoreLocks` can be used on cluster setups with a shared file system for `MDStoreDir` to order activation of renewed certificates when several cluster nodes are restarted at the same time. Store locks are not enabled by default. Restored curl_easy cleanup behaviour from v2.4.14 and refactored the use of curl_multi for OCSP requests to work with that. Fixes <icing/mod_md#293>. *) core: Avoid an overflow on large inputs in ap_is_matchexp. *) mod_heartmonitor: Allow "HeartbeatMaxServers 0" to use file based storage instead of slotmem. Needed after setting HeartbeatMaxServers default to the documented value 10 in 2.4.54. *) mod_dav: DAVlockDiscovery option to disable WebDAV lock discovery This is a game changer for performances if client use PROPFIND a lot.
netbsd-srcmastr
pushed a commit
to NetBSD/pkgsrc
that referenced
this issue
Mar 4, 2023
www/apache24: security update
Revisions pulled up:
- www/apache24/Makefile 1.115
- www/apache24/PLIST 1.36
- www/apache24/distinfo 1.54
- www/apache24/patches/patch-configure 1.3
-------------------------------------------------------------------
Module Name: pkgsrc
Committed By: adam
Date: Fri Jan 20 14:03:16 UTC 2023
Modified Files:
pkgsrc/www/apache24: Makefile PLIST distinfo
pkgsrc/www/apache24/patches: patch-configure
Log Message:
apache24: updated to 2.4.55
Changes with Apache 2.4.55
*) SECURITY: CVE-2022-37436: Apache HTTP Server: mod_proxy prior to
2.4.55 allows a backend to trigger HTTP response splitting
(cve.mitre.org)
Prior to Apache HTTP Server 2.4.55, a malicious backend can
cause the response headers to be truncated early, resulting in
some headers being incorporated into the response body. If the
later headers have any security purpose, they will not be
interpreted by the client.
Credits: Dimas Fariski Setyawan Putra (@nyxsorcerer)
*) SECURITY: CVE-2022-36760: Apache HTTP Server: mod_proxy_ajp
Possible request smuggling (cve.mitre.org)
Inconsistent Interpretation of HTTP Requests ('HTTP Request
Smuggling') vulnerability in mod_proxy_ajp of Apache HTTP Server
allows an attacker to smuggle requests to the AJP server it
forwards requests to. This issue affects Apache HTTP Server
Apache HTTP Server 2.4 version 2.4.54 and prior versions.
Credits: ZeddYu_Lu from Qi'anxin Research Institute of Legendsec
at Qi'anxin Group
*) SECURITY: CVE-2006-20001: mod_dav out of bounds read, or write
of zero byte (cve.mitre.org)
A carefully crafted If: request header can cause a memory read,
or write of a single zero byte, in a pool (heap) memory location
beyond the header value sent. This could cause the process to
crash.
This issue affects Apache HTTP Server 2.4.54 and earlier.
*) mod_dav: Open the lock database read-only when possible.
*) mod_proxy_http2: apply the standard httpd content type handling
to responses from the backend, as other proxy modules do.
*) mod_dav: mod_dav overrides dav_fs response on PUT failure.
*) mod_proxy_hcheck: Honor worker timeout settings. [Yann Ylavic]
*) mod_http2: version 2.0.10 of the module, synchronizing changes
with the gitgub version. This is a partial rewrite of how connections
and streams are handled.
- an APR pollset and pipes (where supported) are used to monitor
the main connection and react to IO for request/response handling.
This replaces the stuttered timed waits of earlier versions.
- H2SerializeHeaders directive still exists, but has no longer an effect.
- Clients that seemingly misbehave still get less resources allocated,
but ongoing requests are no longer disrupted.
- Fixed an issue since 1.15.24 that "Server" headers in proxied requests
were overwritten instead of preserved.
- A regression in v1.15.24 was fixed that could lead to httpd child
processes not being terminated on a graceful reload or when reaching
MaxConnectionsPerChild. When unprocessed h2 requests were queued at
the time, these could stall.
- Improved information displayed in 'server-status' for H2 connections when
Extended Status is enabled. Now one can see the last request that IO
operations happened on and transferred IO stats are updated as well.
- When reaching server limits, such as MaxRequestsPerChild, the HTTP/2 connection
send a GOAWAY frame much too early on new connections, leading to invalid
protocol state and a client failing the request.
The module now initializes the HTTP/2 protocol correctly and allows the
client to submit one request before the shutdown via a GOAWAY frame
is being announced.
- :scheme pseudo-header values, not matching the
connection scheme, are forwarded via absolute uris to the
http protocol processing to preserve semantics of the request.
Checks on combinations of pseudo-headers values/absence
have been added as described in RFC 7540. Fixes #230.
- A bug that prevented trailers (e.g. HEADER frame at the end) to be
generated in certain cases was fixed. See #233 where it prevented
gRPC responses to be properly generated.
- Request and response header values are automatically stripped of leading
and trialing space/tab characters. This is equivalent behaviour to what
Apache httpd's http/1.1 parser does.
The checks for this in nghttp2 v1.50.0+ are disabled.
- Extensive testing in production done by Alessandro Bianchi (@alexskynet)
on the v2.0.x versions for stability. Many thanks!
*) mod_proxy_http2: fixed #235 by no longer forwarding 'Host:' header when
request ':authority' is known. Improved test case that did not catch that
the previous 'fix' was incorrect.
*) mod_proxy_hcheck: hcmethod now allows for HTTP/1.1 requests
using GET11, HEAD11 and/or OPTIONS11. [Jim Jagielski]
*) mod_proxy: The AH03408 warning for a forcibly closed backend
connection is now logged at INFO level. [Yann Ylavic]
*) mod_ssl: When dumping the configuration, the existence of
certificate/key files is no longer tested. [Joe Orton]
*) mod_authn_core: Add expression support to AuthName and AuthType.
[Graham Leggett]
*) mod_ssl: when a proxy connection had handled a request using SSL, an
error was logged when "SSLProxyEngine" was only configured in the
location/proxy section and not the overall server. The connection
continued to work, the error log was in error.
*) mod_proxy_hcheck: Re-enable workers in standard ERROR state.
*) mod_proxy_hcheck: Detect AJP/CPING support correctly.
*) mod_http2: Export mod_http2.h as public header. [Stefan Eissing]
*) mod_md: a new directive `MDStoreLocks` can be used on cluster
setups with a shared file system for `MDStoreDir` to order
activation of renewed certificates when several cluster nodes are
restarted at the same time. Store locks are not enabled by default.
Restored curl_easy cleanup behaviour from v2.4.14 and refactored
the use of curl_multi for OCSP requests to work with that.
Fixes <icing/mod_md#293>.
*) core: Avoid an overflow on large inputs in ap_is_matchexp.
*) mod_heartmonitor: Allow "HeartbeatMaxServers 0" to use file based
storage instead of slotmem. Needed after setting
HeartbeatMaxServers default to the documented value 10 in 2.4.54.
*) mod_dav: DAVlockDiscovery option to disable WebDAV lock discovery
This is a game changer for performances if client use PROPFIND a lot.
To generate a diff of this commit:
cvs rdiff -u -r1.114 -r1.115 pkgsrc/www/apache24/Makefile
cvs rdiff -u -r1.35 -r1.36 pkgsrc/www/apache24/PLIST
cvs rdiff -u -r1.53 -r1.54 pkgsrc/www/apache24/distinfo
cvs rdiff -u -r1.2 -r1.3 pkgsrc/www/apache24/patches/patch-configure
jperkin
pushed a commit
to TritonDataCenter/pkgsrc
that referenced
this issue
Mar 7, 2023
www/apache24: security update
Revisions pulled up:
- www/apache24/Makefile 1.115
- www/apache24/PLIST 1.36
- www/apache24/distinfo 1.54
- www/apache24/patches/patch-configure 1.3
-------------------------------------------------------------------
Module Name: pkgsrc
Committed By: adam
Date: Fri Jan 20 14:03:16 UTC 2023
Modified Files:
pkgsrc/www/apache24: Makefile PLIST distinfo
pkgsrc/www/apache24/patches: patch-configure
Log Message:
apache24: updated to 2.4.55
Changes with Apache 2.4.55
*) SECURITY: CVE-2022-37436: Apache HTTP Server: mod_proxy prior to
2.4.55 allows a backend to trigger HTTP response splitting
(cve.mitre.org)
Prior to Apache HTTP Server 2.4.55, a malicious backend can
cause the response headers to be truncated early, resulting in
some headers being incorporated into the response body. If the
later headers have any security purpose, they will not be
interpreted by the client.
Credits: Dimas Fariski Setyawan Putra (@nyxsorcerer)
*) SECURITY: CVE-2022-36760: Apache HTTP Server: mod_proxy_ajp
Possible request smuggling (cve.mitre.org)
Inconsistent Interpretation of HTTP Requests ('HTTP Request
Smuggling') vulnerability in mod_proxy_ajp of Apache HTTP Server
allows an attacker to smuggle requests to the AJP server it
forwards requests to. This issue affects Apache HTTP Server
Apache HTTP Server 2.4 version 2.4.54 and prior versions.
Credits: ZeddYu_Lu from Qi'anxin Research Institute of Legendsec
at Qi'anxin Group
*) SECURITY: CVE-2006-20001: mod_dav out of bounds read, or write
of zero byte (cve.mitre.org)
A carefully crafted If: request header can cause a memory read,
or write of a single zero byte, in a pool (heap) memory location
beyond the header value sent. This could cause the process to
crash.
This issue affects Apache HTTP Server 2.4.54 and earlier.
*) mod_dav: Open the lock database read-only when possible.
*) mod_proxy_http2: apply the standard httpd content type handling
to responses from the backend, as other proxy modules do.
*) mod_dav: mod_dav overrides dav_fs response on PUT failure.
*) mod_proxy_hcheck: Honor worker timeout settings. [Yann Ylavic]
*) mod_http2: version 2.0.10 of the module, synchronizing changes
with the gitgub version. This is a partial rewrite of how connections
and streams are handled.
- an APR pollset and pipes (where supported) are used to monitor
the main connection and react to IO for request/response handling.
This replaces the stuttered timed waits of earlier versions.
- H2SerializeHeaders directive still exists, but has no longer an effect.
- Clients that seemingly misbehave still get less resources allocated,
but ongoing requests are no longer disrupted.
- Fixed an issue since 1.15.24 that "Server" headers in proxied requests
were overwritten instead of preserved.
- A regression in v1.15.24 was fixed that could lead to httpd child
processes not being terminated on a graceful reload or when reaching
MaxConnectionsPerChild. When unprocessed h2 requests were queued at
the time, these could stall.
- Improved information displayed in 'server-status' for H2 connections when
Extended Status is enabled. Now one can see the last request that IO
operations happened on and transferred IO stats are updated as well.
- When reaching server limits, such as MaxRequestsPerChild, the HTTP/2 connection
send a GOAWAY frame much too early on new connections, leading to invalid
protocol state and a client failing the request.
The module now initializes the HTTP/2 protocol correctly and allows the
client to submit one request before the shutdown via a GOAWAY frame
is being announced.
- :scheme pseudo-header values, not matching the
connection scheme, are forwarded via absolute uris to the
http protocol processing to preserve semantics of the request.
Checks on combinations of pseudo-headers values/absence
have been added as described in RFC 7540. Fixes #230.
- A bug that prevented trailers (e.g. HEADER frame at the end) to be
generated in certain cases was fixed. See #233 where it prevented
gRPC responses to be properly generated.
- Request and response header values are automatically stripped of leading
and trialing space/tab characters. This is equivalent behaviour to what
Apache httpd's http/1.1 parser does.
The checks for this in nghttp2 v1.50.0+ are disabled.
- Extensive testing in production done by Alessandro Bianchi (@alexskynet)
on the v2.0.x versions for stability. Many thanks!
*) mod_proxy_http2: fixed #235 by no longer forwarding 'Host:' header when
request ':authority' is known. Improved test case that did not catch that
the previous 'fix' was incorrect.
*) mod_proxy_hcheck: hcmethod now allows for HTTP/1.1 requests
using GET11, HEAD11 and/or OPTIONS11. [Jim Jagielski]
*) mod_proxy: The AH03408 warning for a forcibly closed backend
connection is now logged at INFO level. [Yann Ylavic]
*) mod_ssl: When dumping the configuration, the existence of
certificate/key files is no longer tested. [Joe Orton]
*) mod_authn_core: Add expression support to AuthName and AuthType.
[Graham Leggett]
*) mod_ssl: when a proxy connection had handled a request using SSL, an
error was logged when "SSLProxyEngine" was only configured in the
location/proxy section and not the overall server. The connection
continued to work, the error log was in error.
*) mod_proxy_hcheck: Re-enable workers in standard ERROR state.
*) mod_proxy_hcheck: Detect AJP/CPING support correctly.
*) mod_http2: Export mod_http2.h as public header. [Stefan Eissing]
*) mod_md: a new directive `MDStoreLocks` can be used on cluster
setups with a shared file system for `MDStoreDir` to order
activation of renewed certificates when several cluster nodes are
restarted at the same time. Store locks are not enabled by default.
Restored curl_easy cleanup behaviour from v2.4.14 and refactored
the use of curl_multi for OCSP requests to work with that.
Fixes <icing/mod_md#293>.
*) core: Avoid an overflow on large inputs in ap_is_matchexp.
*) mod_heartmonitor: Allow "HeartbeatMaxServers 0" to use file based
storage instead of slotmem. Needed after setting
HeartbeatMaxServers default to the documented value 10 in 2.4.54.
*) mod_dav: DAVlockDiscovery option to disable WebDAV lock discovery
This is a game changer for performances if client use PROPFIND a lot.
To generate a diff of this commit:
cvs rdiff -u -r1.114 -r1.115 pkgsrc/www/apache24/Makefile
cvs rdiff -u -r1.35 -r1.36 pkgsrc/www/apache24/PLIST
cvs rdiff -u -r1.53 -r1.54 pkgsrc/www/apache24/distinfo
cvs rdiff -u -r1.2 -r1.3 pkgsrc/www/apache24/patches/patch-configure
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
For regression testing we use a proprietary HTTP service that is able to simulate a CA authoritiy to mod_md. Since we have updated httpd to 2.4.54 we face segmentation faults with various backtraces.
Following are two examples:
In our tests we use these settings
Tracing the request-response flow between mod_md and CA authority we see following HTTP traffic for the first two requests:
It is interesting that if our HTTP service closes the connection after each response, then the segmentation fault does not occur. However, that behaviour is new since 2.4.54.
The text was updated successfully, but these errors were encountered: