Following on from my previous post about Cloudflare WAF – doing more with less, I thought I would have a play with a beta feature I saw in the “Speed” section called “Observatory”. I couldn’t get it to work until I discovered an interesting interaction between Cloudflare Observatory and Cloudflare WAF.
To get started, Cloudflare Observatory appears to let you run a test against your website. Your guess is as good as mine as to what happens next, although this Cloudflare blog post “How to use Cloudflare Observatory for performance experiments“, seems to indicate recommendations of what Cloudflare services can be used to improve performance.
Out of curiosity, I wanted to investigate and determine where the issue originated. One of the plugins I install for any self-hosted WordPress deployments is Wordfence, an on-host web application firewall and security scanner. One of the nice features of Wordfence is the “ability to “Live Traffic” option, allowing the site admin to see, in near real-time, connections to your WordPress instance. For performance reasons, it is recommended to set the “Traffic Logging mode” to “Security-related traffic only”, but I temporarily set it to “All Traffic mode” and re-attempted the Cloudflare Observatory run and pointed it to a post URL – https://www.theitjuggler.com/architecture/cloudflare-waf-doing-more-with-less/. A quick look in Wordfence found:
This log entry, based on the User Agent containing “Cloudflare/Observatory/1.0” and the HTTP Response code of 200, confirmed that Cloudflare was loading the page successfully. So why was Observatory throwing an error?
As demonstrated in the previous article, I had recently implemented Cloudflare’s Web Application Firewall (WAF) on my hosting. This does have some nice event-logging, which is useful to discover if the WAF is blocking traffic. Lo and behold..
The log event matched the User Agent and the URI/Path I’d used for testing Observatory. One Cloudflare service (WAF) is blocking another Cloudflare service (Observatory) from accessing my site proxied via Cloudflare. “Bot Fight mode” is a baked-in feature of the Cloudflare WAF that stops bots from accessing a “proxied” website behind Cloudflare, a useful way of limiting the background automated chatter and noise sniffing around a website. How do you turn off “Bot fight mode”? You can’t, at least not in the Free Cloudflare plan – it is part of the “Managed Rules”. You could disable “Proxied” on your website, but then lose out on the WAF and all the other performance goodness that Cloudflare provides.
If anyone finds a way around this, please let me know as I’d be curious to see what Observatory can tell me.