Problem: legitimate traffic to my site is receiving "403 Forbidden" errors.
Example: A request from 199.16.156.125 to GET /robots.txt HTTP/1.1 from user-agent Twitterbot/1.0 should be allowed, but is denied.
Suspected culprit: my root .htaccess file implements the 5G Firewall from PerishablePress; additional deny rules from various WordPress security plugins; a couple of deny rules that I have hacked together myself; and a long list of banned CIDRs.
Sample code (truncated):
# ban all requests via HTTP/1.0
# UNLESS they are requests to either wp-cron or admin-ajax.
<ifModule mod_rewrite.c>
RewriteCond %{THE_REQUEST} !(HTTP/1\.1)$ [NC]
RewriteCond %{REQUEST_URI} !^(/wp-admin/admin-ajax\.php.*|/wp-cron\.php.*)
RewriteRule ^(.*)$ - [F,L]
</ifModule>
# 5G:[USER AGENTS]
<IfModule mod_setenvif.c>
SetEnvIfNoCase User-Agent ^$ keep_out
SetEnvIfNoCase User-Agent (casper|cmsworldmap|diavol|dotbot) keep_out
SetEnvIfNoCase User-Agent (flicky|ia_archiver|kmccrew) keep_out
SetEnvIfNoCase User-Agent (libwww|planetwork|pycurl|skygrid) keep_out
SetEnvIfNoCase User-Agent (SISTRIX|AhrefsBot|YandexBot|Baidu|SiteExplorer|MJ12bot|Browserlet|msnbot\-media) keep_out
SetEnvIfNoCase User-Agent (Java|Wget|CPython|ruby|panscient\.com) keep_out
<Limit GET POST PUT>
Order Allow,Deny
Allow from all
Deny from env=keep_out
</Limit>
</IfModule>
#allowed IP (example)
allow from 255.255.255.255
#banned networks
deny from 199.119.224.0/22
deny from 199.30.48.0/21
deny from 99.198.96.0/19
There's more to it than that, but I suspect the problem may be in the above sample code.
What have I tried: 1. edit .htaccess rules to attempt to resolve issue. 2. create 403 logger script to provide additional details about requests that receive a 403 response. (The hypothesis was that the forbidden requests might include additional X-Forwarded-For headers that contain IP addresses from banned networks; however, it turned out that this hypothesis was not correct.) 3. bang head against desk.
EDIT
If the general consensus is that the posted snippet could not be causing the problem I'm describing, I will accept an answer that helps me to track down the actual cause of the problem through more effective troubleshooting.
UPDATE
I appreciate the community's input and thought I would provide an update.
Eventually, I contacted my web host for help. After a little back-and-forth, I learned that my host had indeed banned Twitterbot from their network for being a resource hog, apparently by visiting sites running poorly optimized WordPress plugins. So that specific result was not, in fact, an .htaccess issue after all.
However, I still have a client who can't access my site, gets 403 forbidden message instead: legitimate traffic is getting denied, as described above. My host confirmed they are not blocking my client's IP address. I still haven't found a resolution for this. I have even listed his dedicated IP in an "allow" directive before my list of "deny" directives, and still he's denied. I edited the code example above to demonstrate.
