While debugging URLs in Facebook Debugger, you may experience a Response Code 403 error however good or valid the URL is:

Response code reason: This response code could be due to a robots.txt block. Please allowlist facebookexternalhit on your sites robots.txt config to utilize Facebook scraping

This error on Facebook Debugger indicates that Facebook's crawler, facebookexternalhit, is being blocked by your robots.txt file. While it may seem to be assumed as allowed as indicated by the user agent asterisk for wildcard or all (*) in robots.txt, it seems the advice is you still need to include and allow it implicitly. See the next section for the steps on how to fix this.

This issue seems to be a reported bug from way back 2014, but have seemed to resurfaced this year 2024 starting around March up to this writing.

What is Response Code 403?

A response code 403, known as "Forbidden," means that the server understands the request but refuses to authorize it. This typically occurs due to permissions issues. Common reasons for a 403 error include:

• Incorrect server or domain configurations - the domain points to an incorrect IP address
• Authentication and access permissions - the client is attempting to access an invalid or forbidden URL, or the requested content is forbidden for all clients, regardless of authorization; the server may be set to deny access to certain resources; or the server is blocking specific bots, IP addresses or ranges; authentication may be required but not provided, or the provided credentials are insufficient.
• The application logic, such as insufficient rights to a resource, is preventing access

What is this bot facebookexternalhit?

"Facebookexternalhit" is the user agent string for Facebook's crawler, which is a robot (or bot) that analyzes web pages shared on Facebook or Messenger. The crawler fetches data like page titles, meta descriptions, and thumbnail images to generate previews. Facebook also uses crawlers to analyze links shared for security reasons.

Warnings

While allowing the FB bot access to all resources as indicated by the "Allow: /", there are known reports of the bot traffic as intensively heavy for some servers or server setups.

While facebookexternalhit itself isn't inherently bad, frequent requests from this crawler can contribute to increased server load, potentially impacting performance. This is especially relevant for websites with limited server resources or high traffic volumes. To manage this:

• Rate Limiting: Implement rate limiting server-side to control the frequency of requests from facebookexternalhit. 
  Take note: there are confirmed reports that Crawl-delay is not recognized by Facebook bot at the moment, and will therefore not work website or robots.txt level
• Caching: Use caching mechanisms to reduce the load on your server by serving cached content to the crawler.
   
• Robust Hosting: Ensure your hosting plan can handle the additional load.

Balancing these strategies can mitigate performance issues while still benefiting from proper Facebook link previews. Monitor and review results and tests. Try to also test if allowing only certain URLs according to URL patterns might help ease the traffic from the bot. In the end, though, blocking the bot might do more good than not if this becomes a troublesome issue.

Some recent and old reports on this issue from Facebook Meta for Developers and Stack Overflow forums:

• https://developers.facebook.com/support/bugs/1085796049211789/
• https://developers.facebook.com/community/threads/825654679324994/
• https://developers.facebook.com/support/bugs/468422429851941/
• https://stackoverflow.com/questions/11521798/excessive-traffic-from-facebookexternalhit-bot

We hope this helps. Overall, and when in doubt, ask or look for proper web developer support. Contact us if you need help in any way. 

How to fix this error

In our experience, there are generally two (2) options, methods or workaround that usually works:

1. Robots.TXT
2. Facbebook debugger Invalidator

Robots.TXT method

Ensure the URL is publicly published.

To allow the FB bot to scrape your URLs, you need to modify your robots.txt file to allow Facebook's crawler by entering below anywhere in your robots.txt:

User-agent: facebookexternalhit
Allow: /

How to find and edit your robots.txt

If you are not a web developer, find support for one. If you are a technical guy and can navigate and perform some advanced tasks yourself, proceed with the below to find and update your robots.txt file on your website:

1. Direct URL Access:

   • Open a web browser.
   • Enter your website's URL followed by /robots.txt.
   • Example: https:// www. example.com/robots.txt.

2. Website Root Directory:

   • Access your website’s root directory via FTP or your hosting control panel.
   • Look for the robots.txt file in the root directory (public_html or www).

3. Content Management System (CMS):

   • If you use a CMS like WordPress, you can often view and edit the robots.txt file through SEO plugins like Yoast, RankMath, or directly from the CMS settings accordingly.

Facebook Debugger Method

From the Facebook debugger https://developers.facebook.com/tools/debug/, switch from Sharing Debugger tab to Batch Invalidator. Enter the URL or URLs, then click debug.

This will now forcefully "invalidate", or remove any existing cached social media and meta information about the URL in Facebook.

Wait a few minutes, then go back to the previous Sharing Debugger then proceed with re-submitting/scraping the URL again (Scrape again).

This normally fixes the issue. You can verify with the updated time last scraped. 

Usually, the 403 error and issue in question will still appear, but in our experience this now updates and allowed the changes and preview to be updated.

Test with an actual posting on your social media. Clearing your browser cache every time helps in the testing (just don't include the Cookies in clearing browser cache so you will not get logged out of any browser session you are logged in).