Teilweiser Ausfallshare-external
Why is the facebookexternalhit crawler DDoSing our server?
3

Our IP 81.31.37.22 is being bombarded with a huge number of requests from the facebookexternalhit bot. Does it respect any limits? Google and Microsoft are scraping with a fraction of the intensity compared to Facebook.

Michal
Gefragt vor etwa 6 Monaten
Ausgewählte Antwort
1

DDossing :) Check if those requests actually originate from a registered FB network.

7. Juni um 05:05
Lars
Michal

Yes, they all originate from the FB network. With ipV6, it’s visible at first glance, for example: ‘2a03:2880:21ff:c::face:b00c’. We are trying to rate limit. But then the page previews stop working when sharing posts on FB.

7. Juni um 05:25
Lars

The OG cache most likely only needs to be set once, so it's safe to block any requests after the initial request to a specific URL

7. Juni um 06:07
Michal

We have no problem with multiple requests for a specific url. Our application runs thousands of domains. FB seems to scrape them all and completely. The problem is intensity.

7. Juni um 06:48
Lars

Could be bad actors abusing the Sharing Debugger or the API that allows requesting a cache refresh. However, FB won't tell you so rate limiting the user agent is your best/only option.

7. Juni um 07:01