สัญญาณขาดหายบางส่วนshare-external
Why is the facebookexternalhit crawler DDoSing our server?
3

Our IP 81.31.37.22 is being bombarded with a huge number of requests from the facebookexternalhit bot. Does it respect any limits? Google and Microsoft are scraping with a fraction of the intensity compared to Facebook.

Michal
ถามแล้ว ประมาณ​ 6 เดือนที่แล้ว
คำตอบที่เลือก
1

DDossing :) Check if those requests actually originate from a registered FB network.

7 มิถุนายน เวลา 05:05 น.
Lars
Michal

Yes, they all originate from the FB network. With ipV6, it’s visible at first glance, for example: ‘2a03:2880:21ff:c::face:b00c’. We are trying to rate limit. But then the page previews stop working when sharing posts on FB.

7 มิถุนายน เวลา 05:25 น.
Lars

The OG cache most likely only needs to be set once, so it's safe to block any requests after the initial request to a specific URL

7 มิถุนายน เวลา 06:07 น.
Michal

We have no problem with multiple requests for a specific url. Our application runs thousands of domains. FB seems to scrape them all and completely. The problem is intensity.

7 มิถุนายน เวลา 06:48 น.
Lars

Could be bad actors abusing the Sharing Debugger or the API that allows requesting a cache refresh. However, FB won't tell you so rate limiting the user agent is your best/only option.

7 มิถุนายน เวลา 07:01 น.