- Community
- Question
It is solely due to the attacks coming from these IPs and these are coming from facebookexternalhit ┌─────────────────────────┐ │ Top IPs │ ├─────────────────┬───────┤ │ IP │ COUNT │ ├─────────────────┼───────┤ │ 173.252.83.9 │ 2930 │ │ 173.252.83.16 │ 2901 │ │ 173.252.83.10 │ 2891 │ │ 173.252.83.24 │ 2868 │ │ 173.252.83.2 │ 2856 │ │ 173.252.83.3 │ 2854 │ │ 173.252.83.12 │ 2854 │ │ 173.252.83.7 │ 2837 │ └─────────────────┴───────┘ I have already added the rule within the robots.txt file. However, it appears that they are not adhering to this directive.
What should we do?
Same here, that amount is pretty low, we get 180K hits on 1 day.
The crawler does not obey robots.txt because it's not really a 'crawler' https://developers.facebook.com/docs/sharing/webmasters/crawler/
Our server load went up to 15-20 where 0.5 is normal, websites came unresponsive or slow. I ended up rate-limit requests from the crawler user-agent, but that has side effects that previews when sharing a link also end up in the rate-limit ending up in a 429 from my server.
Facebook is unreachable or not reply'ing to our e-mail or closing the tickets without even looking.
My guess is that the API is being misused from external crawlers query'ing the FB api (not the first time)
Hopefully somebody is going to look into soon, my requests/tickets are still open.
