facebookexternalhit & Python/3.10 aiohttp/3.9.3
2

Hi everyone,

We are seeing millions of requests from Facebook IPs and a combination of user agents both "facebookexternalhit" and "Python/3.10 aiohttp/3.9.3"

Other message boards are reporting similar activity. Does anyone have any idea what they are doing or what this surge is about? How are you handling the increased traffic?

Dan
تم طرح السؤال منذ حوالي ‏٥‏ أشهر
Dan

I do want to credit the solution mentioned in this thread. Looking for those specific headers with facebookexternalhit did slow things down. https://developers.facebook.com/community/threads/974370274080457/

‏١٨ يونيو‏‏٧:٣٣ ص‏111
الإجابة المحددة
1

Make sure those requests orignate from any of the registered FB networks, otherwise just block them. If it's the FB crawler rate limit the user agent to 1 request per URL per hour or something.

‏١٨ يونيو‏‏٨:٠٩ ص‏111
Lars
Dan

Thanks for the reply, the IPs are FB. We can't outright block as we want shared URLs to have the full preview FB generates. A blend of measures is in place and we are at in a decent spot right now, but the activity is being noticed and no one knows details. https://www.abuseipdb.com/check/173.252.83.112 https://www.abuseipdb.com/check/69.171.230.13

‏١٨ يونيو‏‏٨:٢٣ ص‏111
Lars

Just allow one requests per URL for this UA in a specified amount of time

‏١٨ يونيو‏‏٨:٢٥ ص‏111
Dan

Again, a blend of measures is in place for our platform - back to my original question -> Does anyone have any idea what they are doing or what this surge is about?

‏١٨ يونيو‏‏٨:٢٧ ص‏111
Lars

Like mentioned already: Please try a forum search first, it's not like this hasn't been noticed before

‏١٨ يونيو‏‏٨:٢٨ ص‏111
Dan

I did and no one is talking about aiohttp in this context: https://developers.facebook.com/community/search?text=aiohttp

‏١٨ يونيو‏‏٨:٣٢ ص‏111
Lars

I'm referring to the other threads that see a huge spike in such requests.

‏١٨ يونيو‏‏٨:٣٤ ص‏111
Lars

Also feel free to add example logs if you see a different user agent than other users.

‏١٨ يونيو‏‏٨:٣٨ ص‏111
Dan

One example FB IP we've recorded over 19,000 hits today in our security layer with the 'Python' user agent. "REMOTE_ADDR": "69.171.249.115", "HTTP_USER_AGENT": "Python/3.10 aiohttp/3.9.3"

88 requests from this same IP have the 'facebookexternalhit' user agent. "REMOTE_ADDR": "69.171.249.115", "HTTP_USER_AGENT":"facebookexternalhit\/1.1 (+http:\/\/www.facebook.com\/externalhit_uatext.php)"

‏١٨ يونيو‏‏١٠:٢٣ ص‏111
1

Side note: please try a forum search first

‏١٨ يونيو‏‏٨:١٤ ص‏111
Lars