/filterSearch crashing our site

We have been having instability issues for the last couple weeks and finally was able to find the culprit. We have someone trying to crawl/scrape our site and they are hitting about 15-20k /filterSearch?q= pages in a matter of an hour or two. When they do that our site completely crashes our CPU pegs to 99%. Do you know what potentially could be causing the page not to be able to handle the request and peg our CPU to 99%? Obviously that is alot of requests and we are trying to figure out how to stop them but just curious your thoughts?

Hi richhoward,

The best you can do is to block the IPs of these requests as too many requests on a potentially heavy page could break your website down. The search page is a page where you have to filter all the products in the store based on a search keyword and it hits the database for sure and could be considered "heavy" page.
Simply block these IP addresses to prevent such attacks to your website.

The ip addresses keep changing and some of these requests are from googlebot but what can we do to make the page not so heavy? We have about 110k products in our database, we turned off full text search.

We just upgraded to 3.9 and we are still having the same problem, is there anything we could do to tweak the /filtersearch page to not make it so heavy?

Hi Rich,

It is not about the /filtersearch page only. Even if you remove the Ajax Filters and leave the default ones they can open the /search page and will also make a heavy query that will search all products in your store for the specified search term.
Is the URL they request the same one all the time? If so then we can somehow ignore this specified search and do not search the database for that keyword at all i.e "pages" as in your example with /filterSearch?q= pages

No it is random product names and upc's they search like 17k versions