Background:

We use solr crawler and clean command for project with multiregional page. Multiregional page had custom backend geolocation redirect functionality - if user from europe visit us website, he is redirected to europe website. Because we still need to crawler the website or other robots to be able to visit page they are visiting without being redirected (e.g. redirection only for real users), we have ignore list based on user-agents. one of the regexp in list is 'crawler' which ignores solr crawler. so crawling of the website is working fine and our crawler is able to crawl any regional page. Unfortunately this is not case for solr clean command which is getting 302 redirect on every regional page which is different form magnolia author instance region. After deeper investigation we find out solr clean command doesn't have any user-agent.

Solution:

Please add same user-agent as has crawler also for clean command.