While trying to establish the Search Crawl on a UAT server, I was having a recurring error :
The crawler could not communicate with the server. Check that the server is available and that the firewall access is configured correctly
This led me to check a number of things :
- Default Search Crawl Account – check – it HAS access to the server
- Can connect using a browser (with the account) – check – that works – while LOCAL logged on (RDP)
- Dump index – and re-crawl – no, didn’t fix it.
- Try crawling another site collection (eg. Central Admin) – ok – that works !!
I was beginning to run out of ideas, when I suddenly clicked on the page url that was shown in the error log. And, I was shown an ACCESS DENIED page (proper ASP.NET stack trace) – for IUSR.
Hang on a sec – that’s the ANON account (!!!)
Yes – my site (intranet) has been extended to also allow anonymous users.
And – here’s the clincher – the page that was the HOME page was not anonymously accessible…
My home page was set to be “/SitePages/Home.aspx”.
I changed this to be “/Pages/Home.aspx” – with a new, blank page – checked-in and published – and then tried to re-crawl the content.
The simple reasoning is (as far as I can tell) :
- Search will try to hit the home page in ANON mode first – when crawling.
- If this fails (denied) – then crawling STOPS.
- If it succeeds, it flips to AUTH mode, and crawls all content happily.
- So – make sure your home page is ANON accessible – only if you need ANON access (of course).
- If you don’t have it turned on, this is a non-issue.
It was a wierd error for sure – I’ll probably never encounter it again – but hope this helps someone !