Had a client today (last week now) who broke search all the way. And in their attempts to straighten it out they changed some pieces that weren’t broken. Then while they were in the process of trying to put it all back together I called and said let me at it so they just stopped. Needless to say I picked up the farm in an odd state or more exactly Search was dead.
So the first step always when Search is broke is to go to the SSP Admin and check out things.
-
Open the SSP Administration page
-
Click on Search Administration and see what it has to say. (if you don’t see Search Administration this means you have not installed the infrastructure update. I would highly recommend at a minimum you have that installed. Get the latest update install guide here)
When I opened the page I saw a Crawl status of Error. That is about worthless.
That is pretty much as generic as they come. You get the same Error when the server is on fire as you do when there is small hiccup. So a much better thing to do is:
-
Go back to the SSP administration page
-
Click on Search Settings (which is what we used pre infrastructure update)
This page does a much better job of giving you tangible errors. Here is what I got:
Error: An indexer is not assigned to the Shared Services Provider ‘SharedServices1′.
Link to: Configure an indexer and a search database for this Shared Services Provider
Well that is fixable but how did they end up like this? They stopped the Indexing service in the farm by:
-
Go to Central Admin
-
Click on Operations
-
Click Services on Server
-
They choose their Index server
-
Then clicked Stop to the right of Office SharePoint Search Service
This doesn’t just stop the service. This actually removes the service completely. This also removes the Index server from any SSP configured to use it. Now if you did want to just start and stop the service there is a way to do this:
-
Open a command prompt
-
Type net stop osearch and press enter
-
Type net start osearch and press enter
This will cycle the search service. Usually the only time you need to do something like this is after installing a new ifilter but sometimes it makes you feel better to give it a shot and see if that helps your problem. I do it more often than I should just for that reason.
Back to the task at hand clearing up that error! I double checked and they had already reconfigured the Office SharePoint Search Service on the Index server so all I need to do is go back to the Index server and re-associate the indexer.
-
From Central Administration click on Shared Services Administration from the left hand side of the page.
-
Hover over the SSP name, click the drop down arrow and click Edit properties
-
Scroll to the bottom of the page and select your Index server from the Index Server dropdown. If you see No Indexers in red you need to go back to your Services on Server and make sure you have the Office SharePoint Search service started and configured for the Index role.
-
Confirm that you have the correct index location. Usually the C: drive is less than ideal.
-
Click Ok
The SSP is now configured with an Indexer. Let’s go make sure Search is happy.
-
Now click on the Shared Services Provider name to open the SSP admin site.
-
Click Search Settings
Don’t be surprised if you get this error:
Error: The search service is currently offline. Visit the Services on Server page in SharePoint Central Administration to verify whether the service is enabled. This might also be because an indexer move is in progress.
Typically this is because the wheels of Search can move slowly. I have seen this error come up for 10 minutes or so in some farms. What Search is really telling you is it is busy getting the index and the database ready to go so you can start indexing. Be patient grass hopper. At the client this was gone after about 2 minutes.
Once I was able to get to a happy Search Settings page I went ahead and reset the Index back to zero. Not always necessary but they had 33,000 items in the index and 140,000 or errors. I thought better to start everything back to 0.
In order to reset the Index.
-
From the SSP admin screen click Search Administration
-
From the left hand column (quick launch for those who know terminology) click Reset all crawled content
-
Select Deactivate search alerts during reset
-
Click Reset now
Now you have a completely blank index. Why did we choose to deactivate search alerts? This is to keep from annoying the users. We don’t want them all to get new alerts when new content is discovered when we recrawl in a minute. Once the index is back to normal we will re enable the alerts for them.
Ok so now the next step should be doing a full crawl. So let’s try that.
-
From your SSP Administration home page click Search Administration
-
From the Quick Launch bar (on the left) click Content Sources
-
Hover over your Content Source, click the drop down arrow, and select full crawl
-
Now go back to the home page of Search Administration and watch to see if the crawl is running
Unfortunately in our case after about a minute I was left with 0 items in the index and 3 errors. After checking the errors I got Access Denied. L If you haven’t done any monkeying around with changing your default content access account then it should have been automatically granted full access to your content source. You can confirm this by checking your Policy for web application in Central administration. If you forget how to do that check this blog post for a reminder. http://msmvps.com/blogs/shane/archive/2007/01/21/become-administrator-of-the-entire-web-application.aspx
If that checked out ok then the next thing I would check is to make sure your web application is set to integrated authentication and not basic authentication. MOSS will not pass basic authentication by default. So if you changed your web application from integrated to basic, so people users don’t have to enter their domain for example, then you need to setup a custom crawl rule to pass basic authentication.
-
From your SSP Administration home page click Search Administration
-
In the Quick Launch bar click Crawl rules
-
Click New Crawl Rule
-
For path enter your web app URL ex: http://portal.company.com/*
-
For Crawl Configuration select Include all items in this path
-
For Specify Authentication select Specify a different content access account
-
Now fill in username and password remembering your domain\username form. I would recommended using your normal search account as you know it already has read access to the content.
-
Key step de-select the box to Do not allow Basic Authentication
-
Now do a full crawl. Also, remember if you have multiple web apps you may need more than one of this rules.
For the client this was not this issue but it is an important and often over looked troubleshooting step so I thought throwing it in here would be helpful.
The next thing I take a look at is the dreaded loopback fix. http://www.toddklindt.com/blog/Lists/Posts/Post.aspx?ID=107 It is all but a guarantee these days if you have the WFE and Index role on the same server you are going to need to do this. A lot of farms have ran fine for a long time and just recently they have started requiring it. Must have been a Windows update that is causing this to be needed more but I haven’t identified it. Another note even though this fix is only listed as applying to Windows 2003 it also applies to Windows 2008, had a different client need it last week.
Loopback fix in and the server rebooted I tried another Full crawl. Success! Seems this was the root of their issues but as is often the case that happens to all of us, trying to fix it only made the problem worse. LOL
Don’t forget to re-enable those search alerts.
-
From your SSP administration home page click Search Administration
-
In the System Status section in the center of the page click Search alerts status Enable
Another troubleshooting step I skipped, because the client had already done it was resetting search permissions. Read the blog post John Ross did summing up the steps to get permissions back on the up and up for the Search Service. http://www.sharepoint911.com/blogs/john/archive/2009/04/03/change-to-group-policy-broke-sharepoint-search-–-thanks-conficker-scare.aspx
Something I learned that was new
I am guessing since I didn’t realize this is an option (or more probably I knew and forgot) you probably didn’t either. So run stsadm –o help like below and take a look at the output.
Use Stsadm.exe from the 12 hive (c:\program files\common files\Microsoft shared\web server extensions\12\). Actually 12\bin to be exact.
C:\ >stsadm -help osearch
stsadm -o osearch
[-action <list|start|stop|showdefaultsspadmin>] required parameters for ’start’ (if not already set): role, farmcontactemail, service credentials
[-f (suppress prompts)]
[-role <Index|Query|IndexQuery>]
[-farmcontactemail <email>]
[-farmperformancelevel <Reduced|PartlyReduced|Maximum>]
[-farmserviceaccount <DOMAIN\name> (service credentials)]
[-farmservicepassword <password>]
[-defaultindexlocation <directory>]
[-propagationlocation <directory>]
[-cleansearchdatabase <true|false>]
[-ssp <ssp name>] required parameter for ‘cleansearchdatabase’
So really very similar to the options you have available to you from the GUI. The reason I used it was one of the Query servers was stuck in the starting state. In the GUI there is no stop until the service gets too started, not even a reboot will help. With stsadm you can do a stop and get out of the perpetual starting. J A very helpful trick.
