Lesson Learned - SharePoint 2010 Search Service Application

Recently I was having issues with the Search Service Application on SP 2010 farm.  I decided to remove it and re-create it from scratch.  During the removal, however, the process seems to have hung up for some reason.  I went ahead any manually clean things by doing the following:

• Ensure the old SSA instance is gone (Stop-SPServiceInstance)
• Remove old SSA if still there (Stsadm -o deleteconfigurationobject -id {GUID})
• Delete old service application pool (Remove-SPServiceApplicationPool)
• Delete the older search databases from SP (Get-SPDatabase) and from SQL.

Since I have successfully done this many times in the past, re-creating the SSA should be breeze right? Wrong!!!!

PROBLEM

My initial attempt is to go thru Central Admin to create the SSA but the creation process just stays or hangs until it times out hours later.  I have tried using powershell script and the result was the same.  When I look at the Service App, I do see the Search Admin Web Service get created/started.  I also search the Search Service Application... but with the following error:

"The search service is not able to connect to the machine that hosts the administration component. Verify that the administration component 'cae20adf-f859-4a81-be8b-87b47c7921fa' in search application 'Server Search Application' is in a good state and try again."

Each time, I made sure to clean up the services apps, databases, service instances, and proxy before attempting to re-create.  I am using a managed account for the search service.  On the database site, I am seeing 

"Login failed for user 'DOMAIN\FARMACCOUNT'. Reason: Failed to open the explicitly specified database."

When I click on the "Modify Topology", I get an exception.  Below is the entries in the log:

Unhandled page level exception. Path: /_admin/search/searchconfigwizardtopologyconfig.aspx, Error: Exception has been thrown by the target of an invocation., Details: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.    

 at Microsoft.Office.Server.Search.WebControls.TopologyWebControlDataLayer.AddAdminTopologySettingsToTable()  
 at Microsoft.Office.Server.Search.WebControls.TopologyWebControlDataLayer.GetComponents(String searchAppGuidParam, String viewTypeParam)     -
 -- End of inner exception stack trace ---  
 at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)  
 at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)  
 at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)  
 at System.Web.UI.WebControls.ObjectDataSourceView.InvokeMethod(ObjectDataSourceMethod method, Boolean disposeInstance, Object& instance)  
 at System.Web.UI.WebControls.ObjectDataSourceView.ExecuteSelect(DataSourceSelectArguments arguments)  
 at System.Web.UI.DataSourceView.Select(DataSourceSelectArguments arguments, DataSourceViewSelectCallback callback)  
 at System.Web.UI.WebControls.DataBoundControl.PerformSelect()  
 at Microsoft.Office.Server.Search.WebControls.TopologyWebControl.RefreshGridView()  
 at Microsoft.Office.Server.Search.WebControls.TopologyWebControl.Initialize()  
 at Microsoft.Office.Server.Search.WebControls.TopologyWebControl.CreateChildControls()  
 at System.Web.UI.Control.EnsureChildControls()  
 at System.Web.UI.Control.PreRenderRecursiveInternal()  
 at System.Web.UI.Control.PreRenderRecursiveInternal()  
 at System.Web.UI.Control.PreRenderRecursiveInternal()  
 at System.Web.UI.Control.PreRenderRecursiveInternal()  
 at System.Web.UI.Control.PreRenderRecursiveInternal()  
 at System.Web.UI.Control.PreRenderRecursiveInternal()  
 at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.  
 at Microsoft.Office.Server.Search.WebControls.TopologyWebControlDataLayer.AddAdminTopologySettingsToTable()  
 at Microsoft.Office.Server.Search.WebControls.TopologyWebControlDataLayer.GetComponents(String searchAppGuidParam, String viewTypeParam)     -
 -- End of inner exception stack trace ---  
 at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)  
 at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)  
 at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)  
 at System.Web.UI.WebControls.ObjectDataSourceView.InvokeMethod(ObjectDataSourceMethod method, Boolean disposeInstance, Object& instance)  
 at System.Web.UI.WebControls.ObjectDataSourceView.ExecuteSelect(DataSourceSelectArguments arguments)  
 at System.Web.UI.DataSourceView.Select(DataSourceSelectArguments arguments, DataSourceViewSelectCallback callback)  
 at System.Web.UI.WebControls.DataBoundControl.PerformSelect()  
 at Microsoft.Office.Server.Search.WebControls.TopologyWebControl.RefreshGridView()  
 at Microsoft.Office.Server.Search.WebControls.TopologyWebControl.Initialize()  
 at Microsoft.Office.Server.Search.WebControls.TopologyWebControl.CreateChildControls()  
 at System.Web.UI.Control.EnsureChildControls()  
 at System.Web.UI.Control.PreRenderRecursiveInternal()  
 at System.Web.UI.Control.PreRenderRecursiveInternal()  
 at System.Web.UI.Control.PreRenderRecursiveInternal()  
 at System.Web.UI.Control.PreRenderRecursiveInternal()  
 at System.Web.UI.Control.PreRenderRecursiveInternal()  
 at System.Web.UI.Control.PreRenderRecursiveInternal()  
 at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

SOLUTION

After banging my heads for several days, I decided to contact MS Support for help.  In addition to the cleanup steps I've mentioned earlier, we were able to resolve the issues by taking additional steps:

1. Clear out the Cache: Stop the Sharepoint Timer services on all WFE and App servers.  Go to "C:\ProgramData\Microsoft\SharePoint\Config\[GUID]".  Other than Cache.ini, delete everything  under C:\ProgramData\Microsoft\SharePoint\Config\[GUID] directory.   Update the cache.ini and change the number to 1 and save it.  Start the Timer services again.
2. Review Job Definition: Go to http://CentralAdmin/_admin/ServiceJobDefinitions.aspx page.  See if there's any phantom jobs (job with only GUID but no details) and remove them if exist.
   
3. Stop and restart "Sharepoint Server Search 14"
4. Stop and restart Search Query and Site Settings Service

One you have verified all the above.  Trying creating the SSA again.  Hope this helps someone.