Posted on:
Categories: SharePoint
Description:

We were recently called to troubleshoot a SharePoint server services down situation on a client's staging farm.
All browsing of SharePoint sites resulted in correlation errors on the configured web front end server.
Browsing sites on the configured application server also hosting the SharePoint Foundation Web Application service, were fully functional.
This web front end server is the primary web application server servicing client request and therefore was the only server with the Forefront product installed and configured.
This identified the WFE as the problematic server for further investigation.

Symptoms:

  • Correlation errors when browsing any site protected by Microsoft Forefront Protection 2010 for SharePoint
  • Source : SharePoint Foundation / ULS & Application Error EventID 1 (Antivirus Scanner timed out) logged immediately when clients request web services from web application services
  • Source : FSCEventing Application Information EventID 1076 (Forefront Protection Eventing Service has stopped) logged continuously on server running Microsoft Forefront Protection 2010 for SharePoint
  • Microsoft Forefront Server Protection Controller windows service will not start due to "dependency" errors
  • Microsoft Forefront Server Protection console will hang/freeze upon launch from the start menu

Cause:

These symptoms can be caused when the Forefront Protection 2010 for SharePoint will not start successfully.

Specifically the "Microsoft Forefront Server Protection Controller" service will not start.

 

When the Forefront Protection 2010 for SharePoint is installed, this service processes all incoming browsing requests through the Forefront Antivirus scanning engine.

When the product is enabled, SharePoint browsing will not function without this request handler. The services must be running. In this way, it ensures protection of the farm.

 

In this situation the root cause of the service failure was a corrupted config file in the Forefront Protection 2010 for SharePoint application directory.

"C:\Program Files\Microsoft Forefront Protection for SharePoint\Data\Configuration.xml"

The application periodically creates a backup of this file (Configuration.bak), in this case the backup was also corrupted. This was likely due to a hard shutdown of the server.

Resolution:

The final resolution of this issue was using a functioning "Configuration.xml" file "C:\Program Files\Microsoft Forefront Protection for SharePoint\Data\Configuration.xml" from a working farm (our errors were on a staging farm so fortunately we had a producion farm to copy the file from).

The steps to resolve and tools used to troubleshoot the issue are below.

Implementing the fix:

Stop all SharePoint services on the server with the failing Forefront product. In out two server farm, the web front end was the only server with the product/services running.

Stop the windows SharePoint services in order:

  1. SharePoint Administration Services
  2. SharePoint Timer Services
  3. SharePoint Tracing Services
  4. World Wide Web Publishing services

 

Once these services have stopped successfully, we disabled the Forefront services by using the FSCUtility.exe utility from the application program directory.

C:\Program Files\Microsoft Forefront Protection for SharePoint\FSCUtility.exe /disable

Ref : http://technet.microsoft.com/en-ca/library/dd639437.aspx

 

Replace the corrupted "Configuration.xml" file with a known good, or file from backup.

 

Re enable Forefront services

C:\Program Files\Microsoft Forefront Protection for SharePoint\FSCUtility.exe /enable

 

Start the windows SharePoint services in order:

  1. SharePoint Administration Services
  2. SharePoint Timer Services
  3. SharePoint Tracing Services
  4. World Wide Web Publishing services

 

Start the "Microsoft Forefront Server Protection Controller" windows service on the server. This will start all required subordinate Forefront services.

Outcome:

In our scenario, this restored all Forefront services as well as all SharePoint services and brought the farm back to a functional state.

The Forefront console was then fully functional and the product was fully configurable.

Troubleshooting Tools used:

SharePoint ULS Log Viewer:

Reviewing the ULS logs on the web front end server revealed it as the only problematic server as well as identified the "Antivirus Scanner timed out" errors immediately on browsing the sites. This pointed us to investigate the Forefront product as the cause.

Sysinternals Procmon:

Using procmon on the server while starting the Forefront Server Protection Controller windows service illustrated high numbers of reads/failures on the "C:\Program Files\Microsoft Forefront Protection for SharePoint\Data\Configuration.xml" file on the file system.

We also noticed continued reading/writing to the "C:\Program Files\Microsoft Forefront Protection for SharePoint\Data\Configuration.bak" configuration backup file.

Forefront Management Shell

The Microsoft Forefront Protection 2010 for SharePoint product has a PowerShell console that allows for non-gui administration of the Forefront product.

Ref: http://technet.microsoft.com/en-us/library/dd639426.aspx

I hoped to elevate the tracing levels of the product for further troubleshooting with "Set-FsspTracing -Level Verbose"

however, this would error out likely due to the controller services failing.