Wednesday, May 23, 2012

SharePoint Issues Caused by Inconsistent Times

Today we got some problems in our SharePoint environment after a few Windows patches installed by operation team:

1.Missing Search Bar on main page.
2.No search results when Find an Associate is used.
3.User Audiences not working (all web parts appearing on main page).
4.Missing Navigation/Links Bar on left.

It looks like something wrong with the application service at the first glance. By looking at the SharePoint log, one specific error caught my attention:

...Exception occured while connecting to WCF endpoint: System.ServiceModel.Security.MessageSecurityException: The security timestamp is invalid because its creation time ('2012-05-23T13:53:00.168Z') is in the future...

It tells there were some time-related issue in the farm. When the system admin logged in to front-end and application servers, he found there's 10 minutes difference between two servers! After syncing the time all SharePoint issues were gone and the portal was back to normal immediately.

So the root cause was the time inconsistency in the SharePoint farm, nothing related to the Windows patches. When the front-end server got a reply from the application server, it noticed response timestamp was far ahead of its local time, and then it throw a security exception and terminated further processing.

Why that time variance happened? One major reason is that our two SharePoint front-end servers are actually virtual machines, while the application server is a physical machine. The virtual machines would lose some time when hibernate or reboot if time service is not set properly.