SharePoint ULS logs

Recently I wrote a PowerShell script to collect all Critical, Unexpected and Monitorable messages in the ULS logs.

With this script I’m creating an item in a SharePoint list. Gosh this does create a lot of items. Things I wasn’t aware of are now visible. I thought that we had quite a healthy SharePoint 2013 farm.

If you are interested in the script then please contact me or you can now download it from codeplex at https://github.com/Pieter-Veenstra/ULS-Monitor.

I have now installed ULSMonitor at some of my customers. If you want me to help you too then please get in touch.

First what do all the different levels mean:

Unexpected

These are the error that should not really happen. Something has gone wrong and some code is not handling the specific error. So this looks a bit like

try

{

… Some dodgy code …

}

catch (Exception ex)

{

… Log error ex in the ULS logs …

}

So a catch all is collecting all errors rather than specific errors being handled.

Critical

Something has gone wrong. Microsoft knows about this and the code is reporting this issue to you. Something should be done by a system administrator.

Warning

The health Analyser has found an issue (e.g. running out of disk space soon but not yet). In general you shouldn’t leave these issues there.

Monitorable

Something is happening but it’s not always clear what has gone wrong.

The errors I found so far

A lot of the below errors I might not have a solution for so if you have the same error and you do have a solution then please feel free to contact me and I will update this blog. I will update this blog as I go along fixing the issues.

 

Critical 1

The SharePoint Health Analyzer detected an error. Drives are running out of free space. Available drive space is less than twice the value of physical memory. This is dangerous because it does not provide enough room for a full memory dump with continued operation. It also could cause problems with the Virtual Memory swap file: OS (SPHOST – C:\), LOGS (SPHOST – E:\). Examine the failing servers and delete old logs or free space on the drives. For more information about this rule, see “http://go.microsoft.com/fwlink/?LinkID=142688”.

Issue

SharePoint wants to have enough spare disk space is case of a system crash.

Solution

In our case we have disabled memory dumps to be written to the disks mentioned however SharePoint still sees the lack of disk space as a critical issue.

 

Critical 2

Table SPDistributedCacheCalls_Partition14 has 461012992 bytes that has exceeded the max bytes 460175067

Issue

Too much unnecessary usage data.

Solution

http://stevemannspath.blogspot.co.uk/2013/04/sharepoint-2013-sql-partition-exceeded.html

Critical 3

Unknown SQL Exception 53 occurred. Additional error information from SQL Server is included below. A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 – Could not open a connection to SQL Server)

Issue

Unknown

Solution

Unknown. Related articles found:

http://jakejacobsen.net/2010/01/06/sharepoint-configuration-wizard-failing-with-sql-exception-a-number-of-possible-fixes/

 

Critical 4

A certificate validation operation took 15002.4863 milliseconds and has exceeded the execution time threshold. If this continues to occur, it may represent a configuration issue. Please see http://go.microsoft.com/fwlink/?LinkId=246987 for more details.

Solution

For this problem I decided to create a separate post

 

Unexpected 1

SPReaderWriterLock named [SPInitializeOnceLock] held for 3166 milliseconds. Call stack: at Microsoft.SharePoint.Utilities.SPReaderWriterLock.SPReaderWriterLockScope.Dispose() at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.get_ReferencedAssemblies() at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.SubCompiler.AddReferenceableAssemblies(WorkflowCompilerParameters theParameters) at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.SubCompiler.DoCompile(WorkflowCompilerParameters parameters, String xomlSource, String assemblyName, CompilationPacket& packet, DirectoryInfo& tempDir)

Read more about SPReaderWriterLock

Unexpected 2

An SPPerformanceCounter was not properly disposed. This could cause excessive memory use.

Issue

An SPSite, SP Web or any other object that needs to be disposed isn’t disposed properly. Check your code and make sure that everything is cleaned up properly before the garbage collection kicks in.

Solution

https://msdn.microsoft.com/en-us/library/aa973248(v=office.12).aspx

 

Unexpected 3

the User profile service returns the following unexpected errors:
ActivityManager.ProcessUserPostCollection (Setting Current User), AccountName:{0}, Exception: {1}.
SPSocialFeedManager.ProcessUserPost: Exception: Microsoft.Office.Server.UserProfiles.SocialDataStoreException: PersonalSiteNotFound : Personal site does not exist. at Microsoft.Office.Server.UserProfiles.SPS2SAppUtility.GetPersonalUrl(UserProfile& profile) at Microsoft.Office.Server.UserProfiles.SPS2SAppExecutionContext.InitializeForProfile() at Microsoft.Office.Server.UserProfiles.SPS2SAppExecutionContext.EnsureInitialized() at Microsoft.Office.Server.Social.SPSocialFeedManager.S2SInvokeWithTrustedFeedManager(String name, UserProfile profile, SPServiceContext serviceContext, Action`1 func)

Issue

Unknown

Solution

Unknown

Unexpected 4

95cdef35-56e0-422a-a953-0e44661055a5 Stack trace: at Microsoft.SharePoint.SPListCollection.GetListByName(String strListName, Boolean bThrowException) at CallSite.Target(Closure , CallSite , Object , Object ) at System.Management.Automation.Interpreter.DynamicInstruction`3.Run(InterpretedFrame frame) at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame) at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame) at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame) at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame) at System.Management.Automation.Interpreter.EnterTry…

Issue

I’m getting this unexpected error when I load my list using PwoerShell code like this:

$list = $web.Lists[“My List”]

Solution

use

$list = $web.GetList(“MyList”)

instead gives a better result. Using SPWeb.GetList makes warning go away. It looks like something isn’t handled properly.

Unexpected 5

The Legacy Worklfow Infrastructure returns the following Unexpected error:
AutoStart Workflow: Microsoft.SharePoint.SPException: Recursive workflows are not permitted. at Microsoft.SharePoint.Workflow.SPWorkflowAutostartEventReceiver.<>c__DisplayClass1.b__0(SPSite superUserSite, SPWeb superUserWeb)

Issue

Workflows can start/trigger other workflows. When workflows however start each other over and over again then SharePoint seems to protect itself. And cancels the process.

Solution

Develop better workflows 🙂

Unexpected 7

SPReaderWriterLock named [SPInitializeOnceLock] held for 406 milliseconds. Call stack: at Microsoft.SharePoint.Utilities.SPReaderWriterLock.SPReaderWriterLockScope.Dispose() at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.get_ReferencedAssemblies() at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.SubCompiler.AddReferenceableAssemblies(WorkflowCompilerParameters theParameters) at Microsoft.SharePoint.Workflow.SPNoCodeXomlCompiler.SubCompiler.DoCompile(WorkflowCompilerParameters parameters, String xomlSource, String assemblyName, CompilationPacket& packet, DirectoryInfo& tempDir)

Issue

Unknown

Solution

Unknown

Unexpected 8

Heavily fragmented indexes can degrade crawl performance.. Automatic repair is being attempted.

Issue

Unknown

Solution

Unknown

Unexpected 9

Unexpected exception in FeedCacheService.IsRepopulationNeeded: Connection to the server terminated,check if the cache host(s) is running .: at Microsoft.Office.Server.DistributedCaching.SPDistributedCache.GetObject(String key, String regionName) at Microsoft.Office.Server.FeedCache.FeedCacheDataCacheWrapper.GetObject(String key, String regionName) at Microsoft.Office.Server.FeedCache.FeedCacheImplementation.IsRepopulationNeeded(Guid callerID) at Microsoft.Office.Server.UserProfiles.FeedCacheService.Microsoft.Office.Server.UserProfiles.IFeedCacheService.IsRepopulationNeeded(Guid callerID)

Issue

Unknown

Solution

Unknown

Unexpected 10

Distributed Cache returns the following Unexpected error:
Unexpected error occurred in method ‘GetObject’ , usage ‘FeedCache’ – Exception ‘Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode:SubStatus:The connection was terminated, possibly due to server or network problems or serialized Object size is greater than MaxBufferSize on server. Result of the request is unknown. —> System.TimeoutException: The socket was aborted because an asynchronous receive from the socket did not complete within the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout. —> System.IO.IOException: The read operation failed, see inner exception. —> System.TimeoutException: The socket was aborted because an asynchronous receive from the socket did not complete within the allo…

Issue

Unknown

Solution

Unknown

Unexpected 11

When items in the health analyzer rules list are added or changed, new timer jobs must be created to execute the rules. For example, if a rule is rescheduled to run weekly, instead of daily, a new weekly timer job may be needed to execute the rule. The repair method of this rule ensures that all the appropriate jobs exist for all rules in the list.. Automatic repair is being attempted.

Issue

Unknown

Solution

Unknown

 

Unexpected 12

WinWF Internal Error, terminating workflow Id# c9ac6a7e-7c37-4547-a46d-5b583ac62ecf

Issue

Unknown

Solution

Unknown

Unexpected 14

SPSocialDataStore.OpenList: List not found: System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)) at Microsoft.SharePoint.SPGlobal.HandleUnauthorizedAccessException(UnauthorizedAccessException ex) at Microsoft.SharePoint.Library.SPRequest.GetMetadataForUrl(String bstrUrl, Int32 METADATAFLAGS, Guid& pgListId, Int32& plItemId, Int32& plType, Object& pvarFileOrFolder) at Microsoft.SharePoint.SPWeb.GetList(String strUrl) at Microsoft.Office.Server.UserProfiles.SPSocialDataStore.OpenList(String name)

Issue: During the User Profile Synchronisation process SharePoint tried to access the SPSocialDataStore and it is not able to do this. In my case I found the following related message in the logs

STS Call Claims Windows: Successfully requested sign-in claim identity for user ‘DOMAIN\spfarm’

For some reason the Object Cache was configured to use the farm account for both super reader and super user account. Solution: Configure the Object cache with a super reader and super user account (http://technet.microsoft.com/en-gb/library/ff758656(v=office.15).aspx)

Unexpected 15

Unexpected Exception in SPDistributedCachePointerWrapper::InitializeDataCacheFactory for usage ‘DistributedLogonTokenCache’ – Exception ‘Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.). Additional Information : The client was trying to co…

Issue

Unknown

Solution

Unknown

Unexpected 16

Token Cache: Failed to initialize SPDistributedSecurityTokenCache Exception: ‘Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.). Additional Information : The client was trying to communicate with the server : net.tcp://CENTRALADMINHOST:22233 …

Issue:

Unknown

Solution:

Unknown

Unexpected 17

CannotMakeBrowserCacheSafeLayoutsUrl: 15/1033/styles/calendar.css

Issue

Unknown

Solution

Unknown

 

Unexpected 18

Exception occured while updating the metadata hub timer job: System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. at Microsoft.SharePoint.Administration.SPSiteCollection.get_Item(Int32 index) at Microsoft.SharePoint.Taxonomy.UpdateHiddenListJobDefinition.Execute(Guid targetInstanceId)

 

Issue

Unknown

Solution

Unknown

Unexpected 19

ActivityManager.ProcessUserPostCollection (Setting Current User), AccountName:{0}, Exception: {1}.

Issue

Unknown

Solution

Unknown

Unexpected 20

Culture_Client::Constructing CultureInfo from ‘-1’ failed: System.ArgumentOutOfRangeException: Positive number required. Parameter name: culture at System.Globalization.CultureInfo..ctor(Int32 culture, Boolean useUserOverride) at Microsoft.Office.Server.Search.Query.Query.set_Culture_Client(Int32 value)

Issue

Unknown

Solution

Unknown

Unexpected 21

Unexpected HttpException while client is still connected. System.Web.HttpException (0x80004005): An error occurred while communicating with the remote host. The error code is 0x800703E3. —> System.Runtime.InteropServices.COMException (0x800703E3): The I/O operation has been aborted because of either a thread exit or an application request. (Exception from HRESULT: 0x800703E3) at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect) at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush() at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async) at Microsoft.SharePoint.Publishing.BlobCache.c__DisplayClass41.b__40() StackTrace: at Microsoft.Office.Server.Native.dll: (sig=80530fbd-8cb8-46a1-ba61-2bfa…

Issue

Unknown

Solution

Unknown

Unexpected 22

System.Web.HttpException: Server cannot append header after HTTP headers have been sent. at System.Web.HttpResponse.AppendHeader(String name, String value) at Microsoft.SharePoint.ApplicationRuntime.SPRequestModule.PostResolveRequestCacheHandler(Object oSender, EventArgs ea) at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Issue

This happens when a page is loaded with a video when the video is stored in a library in SharePoint which used to have content approval enabled. The videos affected were uploaded before the approval feature was disabled.

Solution

Check in the affected videos again.

Monitorable 1

Failed to close ETW trace. Error is 6

Issue

Unknown

Solution

Unknown

Monitorable 2

RerunDiscovery Timer Job succeeded for all servers after 1 attempt(s).

Issue

Unknown

Solution

Unknown

Monitorable 3

security token ‘S-1-1-11-1111111111-111111111-11111’ is found in the local cache, but it is expired. Returing Null..

Issue

Unknown

Solution

Unknown

Monitorable 3

the following Monitorable messages seem to be the same issues for different types of fields:
Unable to get system type for data type URL. (Build:3070)
Unable to get system type for data type Person. (Build:3070)
Unable to get system type for data type Email. (Build:3070)

Issue

Unknown

Solution

Unknown

Monitorable 4

A large block of literal text was sent to sql (length 122415). This can result in blocking in sql and excessive memory use on the front end. Verify that no binary parameters are being passed as literals, and consider breaking up batches into smaller components. If this request is for a SharePoint list or list item, you may be able to resolve this by reducing the number of fields. Call stack: at Microsoft.SharePoint.Utilities.SqlSession.OnPostExecuteCommand(SqlCommand command, SqlQueryData monitoringData) at Microsoft.SharePoint.Utilities.SqlSession.ExecuteNonQuery(SqlCommand command) at Microsoft.SharePoint.Administration.SPServerProductInfo.UpdateProductInfoInDatabase(Guid serverGuid) at Microsoft.SharePoint.Administration.SPProductVersionJobDefinition.Execute(Guid targe…

Issue

Unknown

Solution

Unknown

Monitorable 5

ViewStateLog: Failed to write to the velocity cache http://centraladminhost:2013/default.aspx

Issue

Unknown

Solution

Unknown

Monitorable 6

Token Cache: Reverting to local cache to Add the token for ‘0).w|s-1-5-21-1482476501-839522115-725345543-18620’.

Issue

Unknown

Solution

Unknown

Monitorable 7

Token Cache: Failed to add token from distributed cache for ‘0).w|s-1-5-21-1482476501-839522115-725345543-18620’.(This is expected during the process warm up or if data cache Initialization is getting done by some other thread).

Issue

Unknown

Solution

Unknown

Monitorable 8

Exception occured while updating the metadata hub timer job: System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. at Microsoft.SharePoint.Administration.SPSiteCollection.get_Item(Int32 index) at Microsoft.SharePoint.Taxonomy.UpdateHiddenListJobDefinition.Execute(Guid targetInstanceId)

Issue

Unknown

Solution

Unknown

 

 

Monitorable 9

The following error is generated by a PowerShell script:

Leaked record detected with address 000000001D5ED488, name (unknown), and thread local refcount 73. To see AddRef/Release callstacks for this leak, run stsadm -o setproperty -pn refcountcallstacktrackingenabled -pv true.

Issue

First we looked into when this error occurred and identified the PowerShell in question.

Solution

For this problem I decided to create a separate post

Monitorable 10

Orphaning WebPart with id g_0cdafa54_8976_4497_b309_43d1c0d8c3b1 on Page https://mycorp.com/admin/humancapital/Pages/Home.aspx.

Issue

The problem here is that changes to an existing page have not been checked in. So a page editor has added a new web part to a page and a normal user then visits the published version of the page.

Solution

This can be ignored

Unexpected 13

11/21/2014 07:11:52.93 OWSTIMER (0x0ABC) 0x0F00 SharePoint Foundation Upgrade SPUpgradeSession ajxmj ERROR IsBackwardsCompatible [SPConfigurationDatabase] failed. 00000000-0000-0000-0000-000000000000

Issue

Unknown

Solution

Unknown

 

 

Warning 1

The SharePoint Health Analyzer detected a condition requiring your attention. Databases running in compatibility range, upgrade recommended. The following databases have versions that are older than the current SharePoint software, but are within the backwards compatible range: SP2013_PRD_Services_BDC. To achieve optimal results from these databases, use Upgrade-SPContentDatabase to upgrade Content databases, or psconfig.exe to upgrade other databases. For more information about this rule, see “http://go.microsoft.com/fwlink/?LinkID=142697&#8221;.

Issue

Recently updates were installed. however the BDC database wasn’t upgraded and is still running in previous mode

Solution

It looks like the Products Configuration wizard doesn’t upgrade the BDC database. In our case we’re not using BDC anyway so we ignored this.

 

 

If you still haven’t had enough

Some errors that should be fixed today

Advertisements