TFS 2010 RTM throws TF221122 error when processing OLAP cube

olap_cube

Background

I was working with a client recently.  We were installing and configuring a new Team Foundation Server 2010 installation.  As usual, we started checking out the environment’s stability the day after installation.  We checked Event Logs and ran the TFS 2010 Best Practices Analyzer tool (part of the TFS 2010 Power Tools) and noticed a series of errors from the TFSJobAgent service that repeated each hour.  The full entry is pasted (and scrubbed) at the end of this article so that it will be indexed by search engines.

Environment

The installation environment consisted of:

  • Server 1: App Tier – TFS 2010 RTM, SSRS (SQL Server 2008 SP2)
    • App tier running under CORPTFSService account
  • Server 2: Data Tier – Database and Analysis Services engine (SQL Server 2008 SP2)
    • Database and Analysis Services running under server’s NETWORK SERVICE account
  • Server 3: SharePoint – MOSS 2010 RTM

Note: The MOSS server in this installation has no bearing on the problem described here.  The issue would have still arisen even if WSS 3.0 was installed on the App Tier. 

Issue

Our reports weren’t being updated with new data and numerous errors were seen in the event log.  The messages we were getting indicated that the TFS Job Service (running as the TFSService service account) couldn’t access the Analysis Services service on the Data Tier. 

OLE DB error: OLE DB or ODBC error: A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.; 08001; Client unable to establish connection; 08001; Encryption not supported on the client.; 08001.
Errors in the high-level relational engine. A connection could not be made to the data source with the DataSourceID of ‘Tfs_AnalysisDataSource’, Name of ‘Tfs_AnalysisDataSource’.

What is happening is that the TFS Warehouse (database engine) is trying to connect to the Cube (analysis services engine) on the same machine but can’t connect successfully.

Investigation

Over the course of a couple of hours we tried many things, including checking that the NetworkService account had the proper permissions on the SQL Server database and analysis engines.  All looked good. 

To try to rule out a permissions issue we changed SQL Server and Analysis Services service accounts to LocalSystem and rebuilt warehouse. In this case, processing the warehouse and cube was successful but this isn’t a viable solution because we have a dual-tier environment and the LocalSystem account has no network access.

We then changed SQL Server and Analysis Services service accounts back to NetworkService and rebuilt the warehouse. We processed the warehouse successfully but now the cube processing failed like it did before.

We contacted Microsoft Support to see if they could give us a hand.  We gave our contact Wendell a very thorough run down of our environment and troubleshooting steps taken so far.  He was very thorough and nice enough to trust that we knew what we were talking about so we didn’t repeat all the basic troubleshooting steps.  (That was a way better experience than I had talking with the folks at my cable company when they messed up my connectivity.) 

Resolution

After a bit more troubleshooting Wendell and I determined that this was a full-scale “conundrum” so he brought in a couple of other Support folks into the conversation.  One of them had seen something weirdly similar the prior weekend and suggested that we change the SQL database and analysis services account from NetworkService to a domain account.  The TFS Installation Guide states that you can use NetworkService or a domain service account for this purpose and it should work the same way.  In fact, it is my understanding that Microsoft uses NetworkService on its Pioneer Server (dogfooding TFS server) with great results.

We changed both services to run with the CorpTFSService service account and rebuilt the warehouse. Processing was now successful for both the warehouse and cube.  Changing to a domain account fixed the issue.

Just to satisfy our curiosity, we changed the service accounts back to NetworkService and the processing went back to failure so we know that this is a reproducible issue.

Follow-up

We monitored the Event Logs on the App Tier for the next couple of days to see if the errors reappeared.  They did not so we judged that this fix was successful.

Commentary

Don’t take this post as reason to use a domain service account for the Data Tier services over NetworkService.  This was a very specific case where we were installing on a new AD Domain that was not completely configured.  You may not have the same problem that we had here.

With that said, I usually recommend that all services accounts in a TFS installation be domain service accounts instead of NetworkService so that if you need to configure limited permissions for the account you can do so.


Event Log Entry

Log Name: Application
Source: TFS Services
Date: 6/09/2010 2:49:09 PM
Event ID: 3305
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: tfsat1.corp.mycompany.com
Description:
TF53010: The following error has occurred in a Team Foundation component or extension:
Date (UTC): 5/13/2010 6:49:09 PM
Machine: tfsat1
Application Domain: TfsJobAgent.exe
Assembly: Microsoft.TeamFoundation.Framework.Server, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727
Service Host:
Process Details:
Process Name: TFSJobAgent
Process Id: 1748
Thread Id: 2948
Account name: CORPtfsservice
Detailed Message: TF221122: An error occurred running job Full Analysis Database Sync for team project collection or Team Foundation server TEAM FOUNDATION.
Exception Message: Failed to Process Analysis Database ‘Tfs_Analysis’. (type WarehouseException)
Exception Stack Trace: at Microsoft.TeamFoundation.Warehouse.TFSOlapProcessComponent.ProcessOlap(AnalysisDatabaseProcessingType processingType, WarehouseChanges warehouseChanges, Boolean lastProcessingFailed, Boolean cubeSchemaUpdateNeeded)
at Microsoft.TeamFoundation.Warehouse.AnalysisDatabaseSyncJobExtension.RunInternal(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage)
at Microsoft.TeamFoundation.Warehouse.WarehouseJobExtension.Run(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage)
Inner Exception Details:
Exception Message: Internal error: The operation terminated unsuccessfully.
Internal error: The operation terminated unsuccessfully.
OLE DB error: OLE DB or ODBC error: A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.; 08001; Client unable to establish connection; 08001; Encryption not supported on the client.; 08001.
Errors in the high-level relational engine. A connection could not be made to the data source with the DataSourceID of ‘Tfs_AnalysisDataSource’, Name of ‘Tfs_AnalysisDataSource’.
Errors in the OLAP storage engine: An error occurred while the dimension, with the ID of ‘Result’, Name of ‘Test Result’ was being processed.
Errors in the OLAP storage engine: An error occurred while the ‘Resolution State’ attribute of the ‘Test Result’ dimension from the ‘Tfs_Analysis’ database was being processed.
Server: The operation has been cancelled.
Internal error: The operation terminated unsuccessfully.
Internal error: The operation terminated unsuccessfully.
Internal error: The operation terminated unsuccessfully.
Internal error: The operation terminated unsuccessfully.
Internal error: The operation terminated unsuccessfully.
Internal error: The operation terminated unsuccessfully.
OLE DB error: OLE DB or ODBC error: A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.; 08001; Client unable to establish connection; 08001; Encryption not supported on the client.; 08001.
Errors in the high-level relational engine. A connection could not be made to the data source with the DataSourceID of ‘Tfs_AnalysisDataSource’, Name of ‘Tfs_AnalysisDataSource’.
Errors in the OLAP storage engine: An error occurred while the dimension, with the ID of ‘Result’, Name of ‘Test Result’ was being processed.
Errors in the OLAP storage engine: An error occurred while the ‘Failure Type Id’ attribute of the ‘Test Result’ dimension from the ‘Tfs_Analysis’ database was being processed.
Internal error: The operation terminated unsuccessfully.
OLE DB error: OLE DB or ODBC error: A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.; 08001; Client unable to establish connection; 08001; Encryption not supported on the client.; 08001.
Errors in the high-level relational engine. A connection could not be made to the data source with the DataSourceID of ‘Tfs_AnalysisDataSource’, Name of ‘Tfs_AnalysisDataSource’.
Errors in the OLAP storage engine: An error occurred while the dimension, with the ID of ‘Result’, Name of ‘Test Result’ was being processed.
Errors in the OLAP storage engine: An error occurred while the ‘Readiness State Id’ attribute of the ‘Test Result’ dimension from the ‘Tfs_Analysis’ database was being processed.
Internal error: The operation terminated unsuccessfully.
Internal error: The operation terminated unsuccessfully.
OLE DB error: OLE DB or ODBC error: A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.; 08001; Client unable to establish connection; 08001; Encryption not supported on the client.; 08001.
Errors in the high-level relational engine. A connection could not be made to the data source with the DataSourceID of ‘Tfs_AnalysisDataSource’, Name of ‘Tfs_AnalysisDataSource’.
Errors in the OLAP storage engine: An error occurred while the dimension, with the ID of ‘Result’, Name of ‘Test Result’ was being processed.
Errors in the OLAP storage engine: An error occurred while the ‘Readiness State’ attribute of the ‘Test Result’ dimension from the ‘Tfs_Analysis’ database was being processed.
Internal error: The operation terminated unsuccessfully.
OLE DB error: OLE DB or ODBC error: A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.; 08001; Client unable to establish connection; 08001; Encryption not supported on the client.; 08001.
Errors in the high-level relational engine. A connection could not be made to the data source with the DataSourceID of ‘Tfs_AnalysisDataSource’, Name of ‘Tfs_AnalysisDataSource’.
Errors in the OLAP storage engine: An error occurred while the dimension, with the ID of ‘Result’, Name of ‘Test Result’ was being processed.
Errors in the OLAP storage engine: An error occurred while the ‘Test Result Priority’ attribute of the ‘Test Result’ dimension from the ‘Tfs_Analysis’ database was being processed.
OLE DB error: OLE DB or ODBC error: A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.; 08001; Client unable to establish connection; 08001; Encryption not supported on the client.; 08001.
Errors in the high-level relational engine. A connection could not be made to the data source with the DataSourceID of ‘Tfs_AnalysisDataSource’, Name of ‘Tfs_AnalysisDataSource’.
Errors in the OLAP storage engine: An error occurred while the dimension, with the ID of ‘Result’, Name of ‘Test Result’ was being processed.
Errors in the OLAP storage engine: An error occurred while the ‘Outcome Id’ attribute of the ‘Test Result’ dimension from the ‘Tfs_Analysis’ database was being processed.
Internal error: The operation terminated unsuccessfully.
Internal error: The operation terminated unsuccessfully.
Internal error: The operation terminated unsuccessfully.
Internal error: The operation terminated unsuccessfully.
OLE DB error: OLE DB or ODBC error: A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.; 08001; Client unable to establish connection; 08001; Encryption not supported on the client.; 08001.
Errors in the high-level relational engine. A connection could not be made to the data source with the DataSourceID of ‘Tfs_AnalysisDataSource’, Name of ‘Tfs_AnalysisDataSource’.
Errors in the OLAP storage engine: An error occurred while the dimension, with the ID of ‘Result’, Name of ‘Test Result’ was being processed.
Errors in the OLAP storage engine: An error occurred while the ‘Failure Type’ attribute of the ‘Test Result’ dimension from the ‘Tfs_Analysis’ database was being processed.
OLE DB error: OLE DB or ODBC error: A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.; 08001; Client unable to establish connection; 08001; Encryption not supported on the client.; 08001.
Errors in the high-level relational engine. A connection could not be made to the data source with the DataSourceID of ‘Tfs_AnalysisDataSource’, Name of ‘Tfs_AnalysisDataSource’.
Errors in the OLAP storage engine: An error occurred while the dimension, with the ID of ‘Result’, Name of ‘Test Result’ was being processed.
Errors in the OLAP storage engine: An error occurred while the ‘Test Result Executed By’ attribute of the ‘Test Result’ dimension from the ‘Tfs_Analysis’ database was being processed.
OLE DB error: OLE DB or ODBC error: A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.; 08001; Client unable to establish connection; 08001; Encryption not supported on the client.; 08001.
Errors in the high-level relational engine. A connection could not be made to the data source with the DataSourceID of ‘Tfs_AnalysisDataSource’, Name of ‘Tfs_AnalysisDataSource’.
Errors in the OLAP storage engine: An error occurred while the dimension, with the ID of ‘Result’, Name of ‘Test Result’ was being processed.
Errors in the OLAP storage engine: An error occurred while the ‘Outcome’ attribute of the ‘Test Result’ dimension from the ‘Tfs_Analysis’ database was being processed.
Internal error: The operation terminated unsuccessfully.
Internal error: The operation terminated unsuccessfully.
OLE DB error: OLE DB or ODBC error: A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.; 08001; Client unable to establish connection; 08001; Encryption not supported on the client.; 08001.
Errors in the high-level relational engine. A connection could not be made to the data source with the DataSourceID of ‘Tfs_AnalysisDataSource’, Name of ‘Tfs_AnalysisDataSource’.
Errors in the OLAP storage engine: An error occurred while the dimension, with the ID of ‘Result’, Name of ‘Test Result’ was being processed.
Errors in the OLAP storage engine: An error occurred while the ‘Test Result Owner’ attribute of the ‘Test Result’ dimension from the ‘Tfs_Analysis’ database was being processed.
Internal error: The operation terminated unsuccessfully.
Internal error: The operation terminated unsuccessfully.
Warning: Parser: Out of line object ‘Binding’, referring to ID(s) ‘Tfs_Analysis, Team System, FactCurrentWorkItem’, has been specified but has not been used.
Warning: Parser: Out of line object ‘Binding’, referring to ID(s) ‘Tfs_Analysis, Team System, FactWorkItemHistory’, has been specified but has not been used.
Warning: Parser: Out of line object ‘Binding’, referring to ID(s) ‘Tfs_Analysis, Team System, v Fact WorkItem To Tree’, has been specified but has not been used.
Warning: Parser: Out of line object ‘Binding’, referring to ID(s) ‘Tfs_Analysis, Team System, v Fact Linked Current WorkItem’, has been specified but has not been used.
Warning: Parser: Out of line object ‘Binding’, referring to ID(s) ‘Tfs_Analysis, Team System, Fact WorkItem To Category’, has been specified but has not been used.
Warning: Parser: Out of line object ‘Binding’, referring to ID(s) ‘Tfs_Analysis, Team System, v Fact WorkItem Changeset’, has been specified but has not been used.
Warning: Parser: Out of line object ‘Binding’, referring to ID(s) ‘Tfs_Analysis, Team System, Fact Build Project’, has been specified but has not been used.
Warning: Parser: Out of line object ‘Binding’, referring to ID(s) ‘Tfs_Analysis, Team System, Fact Build Details’, has been specified but has not been used.
Warning: Parser: Out of line object ‘Binding’, referring to ID(s) ‘Tfs_Analysis, Team System, Fact Code Churn’, has been specified but has not been used.
Warning: Parser: Out of line object ‘Binding’, referring to ID(s) ‘Tfs_Analysis, Team System, v Fact Test Result Overlay’, has been specified but has not been used.
Warning: Parser: Out of line object ‘Binding’, referring to ID(s) ‘Tfs_Analysis, Team System, Fact Build Changeset’, has been specified but has not been used.
Warning: Parser: Out of line object ‘Binding’, referring to ID(s) ‘Tfs_Analysis, Team System, Fact Build Coverage’, has been specified but has not been used.
Warning: Parser: Out of line object ‘Binding’, referring to ID(s) ‘Tfs_Analysis, Team System, v Fact WorkItem Test Result’, has been specified but has not been used.
Warning: Parser: Out of line object ‘Binding’, referring to ID(s) ‘Tfs_Analysis, Team System, v Fact Linked Current Work Item Test Case’, has been specified but has not been used.
Warning: Parser: Out of line object ‘Binding’, referring to ID(s) ‘Tfs_Analysis, Team System, Fact Run Coverage’, has been specified but has not been used. (type WarehouseException)
Exception Stack Trace: at Microsoft.TeamFoundation.Warehouse.TFSOlapProcessComponent.ExecuteXmla(String finalXmla)
at Microsoft.TeamFoundation.Warehouse.TFSOlapProcessComponent.ProcessOlap(AnalysisDatabaseProcessingType processingType, WarehouseChanges warehouseChanges, Boolean lastProcessingFailed, Boolean cubeSchemaUpdateNeeded)