by Martin Laukkanen | Sep 23, 2011 | Troubleshooting
I hate log noise, errors like the following:
Log Name: Application
Source: Microsoft-SharePoint Products-Project Server
Date: 23/09/2011 11:16:05 AM
Event ID: 2077
Task Category: General
Level: Error
Keywords:
User: NT AUTHORITYIUSR
Computer: server.domain.local
Description:
Failed to get language information for Project Server (https://server/PWA)
Apparently this is an error that can be ignored, hence my term “log noise”, however if you are managing large farms having hundreds or even thousands of these occur each night across a number of servers is clearly a big problem.
So I set out to resolve this one, and the cause turned out to be rather simple.
The errors occur nightly during the same window as the Full Search Crawl, so it’s not hard to find the culprit, a bit of searching in the ULS logs correlates the above with some or all of the following errors:
09/23/2011 11:16:05.26 w3wp.exe (0x194C) 0x2A74 Project Server General auvt Exception System.Web.Services.Protocols.SoapException: ProjectServerError(s) LastError=GeneralSecurityAccessDenied Instructions: Pass this into PSClientError constructore to access all error information at Microsoft.Office.Project.Server.WebServiceProxy.Admin.ListInstalledLanguages() at Microsoft.Office.Project.PWA.PJLanguageInfo.EnsureLanguageInfo
Cause
The cause is simply that the Search Default Content Access Account does not have permissions in PWA. Typically said accounts would be granted a Web Application “Full Read” policy in order to index content, however although this allows the account to index all the content without issue it does result in these errors still being thrown up by the PSI.
In that sense this can be safely ignored, however if like me you need to get rid of these the solution is simple; just setup the search account in PWA. In my case on a default test PWA instance granting Team Member rights resolved 14 out of 16 of these, so some level higher than that is required to fully resolve this.
by Martin Laukkanen | Sep 3, 2011 | Project 2010, Troubleshooting
A customer of mine had this one after completing the upgrade to SP1 recently, basically all cube building would fail with the following error:
Your CBSRequest job failed. Its current state is FailedNotBlocking. It was 0% complete. It entered the queue at 09/02/2011 11:00:26.
[snip…]
The errors returned from the queue are as follows:
Error ID: 17007
Error ID: 26000
[snip…]
<class name="CBS message processor failed">
<error id="17007" name="CBSOlapDatabaseSetupFailure" uid="7d14c29d-a133-492b-baea-2e7c0bec444b" QueueMessageBody="Setting UID=00007829-4392-48b3-b533-5a5a4797e3c9 ASServerName=server ASDBName=FullCube1 ASExtraNetAddress= RangeChoice=0 PastNum=1 PastUnit=0 NextNum=1 NextUnit=0 FromDate=01/04/2010 00:00:00 ToDate=12/31/2011 00:00:00 HighPriority=True" Error="Error Setting Olap Database ‘FullCube1’ roles: Error: This method can only convert identity claims, and only when a logical conversion exists.
Parameter name: encodedClaim" />
This customer is using Claims-NTLM (ie AD users via Claims) for logins and that seems to be the cause of this issue. The give away is clearly in the error: “This method can only convert identity claims”.
Solution
Fortunately the solution turned out to be rather simple, it seems that SP1 does some additional checking when setting up the Cube Roles, as a result when users in the Project Server have issues then this error is caused.
A little more info can be seen in the ULS log:
09/02/2011 12:00:33.18 Microsoft.Office.Project.Server (0x17B0) 0x1A04 SharePoint Foundation Claims Authentication d01p Medium ConvertWindowsClaimToWindowsPrincipalName() encountered error: Some or all identity references could not be translated.
As it turns out a number of users in PWA have left the company and in this case as no AD-sync is used some of the accounts had been deleted from Active Directory but not updated in PWA server settings.
It seems also that accounts set as “inactive” also caused this error if they were in a group with the Global “View OLAP Cubes” permission.
The fix was to remove those user accounts from the PWA groups ‘Project Managers’, ‘Portfolio Managers’ and ‘Executives’ (and any others with the above permission).
Finally I just tested this in a non-Claims lab, and the problem doesn’t occur, in fact the cube log identifies and lists the invalid account then continues processing, so technically I would call this one a code defect.
Hope that helps someone else out there!
by Martin Laukkanen | Jul 16, 2011 | Exchange, Troubleshooting
Recently I worked with a large customer getting to the bottom of some issues experienced with Exchange Sync in Project Server 2010, specifically the issues originated due to the large multi-forest nature of the Active Directory environment in which the Project Server was deployed.
Symptoms
Exchange Sync for resources not working, little is logged in the ULS without verbose logging other than the following:
06/30/2011 12:25:49.04 Microsoft.Office.Project.Server (0x0E3C) 0x08C4 Project Server Queue 954k Medium PWA:http://pwa.something/PWA, ServiceApp:Project Service Application, User DOMAINServiceAcc, PSI: [QUEUE] Retry: 1 ExchangeSyncTasks Microsoft.Office.Project.Server.BusinessLayer.QueueMsg.ExchangeSyncTasks
To get to the bottom of what was happening Verbose logging on Project Server Exchange Tasks was required which showed far more detail which I will summaries here:
06/30/2011 12:25:48.00 Microsoft.Office.Project.Server (0x0E3C) 0x16D4 Project Server Exchange Sync fux2 Verbose -! Info: System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond xxx.xxx.xxx.xxx:443 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at … …
06/30/2011 12:25:48.00 Microsoft.Office.Project.Server (0x0E3C) 0x16D4 Project Server Exchange Sync fux2 Verbose =? Trying to Autodiscover using email at ‘https://autodiscover.companydomain.com/autodiscover/autodiscover.xml’
06/30/2011 12:25:48.04 Microsoft.Office.Project.Server (0x0E3C) 0x16D4 Project Server Exchange Sync fux2 Verbose -! Info: System.Net.WebException: The remote name could not be resolved: ‘autodiscover.companydomain.com’ at System.Net.HttpWebRequest.GetRequestStream … …
06/30/2011 12:25:48.04 Microsoft.Office.Project.Server (0x0E3C) 0x16D4 Project Server Exchange Sync 1zpd Verbose Error is: ExchangeSyncEWSUrlFailed. Details: Attributes: a6a779c4-a930-461f-a738-a3c78e8e826a . Standard Information: PSI Entry Point: Project User: DOMAINusername Correlation Id: a1e43a95-1a78-44a0-be1b-e8b3aa267074 PWA Site URL: SSP Name: Project Service Application PSError: ExchangeSyncEWSUrlFailed (40509)
06/30/2011 12:25:48.04 Microsoft.Office.Project.Server (0x0E3C) 0x16D4 Project Server Exchange Sync 9fbi Verbose Error is: ExchangeSyncGeneralProcessingFailure. Details: Attributes: a6a779c4-a930-461f-a738-a3c78e8e826a Microsoft.Office.Project.Server.BusinessLayer.Queue.
ExchangeSyncEmailAddressInvalidException: Could not find Exchange server for resource a6a779c4-a930-461f-a738-a3c78e8e826a at Microsoft.Office.Project.Server.BusinessLayer.Queue.
ProcessExchangeSyncMessage.ExecuteSync … …
06/30/2011 12:25:49.04 Microsoft.Office.Project.Server (0x0E3C) 0x0DD0 Project Server Exchange Sync fux2 Verbose ?? Starting SCP lookup for domainName=’companydomain.com’, root path=”
[cut out multiple variations of the above as Autodiscover tries in vain to find the account]
If you’ve read through that, you can see that Project Server is attempting to use the Exchange Autodiscover service to locate the users Exchange details, as per the Autodiscover protocol which very very basically is something like this:
- Look for an Autodiscover service on the host (then look on the same host without the autodiscover bit)
- If not found try an SCP (Service Connection Point) lookup in Active Directory to find the resources
This can fail in a multi-forest Active Directory environment if your Exchange Client Access Servers (CAS) are located in a different forest. As was the case with my customer.
Solution
This is actually a well understood problem in Exchange circles, and the full solution is well documented:
How to Configure the Autodiscover Service for Multiple Forests
However if like in my case you don’t have the time to get a significant change such as the above completed then an alternative is needed.
Workaround
Looking at the above autodiscover procedure an easy “workaround” (I call this one a ‘hack’) is clear;
- Add an entry to the HOSTS file on all Project Server application servers pointing to autodiscover.companyemaildomain.com which points to the actual autodiscover service running on the Exchange CAS.
However now you’ll see something like this in your Verbose ULS:
06/30/2011 12:40:04.17 Microsoft.Office.Project.Server (0x0E3C) 0x1930 Project Server Exchange Sync fux2 Verbose -! Info: System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. —> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
This is because your Exchange CAS has something like as the Subject CN in its SSL certificate which would be expected, so what you need to do add a Subject Alternative Name to your certificate, see the following article on doing this:
Configure SSL Certificates to Use Multiple Client Access Server Host Names
Now you’re almost there, just one last thing! In order for the Exchange and Project Server to both authenticate these cross-forest users you need to make some changes on the service account on both forests by configuring the msExchMasterAccountSid property on the Exchange forest. See the following similar technet forum solution to achieve this:
EWS returns error "Failed to get valid Active Directory information for the calling account"
Done.
Hope that helps someone else out there!
by Martin Laukkanen | Jul 1, 2011 | SharePoint 2010, Troubleshooting
Had some time today to upgrade my lab to SP1, and thought it worth a quick post here about my experience.
For many people SP1 will be the first update applied to Project Server and SharePoint 2010, so with up to a year or more of production use it’s very possible that some features / solutions have been installed and removed which might cause some problems for the SP1 install.
In my case on my development lab this was most certainly the case with literally dozens of (often half developed) solutions in various states of deployment!
So in this case you may have the configuration Wizard fail after SP1 setup with a message like this:
Upgrade Timer job is exiting due to exception: Microsoft.SharePoint.Upgrade.SPUpgradeException: Upgrade completed with errors. Review the upgrade log file located in C:Program FilesCommon FilesMicrosoft SharedWeb Server Extensions14LOGSUpgrade-20110701-160359-545.log. The number of errors and warnings is listed at the end of the upgrade log file.
Digging into the log mentioned reveals something like:
[ERROR] [7/1/2011 4:17:20 PM]: The feature with Id d67567a3-4946-412f-9428-1ca6061a5189 is referenced in the database [WSS_Content], but is not installed on the current farm. The missing feature may cause upgrade to fail.
(A very helpful and surprisingly accurate message)
The problem as you can see is a missing feature, however using the standard PowerShell get-SPFeature commands don’t reveal anything. This is due to the solution already being uninstalled (or perhaps never installed in the case of a migrated database). Fortunately there is a solution:
Thanks Phil, using the provided scripts I was quickly able to identify and remove the offending feature. Once done a rerun of the PSCONFIG wizard (in my case using the command line “PSCONFIG –cmd upgrade –inplace b2b”) completes without errors.
Hope that helps someone else out there.
by Martin Laukkanen | Feb 26, 2011 | Project 2010, SharePoint 2010, Troubleshooting
I came across an unexplained issue recently with an ‘old’ Project Server 2010 customer, in short what happened was many of the permissions across the Project workspace sites were lost. Unfortunately the loss was triggered by performing a Playbooks (2010 version) backup of the server configuration which was even more concerning, considering that is such a commonly used tool.
In short we needed a way to restore all of the workspace site permissions to what they should be, without having to manually action each site.
Some background:
As it turns out Project Server 2010 has (once again) drastically changed the way that workspace site permissions are managed, fortunately changed for the better as now the behaviour is something like this:
- On first publish or creation of the workspace all appropriate users are granted permissions based on their group memberships and if they are a member of the project team. (Just as it previously worked)
- Subsequent publish jobs trigger another ‘Workspace Create’ job which now only adds new permissions but does not remove any existing ones, such as when a new team member is added to the plan. (This is similar to 2007 SP2)
- Active Directory Sync jobs trigger a permissions sync which acts just like #2.
The big differences are the following:
- During the sync NO permissions are removed. Thus correcting the old situation where an AD sync during business hours could result temporary permission losses.
- Manually changed permissions at the SharePoint level are respected. For instance if you remove a particular user from a SharePoint site, Project Server will not add it back in either of the standard permissions sync jobs above.
The Solution: To Restore All Permissions
As you may have found for yourself due to the new behaviour if a user is deleted from the SharePoint site, re-publishing a project will no longer fix the permissions. As such a new solution was needed in my case where permissions were lost across all sites:
- Create a new Project Server group containing all users.
- Assign My Organisation Category to that group.
- Assign Permission ‘View Project Site’ to My Organisation category.
- Save the new group (and wait for queue jobs to complete).
- Once confirmed permissions working delete the group (and again wait for the queue).
This will force a full refresh of the workspace site permissions across all users and all sites. Due the the new method of handling permissions it will only add each user with a minimum of ‘Reader’ unless they have another group which applies like Project Manager or Administrator, once you complete step 5 those unnecessary Readers will be removed, leaving only the correct permissions.
HTH,
by Martin Laukkanen | Jan 12, 2011 | Reporting, SharePoint 2010, Troubleshooting
Update 21/01/2011: Update below regarding Excel Services and Multi-tenancy!
One of the exciting new features of SharePoint 2010 is multi-tenancy, if you’re working in hosted or shared environments then it is no less than a must-have feature.
Unfortunately in the real world of software nothing is perfect at RTM, and this would have to be one of those cases!
In short neither Project Server or PerformancePoint appear to support multi-tenancy in SharePoint 2010, it would seem that the feature has yet to be fully implemented for either service application, however the situation is not as bad as you might fear.
Project Server in a multi-tenanted environment
Firstly although Project Server does not accept any subscription related parameters when provisioning a new instance using PowerShell or the Central Admin, it does appear to work in a tenanted environment. Basically the fact that any provisioned PWA site is a site collection of its own means that once you have provisioned your PWA instance you can use the following PowerShell command to associate your PWA site collection with your tenant subscription:
Set-SpSite $PWASiteUrl -SiteSubscription $subscription
More good news is that once provisioned a PWA instance is able to communicate with other service applications belonging to the same subscription. Most importantly: Secure Store Service. Without that Excel Services wouldn’t work!
All is not good though, especially if you like to use the full feature set of the 2010 product, read on..
PerformancePoint in a multi-tenanted environment
This is where the news gets bad, it would appear that PerformancePoint in 2010 does not support multi-tenancy at all, it actually doesn’t appear to respect tenant subscriptions and so as a result you might end up with errors like the following when attempting to run Dashboard Designer or configuring the service application unattended account;
w3wp.exe (0x04E4) 0x0ED8 Secure Store Service Secure Store 7557 Critical The Secure Store Service application Secure Store Service Proxy is not accessible. The full exception text is: Access is denied. adfdd2a3-b6e5-4d92-8c5e-5a44fd821969
w3wp.exe (0x04E4) 0x0ED8 Secure Store Service Secure Store d9ld Unexpected Unexpected exception from endpoint address
w3wp.exe (0x04E4) 0x0ED8 Secure Store Service Secure Store d9le Unexpected Logging unknown/unexpected client side exception: SecurityAccessDeniedException. This will cause this application server to be removed from the load balancer queue. Exception: System.ServiceModel.Security.SecurityAccessDeniedException: Access is denied
Some searching will lead you to a number of sources talking about the lack of a Master Key being created in the Secure Store which is most certainly the cause of the issue, unfortunately it appears that in the case where your Secure Store is created with tenancy enabled PerformancePoint is unable to see the Secure Store service application (and thus unable to retrieve the master key)!
Some deep reading led me to some TechNet articles detailing the requirements for PerformancePoint (sorry can’t find the link ATM), basically PPS requires that a Secure Store Service Application exists in the default proxy group, and as I’ve found that app also cannot be tenanted.
Fortunately this can also be worked around, maintaining a non-tenanted default Secure Store app dedicated to PPS does not as I see it introduce any security implications, in particular as you will have to provision a separate PPS service for each tenant in your farm (thus losing much of the benefits of multi-tenancy).
All I can say on this one is that I can’t wait for SP1 for PPS, maybe then we’ll even be able to name our databases and lose that BETA PowerShell command line syntax? :)
Update 21/01/2011:
See the following blog for information on how Excel Services impacts the above configurations!
