by Martin Laukkanen | Jan 17, 2014 | Project 2010, Troubleshooting
It’s not often these days I come across what seems to be a new Project Server 2010 error! Having said that, I would expect the same error to occur on either 2007 or 2013 as it relates to AD sync.
Anyway while working with a customer recently I came across the following issue that had started occurring for all new projects being created as well as on any attempt to synchronize the site permissions from Server Settings – Project Sites.
The effect of the error was that users cannot access any of the affected sites.
Notably though the AD sync was not experiencing any errors.
Error Messages
GeneralQueueJobFailed (26000) – SynchronizeMembershipForWssSite.SynchronizeMembershipForWssSiteMessage. Details: id=’26000′ name=’GeneralQueueJobFailed’ uid=’677df8c5-7409-4e20-8a05-e98395aa2af3′ JobUID=’3ca3af00-2e08-49aa-b693-f1314fc09b96′ ComputerName=’…’ GroupType=’SynchronizeMembershipForWssSite’ MessageType=’SynchronizeMembershipForWssSiteMessage’ MessageId=’1′ Stage=”
And in the ULS log we see some more details:
01/14/2014 15:43:07.23 Microsoft.Office.Project.Server (0x1694) 0x1558 SharePoint Foundation General 8kh7 High Access denied. You do not have permission to perform this action or access this resource.<nativehr>0x810200ce</nativehr><nativestack></nativestack> 237e539d-3835-4ec0-84d3-383759fccdb6
01/14/2014 15:43:07.23 Microsoft.Office.Project.Server (0x1694) 0x1AAC Project Server Queu cf0l Critical Standard Information:PSI Entry Point: Project User: …. Correlation Id: 237e539d-3835-4ec0-84d3-383759fccdb6 PWA Site URL: http://server/PWA SSP Name: Project Server Service Application PSError: GeneralQueueJobFailed (26000) A queue job has failed …
Investigation and Resolution
I’ve highlighted the interesting bits of the logs above, and on searching for this specific issue the only references relate to Search Service indicating one of the following causes:
- The FarmAdmin account is not a local admin on the servers.
- One or more AD accounts has been deleted and recreated with the same name.
So after speaking with the AD team it turns out the customer was in the middle of an Active Directory restructure (alarm bells start ringing!) and specifically about the same time when this issue was first reported the AD team had moved a group of PWA users into a new sub-domain in the same AD forest.
Reverting that change immediately corrected the isssue! Phew.
Not completely done yet though, as that change will need to be re-done in the near future further investigation was required. It turns out that the migration of accounts was being done in a staged manner, and specifically service accounts and admin accounts (including our Farm Admin) were NOT moved with the users, which is what caused our issue here.
If anyone else comes across this issue in the future let me know, as I have a strange feeling that this might be the first and last time this particular issue breaks a Project Server. :)
by Martin Laukkanen | Dec 20, 2011 | Project 2010, Project Pro 2010
This issue has recently affected more than one of my customers, so I thought after a few months working with Microsoft support it is definitely worth sharing;
Problem:
When attempting to disable BCM mode from Server Settings in a migrated Project Server 2010 environment, unchecking the BCM mode option leads to corruption in multiple project schedules when opened in MS Project 2010.
The following message is displayed:
An unexpected problem occurred while opening the file.
The file may be damaged. Try using a backup copy.
Cause:
After much investigation it seems that the enterprise custom fields configured as “Workflow Controlled” are at fault here, it seems that when disabling BCM mode some (not necessarily all!) projects which have values set in custom fields that are workflow controlled will become corrupt according to MS Project 2010. No service patch or cumulative update (as yet: Dec/2011) helps, but clearly something in the configuration of those fields gets messed up when BCM is switched off.
For Google and those into debugging WinProj, the internal error is:
The error occurs because the following error is returned when NonCoreProjectData is read.
The queried PID is Bad_PID, and the call winproj!TBkndPropCntr::GetAccessInfoPid cannot get Access information.
Solution:
Fortunately there is one, although a hotfix might come in the future, for my customer(s) working with MS we were able to find a procedure to fix the corruption.
Firstly when you disable BCM (by unchecking Enable Project 2007 Compatibility Mode in Server Settings – Additional Server Settings) the following SQL query can be used to identify any projects that may fail:
select distinct PROJ.PROJ_NAME from MSP_PROJECTS as PROJ
inner join MSP_PROJ_CUSTOM_FIELD_VALUES as PROJCF
on PROJ.PROJ_UID = PROJCF.PROJ_UID
inner join MSP_CUSTOM_FIELDS CF
on CF.MD_PROP_ID = PROJCF.MD_PROP_ID
where MD_PROP_IS_WORKFLOW_CONTROLLED = 1
order by PROJ.PROJ_NAME
In my recent case it was just about every project! But regardless of how many the following steps will correct the issue for those identified projects:
- Backup your 4 x Project Server and 1 x SharePoint databases!
- Copy Workflow Stage configuration from Server Settings – Workflow Stages. Copy Grid data to Excel, the columns required most are; Stage Name, Required Custom Fields and Read Only Custom Fields. (You’ll need this info later!)
- Copy Enterprise Custom Field configuration from Server Settings – Enterprise Custom Fields. Copy Grid to Excel. (Again this is for later reference)
- Open each Workflow Controlled custom field from Server Settings and uncheck “Workflow Controlled” then save, this removes every field from every workflow stage configuration.
- Note: For each field this will REMOVE the Read Only and Required configuration from each workflow stage where this field is used! (Make sure you have your backup from step #1!)
- If required Force Check in all checked out projects, check queue to ensure all jobs complete before continuing.
- Restart MS Project before continuing!
- Open each affected project, save and publish them. (Note a full publish from MS Project is required – nope bulk publish using ProjTool doesn’t help!)
- From Server Setting – Additional Server Settings – Uncheck "Enable Project 2007 Compatibility Mode"
- Now to correct the custom field and workflow configuration, re-open each custom field previously changed and recheck the “Workflow Controlled” setting. (Using the information backed up in step #3)
- Now Reconfigure the required custom fields and read-only custom fields for ALL workflow stages. (Using the information backed up in step #2)
- Restart MS Project before continuing!
- Re-test affected projects.
The process effectively removes the “corruption” caused by the Workflow Controlled attribute in those projects, and fortunately if you are stuck after unchecking the BCM box without a backup using these steps (minus step 6) still should work!
I hope if you have this issue that you are seeing it only in Dev, as there is no better test of a DR procedure than unchecking that one little check box! :)
Hope that helps someone else out there. Update 2/05: Minor re-ordering of the steps above based on some feedback.
by Martin Laukkanen | Nov 24, 2011 | How to, Project 2010
One thing I always have to search for when creating a new EPT is the appropriate icon to use. I have previously found and used some of the out-of-the-box icons that can be found in any default Project Server 2010 installation and thought that I might share them here.
There is in fact a long list which I have included below (forgive me copyright gods in MS!), to use one of these icons, just update the image URL under Server Settings –> Enterprise Project Types, for example to update the default Sample Proposal with any of the icons below, simply modify the image URL under:
For a Master Project icon (taken from the image below), replace the URL above with the following:
/_layouts/inc/pwa/images/CenterMasterProject.png
Full list of icons (16×16 size only):
Enjoy!
by Martin Laukkanen | Sep 23, 2011 | Troubleshooting
I hate log noise, errors like the following:
Log Name: Application
Source: Microsoft-SharePoint Products-Project Server
Date: 23/09/2011 11:16:05 AM
Event ID: 2077
Task Category: General
Level: Error
Keywords:
User: NT AUTHORITYIUSR
Computer: server.domain.local
Description:
Failed to get language information for Project Server (https://server/PWA)
Apparently this is an error that can be ignored, hence my term “log noise”, however if you are managing large farms having hundreds or even thousands of these occur each night across a number of servers is clearly a big problem.
So I set out to resolve this one, and the cause turned out to be rather simple.
The errors occur nightly during the same window as the Full Search Crawl, so it’s not hard to find the culprit, a bit of searching in the ULS logs correlates the above with some or all of the following errors:
09/23/2011 11:16:05.26 w3wp.exe (0x194C) 0x2A74 Project Server General auvt Exception System.Web.Services.Protocols.SoapException: ProjectServerError(s) LastError=GeneralSecurityAccessDenied Instructions: Pass this into PSClientError constructore to access all error information at Microsoft.Office.Project.Server.WebServiceProxy.Admin.ListInstalledLanguages() at Microsoft.Office.Project.PWA.PJLanguageInfo.EnsureLanguageInfo
Cause
The cause is simply that the Search Default Content Access Account does not have permissions in PWA. Typically said accounts would be granted a Web Application “Full Read” policy in order to index content, however although this allows the account to index all the content without issue it does result in these errors still being thrown up by the PSI.
In that sense this can be safely ignored, however if like me you need to get rid of these the solution is simple; just setup the search account in PWA. In my case on a default test PWA instance granting Team Member rights resolved 14 out of 16 of these, so some level higher than that is required to fully resolve this.
by Martin Laukkanen | Jul 16, 2011 | Exchange, Troubleshooting
Recently I worked with a large customer getting to the bottom of some issues experienced with Exchange Sync in Project Server 2010, specifically the issues originated due to the large multi-forest nature of the Active Directory environment in which the Project Server was deployed.
Symptoms
Exchange Sync for resources not working, little is logged in the ULS without verbose logging other than the following:
06/30/2011 12:25:49.04 Microsoft.Office.Project.Server (0x0E3C) 0x08C4 Project Server Queue 954k Medium PWA:http://pwa.something/PWA, ServiceApp:Project Service Application, User DOMAINServiceAcc, PSI: [QUEUE] Retry: 1 ExchangeSyncTasks Microsoft.Office.Project.Server.BusinessLayer.QueueMsg.ExchangeSyncTasks
To get to the bottom of what was happening Verbose logging on Project Server Exchange Tasks was required which showed far more detail which I will summaries here:
06/30/2011 12:25:48.00 Microsoft.Office.Project.Server (0x0E3C) 0x16D4 Project Server Exchange Sync fux2 Verbose -! Info: System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond xxx.xxx.xxx.xxx:443 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at … …
06/30/2011 12:25:48.00 Microsoft.Office.Project.Server (0x0E3C) 0x16D4 Project Server Exchange Sync fux2 Verbose =? Trying to Autodiscover using email at ‘https://autodiscover.companydomain.com/autodiscover/autodiscover.xml’
06/30/2011 12:25:48.04 Microsoft.Office.Project.Server (0x0E3C) 0x16D4 Project Server Exchange Sync fux2 Verbose -! Info: System.Net.WebException: The remote name could not be resolved: ‘autodiscover.companydomain.com’ at System.Net.HttpWebRequest.GetRequestStream … …
06/30/2011 12:25:48.04 Microsoft.Office.Project.Server (0x0E3C) 0x16D4 Project Server Exchange Sync 1zpd Verbose Error is: ExchangeSyncEWSUrlFailed. Details: Attributes: a6a779c4-a930-461f-a738-a3c78e8e826a . Standard Information: PSI Entry Point: Project User: DOMAINusername Correlation Id: a1e43a95-1a78-44a0-be1b-e8b3aa267074 PWA Site URL: SSP Name: Project Service Application PSError: ExchangeSyncEWSUrlFailed (40509)
06/30/2011 12:25:48.04 Microsoft.Office.Project.Server (0x0E3C) 0x16D4 Project Server Exchange Sync 9fbi Verbose Error is: ExchangeSyncGeneralProcessingFailure. Details: Attributes: a6a779c4-a930-461f-a738-a3c78e8e826a Microsoft.Office.Project.Server.BusinessLayer.Queue.
ExchangeSyncEmailAddressInvalidException: Could not find Exchange server for resource a6a779c4-a930-461f-a738-a3c78e8e826a at Microsoft.Office.Project.Server.BusinessLayer.Queue.
ProcessExchangeSyncMessage.ExecuteSync … …
06/30/2011 12:25:49.04 Microsoft.Office.Project.Server (0x0E3C) 0x0DD0 Project Server Exchange Sync fux2 Verbose ?? Starting SCP lookup for domainName=’companydomain.com’, root path=”
[cut out multiple variations of the above as Autodiscover tries in vain to find the account]
If you’ve read through that, you can see that Project Server is attempting to use the Exchange Autodiscover service to locate the users Exchange details, as per the Autodiscover protocol which very very basically is something like this:
- Look for an Autodiscover service on the host (then look on the same host without the autodiscover bit)
- If not found try an SCP (Service Connection Point) lookup in Active Directory to find the resources
This can fail in a multi-forest Active Directory environment if your Exchange Client Access Servers (CAS) are located in a different forest. As was the case with my customer.
Solution
This is actually a well understood problem in Exchange circles, and the full solution is well documented:
How to Configure the Autodiscover Service for Multiple Forests
However if like in my case you don’t have the time to get a significant change such as the above completed then an alternative is needed.
Workaround
Looking at the above autodiscover procedure an easy “workaround” (I call this one a ‘hack’) is clear;
- Add an entry to the HOSTS file on all Project Server application servers pointing to autodiscover.companyemaildomain.com which points to the actual autodiscover service running on the Exchange CAS.
However now you’ll see something like this in your Verbose ULS:
06/30/2011 12:40:04.17 Microsoft.Office.Project.Server (0x0E3C) 0x1930 Project Server Exchange Sync fux2 Verbose -! Info: System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. —> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
This is because your Exchange CAS has something like as the Subject CN in its SSL certificate which would be expected, so what you need to do add a Subject Alternative Name to your certificate, see the following article on doing this:
Configure SSL Certificates to Use Multiple Client Access Server Host Names
Now you’re almost there, just one last thing! In order for the Exchange and Project Server to both authenticate these cross-forest users you need to make some changes on the service account on both forests by configuring the msExchMasterAccountSid property on the Exchange forest. See the following similar technet forum solution to achieve this:
EWS returns error "Failed to get valid Active Directory information for the calling account"
Done.
Hope that helps someone else out there!
