Queue error creating project site or synchronizing site permissions

It’s not often these days I come across what seems to be a new Project Server 2010 error! Having said that, I would expect the same error to occur on either 2007 or 2013 as it relates to AD sync.

Anyway while working with a customer recently I came across the following issue that had started occurring for all new projects being created as well as on any attempt to synchronize the site permissions from Server Settings – Project Sites.

The effect of the error was that users cannot access any of the affected sites.

Notably though the AD sync was not experiencing any errors.

Error Messages

GeneralQueueJobFailed (26000) – SynchronizeMembershipForWssSite.SynchronizeMembershipForWssSiteMessage. Details: id=’26000′ name=’GeneralQueueJobFailed’ uid=’677df8c5-7409-4e20-8a05-e98395aa2af3′ JobUID=’3ca3af00-2e08-49aa-b693-f1314fc09b96′ ComputerName=’…’ GroupType=’SynchronizeMembershipForWssSite’ MessageType=’SynchronizeMembershipForWssSiteMessage’ MessageId=’1′ Stage=”

And in the ULS log we see some more details:

01/14/2014 15:43:07.23         Microsoft.Office.Project.Server (0x1694) 0x1558 SharePoint Foundation General 8kh7 High  Access denied.  You do not have permission to perform this action or access this resource.<nativehr>0x810200ce</nativehr><nativestack></nativestack>        237e539d-3835-4ec0-84d3-383759fccdb6

01/14/2014 15:43:07.23         Microsoft.Office.Project.Server (0x1694) 0x1AAC        Project Server Queu cf0l  Critical Standard Information:PSI Entry Point:   Project User: ….  Correlation Id: 237e539d-3835-4ec0-84d3-383759fccdb6  PWA Site URL: http://server/PWA  SSP Name: Project Server Service Application  PSError: GeneralQueueJobFailed (26000) A queue job has failed …

Investigation and Resolution

I’ve highlighted the interesting bits of the logs above, and on searching for this specific issue the only references relate to Search Service indicating one of the following causes:

  • The FarmAdmin account is not a local admin on the servers.
  • One or more AD accounts has been deleted and recreated with the same name.

So after speaking with the AD team it turns out the customer was in the middle of an Active Directory restructure (alarm bells start ringing!) and specifically about the same time when this issue was first reported the AD team had moved a group of PWA users into a new sub-domain in the same AD forest.

Reverting that change immediately corrected the isssue! Phew.

Not completely done yet though, as that change will need to be re-done in the near future further investigation was required. It turns out that the migration of accounts was being done in a staged manner, and specifically service accounts and admin accounts (including our Farm Admin) were NOT moved with the users, which is what caused our issue here.

 

If anyone else comes across this issue in the future let me know, as I have a strange feeling that this might be the first and last time this particular issue breaks a Project Server. :)

Disabling Backwards Compatibility Mode results in Corruption

This issue has recently affected more than one of my customers, so I thought after a few months working with Microsoft support it is definitely worth sharing;

Problem:

When attempting to disable BCM mode from Server Settings in a migrated Project Server 2010 environment, unchecking the BCM mode option leads to corruption in multiple project schedules when opened in MS Project 2010.

The following message is displayed:

clip_image002

An unexpected problem occurred while opening the file.

The file may be damaged. Try using a backup copy.

 

Cause:

After much investigation it seems that the enterprise custom fields configured as “Workflow Controlled” are at fault here, it seems that when disabling BCM mode some (not necessarily all!) projects which have values set in custom fields that are workflow controlled will become corrupt according to MS Project 2010. No service patch or cumulative update (as yet: Dec/2011) helps, but clearly something in the configuration of those fields gets messed up when BCM is switched off.

For Google and those into debugging WinProj, the internal error is:

The error occurs because the following error is returned when NonCoreProjectData is read.

The queried PID is Bad_PID, and the call winproj!TBkndPropCntr::GetAccessInfoPid cannot get Access information.

 

Solution:

Fortunately there is one, although a hotfix might come in the future, for my customer(s) working with MS we were able to find a procedure to fix the corruption.

Firstly when you disable BCM (by unchecking Enable Project 2007 Compatibility Mode in Server Settings – Additional Server Settings) the following SQL query can be used to identify any projects that may fail:

select distinct PROJ.PROJ_NAME from MSP_PROJECTS as PROJ

inner join MSP_PROJ_CUSTOM_FIELD_VALUES as PROJCF

on PROJ.PROJ_UID = PROJCF.PROJ_UID

inner join MSP_CUSTOM_FIELDS CF

on CF.MD_PROP_ID = PROJCF.MD_PROP_ID

where MD_PROP_IS_WORKFLOW_CONTROLLED = 1

order by PROJ.PROJ_NAME

 

In my recent case it was just about every project! But regardless of how many the following steps will correct the issue for those identified projects:

  1. Backup your 4 x Project Server and 1 x SharePoint databases!
  2. Copy Workflow Stage configuration from Server Settings – Workflow Stages. Copy Grid data to Excel, the columns required most are; Stage Name,  Required Custom Fields and Read Only Custom Fields. (You’ll need this info later!)
  3. Copy Enterprise Custom Field configuration from Server Settings – Enterprise Custom Fields. Copy Grid to Excel. (Again this is for later reference)
  4. Open each Workflow Controlled custom field from Server Settings and uncheck “Workflow Controlled” then save, this removes every field from every workflow stage configuration.
    1. Note: For each field this will REMOVE the Read Only and Required configuration from each workflow stage where this field is used! (Make sure you have your backup from step #1!)
  5. If required Force Check in all checked out projects, check queue to ensure all jobs complete before continuing.
  6. Restart MS Project before continuing!
  7. Open each affected project, save and publish them. (Note a full publish from MS Project is required – nope bulk publish using ProjTool doesn’t help!)
  8. From Server Setting – Additional Server Settings – Uncheck "Enable Project 2007 Compatibility Mode"
  9. Now to correct the custom field and workflow configuration, re-open each custom field previously changed and recheck the “Workflow Controlled” setting. (Using the information backed up in step #3)
  10. Now Reconfigure the required custom fields and  read-only custom fields for ALL workflow stages. (Using the information backed up in step #2)
  11. Restart MS Project before continuing!
  12. Re-test affected projects.

 

The process effectively removes the “corruption” caused by the Workflow Controlled attribute in those projects, and fortunately if you are stuck after unchecking the BCM box without a backup using these steps (minus step 6) still should work!

I hope if you have this issue that you are seeing it only in Dev, as there is no better test of a DR procedure than unchecking that one little check box! :)

 

Hope that helps someone else out there. Update 2/05: Minor re-ordering of the steps above based on some feedback.

Enterprise Project Type Icons Available

One thing I always have to search for when creating a new EPT is the appropriate icon to use. I have previously found and used some of the out-of-the-box icons that can be found in any default Project Server 2010 installation and thought that I might share them here.

There is in fact a long list which I have included below (forgive me copyright gods in MS!), to use one of these icons, just update the image URL under Server Settings –> Enterprise Project Types, for example to update the default Sample Proposal with any of the icons below, simply modify the image URL under:

eptconf

For a Master Project icon (taken from the image below), replace the URL above with the following:

/_layouts/inc/pwa/images/CenterMasterProject.png

 

Full list of icons (16×16 size only):

Image1

Image2

Image3

 

Enjoy!

Error: Failed to get language information for Project Server

I hate log noise, errors like the following:

Log Name: Application

Source: Microsoft-SharePoint Products-Project Server

Date: 23/09/2011 11:16:05 AM

Event ID: 2077

Task Category: General

Level: Error

Keywords:

User: NT AUTHORITYIUSR

Computer: server.domain.local

Description:

Failed to get language information for Project Server (https://server/PWA)

 

Apparently this is an error that can be ignored, hence my term “log noise”, however if you are managing large farms having hundreds or even thousands of these occur each night across a number of servers is clearly a big problem.

So I set out to resolve this one, and the cause turned out to be rather simple.

The errors occur nightly during the same window as the Full Search Crawl, so it’s not hard to find the culprit, a bit of searching in the ULS logs correlates the above with some or all of the following errors:

09/23/2011 11:16:05.26    w3wp.exe (0x194C)    0x2A74    Project Server    General    auvt    Exception    System.Web.Services.Protocols.SoapException: ProjectServerError(s) LastError=GeneralSecurityAccessDenied Instructions: Pass this into PSClientError constructore to access all error information     at Microsoft.Office.Project.Server.WebServiceProxy.Admin.ListInstalledLanguages() at Microsoft.Office.Project.PWA.PJLanguageInfo.EnsureLanguageInfo

 

Cause

The cause is simply that the Search Default Content Access Account does not have permissions in PWA. Typically said accounts would be granted a Web Application “Full Read” policy in order to index content, however although this allows the account to index all the content without issue it does result in these errors still being thrown up by the PSI.

In that sense this can be safely ignored, however if like me you need to get rid of these the solution is simple; just setup the search account in PWA. In my case on a default test PWA instance granting Team Member rights resolved 14 out of 16 of these, so some level higher than that is required to fully resolve this.

Exchange Sync Issues with large Active Directory

Recently I worked with a large customer getting to the bottom of some issues experienced with Exchange Sync in Project Server 2010, specifically the issues originated due to the large multi-forest nature of the Active Directory environment in which the Project Server was deployed.

 

Symptoms

Exchange Sync for resources not working, little is logged in the ULS without verbose logging other than the following:

06/30/2011 12:25:49.04        Microsoft.Office.Project.Server (0x0E3C)        0x08C4        Project Server        Queue        954k   Medium  PWA:http://pwa.something/PWA, ServiceApp:Project Service Application, User DOMAINServiceAcc, PSI: [QUEUE] Retry: 1 ExchangeSyncTasks Microsoft.Office.Project.Server.BusinessLayer.QueueMsg.ExchangeSyncTasks 

To get to the bottom of what was happening Verbose logging on Project Server Exchange Tasks was required which showed far more detail which I will summaries here:

06/30/2011 12:25:48.00        Microsoft.Office.Project.Server (0x0E3C)        0x16D4        Project Server        Exchange Sync    fux2     Verbose        -! Info: System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond xxx.xxx.xxx.xxx:443 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at … …

06/30/2011 12:25:48.00        Microsoft.Office.Project.Server (0x0E3C)        0x16D4        Project Server        Exchange Sync   fux2  Verbose        =? Trying to Autodiscover using email at ‘https://autodiscover.companydomain.com/autodiscover/autodiscover.xml’ 

06/30/2011 12:25:48.04        Microsoft.Office.Project.Server (0x0E3C)        0x16D4        Project Server        Exchange Sync   fux2  Verbose        -! Info: System.Net.WebException: The remote name could not be resolved: ‘autodiscover.companydomain.com’ at System.Net.HttpWebRequest.GetRequestStream … …

06/30/2011 12:25:48.04        Microsoft.Office.Project.Server (0x0E3C)        0x16D4        Project Server        Exchange Sync   1zpd  Verbose        Error is: ExchangeSyncEWSUrlFailed. Details: Attributes: a6a779c4-a930-461f-a738-a3c78e8e826a . Standard Information: PSI Entry Point: Project User: DOMAINusername Correlation Id: a1e43a95-1a78-44a0-be1b-e8b3aa267074 PWA Site URL: SSP Name: Project Service Application PSError: ExchangeSyncEWSUrlFailed (40509)

06/30/2011 12:25:48.04        Microsoft.Office.Project.Server (0x0E3C)        0x16D4        Project Server        Exchange Sync  9fbi  Verbose        Error is: ExchangeSyncGeneralProcessingFailure. Details: Attributes: a6a779c4-a930-461f-a738-a3c78e8e826a Microsoft.Office.Project.Server.BusinessLayer.Queue.
ExchangeSyncEmailAddressInvalidException: Could not find Exchange server for resource a6a779c4-a930-461f-a738-a3c78e8e826a at Microsoft.Office.Project.Server.BusinessLayer.Queue.
ProcessExchangeSyncMessage.ExecuteSync … …

06/30/2011 12:25:49.04        Microsoft.Office.Project.Server (0x0E3C)        0x0DD0        Project Server        Exchange Sync  fux2  Verbose        ?? Starting SCP lookup for domainName=’companydomain.com’, root path=”   

[cut out multiple variations of the above as Autodiscover tries in vain to find the account]

If you’ve read through that, you can see that Project Server is attempting to use the Exchange Autodiscover service to locate the users Exchange details, as per the Autodiscover protocol which very very basically is something like this:

  1. Look for an Autodiscover service on the host (then look on the same host without the autodiscover bit)
  2. If not found try an SCP (Service Connection Point) lookup in Active Directory to find the resources

This can fail in a multi-forest Active Directory environment if your Exchange Client Access Servers (CAS) are located in a different forest. As was the case with my customer.

 

Solution

This is actually a well understood problem in Exchange circles, and the full solution is well documented:

How to Configure the Autodiscover Service for Multiple Forests

However if like in my case you don’t have the time to get a significant change such as the above completed then an alternative is needed.

 

Workaround

Looking at the above autodiscover procedure an easy “workaround” (I call this one a ‘hack’) is clear;

  • Add an entry to the HOSTS file on all Project Server application servers pointing to autodiscover.companyemaildomain.com which points to the actual autodiscover service running on the Exchange CAS.

However now you’ll see something like this in your Verbose ULS:

06/30/2011 12:40:04.17        Microsoft.Office.Project.Server (0x0E3C)        0x1930        Project Server        Exchange Sync        fux2        Verbose        -! Info: System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. —> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

This is because your Exchange CAS has something like as the Subject CN in its SSL certificate which would be expected, so what you need to do add a Subject Alternative Name to your certificate, see the following article on doing this:

Configure SSL Certificates to Use Multiple Client Access Server Host Names

Now you’re almost there, just one last thing! In order for the Exchange and Project Server to both authenticate these cross-forest users you need to make some changes on the service account on both forests by configuring the msExchMasterAccountSid property on the Exchange forest. See the following similar technet forum solution to achieve this:

EWS returns error "Failed to get valid Active Directory information for the calling account"

 

Done.

Hope that helps someone else out there!