Queue error creating project site or synchronizing site permissions

It’s not often these days I come across what seems to be a new Project Server 2010 error! Having said that, I would expect the same error to occur on either 2007 or 2013 as it relates to AD sync.

Anyway while working with a customer recently I came across the following issue that had started occurring for all new projects being created as well as on any attempt to synchronize the site permissions from Server Settings – Project Sites.

The effect of the error was that users cannot access any of the affected sites.

Notably though the AD sync was not experiencing any errors.

Error Messages

GeneralQueueJobFailed (26000) – SynchronizeMembershipForWssSite.SynchronizeMembershipForWssSiteMessage. Details: id=’26000′ name=’GeneralQueueJobFailed’ uid=’677df8c5-7409-4e20-8a05-e98395aa2af3′ JobUID=’3ca3af00-2e08-49aa-b693-f1314fc09b96′ ComputerName=’…’ GroupType=’SynchronizeMembershipForWssSite’ MessageType=’SynchronizeMembershipForWssSiteMessage’ MessageId=’1′ Stage=”

And in the ULS log we see some more details:

01/14/2014 15:43:07.23         Microsoft.Office.Project.Server (0x1694) 0x1558 SharePoint Foundation General 8kh7 High  Access denied.  You do not have permission to perform this action or access this resource.<nativehr>0x810200ce</nativehr><nativestack></nativestack>        237e539d-3835-4ec0-84d3-383759fccdb6

01/14/2014 15:43:07.23         Microsoft.Office.Project.Server (0x1694) 0x1AAC        Project Server Queu cf0l  Critical Standard Information:PSI Entry Point:   Project User: ….  Correlation Id: 237e539d-3835-4ec0-84d3-383759fccdb6  PWA Site URL: http://server/PWA  SSP Name: Project Server Service Application  PSError: GeneralQueueJobFailed (26000) A queue job has failed …

Investigation and Resolution

I’ve highlighted the interesting bits of the logs above, and on searching for this specific issue the only references relate to Search Service indicating one of the following causes:

  • The FarmAdmin account is not a local admin on the servers.
  • One or more AD accounts has been deleted and recreated with the same name.

So after speaking with the AD team it turns out the customer was in the middle of an Active Directory restructure (alarm bells start ringing!) and specifically about the same time when this issue was first reported the AD team had moved a group of PWA users into a new sub-domain in the same AD forest.

Reverting that change immediately corrected the isssue! Phew.

Not completely done yet though, as that change will need to be re-done in the near future further investigation was required. It turns out that the migration of accounts was being done in a staged manner, and specifically service accounts and admin accounts (including our Farm Admin) were NOT moved with the users, which is what caused our issue here.

 

If anyone else comes across this issue in the future let me know, as I have a strange feeling that this might be the first and last time this particular issue breaks a Project Server. :)

ADFS Login Failure on one SharePoint site collection

After migrating a Project Server (or any other SharePoint web app) to a new SharePoint 2010 farm using ADFS for federated authentication and multi-tenancy you receive the below error message when trying to access some of the migrated site collections. In my case the root site worked but /PWA failed.

Also if you reprovision a new site collection (include a PWA) in the same web application it also works fine.

 

Error message from ADFS

clip_image001

There was a problem accessing the site. Try to browse to the site again.

If the problem persists, contact the administrator of this site and provide the reference number to identify the problem.

Reference number: [GUID]

 

Event log on ADFS server

Log Name: AD FS 2.0/Admin
Source: AD FS 2.0
Date: 5/11/2012 10:00:43 PM
Event ID: 364
Task Category: None
Level: Error
Keywords: AD FS

Description:

Encountered error during federation passive request.
Additional Data

Exception details:
Microsoft.IdentityServer.Web.InvalidRequestException: MSIS7042: The same client browser session has made ‘6’ requests in the last ‘5’ seconds. Contact your administrator for details.

at Microsoft.IdentityServer.Web.FederationPassiveAuthentication. UpdateLoopDetectionCookie()

at Microsoft.IdentityServer.Web.FederationPassiveAuthentication. SendSignInResponse(MSISSignInResponse response)

 

Cause

What’s happening is that while authentication works as expected on some sites, when you open a particular site collection the authentication goes into a loop, eventually failing when the ADFS server detects this redirect loop.

After a fair amount of digging the problem turned out to be in the multi-tenancy configuration of this particular farm. Specifically the site collections for some reason did not all have a subscription.

Using the PowerShell command ‘Get-SPSite https://site/sitecollection1’ against two site collections to compare the settings turned up the key difference:

image

 

This makes sense as to why the authentication is failing; the different site subscription overrides the FedAuth authentication cookie assigned by ADFS as soon as it is assigned, causing the page to redirect back to re-authenticate.

 

Resolution

Fortunately once the above was found the fix is simple, run the following command in PowerShell:

Get-SPSite | Set-SPSite 
–SiteSubscription [Guide of site subscription]

 

Now after a quick IISRESET all should be working!

Recovering from Project Server corruption

A customer of mine recently had to recover from a worst case corruption scenario; one in which reverting to a backup was not an option. In short a hardware change resulted in progressive database corruption that was not picked up for over two weeks(!), leaving few options but to attempt to recover the corruption in-place.

Firstly though we had to fix the SQL databases, for that the following command was needed;

alter database ProjectServer_Published SET SINGLE_USER WITH ROLLBACK IMMEDIATE

DBCC CHECKDB(‘ProjectServer_Published’, REPAIR_ALLOW_DATA_LOSS)

alter database ProjectServer_Published SET MULTI_USER;

Yes that parameter does say “Allow Data Loss”, once you run this there is no going back, the databases are now free of consistency errors and other corruptions but in order to reach that state the DBCC command probably had to delete data!

So what now?

Well (fortunately for once!!) Project Server and Project Professional both are no stranger to application level corruption (*cough* 2007 *cough*), so despite DBCC repairing 7000+ consistency errors in the published database alone we are not without options.

So let’s look at some of the errors we had to deal with, firstly the MS Project client side errors and how we dealt with them.

 

MS Project Errors

 

Error 1. There is a circular relationship in task …

image

This one was surprising, but a sign that most likely the last save attempt for this particular schedule wrote corruption to some of the tasks.

To correct this you need to identify the task and correct or delete the predecessors, in my experience the circular reference is pretty obvious so fixing it is quick, however if that’s not possible then I’ve previously found this procedure helpful: http://www.domorethanmanage.com/articles/2008/05/29/Resolvingcircularreferenc.html

 

Error 2. Save Error’s 9000(0x2328) and 26005(0x6595)

clip_image001

clip_image002[6]

I’ve put these two together as the solution is the same, in fact this procedure also applies to the Publish errors that you may get also see.

If you see the above, the first thing to try is an administrative restore, however if that fails there is one last resort;

The “XML Round Trip” method:

1. Open the project in MS Project.

2. File, Save As, Save As File.

3. When prompted select to save with “All Enterprise Custom Fields”.

4. In the file dialog select the Save as type: XML

image

5. Save the file to your local computer.

6. Once saved, close the project (don’t forget to check in!) and reopen the saved XML file, when prompted choose to import ‘As a new project’.

7. Once opened, select Save As and save the project to the project server using the exact same project name.

8. When prompted say yes to overwrite the existing project (you did check it in right?)

9. Publish and your done.

Note; This process will overwrite any previously saved data for that project, the impacts of this are as follows;

  • In-progress timesheets will be affected, remember that internally all tasks and assignments are recreated, so existing OPEN timesheet periods will be updated. This may result in un-submitted timesheets losing actual work, or worse in progress timesheets failing to submit.
  • Reports or custom add-in’s relying on GUID’s will likely have issues.

 

Error 3: Unexpected problem occurred while opening the file

image

If you see this, then hopefully one of the following works, if not you might be out of luck!

  • Open the project from the local project cache.
  • Open the PUBLISHED version of the project by selecting the appropriate Store from the open dialog.

image

  • Restore the project from administrative backup.

 

PWA Errors

Next, a number of errors in PWA can show up as a result of corruption, it’s worth mentioning that pretty much all of the data in PWA comes solely from the Published database. This is good to know because typically 99% of them can be fixed by simply re-publishing!

That’s all good when dealing with a single project, however when your dealing with multiple project corruption, or simply more commonly when a view fails and you don’t know *what* project caused it this method of bulk-publishing will come in handy:

 

Bulk Publishing Projects using ProjTool

Firstly get yourself ProjTool 2010: http://blogs.msdn.com/b/project_programmability/archive/2010/11/03/projtool-for-project-server-2010.aspx

This tool is extremely powerful (IE destructive!) and should be used with caution, but just completing these steps you don’t need to get too deep into the tool so your not risking too much:

1. Open ProjTool, enter project server URL and select Windows for Authentication.

2. Select the projects to republish (multi-select by holding shift).

3. Click Publish from the Toolbar, when prompted select YES for a FULL Publish.

image

From experience selecting over a couple of hundred projects will cause this to fail, but I have often used this to queue up 80+ projects for a republish.

Note however that almost ALWAYS ProjTool will report a problem in the queue, that’s because it only waits for a fixed time period for the queue jobs to complete and that never seems to be enough.

 

PWA View Errors

image

(The view failed to load. Press OK to reload…)

image

(An unknown error has occurred.)

These types of errors are likely caused by one of the following:

  1. Corruption in the project data being viewed
  2. Basic corruption in the view configuration.

For #1 this can be a custom field value that is filtered where one project has an invalid (null?) value for the field in question, so often just removing any filters configured on the view will fix this.

If you identify the filter in question then see the tips above about republishing (or possibly re-saving AND re-publishing) some or all projects to correct the data.

For #2 there are few options other than to recreate the view, copying the current view sometimes helps but not reliably.

 

Project Workspace Permissions Issues

I’m not going to cover how to restore document or general SharePoint data corruption in this blog, but in terms of Project Server usage you’ll most likely have issues relating to to the project permissions synchronisation, for this see the following link: https://nearbaseline.com/blog/2011/02/resetting-lost-permissions-in-project-server-2010/

 

Other Issues

Duplication of Custom Field values in PDPs:

image

This one turned out to be just an odd co-incidence in timing see the following recent Microsoft kb article if you see this issue;

http://support.microsoft.com/kb/2598251

 

Finally Reporting Database Refresh

Once all the above has been corrected you may still see some errors in Reporting jobs in the queue;

ReportingProjectChangeMessageFailed (24006) – The INSERT statement conflicted with the FOREIGN KEY constraint

There are a few options available for this problem, two are well described here: http://www.epmpartners.com.au/blog/insert-statement-conflicted-with-the-foreign-key-constraint/

I’d suggest the RDB Refresh (Option 2 in the blog linked), this will take some time (hours typically), and any failures will result in projects NOT being listed in the reporting database at all. This is because the job removes and resynchronises the projects, so typically after running this job I usually use ProjTool to do a bulk publish (see above).

Final Words

The moral of this story is simple: Make sure your SQL DBA regularly uses DBCC CHECKDB (http://www.sql-server-pro.com/dbcc-checkdb.html) to ensure you don’t get into this sort of mess!

Disabling Backwards Compatibility Mode results in Corruption

This issue has recently affected more than one of my customers, so I thought after a few months working with Microsoft support it is definitely worth sharing;

Problem:

When attempting to disable BCM mode from Server Settings in a migrated Project Server 2010 environment, unchecking the BCM mode option leads to corruption in multiple project schedules when opened in MS Project 2010.

The following message is displayed:

clip_image002

An unexpected problem occurred while opening the file.

The file may be damaged. Try using a backup copy.

 

Cause:

After much investigation it seems that the enterprise custom fields configured as “Workflow Controlled” are at fault here, it seems that when disabling BCM mode some (not necessarily all!) projects which have values set in custom fields that are workflow controlled will become corrupt according to MS Project 2010. No service patch or cumulative update (as yet: Dec/2011) helps, but clearly something in the configuration of those fields gets messed up when BCM is switched off.

For Google and those into debugging WinProj, the internal error is:

The error occurs because the following error is returned when NonCoreProjectData is read.

The queried PID is Bad_PID, and the call winproj!TBkndPropCntr::GetAccessInfoPid cannot get Access information.

 

Solution:

Fortunately there is one, although a hotfix might come in the future, for my customer(s) working with MS we were able to find a procedure to fix the corruption.

Firstly when you disable BCM (by unchecking Enable Project 2007 Compatibility Mode in Server Settings – Additional Server Settings) the following SQL query can be used to identify any projects that may fail:

select distinct PROJ.PROJ_NAME from MSP_PROJECTS as PROJ

inner join MSP_PROJ_CUSTOM_FIELD_VALUES as PROJCF

on PROJ.PROJ_UID = PROJCF.PROJ_UID

inner join MSP_CUSTOM_FIELDS CF

on CF.MD_PROP_ID = PROJCF.MD_PROP_ID

where MD_PROP_IS_WORKFLOW_CONTROLLED = 1

order by PROJ.PROJ_NAME

 

In my recent case it was just about every project! But regardless of how many the following steps will correct the issue for those identified projects:

  1. Backup your 4 x Project Server and 1 x SharePoint databases!
  2. Copy Workflow Stage configuration from Server Settings – Workflow Stages. Copy Grid data to Excel, the columns required most are; Stage Name,  Required Custom Fields and Read Only Custom Fields. (You’ll need this info later!)
  3. Copy Enterprise Custom Field configuration from Server Settings – Enterprise Custom Fields. Copy Grid to Excel. (Again this is for later reference)
  4. Open each Workflow Controlled custom field from Server Settings and uncheck “Workflow Controlled” then save, this removes every field from every workflow stage configuration.
    1. Note: For each field this will REMOVE the Read Only and Required configuration from each workflow stage where this field is used! (Make sure you have your backup from step #1!)
  5. If required Force Check in all checked out projects, check queue to ensure all jobs complete before continuing.
  6. Restart MS Project before continuing!
  7. Open each affected project, save and publish them. (Note a full publish from MS Project is required – nope bulk publish using ProjTool doesn’t help!)
  8. From Server Setting – Additional Server Settings – Uncheck "Enable Project 2007 Compatibility Mode"
  9. Now to correct the custom field and workflow configuration, re-open each custom field previously changed and recheck the “Workflow Controlled” setting. (Using the information backed up in step #3)
  10. Now Reconfigure the required custom fields and  read-only custom fields for ALL workflow stages. (Using the information backed up in step #2)
  11. Restart MS Project before continuing!
  12. Re-test affected projects.

 

The process effectively removes the “corruption” caused by the Workflow Controlled attribute in those projects, and fortunately if you are stuck after unchecking the BCM box without a backup using these steps (minus step 6) still should work!

I hope if you have this issue that you are seeing it only in Dev, as there is no better test of a DR procedure than unchecking that one little check box! :)

 

Hope that helps someone else out there. Update 2/05: Minor re-ordering of the steps above based on some feedback.

Enterprise Project Type Icons Available

One thing I always have to search for when creating a new EPT is the appropriate icon to use. I have previously found and used some of the out-of-the-box icons that can be found in any default Project Server 2010 installation and thought that I might share them here.

There is in fact a long list which I have included below (forgive me copyright gods in MS!), to use one of these icons, just update the image URL under Server Settings –> Enterprise Project Types, for example to update the default Sample Proposal with any of the icons below, simply modify the image URL under:

eptconf

For a Master Project icon (taken from the image below), replace the URL above with the following:

/_layouts/inc/pwa/images/CenterMasterProject.png

 

Full list of icons (16×16 size only):

Image1

Image2

Image3

 

Enjoy!

Cube Building Errors after SP1 Installation

A customer of mine had this one after completing the upgrade to SP1 recently, basically all cube building would fail with the following error:

Your CBSRequest job failed. Its current state is FailedNotBlocking. It was 0% complete. It entered the queue at 09/02/2011 11:00:26.

[snip…]

The errors returned from the queue are as follows:

Error ID: 17007

Error ID: 26000

[snip…]

<class name="CBS message processor failed">

<error id="17007" name="CBSOlapDatabaseSetupFailure" uid="7d14c29d-a133-492b-baea-2e7c0bec444b" QueueMessageBody="Setting UID=00007829-4392-48b3-b533-5a5a4797e3c9 ASServerName=server ASDBName=FullCube1 ASExtraNetAddress= RangeChoice=0 PastNum=1 PastUnit=0 NextNum=1 NextUnit=0 FromDate=01/04/2010 00:00:00 ToDate=12/31/2011 00:00:00 HighPriority=True" Error="Error Setting Olap Database ‘FullCube1’ roles: Error: This method can only convert identity claims, and only when a logical conversion exists.&#xD;&#xA;Parameter name: encodedClaim" />

This customer is using Claims-NTLM (ie AD users via Claims) for logins and that seems to be the cause of this issue. The give away is clearly in the error: “This method can only convert identity claims”.

Solution

Fortunately the solution turned out to be rather simple, it seems that SP1 does some additional checking when setting up the Cube Roles, as a result when users in the Project Server have issues then this error is caused.

A little more info can be seen in the ULS log:

09/02/2011 12:00:33.18  Microsoft.Office.Project.Server (0x17B0) 0x1A04  SharePoint Foundation   Claims Authentication d01p Medium  ConvertWindowsClaimToWindowsPrincipalName() encountered error: Some or all identity references could not be translated.

As it turns out a number of users in PWA have left the company and in this case as no AD-sync is used some of the accounts had been deleted from Active Directory but not updated in PWA server settings.

It seems also that accounts set as “inactive” also caused this error if they were in a group with the Global “View OLAP Cubes” permission.

The fix was to remove those user accounts from the PWA groups ‘Project Managers’, ‘Portfolio Managers’ and ‘Executives’ (and any others with the above permission).

 

Finally I just tested this in a non-Claims lab, and the problem doesn’t occur, in fact the cube log identifies and lists the invalid account then continues processing, so technically I would call this one a code defect.

 

Hope that helps someone else out there!