Disabling Backwards Compatibility Mode results in Corruption

This issue has recently affected more than one of my customers, so I thought after a few months working with Microsoft support it is definitely worth sharing;

Problem:

When attempting to disable BCM mode from Server Settings in a migrated Project Server 2010 environment, unchecking the BCM mode option leads to corruption in multiple project schedules when opened in MS Project 2010.

The following message is displayed:

clip_image002

An unexpected problem occurred while opening the file.

The file may be damaged. Try using a backup copy.

 

Cause:

After much investigation it seems that the enterprise custom fields configured as “Workflow Controlled” are at fault here, it seems that when disabling BCM mode some (not necessarily all!) projects which have values set in custom fields that are workflow controlled will become corrupt according to MS Project 2010. No service patch or cumulative update (as yet: Dec/2011) helps, but clearly something in the configuration of those fields gets messed up when BCM is switched off.

For Google and those into debugging WinProj, the internal error is:

The error occurs because the following error is returned when NonCoreProjectData is read.

The queried PID is Bad_PID, and the call winproj!TBkndPropCntr::GetAccessInfoPid cannot get Access information.

 

Solution:

Fortunately there is one, although a hotfix might come in the future, for my customer(s) working with MS we were able to find a procedure to fix the corruption.

Firstly when you disable BCM (by unchecking Enable Project 2007 Compatibility Mode in Server Settings – Additional Server Settings) the following SQL query can be used to identify any projects that may fail:

select distinct PROJ.PROJ_NAME from MSP_PROJECTS as PROJ

inner join MSP_PROJ_CUSTOM_FIELD_VALUES as PROJCF

on PROJ.PROJ_UID = PROJCF.PROJ_UID

inner join MSP_CUSTOM_FIELDS CF

on CF.MD_PROP_ID = PROJCF.MD_PROP_ID

where MD_PROP_IS_WORKFLOW_CONTROLLED = 1

order by PROJ.PROJ_NAME

 

In my recent case it was just about every project! But regardless of how many the following steps will correct the issue for those identified projects:

  1. Backup your 4 x Project Server and 1 x SharePoint databases!
  2. Copy Workflow Stage configuration from Server Settings – Workflow Stages. Copy Grid data to Excel, the columns required most are; Stage Name,  Required Custom Fields and Read Only Custom Fields. (You’ll need this info later!)
  3. Copy Enterprise Custom Field configuration from Server Settings – Enterprise Custom Fields. Copy Grid to Excel. (Again this is for later reference)
  4. Open each Workflow Controlled custom field from Server Settings and uncheck “Workflow Controlled” then save, this removes every field from every workflow stage configuration.
    1. Note: For each field this will REMOVE the Read Only and Required configuration from each workflow stage where this field is used! (Make sure you have your backup from step #1!)
  5. If required Force Check in all checked out projects, check queue to ensure all jobs complete before continuing.
  6. Restart MS Project before continuing!
  7. Open each affected project, save and publish them. (Note a full publish from MS Project is required – nope bulk publish using ProjTool doesn’t help!)
  8. From Server Setting – Additional Server Settings – Uncheck "Enable Project 2007 Compatibility Mode"
  9. Now to correct the custom field and workflow configuration, re-open each custom field previously changed and recheck the “Workflow Controlled” setting. (Using the information backed up in step #3)
  10. Now Reconfigure the required custom fields and  read-only custom fields for ALL workflow stages. (Using the information backed up in step #2)
  11. Restart MS Project before continuing!
  12. Re-test affected projects.

 

The process effectively removes the “corruption” caused by the Workflow Controlled attribute in those projects, and fortunately if you are stuck after unchecking the BCM box without a backup using these steps (minus step 6) still should work!

I hope if you have this issue that you are seeing it only in Dev, as there is no better test of a DR procedure than unchecking that one little check box! :)

 

Hope that helps someone else out there. Update 2/05: Minor re-ordering of the steps above based on some feedback.

Share and Enjoy !

Shares

Cube Building Errors after SP1 Installation

A customer of mine had this one after completing the upgrade to SP1 recently, basically all cube building would fail with the following error:

Your CBSRequest job failed. Its current state is FailedNotBlocking. It was 0% complete. It entered the queue at 09/02/2011 11:00:26.

[snip…]

The errors returned from the queue are as follows:

Error ID: 17007

Error ID: 26000

[snip…]

<class name="CBS message processor failed">

<error id="17007" name="CBSOlapDatabaseSetupFailure" uid="7d14c29d-a133-492b-baea-2e7c0bec444b" QueueMessageBody="Setting UID=00007829-4392-48b3-b533-5a5a4797e3c9 ASServerName=server ASDBName=FullCube1 ASExtraNetAddress= RangeChoice=0 PastNum=1 PastUnit=0 NextNum=1 NextUnit=0 FromDate=01/04/2010 00:00:00 ToDate=12/31/2011 00:00:00 HighPriority=True" Error="Error Setting Olap Database ‘FullCube1’ roles: Error: This method can only convert identity claims, and only when a logical conversion exists.&#xD;&#xA;Parameter name: encodedClaim" />

This customer is using Claims-NTLM (ie AD users via Claims) for logins and that seems to be the cause of this issue. The give away is clearly in the error: “This method can only convert identity claims”.

Solution

Fortunately the solution turned out to be rather simple, it seems that SP1 does some additional checking when setting up the Cube Roles, as a result when users in the Project Server have issues then this error is caused.

A little more info can be seen in the ULS log:

09/02/2011 12:00:33.18  Microsoft.Office.Project.Server (0x17B0) 0x1A04  SharePoint Foundation   Claims Authentication d01p Medium  ConvertWindowsClaimToWindowsPrincipalName() encountered error: Some or all identity references could not be translated.

As it turns out a number of users in PWA have left the company and in this case as no AD-sync is used some of the accounts had been deleted from Active Directory but not updated in PWA server settings.

It seems also that accounts set as “inactive” also caused this error if they were in a group with the Global “View OLAP Cubes” permission.

The fix was to remove those user accounts from the PWA groups ‘Project Managers’, ‘Portfolio Managers’ and ‘Executives’ (and any others with the above permission).

 

Finally I just tested this in a non-Claims lab, and the problem doesn’t occur, in fact the cube log identifies and lists the invalid account then continues processing, so technically I would call this one a code defect.

 

Hope that helps someone else out there!

Share and Enjoy !

Shares

Exchange Sync Issues with large Active Directory

Recently I worked with a large customer getting to the bottom of some issues experienced with Exchange Sync in Project Server 2010, specifically the issues originated due to the large multi-forest nature of the Active Directory environment in which the Project Server was deployed.

 

Symptoms

Exchange Sync for resources not working, little is logged in the ULS without verbose logging other than the following:

06/30/2011 12:25:49.04        Microsoft.Office.Project.Server (0x0E3C)        0x08C4        Project Server        Queue        954k   Medium  PWA:http://pwa.something/PWA, ServiceApp:Project Service Application, User DOMAINServiceAcc, PSI: [QUEUE] Retry: 1 ExchangeSyncTasks Microsoft.Office.Project.Server.BusinessLayer.QueueMsg.ExchangeSyncTasks 

To get to the bottom of what was happening Verbose logging on Project Server Exchange Tasks was required which showed far more detail which I will summaries here:

06/30/2011 12:25:48.00        Microsoft.Office.Project.Server (0x0E3C)        0x16D4        Project Server        Exchange Sync    fux2     Verbose        -! Info: System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond xxx.xxx.xxx.xxx:443 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at … …

06/30/2011 12:25:48.00        Microsoft.Office.Project.Server (0x0E3C)        0x16D4        Project Server        Exchange Sync   fux2  Verbose        =? Trying to Autodiscover using email at ‘https://autodiscover.companydomain.com/autodiscover/autodiscover.xml’ 

06/30/2011 12:25:48.04        Microsoft.Office.Project.Server (0x0E3C)        0x16D4        Project Server        Exchange Sync   fux2  Verbose        -! Info: System.Net.WebException: The remote name could not be resolved: ‘autodiscover.companydomain.com’ at System.Net.HttpWebRequest.GetRequestStream … …

06/30/2011 12:25:48.04        Microsoft.Office.Project.Server (0x0E3C)        0x16D4        Project Server        Exchange Sync   1zpd  Verbose        Error is: ExchangeSyncEWSUrlFailed. Details: Attributes: a6a779c4-a930-461f-a738-a3c78e8e826a . Standard Information: PSI Entry Point: Project User: DOMAINusername Correlation Id: a1e43a95-1a78-44a0-be1b-e8b3aa267074 PWA Site URL: SSP Name: Project Service Application PSError: ExchangeSyncEWSUrlFailed (40509)

06/30/2011 12:25:48.04        Microsoft.Office.Project.Server (0x0E3C)        0x16D4        Project Server        Exchange Sync  9fbi  Verbose        Error is: ExchangeSyncGeneralProcessingFailure. Details: Attributes: a6a779c4-a930-461f-a738-a3c78e8e826a Microsoft.Office.Project.Server.BusinessLayer.Queue.
ExchangeSyncEmailAddressInvalidException: Could not find Exchange server for resource a6a779c4-a930-461f-a738-a3c78e8e826a at Microsoft.Office.Project.Server.BusinessLayer.Queue.
ProcessExchangeSyncMessage.ExecuteSync … …

06/30/2011 12:25:49.04        Microsoft.Office.Project.Server (0x0E3C)        0x0DD0        Project Server        Exchange Sync  fux2  Verbose        ?? Starting SCP lookup for domainName=’companydomain.com’, root path=”   

[cut out multiple variations of the above as Autodiscover tries in vain to find the account]

If you’ve read through that, you can see that Project Server is attempting to use the Exchange Autodiscover service to locate the users Exchange details, as per the Autodiscover protocol which very very basically is something like this:

  1. Look for an Autodiscover service on the host (then look on the same host without the autodiscover bit)
  2. If not found try an SCP (Service Connection Point) lookup in Active Directory to find the resources

This can fail in a multi-forest Active Directory environment if your Exchange Client Access Servers (CAS) are located in a different forest. As was the case with my customer.

 

Solution

This is actually a well understood problem in Exchange circles, and the full solution is well documented:

How to Configure the Autodiscover Service for Multiple Forests

However if like in my case you don’t have the time to get a significant change such as the above completed then an alternative is needed.

 

Workaround

Looking at the above autodiscover procedure an easy “workaround” (I call this one a ‘hack’) is clear;

  • Add an entry to the HOSTS file on all Project Server application servers pointing to autodiscover.companyemaildomain.com which points to the actual autodiscover service running on the Exchange CAS.

However now you’ll see something like this in your Verbose ULS:

06/30/2011 12:40:04.17        Microsoft.Office.Project.Server (0x0E3C)        0x1930        Project Server        Exchange Sync        fux2        Verbose        -! Info: System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. —> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

This is because your Exchange CAS has something like as the Subject CN in its SSL certificate which would be expected, so what you need to do add a Subject Alternative Name to your certificate, see the following article on doing this:

Configure SSL Certificates to Use Multiple Client Access Server Host Names

Now you’re almost there, just one last thing! In order for the Exchange and Project Server to both authenticate these cross-forest users you need to make some changes on the service account on both forests by configuring the msExchMasterAccountSid property on the Exchange forest. See the following similar technet forum solution to achieve this:

EWS returns error "Failed to get valid Active Directory information for the calling account"

 

Done.

Hope that helps someone else out there!

Share and Enjoy !

Shares

Manually removing a missing feature after Config Wizard error after SP1 install

Had some time today to upgrade my lab to SP1, and thought it worth a quick post here about my experience.

For many people SP1 will be the first update applied to Project Server and SharePoint 2010, so with up to a year or more of production use it’s very possible that some features / solutions have been installed and removed which might cause some problems for the SP1 install.

In my case on my development lab this was most certainly the case with literally dozens of (often half developed) solutions in various states of deployment!

So in this case you may have the configuration Wizard fail after SP1 setup with a message like this:

Upgrade Timer job is exiting due to exception: Microsoft.SharePoint.Upgrade.SPUpgradeException: Upgrade completed with errors.  Review the upgrade log file located in C:Program FilesCommon FilesMicrosoft SharedWeb Server Extensions14LOGSUpgrade-20110701-160359-545.log.  The number of errors and warnings is listed at the end of the upgrade log file.

Digging into the log mentioned reveals something like:

[ERROR] [7/1/2011 4:17:20 PM]: The feature with Id d67567a3-4946-412f-9428-1ca6061a5189 is referenced in the database [WSS_Content], but is not installed on the current farm. The missing feature may cause upgrade to fail.

(A very helpful and surprisingly accurate message)

The problem as you can see is a missing feature, however using the standard PowerShell get-SPFeature commands don’t reveal anything. This is due to the solution already being uninstalled (or perhaps never installed in the case of a migrated database). Fortunately there is a solution:

Thanks Phil, using the provided scripts I was quickly able to identify and remove the offending feature. Once done a rerun of the PSCONFIG wizard (in my case using the command line “PSCONFIG –cmd upgrade –inplace b2b”) completes without errors.

 

Hope that helps someone else out there.

Share and Enjoy !

Shares

Extending Project Server 2010 for Extranet Users

A common requirement for many SharePoint and Project Server deployments is to have an external facing interface using an alternate Forms based authentication method, the most common is using an ASP.NET SQL provider which since the 2007 version is the one of the simplest options. With 2010 however and Claims authentication this requires that you convert your existing Windows authenticated Web Applications to Claims based NTLM authentication, this process is well documented online for SharePoint 2010, however I have recently found that with Project Server 2010 a number of problems can be encountered.

Here is a typical error with the Project Details webpart after conversion:

image

[Hi Google: (An unknown error has occurred)]

In this blog I’m not going to cover the whole process of setting up Extranet access, I’ll leave it to you to read some of the other excellent blogs on the topic linked below. What I will cover is how to do the not-so-well documented parts.

 

To Start With: Procedure to Setup Extranet Access

The general procedure for extending a Project Server application for extranet users is something like this:

  1. Provision a ASP.NET SQL membership Provider.
  2. Extend the Web Application to add an Extranet zone.
  3. Configure the Claims membership providers in the web.config files.

References for these steps:

  1. MMohanty: http://blogs.technet.com/b/mahesm/archive/2010/04/07/configure-forms-based-authentication-fba-with-sharepoint-2010.aspx
  2. Geoff Varosky: http://gvaro.wordpress.com/2011/03/28/planning-and-configuring-extranets-in-sharepoint-2010part-1/
  3. Geoff Varosky: http://gvaro.wordpress.com/2011/04/01/planning-and-configuring-extranets-in-sharepoint-2010part-2/

If you have followed those steps discussed in the links, then when you will likely have found that you are seeing all sorts of security issues after changing your Classic Web App to Claims.

In short the error above originates from the procedure used to migrate the application to Claims:

image

(WARNING: Don’t use the above commands!)

Unfortunately this does not fully migrate the SharePoint application and from my experience will lead to errors such as the first one above.

 

Migrating the Web Application to Claims Without Issues

Fortunately Microsoft has a well documented procedure: http://technet.microsoft.com/en-us/library/gg251985.aspx

$WebAppName = "http:// yourWebAppUrl"
$account = "yourDomainyourUser"
$wa = get-SPWebApplication $WebAppName

Set-SPwebApplication $wa -AuthenticationProvider `
  (New-SPAuthenticationProvider) -Zone Default

Note: that the above commands migrate ALL Web Applications sharing that URL, it is not possible to only migrate your Extended application!

That command will migrate the web application fully and prepare it for Claims authentication, and don’t forget the TechNet article discusses the need to update your portalsuperreaderaccount and portalsuperuseraccount accounts if they have been set, which can be done easily using the following commands:

$wa.Properties["portalsuperuseraccount"] = "i:0#.w|domainapppool"

$wa.Properties["portalsuperreaderaccount"] = "i:0#.w|domainapppool"

$wa.Update()

 

Migrating Users to Re-enable Login

Once your Web App is in Claims mode then you will still need to migrate your AD users, this part is not so well documented.

Essentially you need to use the PowerShell command Move-SPUser to move all of your existing “DOMAINusername” users to the new Claims-NTLM identity “i0#.w|domainusername”.

Before you can do this you need to ensure that your admin user has permissions to access this site, this can be done from Central Admin by updating the Web Application Policy:

image

 

Add or re-add your user account (DOMAINadminuser) with Full Control to the web application in question, before proceeding with the following steps. (Note: the above Policy should now show your Admin user with the User Name like so; “i:0#.w|domainadminuser”)

Next, here is the command to migrate a single user:

Get-SPUser -web http://server/pwa -identity "DOMAINuser" | Move-SPUser -NewAlias "i:0#.w|domainuser" –IgnoreSID

The command will actually give the following error:

image

However if you then use Get-SPUser you will note that the account has actually been migrated.

So for my purposes I wrote the following script which will migrate all users without the claims prefix “i:0#.w|” (Yep ignore those errors!):

$UsersToMigrate = Get-SPUser -web http://server/pwa | `

  where {$_.UserLogin –like ‘DOMAIN*’ }

 

ForEach ($user in $UsersToMigrate)

{

  Get-SPUser -web http://server/pwa -identity $user | Move-SPUser `

    -NewAlias ("i:0#.w|"+$user.UserLogin.toLower()) -IgnoreSID

}

After running the above, users should now be able to login to your migrated Web Application with any AD user!

If you see other errors running either that script or the above command by itself, make certain that you can do the following without errors:

Get-SPUser –web http://server/pwa

If not check your web policy again as above.

 

Finally Adding Your Claims Users to PWA

Now we have a fully migrated Claims-NTLM Web Application in addition to a newly created Extranet Claims Web Application which is attempting to use a Forms membership provider (SQL-MembershipProvider if you following the steps linked above).

The next steps are again well documented in the links above, configure your web.config files and then setup your SQL users.

The final problem you may face when attempting to add your new forms users to PWA, assuming (if using SQL that your ASPNETDB database permissions are correctly set) then adding the users to SharePoint will be easy using Site Actions – Site Permissions – Grant Permission, however what you will may see when attempting to add to Users to PWA is an error like the following:

image

(Error Message: The NT account specified is invalid. …)

This is because the user has not yet been added to the SharePoint site collection which fortunately is easy enough to fix, just add the user to SharePoint from Site Permissions first!

From Site Actions –> Site Permissions:

image

image

As long as SharePoint can find the user (make sure to use the full username!) then once you hit Ok the user identity will be added to the Site Collection, and then you will be able to add the user to PWA!

FYI if like me you like doing things in bulk here’s the PowerShell command to do the above:

New-SPUser -UserAlias "i:0#.f|SQLMembershipProvider|JaneDoe" -Web http://server/pwa -DisplayName "Jane Doe" -Email [email protected]

 

All done, enjoy your forms membership provider!

Share and Enjoy !

Shares