PrincipalObjectAccess and double mailboxes

PrincipalObjectAccess table – The stuff that nightmares are made of 🙂

Ok So first of all if you don’t know what PrincipalObjectAccess is (POA from now on), Go ahead and write it down in your favorite search engine, do some reading in the endless blog posts and articles available out there – and then get back here, as I’m going to assume that you already know what it is and how it works.

Think of the following scenario – You have 10 users in your Organization but want to use only 1 mailbox, and you want all the users to use the same email address and see the same emails in the system. Yes you should definitely use a Queue for this, but maybe you don’t feel like using a queue or just don’t know exactly how it works and how to set it up – and hey, the system does not prevent you from setting the same email address for multiple users, right…?

So can you do it this way? Probably yes
Will it actually work? Likely it will
Is it a good idea? Nope

Here’s what will happen
** Obviously this is for demonstration only – DO NOT DO THIS IN YOUR ENVIRONMENT **

Step 1: Assign 10 users the same email address. (I used a yahoo.com email for testing – feel free to spam my email address :))

Step 2: Create your server-side-sync profile and assign to the users, then activate, approve, test & enable all the mailboxes – Basically do all the steps you need to do for a mailbox to start working.

Step 3: Change all the user’s settings to automatically track all emails in the mailbox (the scenario would also work for emails in reply to CRM emails).

Step 4: Send an email to the newly crated mailbox and wait a few minutes.

Step 5: Check your POA table by running a query on your DB:

SELECT TOP 100 * FROM PrincipalObjectAccess
ORDER BY ChangedOn DESC

Results:
The email that entered the system receives a POA share record for every user that owns the email address. Not so great!

This is a small example of what will happen. I found it at a customer with more then 400 users that were assigned the same mailbox and something in between a few hundreds to thousand email threads – every day!
Needless to say that in this scenario their POA table grew at a rate of about 500,000 records per day.

This could definitely be causing additional side effects but I didn’t bother to check any further 🙂

The fact that the system allows you to do things in a certain way does not always mean it’s a good practice, and if there is a mechanism built in the system to address a specific scenario – you should probably use it as there is a reason behind it.

If this post prevents from even one person setting up a system in this way – I’ve done my job 🙂

Happy POA’ing

Michael

Improving Server Side Sync performance

Leave a reply

The following blog was created after helping a customer of mine to drastically improve their server-side-sync performance by modifying the polling intervals of the mailboxes. let’s go 🙂

Understanding the mechanism

Server side sync polls mailboxes for Emails and ACT’s (Accounts, Contacts, Tasks) in sync cycles. In each sync cycle a mailbox will be inspected for new items and according to your settings for that specific mailbox it will sync these items to Dynamics.

This mechanism has an internal prioritization logic that increases and decreases the time between each polling for each mailbox according to activity that is observed on that specific mailbox. This behavior is described briefly in the SSS whitepapers and I will expand on this in this post.

A busy mailbox that has constant activity on it should be polled for items (emails) approx. every 5 minutes. But when there is no activity on the mailbox for several consecutive cycles – The mechanism will kick in and start increasing the time between each poll. At this point the mailbox enters a state of an IdleMailbox – and for these mailboxes type the sync cycle can increase to up to 6 Hours. That’s right, 6 hours. This same behavior is also relevant for ACT’s and has a separate setting with separate intervals.

This means that you can end up in a scenario in which a mailbox becomes Idle at 7:00 AM because there were no emails flowing in, and from that point the mailbox will be polled again only at 13:00 (1:00 PM) 6 hours later in the worst case scenario.

This mechanism is in place for a reason – to decrease the utilization on the email integration servers and unnecessary calls to EWS. Without it a customer that has for example 5,000 configured mailboxes but only few of them actually active – would end up with massive utilization of the servers and huge amount of calls to EWS. ((5000 x 12 email polls per hour) + (5000 x 5 ACT polls per hour)) = 85,000 polls per hour.

Luckily – we can control these settings, and it helped me solve an issue for a customer that actually needed to poll ~1000 mailboxes at a very high and consistent rate, without any delays or surprises. As explained above changing the setting caused the Async servers to soar in terms of resource consumption, so this is something you need to take into account and make sure your infrastructure can handle the change.

Explaining the parameters

The actual polling settings are stored in the DeploymentProperties table in the MSCRM_CONFIG database and are represented in seconds.

Although the Minimum values for Emails and ACT’s are 1 minute & 5 minutes – In reality Iv’e always seen that the MaximumBackoff values for polls are being used for the Active mailboxes.

Changing the setting

You can use PowerShell on your Dynamics servers to adjust the settings. In this example we will change the IdleMailboxMaximumBackoff time from 21600 (6 Hours) seconds to 1800 seconds (30 minutes)

Add-PSSnapin Microsoft.Crm.PowerShell
Get-CrmSetting -SettingType ServerSideSyncEmailSettings
$set = Get-CrmSetting -SettingType ServerSideSyncEmailSettings
$set.IdleMailboxMaximumBackoff = “1800”
Set-CrmSetting -Setting $set
Get-CrmSetting -SettingType ServerSideSyncEmailSettings

Result after change
* not that the column name is ECidlemailboxMaximumBackoff

You could also change those settings on the DB but for safety and supportability reasons it would be a better to do it VIA PowerShell.

MailboxstatisticsBase

The MailboxStatisticsBase table is an excellent source of insights regarding the internal works of the polling mechanism. Download and run This query to see all the polls that were done on all the mailboxes and how many items were processed in each poll. You can also filter it by a time interval to show you all the times in which a single sync cycle on any mailbox took more then X minutes – This is very useful when you need to troubleshoot sync issues. Just read the comments in the SQL query.

Additional Notes

If your MailboxStatisticsBase table is empty and not populating then it’s probably disabled for data collection – You can enable it with the OrgDBSettings tool by setting the MailboxStatisticsPersistenceTimeInDays to the number of days you want to save data for (Lot’s of data!) 0 means no data is collected.

Needless to mention – Those changes are only applicable for Dynamics on-premises deployments. And as mentioned above – If you make changes be sure you are ready for the extra resource consumption on the servers.

It’s been a long post! hope you find this useful 🙂

Michael

Back to business…

Leave a reply

It’s been a while,

2.5 years to be precise since I wrote my last post here, and quite a few things have changed. First of all I started to work for Microsoft – a big accomplishment and a personal goal that I was very happy and proud to achieve. I’m a Dynamics 365 Premier field engineer and am enjoying my work very much – So an update to my “About” section is also coming. And Of course many changes to the product, the movement to Dynamics 365 online with it’s Azure & Office 365 ecosystem, New versions and features for on-premise versions, The relatively new V9 and Unified interface, App for Outlook for both online and on-premises deployments and much much more – In other words, lot’s of stuff to write about 🙂

So as the headline above implies, I’m going to start writing again and already have some great ideas in mind that will hopefully help you out with your Dynamics 365 ventures, whether it’s online or on-premise

Watch this space & see you soon!

Michael

SYSTEM user account not found when trying to activate CRM Encryption Key

Leave a reply

I stumbled upon a weird error after upgrading CRM 2013 to CRM 2016, After enabling the option to view the encryption key in an HTTP deployment:

UPDATE [MSCRM_CONFIG].[dbo].[DeploymentProperties]
SET [BitColumn]=1
WHERE ColumnName='DisableSSLCheckForEncryption'

I received the following error when trying to open the encryption key form:

Event Log on the front end server had the following error:

Exception type: CrmException

Exception message: SYSTEM user account not found.

It took me a while to figure it out but it seems that the problem in my deployment was that i had TWO rows of the CRM SYSTEM user in the SystemUserBase table! all the way from 2013 to 2016 and it looked like that:

select * from SystemUserBase where fullname like '%system%'

They were both created at the same second just a few milliseconds apart – Have no idea how & why.

Solution:

Delete the second user using the following query:

Delete FROM systemuserbase 
WHERE SystemUserId ='youruserguid'

After that – I was able to open the form and insert an encryption key properly.

Important: Tempering with the DB is unsupported by Microsoft so do this at your OWN RISK – Don’t forget to backup your DB before!

Michael

AX service won’t start after migrating to a new SQL Server

2 Replies

After migrating the AX databases to a new SQL server you might end up having issues starting the service. If you will search the even log you will probably find the following errors:

Object Server 01: Fatal SQL condition during login. Error message: “[Microsoft][SQL Server Native Client 10.0][SQL Server]Login failed for user ‘domain\user’.”

Object Server 01: The database SQLSERVER\INSTANCE is not recognized as a model store.

The domain\user will be the service account you are using to run the AX service.
Providing this account even with sysadmin on the SQL server will not help you, as the problem is a missing stored procedure on the Master database that is created during the initial AX setup – This stored procedure is missing from your new SQL server and needs to be created manually.

1. Run the following query on your new SQL server (Backup first!)

USE [master]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE procedure [dbo].[CREATETEMPDBPERMISSIONS_AX_SERVER_NAME_01] as begin exec (‘USE tempdb; declare @dbaccesscount int; exec sp_grantlogin ”domain\axserviceaccount”; select @dbaccesscount = COUNT(*) from master..syslogins where name = ”domain\axserviceaccount”; if (@dbaccesscount <> 0) exec sp_grantdbaccess ”domain\axserviceaccount”; ALTER USER [domain\axserviceaccount] WITH DEFAULT_SCHEMA=dbo; exec sp_addrolemember ”db_ddladmin”, ”domain\axserviceaccount”; exec sp_addrolemember ”db_datareader”, ”domain\axserviceaccount”; exec sp_addrolemember ”db_datawriter”, ”domain\axserviceaccount”;’) end
GO
EXEC sp_procoption N'[dbo].[CREATETEMPDBPERMISSIONS_AX_SERVER_NAME_01]’, ‘startup’, ‘1’
GO

* Change “AX_SERVER_NAME” to your AX server name
* Change domain\axserviceaccount to your domain and the account that runs the AX Service (If you are using NETWORK SERVICES for the service (Bad…) the use DOMAIN\MACHINENAME$ for the account)

2. In SSMS go to Databases -> System Databases -> Master -> Programmability -> Stored Procedures:
There you will find the newly created SP. Right click -> Properties

ax1
Under permissions – make sure to add the AX service account and grant it the Execute permission

ax2

The service should now start with no issues 🙂

BTW – Don’t forget that the AX service account first needs to be added as a user under security logins and granted the db_datareader, db_datawriter & db_ddladmin roles!

Find active users in NAV using SQL

Leave a reply

Here is a simple query to discover who are the users that are currently connected to your Dynamics NAV instance using SQL Server Management Studio:

USE DatabaseName
GO
SELECT [User ID],[Client Computer Name],[Login Datetime] from [Active Session]

This will return you the user ID, computer name and the login time of the user. You can use * instead if you want to get all the data:

SELECT * from [Active Session]

You can also add a where clause if you are looking for something specific in an instance with large user count, for example:

USE DatabaseName
GO
SELECT [User ID],[Client Computer Name],[Login Datetime] from [Active Session]
WHERE [User ID] LIKE ‘%john%’

Michael

Dynamics CRM 2016 Step-By-Step installation guide

3 Replies

Hi everyone!

To be honest – The installation process & requirements of Dynamics CRM 2016 is practically identical to 2015 – But most people that are new to Dynamics don’t know that.

So here is a new video tutorial for Dynamics CRM 2016 installation

“The server reports that it is from digest” – NAV OData Web Services

7 Replies

Recently I tried to configure Dynamics NAV 2015 OData web services. After the initial configuration i was unable to authenticate with the OData URL – The Credentials prompt just kept popping up time after time. The message i received was:

The server is asking for your username and password
The server reports that it is from digest

It seems that the reason for this issue (Although not documented in NAV requirements) is because I was working on a WORKGROUP server that was not a part of an Active Directory domain, and although Dynamics NAV is supported in a WORKGROUP environment – It seems that OData web services need an Active Directory environment – as they rely on Digest authentication.

In my case – adding the server to a domain or promoting it to a DC – solved the issue!

As i said – i did not find any detailed specifications for this in the Dynamics NAV requirements or technet – I relied on other sources regarding Digest authentication and it solved my issue – If you are having the exact same problem – this might be the solution.

Special thanks to Assaf from https://wirefighter.com/ who pointed me in that direction 🙂

Michael

You recieve a message from the web browser: “A script error has occurred” In NAV load balanced environment

Leave a reply

I have stumbled upon this issue while configuring a Dynamics NAV 2015 Load Balanced solution combined with Forms Authentication.
In this scenario you will randomly get this error while browsing the NAV Web Client:

“A script error has occurred, and the content cannot be displayed. Refresh the page or open a new browser window”

At the same time you will find an ASP.NET warning in the event log on one of the front servers saying:
“An error occurred processing a web or script resource request. The resource identifier failed to decrypt”

So what’s going on here?

Turns out that in order to work with forms authentication with NLB you need to configure Identical Machine Keys on both IIS servers that host your Dynamics NAV Web Client:

Solution:

1. Go to IIS on your first server and go to Machine Keys:

2. Remove the check-box from both “Automatically Generate at runtime” options and then click Generate Keys.

3. Hit apply and copy both keys to notepad.
4. Go to IIS on your second server and paste the keys generated on the first server to the machine keys respectively.
5. Hit apply and do an IISRESET on both servers

This action should solve the script errors

Michael.

Find out who is connected to AX and from which computer

Leave a reply

Here is a very simple query you can run against your Business Database to find out who are the users that have an active sessions in AX:

SELECT DISTINCT USERID, CLIENTCOMPUTER FROM SYSCLIENTSESSIONS WHERE STATUS = 1

This table has some more useful info that you can extract so you can tune the query for your own use.

use SELECT * from SYSCLIENTSESSIONS to see the whole table.

** This works for AX 2009 /2012

Microsoft Dynamics Essentials Blog

Microsoft Dynamics 365 TechBlog

PrincipalObjectAccess and double mailboxes

Improving Server Side Sync performance

Back to business…

SYSTEM user account not found when trying to activate CRM Encryption Key

Exception type: CrmException

Exception message: SYSTEM user account not found.

Find active users in NAV using SQL

Dynamics CRM 2016 Step-By-Step installation guide

“The server reports that it is from digest” – NAV OData Web Services

You recieve a message from the web browser: “A script error has occurred” In NAV load balanced environment

Find out who is connected to AX and from which computer