Public Folder Hierarchy Replication Problems

The moral of this very long post is, "do not put your Exchange server computer accounts in to the Domain Admins group!" (Or any other group that has been denied Exchange permissions!)
__________
I spent a couple of days troubleshooting the "problem from hell". This organization added a new mailbox server to their organization. However, the public folder hierarchy was not replicating to their server; the hierarchy is the names and structure of the public folders in the information store as well as the folder's properties (permissions, replica list, limits, etc..)

If the hierarchy does not replicate, then neither does the contents. I turned up Diagnostics Logging on all of the public folder replication related categories. I saw two events that the Knowledge Base says are not really problems.

Event Type: Error
Event Source: MSExchangeIS Public Store
Event Category: Replication Errors
Event ID: 3093
Computer: MAIL
Description:
Error -2147221233 reading property 0x674b0014 on object type tbtMsgFolder from database "First Storage Group\Public Folder Store (MAIL)".

Event Type: Error
Event Source: MSExchangeIS Public Store
Event Category: Replication Errors
Event ID: 3093
Computer: MAIL
Description:
Error -2147221233 reading property 0x67010003 on object type tbtOwning
Folders from database "First Storage Group\Public Folder Store (MAIL)".

I forced the hierarchy and content replication a number of times and did notice that the messages were being generated and sent. However, on the new mail server, the mail was queued for locally delivery but never submitted to the information store.

Once the message is transmitted, you should see an event id 3018. This indicates that a replication message is being sent.

Event Type: Information
Event Source: MSExchangeIS Public Store
Event Category: Replication Outgoing Messages
Event ID: 3018
Computer: MAIL
Description:
An outgoing replication message was issued.
Type: 0x2
Message ID: <D75E025B70DEF24B969BB110A2CC5A08648A@mail.company.COM>
Database "First Storage Group\Public Folder Store (MAIL)" CN min: 1-254F, CN max: 1-6488 RFIs: 1 1)
FID: 1-19, PFID: 1-18, Offset: 28 NON_IPM_SUBTREE\schema-root\microsoft\exchangeV1 IDCN
Deleted: {0}

On the receiving server, you should see an event id 3028 which indicates the message was received:

Event Type: Information
Event Source: MSExchangeIS Public Store
Event Category: Replication Incoming Messages
Event ID: 3028
Computer: MAIL
Description:
An incoming replication message was processed.
Type: 0x2
Message ID: <0EB40F697909CB42B117859A56E0D12D0B87F7@exchange.company.com>
Database "First Storage Group\Public Folder Store (MAIL)". CN min: 5-B7FF8 CN max: 5-B87F5 RFIs: 41)
FID: 5-B3B10, PFID: b-434E, Offset: 88 IPM_SUBTREE\y2k\Test2) FID: 5-B3B11, PFID: 5-B3B10,
Offset: 838 IPM_SUBTREE\y2k\Test\Test23) FID: 5-B3B14, PFID: 5-B3B10, Offset: 1591 IPM_SUBTREE\y2k\Test\Test31214) FID: 5-B3B19, PFID: 5-B3B14, Offset: 2353 IPM_SUBTREE\y2k\Test\Test3121\Test Report IDCN deleted: {0} Server: /O=COMPANY/OU=ADMINGROUP/cn=Configuration/cn=Servers/
cn=EXCHANGE/cn=Microsoft Public MDB

We were not seeing the 3028 messages on the new server, but the other servers were showing that they were sending the replication messages. Message tracking shows that the messages are being delivered to the server, but not being submitted to the store. At this point, I turned up Diagnostics Logging on the Transport's Categorizer category. I saw a couple of interesting NDR messages relating to this problem including the fact that the sending server did not have permission to send public folder replication to the receiving server.

Event Type: Information
Event Source: MSExchangeTransport
Event Category: Categorizer
Event ID: 9013
Computer: MAIL
Description:A message from 'smtp:EXCHANGE-IS@COMPANY.COM' could not be delivered
because the sender does not have permission to send to recipient 'smtp:MAIL-IS@COMPANY.COM'.
This is due to a delivery restriction configured on the recipient. (Message-ID: <0EB40F697909CB42B117859A56E0D12D0B7FCC@exchange>).
A DSN will be generated.

I turned on network monitoring and noticed that the SMTP traffic for public folders did in include the XEXCH50 verb and the data was being transmitted, but the receiving SMTP server was then responding with a "504 Need to authenticate first" message even though the message WAS being received.

This sure seems to be permissions related. There is a Registry value that you can add to the transport category that will ignore public folder restrictions. In the following Registry key, create a new subkey called Parameters:
HKLM\System\CurrentControlSet\Services\MSExchangeTransport

Once you have created the Parameters subkey, in that subkey, create a REG_DWORD value called SkipPublicMDBRestriction and set it to 0x1. Then restart IIS and the Exchange servers. When I did this, replication mail started to flow and the hierarchy was populated (I did have to wait a while to see the hierarchy completely populated).

This Registry value is a work-around though and it allows unauthenticated public folder replication traffic to be accepted. Not something you want to leave open.

After a couple of more hours of research, I finally found the root cause. Someone had added the old Exchange server computer accounts to the Domain Admins group. Remember, Domain Admins does not have all of the permissions that the Exchange Domain Servers / Exchange Enterprise Servers have. There are some explicit denies such as the Receive As and the Send As permissions.

Once I took the other Exchange servers out of Domain Admins, I removed that Registry key, restarted all the servers and the hierarchy and the content was flowing normally. Whew!