Monday, October 30, 2006

Public Folder Hierarchy Replication Problems

The moral of this very long post is, "do not put your Exchange server computer accounts in to the Domain Admins group!" (Or any other group that has been denied Exchange permissions!)
__________
I spent a couple of days troubleshooting the "problem from hell". This organization added a new mailbox server to their organization. However, the public folder hierarchy was not replicating to their server; the hierarchy is the names and structure of the public folders in the information store as well as the folder's properties (permissions, replica list, limits, etc..)

If the hierarchy does not replicate, then neither does the contents. I turned up Diagnostics Logging on all of the public folder replication related categories. I saw two events that the Knowledge Base says are not really problems.

Event Type: Error
Event Source: MSExchangeIS Public Store
Event Category: Replication Errors
Event ID: 3093
Computer: MAIL
Description:
Error -2147221233 reading property 0x674b0014 on object type tbtMsgFolder from database "First Storage Group\Public Folder Store (MAIL)".

Event Type: Error
Event Source: MSExchangeIS Public Store
Event Category: Replication Errors
Event ID: 3093
Computer: MAIL
Description:
Error -2147221233 reading property 0x67010003 on object type tbtOwning
Folders from database "First Storage Group\Public Folder Store (MAIL)".

I forced the hierarchy and content replication a number of times and did notice that the messages were being generated and sent. However, on the new mail server, the mail was queued for locally delivery but never submitted to the information store.

Once the message is transmitted, you should see an event id 3018. This indicates that a replication message is being sent.

Event Type: Information
Event Source: MSExchangeIS Public Store
Event Category: Replication Outgoing Messages
Event ID: 3018
Computer: MAIL
Description:
An outgoing replication message was issued.
Type: 0x2
Message ID: <D75E025B70DEF24B969BB110A2CC5A08648A@mail.company.COM>
Database "First Storage Group\Public Folder Store (MAIL)" CN min: 1-254F, CN max: 1-6488 RFIs: 1 1)
FID: 1-19, PFID: 1-18, Offset: 28 NON_IPM_SUBTREE\schema-root\microsoft\exchangeV1 IDCN
Deleted: {0}

On the receiving server, you should see an event id 3028 which indicates the message was received:

Event Type: Information
Event Source: MSExchangeIS Public Store
Event Category: Replication Incoming Messages
Event ID: 3028
Computer: MAIL
Description:
An incoming replication message was processed.
Type: 0x2
Message ID: <0EB40F697909CB42B117859A56E0D12D0B87F7@exchange.company.com>
Database "First Storage Group\Public Folder Store (MAIL)". CN min: 5-B7FF8 CN max: 5-B87F5 RFIs: 41)
FID: 5-B3B10, PFID: b-434E, Offset: 88 IPM_SUBTREE\y2k\Test2) FID: 5-B3B11, PFID: 5-B3B10,
Offset: 838 IPM_SUBTREE\y2k\Test\Test23) FID: 5-B3B14, PFID: 5-B3B10, Offset: 1591 IPM_SUBTREE\y2k\Test\Test31214) FID: 5-B3B19, PFID: 5-B3B14, Offset: 2353 IPM_SUBTREE\y2k\Test\Test3121\Test Report IDCN deleted: {0} Server: /O=COMPANY/OU=ADMINGROUP/cn=Configuration/cn=Servers/
cn=EXCHANGE/cn=Microsoft Public MDB

We were not seeing the 3028 messages on the new server, but the other servers were showing that they were sending the replication messages. Message tracking shows that the messages are being delivered to the server, but not being submitted to the store. At this point, I turned up Diagnostics Logging on the Transport's Categorizer category. I saw a couple of interesting NDR messages relating to this problem including the fact that the sending server did not have permission to send public folder replication to the receiving server.

Event Type: Information
Event Source: MSExchangeTransport
Event Category: Categorizer
Event ID: 9013
Computer: MAIL
Description:A message from 'smtp:EXCHANGE-IS@COMPANY.COM' could not be delivered
because the sender does not have permission to send to recipient 'smtp:MAIL-IS@COMPANY.COM'.
This is due to a delivery restriction configured on the recipient. (Message-ID: <0EB40F697909CB42B117859A56E0D12D0B7FCC@exchange>).
A DSN will be generated.

I turned on network monitoring and noticed that the SMTP traffic for public folders did in include the XEXCH50 verb and the data was being transmitted, but the receiving SMTP server was then responding with a "504 Need to authenticate first" message even though the message WAS being received.

This sure seems to be permissions related. There is a Registry value that you can add to the transport category that will ignore public folder restrictions. In the following Registry key, create a new subkey called Parameters:
HKLM\System\CurrentControlSet\Services\MSExchangeTransport

Once you have created the Parameters subkey, in that subkey, create a REG_DWORD value called SkipPublicMDBRestriction and set it to 0x1. Then restart IIS and the Exchange servers. When I did this, replication mail started to flow and the hierarchy was populated (I did have to wait a while to see the hierarchy completely populated).

This Registry value is a work-around though and it allows unauthenticated public folder replication traffic to be accepted. Not something you want to leave open.

After a couple of more hours of research, I finally found the root cause. Someone had added the old Exchange server computer accounts to the Domain Admins group. Remember, Domain Admins does not have all of the permissions that the Exchange Domain Servers / Exchange Enterprise Servers have. There are some explicit denies such as the Receive As and the Send As permissions.

Once I took the other Exchange servers out of Domain Admins, I removed that Registry key, restarted all the servers and the hierarchy and the content was flowing normally. Whew!

5 Comments:

At 12:49 AM, Blogger Oliverio's Family said...

I want to proactively monitor my server environment. What can I do to check that my public folder infrastructure is not encountering any issues?

 
At 8:46 AM, Blogger Roshan said...

Hey Jim,
Just want to say hello.
When is the book on 2007 coming?

Roshan

 
At 9:00 AM, Blogger Joe said...

You may want to take a look at this article as well. http://msexchangeteam.com/archive/2006/01/23/417974.aspx
My problem was identical, but the solution was that the Integrated Windows Authentication was disabled in the SMTP Virtual Server's Authentication properties.

PS. Love your Exchange 2003 book.

 
At 10:54 PM, Blogger Kearts said...

I just fixed a problem with fairly similiar symptoms. This was a small company that had added a new Exchange 2003 server to replace their old 2000 box. The public folder hierarchy was not replicating. Users on a new server in the organization could not see any system folders. I could see the public folder replication messages going out, but they were getting stuck in the outgoing queue on the first server. I saw this post as I was trawling for solutions. The problem in this case had a slightly different cause. It turned out the previous admin had entered a name in the smart host field of the default SMTP virtual server of the old Exchange 2000 server. This was in addition to a smart host entered on a SMTP connector used for delivering Internet e-mail (which worked throughout). I took the smart host off the SMTP virtual server and public folder replication messages started flowing again. The sending server was obviously trying to deliver the messages via the smart host and failing (no authentication would happen, amongst other problems). Once mail was delivered directly, there were no more problems.

 
At 6:36 AM, Blogger Markus said...

Thanks for this Article.
It helped me more as you can ever think.

 

Post a Comment

<< Home