Exchange 2007 RAM recommendations updated

Microsoft has updated the recommended amount of RAM for Exchange 2007 mailbox servers.

Minimum: 2 GB of RAM
Recommended: 2 GB of RAM per server plus 5 megabytes (MB) of RAM per mailbox

This was previously 10MB per mailbox. Unlike Exchange 2003, though, more RAM will improve performance by reducing the IOPS requirements. More on this after the Exchange Connections conference, or if you are there, attend the keynote.

Internet message routing problems to Bridgehead server

I struggled with a problem the other day that I thought was worthy of mentioning. I did find the information I was looking for (eventually), but it was a challenge to narrow the problem down. This organization had just installed a new E2K3 SP2 mailbox server in to their organization. They had an existing mailbox server and an existing front-end/bridgehead server (for inbound and outbound SMTP). Once a mailbox was moved, it could send internal mail, receive internal and external mail, but they received an NDR message when they sent a message to the Internet.

The following recipient(s) could not be reached:
jim@external.net on 10/31/2006 10:57 AM
A configuration error in the e-mail system caused the message to bounce between two servers or to be forwarded between two recipients. Contact your administrator.


We were also seeing the following messages in the event log:

A non-delivery report with a status code of 5.3.5 was generated for recipient rfc822;externaluser@hotmail.com (Message-ID 21BB0C8F0C7EC74EB1BABCA27E16C82A0262A25B@internalserver).
Causes: A looping condition was detected. (The server is configured to route mail back to itself). If you have multiple SMTP Virtual Servers configured on your Exchange server, make sure they are defined by a unique incoming port and that the outgoing SMTP port configuration is valid to avoid looping between local virtual servers.

Solution: Check the configuration of the virtual server's connectors for loops and ensure each virtual server is defined by a unique incoming port.

In addition, when the new mailbox server would create a new temporary link queue to send mail to the SMTP connector, we would see the following message in the Additional Queue Information box at the bottom of the screen.

Unable to deliver the message because the destination address was misconfigured as a mail loop.

I went through the entire configuration a couple of times and missed the problem the first time. There were several issues:

1) The bridgehead server's name was BH.organization.com, but there was also a DNS domain zone (internal) called BH.organization.com.
2) The bridgehead server was registering a "backup network" IP address as well as it's corporate network address.
3) Finally, the SMTP virtual server's name on the bridgehead server had been changed, but that name was not resolvable in DNS internally. Someone had manually fixed this in a HOSTS file on the other mailbox server, but that was long before the current IT folks had possession of the server.

No, why did they change the SMTP virtual server's FQDN? (This is on the SMTP virtual server property pages). When an SMTP virtual server initiates a connection to a remote system, it sends an EHLO or HELO command followed by what it thinks is it's FQDN. It gets that from the SMTP virtual server properties.

Some firewalls and message hygiene systems confirm that this name is resolveable externally. It is not RFC compliant to do this, but they do it anyway. So, sometimes the name that you have for your internal SMTP virtual servers have to be publicly resolveable also. Yuck.

The problem is that Exchange uses this information internally for server-to-server name resolution. In the past, I have gotten around this by creating an internal DNS record so the internal name and IP address can also be resolved.

Public Folder Hierarchy Replication Problems

The moral of this very long post is, "do not put your Exchange server computer accounts in to the Domain Admins group!" (Or any other group that has been denied Exchange permissions!)
__________
I spent a couple of days troubleshooting the "problem from hell". This organization added a new mailbox server to their organization. However, the public folder hierarchy was not replicating to their server; the hierarchy is the names and structure of the public folders in the information store as well as the folder's properties (permissions, replica list, limits, etc..)

If the hierarchy does not replicate, then neither does the contents. I turned up Diagnostics Logging on all of the public folder replication related categories. I saw two events that the Knowledge Base says are not really problems.

Event Type: Error
Event Source: MSExchangeIS Public Store
Event Category: Replication Errors
Event ID: 3093
Computer: MAIL
Description:
Error -2147221233 reading property 0x674b0014 on object type tbtMsgFolder from database "First Storage Group\Public Folder Store (MAIL)".

Event Type: Error
Event Source: MSExchangeIS Public Store
Event Category: Replication Errors
Event ID: 3093
Computer: MAIL
Description:
Error -2147221233 reading property 0x67010003 on object type tbtOwning
Folders from database "First Storage Group\Public Folder Store (MAIL)".

I forced the hierarchy and content replication a number of times and did notice that the messages were being generated and sent. However, on the new mail server, the mail was queued for locally delivery but never submitted to the information store.

Once the message is transmitted, you should see an event id 3018. This indicates that a replication message is being sent.

Event Type: Information
Event Source: MSExchangeIS Public Store
Event Category: Replication Outgoing Messages
Event ID: 3018
Computer: MAIL
Description:
An outgoing replication message was issued.
Type: 0x2
Message ID: <D75E025B70DEF24B969BB110A2CC5A08648A@mail.company.COM>
Database "First Storage Group\Public Folder Store (MAIL)" CN min: 1-254F, CN max: 1-6488 RFIs: 1 1)
FID: 1-19, PFID: 1-18, Offset: 28 NON_IPM_SUBTREE\schema-root\microsoft\exchangeV1 IDCN
Deleted: {0}

On the receiving server, you should see an event id 3028 which indicates the message was received:

Event Type: Information
Event Source: MSExchangeIS Public Store
Event Category: Replication Incoming Messages
Event ID: 3028
Computer: MAIL
Description:
An incoming replication message was processed.
Type: 0x2
Message ID: <0EB40F697909CB42B117859A56E0D12D0B87F7@exchange.company.com>
Database "First Storage Group\Public Folder Store (MAIL)". CN min: 5-B7FF8 CN max: 5-B87F5 RFIs: 41)
FID: 5-B3B10, PFID: b-434E, Offset: 88 IPM_SUBTREE\y2k\Test2) FID: 5-B3B11, PFID: 5-B3B10,
Offset: 838 IPM_SUBTREE\y2k\Test\Test23) FID: 5-B3B14, PFID: 5-B3B10, Offset: 1591 IPM_SUBTREE\y2k\Test\Test31214) FID: 5-B3B19, PFID: 5-B3B14, Offset: 2353 IPM_SUBTREE\y2k\Test\Test3121\Test Report IDCN deleted: {0} Server: /O=COMPANY/OU=ADMINGROUP/cn=Configuration/cn=Servers/
cn=EXCHANGE/cn=Microsoft Public MDB

We were not seeing the 3028 messages on the new server, but the other servers were showing that they were sending the replication messages. Message tracking shows that the messages are being delivered to the server, but not being submitted to the store. At this point, I turned up Diagnostics Logging on the Transport's Categorizer category. I saw a couple of interesting NDR messages relating to this problem including the fact that the sending server did not have permission to send public folder replication to the receiving server.

Event Type: Information
Event Source: MSExchangeTransport
Event Category: Categorizer
Event ID: 9013
Computer: MAIL
Description:A message from 'smtp:EXCHANGE-IS@COMPANY.COM' could not be delivered
because the sender does not have permission to send to recipient 'smtp:MAIL-IS@COMPANY.COM'.
This is due to a delivery restriction configured on the recipient. (Message-ID: <0EB40F697909CB42B117859A56E0D12D0B7FCC@exchange>).
A DSN will be generated.

I turned on network monitoring and noticed that the SMTP traffic for public folders did in include the XEXCH50 verb and the data was being transmitted, but the receiving SMTP server was then responding with a "504 Need to authenticate first" message even though the message WAS being received.

This sure seems to be permissions related. There is a Registry value that you can add to the transport category that will ignore public folder restrictions. In the following Registry key, create a new subkey called Parameters:
HKLM\System\CurrentControlSet\Services\MSExchangeTransport

Once you have created the Parameters subkey, in that subkey, create a REG_DWORD value called SkipPublicMDBRestriction and set it to 0x1. Then restart IIS and the Exchange servers. When I did this, replication mail started to flow and the hierarchy was populated (I did have to wait a while to see the hierarchy completely populated).

This Registry value is a work-around though and it allows unauthenticated public folder replication traffic to be accepted. Not something you want to leave open.

After a couple of more hours of research, I finally found the root cause. Someone had added the old Exchange server computer accounts to the Domain Admins group. Remember, Domain Admins does not have all of the permissions that the Exchange Domain Servers / Exchange Enterprise Servers have. There are some explicit denies such as the Receive As and the Send As permissions.

Once I took the other Exchange servers out of Domain Admins, I removed that Registry key, restarted all the servers and the hierarchy and the content was flowing normally. Whew!

First Baptist Church of Watertown and equal rights

The First Baptist Church of Watertown, NY has fired a female Sunday school teacher, citing Timothy 2:11-14, which says, “A woman should learn in quietness and full submission. I do not permit a woman to teach or to have authority over a man; she must be silent.” Mary Lambert had taught Sunday school for First Baptist for 54 years. (Leonard Pitts, Jr., VNC—Ventura County Star - 8/28/06)

Imaging telling this little old senior citizen that has been teaching Sunday school for 54 years that she can't do it anymore when all she wants to do is spread the good word. What would Jesus say?

Life without electricity - Earthquake recovery

Thanks for the e-mail messages of concern. Yes, here in Hawaii we did have a pretty big Earthquake on Sunday morning around 7:10AM. It actually woke me up (something my mother used to claim required an atomic explosion). I live on Oahu which is at least 100 miles from the epicenter (and 3 islands away). It shook for about 30 seconds and actually increased in intensity towards the end. About 3 minutes later, Hawaiian Electric pulled the plug on our power. A minute after that we had another small tremor, but that was the last aftershock that I felt.

I went back to bed. I got back up around 9:30AM and the power was still out. The city was urging people just to stay home, but I was VERY bored and hungry. My roommate and I drove around a little in the middle of the afternoon. I understand why they were saying "stay home". With the traffic lights out everywhere if there had been much traffic, it would have been like a demolition derby.

Unfortunately, we did not get power back on until after dark that evening. I live up on a hill where I can see from Kahala, Diamond Head, Waikiki, and Downtown. It was VERY strange to look out and see just blackness last night (except for a few car lights.) Finally got e-mail connetivity re-established about 7:00AM this morning. Naturally, I was in withdrawl. :-)

E2K7 - RUS RIP

Yes, when you get moved over to Exchange 2007, the Recipient Update Service (RUS) will be no more. When it worked, we really did not think much about it (except for that delay between creating a new mail-enabled object and the time when it got its e-mail addresses). When it did not work, though, heaven help you! I have heard stories of organizations that wrote their own provisioning tools specifically to get around the shortcomings of the RUS.

Evan Dodds wrote a great Exchange Team blog entry called Good bye RUS. Go forth and read it! And, it includes some good information that you will need to know during a period of interoperability E2K/E2K3 and E2K7)

Cool add-on for Intelligent Message Filter

A pretty common set of feature requests for the Intelligent Message Filter (IMF) includes global safe sender lists, more detailed SCL handling, additional actions for spam, adding content inspection such as SURBLs, white list based on message body, sender, header, etc... SecurExchange has a nice add-on that does most of these things.

Inheritable permissions are being cleared for some users

You know the expression "You learn something new everyday"? Well, if you are in IT, the expression is something like "You learn 1,000 new things everyday". That is what I like and hate about IT. There is always something new to learn. But, there is always something new to learn. Well, the gray hair makes me look dignified. :-)

One of my customers recently applied the Exchange 2003 SP2 hotfix that changes the way "Send As" permissions are delegated. See KB 912918: "Users cannot send e-mail messages from a mobile device or from a shared mailbox in Exchange 2000 Server and in Exchange Server 2003" for more information. After the fix was applied, they delegated their Blackberry server's service account the necessary permissions to the Active Directory.

However, they found that some of the users kept breaking. Upon further investigation, they found that for some users, the "Allow Inheritable Permissions From The Parent To Propagate To This Object And All Child Objects. Include These With Entries Explicitly Defined Here." checkbox was being cleared. It took them a while, but they figured out it was ONLY the members of the Domain Admins group.

My first question to them was "Why are accounts that are members of Domain Admins getting mailboxes in the first place, much less carrying around Blackberries??!!" Answer: They don't separate administrative accounts from the administrator's regular user account. I know, I know. Bad, bad, bad practice!

I found out this was a "feature". This is one of those "double-secret probation" tasks that the domain's PDC emulator does once an hour. If you create an account (or move an account) with Domain Admin (or Operator groups) membership in to an OU with delegated permissions, the "Allow Inheritable Permissions From The Parent...." checkbox will be cleared and the default permissions are applied. This is to make sure that an OU administrator cannot manage a Domain Admin or domain Operators accounts.

For more information, see KB 817433: "Delegated permissions are not available and inheritance is automatically disabled"

And, yes, they should segment their administrative accounts from their user accounts. IMHO, no administrator account should ever have a mailbox.