Data in extension / custom attributes cleared when mailox is removed

Did you know... if you store data about a user account in the extensionAttribute1 - extensionAttribute15 that the data will be cleared when the mailbox is removed.  This is expected behavior and by design.

Filed-based antivirus scanners on Exchange servers

Have I ever mentioned that really dislike file-based antivirus scanners on Exchange Servers?  Really, really, really dislike them.....

For the past 6 weeks, at work we have been fighting weird latency issues for most of our users.  Outlook slows down to a crawl at some points during the day.  We have been blaming WAN congestion, but this past week we noticed that the Test-MapiConnectivity cmdlet was showing much higher than expected latency. (Normal should be less than 20-100ms).  We were seeing spikes on all databases as high as 1,000ms.

The plague of modern IT is LOTS of agents and we are no exception.  Inventory, monitoring, security, antivirus, etc....   The latency went away when we removed Symantec Endpoint Protection.  While the exclusions were all supposed to be in place, we were still seeing the latency until the SEP client was removed entirely.  (A side note, the local firewall component of SEP was disabled.)

So, I guess I'm now questioning whether Exchange servers really need file-based antivirus scanners any longer?  Provided clients are well protected and administrators don't logon to the server console frequently to perform day-to-day administration, do we really need those file-based scanners?

Installing Exchange rollup fixes

While I certainly see the value in Windows 2008 UAC (User Access Control), it can be a big pain, too.  Too often, you can try to run a program only to have the program fail for no reason.  I have seen this recently with Exchange 2010 Rollup fixes.  Today while installing E2K10 SP2 RU3, I had an issue where the installer appeared to run, but failed within a minute or two and gave no obvious reason.

The Windows Application Event log had 2 fairly generic errors:

Event 1036
Windows Installer installed an update. Product Name: Microsoft Exchange Server. Product Version: 14.2.247.5. Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: Update Rollup 3 for Exchange Server 2010 Service Pack 2 (KB2685289) 14.2.309.2. Installation success or error status: 1603.

Event 1024

Product: Microsoft Exchange Server - Update 'Update Rollup 3 for Exchange Server 2010 Service Pack 2 (KB2685289) 14.2.309.2' could not be installed. Error code 1603. Windows Installer can create logs to help troubleshoot issues with installing software packages. Use the following link for instructions on turning on logging support: http://go.microsoft.com/fwlink/?LinkId=23127

The fix for this, though, is pretty simple.  Open up a command prompt as an administrator and simply run the patch from the command line.  In my case, I just typed:

Exchange2010-KB2685289-x64-en.msp

Then the roll-up installs correctly (and it does take 30-45 minutes to complete).

W2K8 / E2K7 Cluster Comms issue

A few weeks ago, someone called me and asked me to help out with an E2K7 CCR cluster running on W2K8.   Regardless of what they tried, the cluster would not achieve quorum.  It had been working up until a few weeks prior and no one had noticed.  The Failover Cluster Administrator suggested that the file share witness could not be contacted. 

However, the CIFS shares on the FSW were accessible and it was ping-able.  The Cluster.log had some interesting errors in it, but the cause was not immediately obvious.  Here are some of the errors that were occuring when the cluster was trying to achieve quorum:

Network Name <Cluster Name>: Unable to Logon. winError 1326
Error 1326 from ResourceControl for resource Cluster Name.
ResourceControl(NETNAME_GET_VIRTUAL_SERVER_TOKEN) to Cluster Name returned 1326.
File Share Witness <File Share Witness (\\FSWSERVER\CLU-01-MNS)>: Failed to get virtual server token from core NetName resource, error 1326.
File Share Witness <File Share Witness (\\FSWSERVER\CLU-01-MNS)>: Failed to retrieve the virtual server token from the core netname resource with 1326. 
RhsCall::Perform_NativeEH: ERROR_LOGON_FAILURE(1326)' because of 'Resource File Share Witness (\\FSWSERVER\CLU-01-MNS): Open call failed.
rcm::RcmAgent::Online: ERROR_LOGON_FAILURE(1326)' because of 'There is a problem with the resource DLL.'
ERROR_LOGON_FAILURE(1326)' because of 'Failed to bring quorum resource e86bd5ca-7bab-4d1c-b9ac-94ef54acdb03 online, status 1326
Signaled NetftRemoteUnreachable  event, local address 10.1.5.210:003853 remote address  
Signaled NetftRemoteUnreachable  event, local address 10.1.5.210:003853 remote address 10.1.5.206:003853
Signaled NetftRemoteUnreachable  event, local address 10.1.5.210:003853 remote address 10.1.5.206:003853

  Frankly, it looked like that problem was with the FSW until the logs suggested that it was the *other* node of the cluster that was not reachable (the IP 10.1.5.206) via port 3853.

  Upon further investigation (which should have been the first thing I looked for), I found my old enemy lurking in the shadows.  Symantec Endpoint Protection with the Network Access protection features enabled.  I checked the SEP firewall logs on the clustered nodes, but it was not showing any errors.   However, once I disabled the Network Access protection component of SEP, the cluster immediately established quorum.

The Exchange support team was unaware the servers were even running SEP.  Their IT security department had deployed SEP to upgrade from an older version of Symantec Antivirus and had not told anyway.

Very cool Exchange organization report generator

I have been meaning to share this for a few weeks.  Very cool Exchange Management Shell script from Exchange MVP Steve Goodman. This script generates a nifty report of some useful information about your Exchange organization including about Exchange / Windows server versions, mailbox databases, sizes, and number of mailboxes.  Management digs this type of information!

Thanks Steve!

E2K10 SP2: Thanks for the new custom attributes!

One of the nice new things that Microsoft introduced in Exchange 2010 Service Pack 2 did not make much news. There are 30 new extension attributes (aka custom attributes). These are msExchExtensionAttribute16 -  msExchExtensionAttribute45.

Note, these are NOT editable from the Exchange Management Console nor the Set-Mailbox cmdlet,  But, they are flagged for Global Catalog replication, they are assigned a mapiID and they can be added to the details templates!

Exchange 2010 SP2 upgrade issue with Discovery Mailbox

I just upgraded my lab servers to Exchange 2010 SP2.  I was somewhat pleased that all I needed to do prior to the upgrade was just run a Microsoft Update and make sure that the recommended and critical updates were applied.  Sometimes, with new Exchange releases, you have to chased down obscure and not-yet-released fixes for things.

I did have one issue as the mailbox role was being upgraded.  Setup crashed repeatedly and included the dump from a script that had failed.

Couldn’t resolve the user or group “volcanosurfboards.com/Microsoft Exchange Security Groups/Discovery Management.”

It is also listed in the eventlog at Event id: 1002: Exchange Server component Mailbox Role failed:

Event ID 1002

Providor Name:MSExchangeSeup

“Couldn’t resolve the user or group /Microsoft Exchange Security Groups/ Discovery Management” If the user or group is a foreign forest principal, you must have either a two-way trust or an outgoing trust.

The trust relationship between the primary domain and the trusted domain failed”

 I chased my tail on this quite a bit assuming it was an Active Directory problem when in fact it was a problem with the Discovery mailbox.  The only solution is to delete the discovery mailbox and recreate it.  My test domain (volcanosurfboards.com) is in the steps below.  Substitute your own domain.

1)   Disable-Mailbox “DiscoverySearchMailbox {D919BA05-46A6-415f-80AD-7E09334BB852}” 

2) Enable-Mailbox “DiscoverySearchMailbox {D919BA05-46A6-415f-80AD-7E09334BB852}” -Arbitration

3) Add-MailboxPermission -Identity:"volcanosurfboards.com/Users/DiscoverySearchMailbox {D919BA05-46A6-415f-80AD-7E09334BB852}” -User:”Discovery Management” -AccessRights:”FullAccess” 

 Note that the user account that is used for the Discovery Search mailbox must be disabled.

The IT Curmudgeon

After a particularly frustrating week at work...

1. There are rarely good technological solutions for bad behavior. (this one comes from Ed Crowley)
2. Any meeting that gets off track more than twice will not achieve its intended goal.
3. Projects that are not well scoped will come in late and over budget.
4. "On time and under budget" are misnomers in IT.
5. An IT department's effectiveness is inversely proportional to the number of layers of management.
6. IT organizations that are afraid / intimidated by their users will inevitably serve those users poorly.
7. Scope creep is the enemy of IT.
8. Good information security practices are important; rigorous information security practices stifle productivity and creativity. 
9. Regularly scheduled meetings diminish in productivity after each meeting occurrence. Beware the recurring meeting!
10. Consultants and vendors almost always act in their own best interest.
11. Complexity and change are the enemies of high availability.
12. Everything has a maintenance / sustainment cost.
13. Fear the IT Manager that tries to get too much in the technical weeds.
14. No good can come from your CIO meeting with your consultant’s “senior practice manager.”
15. No organization ever knows the true cost of their IT infrastructure, services, and operations.
16. Avoid major IT projects using internal staff.  Internal staff gets sidetracked with existing duties. 
17. Beware the IT manager that wants Administrator rights.
18. A Configuration Management team that treats engineering and operations teams like their enemy are encouraging people to find ways to bypass them.
19. People have to see “what’s in it for them” with regards to collaborative tools otherwise they won’t use them.
20. Bureaucrats always expect you to know about their bureaucracy.

Bureaucracy and bureaucrats

"The most annoying thing about bureaucrats is that they expect everyone to understand their little piece of the bureaucracy."- Jim McBee

I really tried to love you, but it is not working out,,, Goodbye OWA 2010 Conversation View

I really tried to love you, but it is not working out.  I'm breaking up with you...   Goodbye little OWA 2010 Conversation View.  And thanks to Tony Redmond for reminding me of how easy it is to turn off...

Click the View button, then clear the checkbox next to Show As Conversations.

OWA and Outlook RPC URLs

A question came up at Exchange Connections this past week about using a separate name space for the OWA/Web Services/ActiveSync URL than you use for the RPC Client Access area.   eg...
owa.domain.com   versus
outlook.domain.com

Microsoft recommends keeping these separate.  While it *does* work if they both point to the same hostname and/or IP address, if they are separate, you reduce the risk of external Outlook clients (Outlook Anywhere / RPC over HTTP) timing out.  

Here is the text from TechNet:

It's important that the (FQDN) specified in the command be only resolvable internally. If the name is also resolvable externally, these external clients will attempt to connect to the array via a TCP connection instead of HTTPS.

 During the session at Connections, I was thinking it was something more complicated than this. 

California AB 155 - It is just bad for business. Period

The California Assembly has passed AB 155 and it is awaiting Gov. Brown's signature.  Though the bill is primarily targeted at Amazon and would require Amazon to pay CA state sales taxes on all sales made to state residents, it has much broader implications for anyone selling online.  Assembly Speaker John Perez says that residents that make purchases online through out of state providers are "tax cheats."

CA Assembly sponsors say they are intent on making California businesses "more competitive" with out-of-state businesses, but that is just hogwash.  This is about hundreds of millions of dollars in potential tax revenue for the State of California.

 If California is successful, surely the other 49 states will follow.

One way or another, the tax is passed on to the consumer. That much is true. But, what about the cost on out-of-state businesses that must charge, collect, record, process, reconcile, and pay those taxes. For a small business that makes maybe 2 or 3 sales in the State of Cali each *month*, I now have a fairly significant monthly, semi-annual, and/or annual burden in the form of labor.  Plus, possible the cost of software and/or services to comply with this law.

A simple analysis of my own suggests that this is going to add 15 to 20% to the cost of operating my business in the form of either a 3rd party service or an accounting clerk that is going to have to track each purchase, its location, the tax rate, and what tax has to be paid for the purchases from that particular state.

Once again, California, you are proving yourself to be unfriendly to business.  Both out-of-state as well as jacking up the price of the merchandise your in-state businesses purchase.

Exchange Connections: The economics of moving to the cloud

The old expression "no one ever gets fired for IBM" shifted a few years ago to "no one ever gets fired for buying Microsoft." We get it. Microsoft technologies are a pretty safe bet for your business. This follows a certain school of thought in business that a technology is not truly legitimate until the big players have an offering. 

I'm not sure you can hear me over the roar of the media and the Microsoft marketing machine, but Office 365 and cloud services are the hot topics.  And, I generally loathe marketing terms such as "cloud". At any rate, cloud services deserve a second (or third) look. 

Cloud or hosted services are certainly nothing new. Since the first large scale mainframes came online, businesses have been outsourcing computer services that they could not effeciently or cost effectively offer themselves. Computerized accounting and billing services were outsourced way back in the 1960s. 

Outsourced e-mail and SharePoint services also not new as of 2011, either. Even in the late 1990s there were providers around the globe offering hosted Exchange services. Microsoft themselves have been in that business with BPOS and other offerings for at least the last 6 years. 

Office 365 is a bit different thought. Aside from the fact that there is a massive marketing effort surrounding Office 365 services, Microsoft seems to be betting the house on these services. The different tiers of service and pricing  seem to be Microsoft's recognition that different customers will have different requirements. They are making it easier than ever for us to move to the cloud including better interoperability with on-premise solutions such as Exchange Server 2010. 

So, even for the most skeptical of cloud curmudgeon, the cloud (and Office 365) should not be considered a viable alternative and not a bleeding edge solution. So, where does that leave us? 

I consider myself an "on premise" kinda guy.  I'm a systems guy at heart and am most comfortable somewhere between an engineering and an operations role. I like to have my hand in both sides of IT. But for many of us, the simple fact is that cloud or outsourced services is in our future and we have to accept that. 

So, I’m a tiny bit conflicted when facing the prospect of cloud-sourcing my favorite part about IT.  But, we have to take a bigger picture view of our jobs. Our first obligation is to provide reliable IT services to our businesses (or non-profit or government entity) and those services must be services that meet our end-user's requirements. But second, and almost important, is to provide those services at a cost that is as affordable as possible. 

When I started thinking about sessions for the Exchange Connections conference that would be of value to today's IT Professional, one of the ideas I had was to explore the economics of moving e-mail services "to the cloud" and what that means to an organization's IT team. We all have visions of mass layoffs, but I'm betting that is not usually the case. 

In order for us to be responsible IT Pros, we need to know not only what our service costs us to offer but also exactly what services we are providing.  In my session "Economics of Cloud Sourcing and what that Means to Your IT Team" I'll explore how you calculate not only the cost of the service you offer (such as the per month per mailbox cost of your system) but also determine what services you are offering to your business and your end users.

This will help you to make an "apples for apples" comparison of costs and services. Some organizations will find that they can do things in the cloud at 25% the price of their current on-premise solution while others are going to find that regardless of the cost savings they cannot duplicate the features or functionality in cloud that are required by their end users. 

IT Pros have a professional responsibility to their employers to approach the cloud with an open mind and eye towards helping their business succeed rather than promoting specific pieces of technology. And, for techies like me, that can be a tough exercise.

I hope you will join me for this session at Exchange Connections in Las Vegas and hear my own experiences and my journey towards weighing costs and services objectively.

Setting the Out-of-Office reply

Very cool!  You can set the Out-of-Office reply on someone's mailbox using the Exchange Management Shell.  Thanks to Bharat Suneja and the Exchange team for sharing this great tip.


Set-MailboxAutoReplyConfiguration bsuneja@e14labs.com –AutoReplyState Scheduled –StartTime “9/8/2011” –EndTime “9/15/2011” –ExternalMessage “External OOF message here” –InternalMessage “Internal OOF message here”

Exchange Connections: 2 new Office 365 sessions from Mike Crowley

We just recently added 2 new Office 365 sessions to the Exchange Connections conference.  Thanks to Mike Crowley from Planet Technologies for these!

EXC22: Exchange Online: Administration
Mike Crowley
Be careful not to fool yourself; Exchange Online (part of Office 365) offloads infrastructure management, but as an administrator, you are still responsible for the administration of your user mailboxes, Internet mail flow, message tracking and more! This session introduces you to the various administrative interfaces of Exchange Online, Forefront, RBAC, provisioning and other operational topics.

EXC23: Exchange Online: Understanding Archiving and Compliance
Mike Crowley
Thinking of moving to Office 365? Whether you are aiming for a period of coexistence or a complete migration, your archival and compliance requirements are not going away! In this session we examine the features and functionality that Microsoft provides around retention, archiving, and search.

Addressing issues uploading photos or writing to logs in Directory Manager or Directory Update

If you are having issues with Directory Update or Directory Manager writing to either the "logs" folder or the "photo" folder, it is probably due to one of two problems.

  First, check the Security properties of the c:\inetpub\wwwroot\DirectoryUpdate\Photos (or .\DirectoryManager\ folder).  This also applies to the Logs folder.  Ensure that the "NETWORK SERVICE" user has "Modify, Read & Execute, List Folder Contents, Read, and Write" permissions to that folder.

  Second, open up IIS Manager, navigate down to the web site on which Directory Update or Directory Manager is installed, view the Advanced Settings of the virtual directory (\DirectoryUpdate or \DirectoryManager) and verify that the Application Pool is "DefaultAppPool" - unless you have created your own dedicated application pool for Directory Update or Directory Manager.)

 Finally, in IIS Manager, navigate to Application Pools, and view the Advanced Settings of the DefaultAppPool (unless you have created your own) and in the Process Model section, make sure that the Identity is set to "NetworkService".

Vote for Directory Update!!!

Directory Update users:  Windows IT Pro has opened voting for the 2011 Community Choice Awards.  Please vote for Ithicos Solutions' Directory Update product in the "Best Active Directory & Group Policy Product" category!!!! 

http://www.surveymonkey.com/s/windowsitpro-communitychoice2011-finalvoting