Filed-based antivirus scanners on Exchange servers

Have I ever mentioned that really dislike file-based antivirus scanners on Exchange Servers?  Really, really, really dislike them.....

For the past 6 weeks, at work we have been fighting weird latency issues for most of our users.  Outlook slows down to a crawl at some points during the day.  We have been blaming WAN congestion, but this past week we noticed that the Test-MapiConnectivity cmdlet was showing much higher than expected latency. (Normal should be less than 20-100ms).  We were seeing spikes on all databases as high as 1,000ms.

The plague of modern IT is LOTS of agents and we are no exception.  Inventory, monitoring, security, antivirus, etc....   The latency went away when we removed Symantec Endpoint Protection.  While the exclusions were all supposed to be in place, we were still seeing the latency until the SEP client was removed entirely.  (A side note, the local firewall component of SEP was disabled.)

So, I guess I'm now questioning whether Exchange servers really need file-based antivirus scanners any longer?  Provided clients are well protected and administrators don't logon to the server console frequently to perform day-to-day administration, do we really need those file-based scanners?