Smartermail 7 and IMAP Mailbox Corruption

Discussion in 'SmarterMail' started by kurt2439, Mar 8, 2011.

  1. kurt2439

    kurt2439 Member

    I'm hoping this thread can be a new gathering point for SmarterMail users and admins who are still experiencing the IMAP mailbox corruption problem that we and others have experienced from SM5 through SM7, and Windows2003 through Windows 2008*. We currently have a ticket open with SM Support and they have agreed to develop a build of SM7 with advanced logging enabled that will supposedly help them track this issue down. Since the corruption happens for us only about every other month, it will most likely be time consuming to generate enough data to debug this -- meanwhile our users and presumably yours are vulnerable to this.

    I'm hoping that others will be interested in running this advanced logging build, which SmarterMail developers assure me will be completely stable. If enough of us are able to help, they should be able to track down the issue far more quickly. Smartermail has told me they will make this build available to anyone who requests it.

    Are there any users out there that have experienced this in the past, or are still experiencing the issue now and want to help?

    *To be clear, I am talking about IMAP mailbox corruption, usually in the form of messages dissapearing from the inbox or showing up with a blank subject line and no data in the e-mail. Often the e-mail view within webmail will say that the message has been deleted from the server, although there is still a message pointer in the Inbox. It will usually be clustered in one day, or over several hours of one day due to SmarterMail arranging each days messages in a seperate .grp file.
  2. Alex Clarke

    Alex Clarke Member

    After upgrading to SM 7.5 I noticed this issue with my mailbox. I've been unable to replicate this... yet, but it happens almost every day.

    It appears to occur when moving emails via Outlook 2010 using an IMAP connection to the server.

    I'll be upgrading to SM 8 over the weekend and, if problems persist, will try the custom build.

    Please let me know if you need any further information.

    All the best,

    Alex
  3. lenjacobs

    lenjacobs New Member

    I am also experiencing this seemingly random problem. At least once per month, even after upgrading to SM7.X, IMAP users have their mailbox corrupted and emails are just lost. Not having seen this as an issue resolved with the latest upgrade to SM8, I am not eager to upgrade until the problem is resolved.

    Apparently one of the developers has offered to provide a build (sometime in the future) with more logging for debugging this problem. That means once the problem occurs and a user loses their emails, there may be evidence in the logs to understand what causes this problem. This seems a danbgerous approach to fixing this long standing and troubling deficiency with SmarterMail. The IMAP corruption issue has been one that has been logged in this forum for several years.

    I also have an open ticket on the problem. The problem has been acknowledged but not resolved. What about the other members of this community who have experienced this issue -- have they simply bailed out of SM altogether? Else, what about setting up a SM server we can all use and hammer on until the system produces the IMAP corruption? The idea of running an experimental build in a production environment seems reckless and unwise.

    Does anyone at SM -- management or developmen teams -- have any better suggestions? We mainly love the product but are really disappointed in the lack of resources and attention given to this defect. Can we as a community of users fined a way to particiapte in the discovery of the cause and resolution of this bug?
  4. fastc

    fastc New Member

    I have the
    same problems using IMAP.
    It seems there are several problems:

    Mail that
    is simply not delivered (I use the archive feature and found mails in the
    archive folder which where never fetched using IMAP but I don’t know if this is
    a IMAP fetch problem or more general). Several weeks ago I had no mail
    delivered for approx. two hours but was
    able to find all in the archive (SM7).

    Outlook
    IMAP error code 0x800CCCDB, I get this hundred times a day (booth SM7 & SM8).

    Mail that
    is corrupted (empty subject line)

    Mail that
    is corrupted and contains several mails in one (when you click on source view
    in webmail you can see that there are several mails merged together).

    The last
    two issues I think are a threading problem. Reducing the number of concurrent
    IMAP sessions from 5 (default) to 1 had resolved this issue for me! I did not test this with SM8.
  5. levelbest

    levelbest Member

    Interesting. I run 2 servers - one this happens on and the other doesn't. I did just upgrade SM7 on the server with the ongoing corruption issues to SM8 hoping it would solve the problem.

    We never had this issue prior to moving to Windows Server 2003R2 that I can recall.

    I had thought it was something to do with the fact I was hitting up the IMAP server from my phone, ipad, web, notebook, outlook and such and it was causing the corruption.

    Happened today just before I ran the upgrade. Blank subject lines or senders, jumbled messages where one will have half the message, another will have the other half, destroyed formatting... very odd stuff indeed.

    Doesn't happen on my 7.5 install on Win2003 (not r2)
  6. fastc

    fastc New Member

    I'm running on Windows 2008 Standard x64 SP2

    What IMAP client are you using and are multiple IMAP clients running at the same time?
  7. lenjacobs

    lenjacobs New Member

    I have been using Outlook 2007 & Outlook 2010 clients, but also Thunderbird 3.1. This happens with only ONE IMAP client/user accessing the server. And losing our users' emails is a totally unacceptable service from a commercial email server that "supports IMAP".

    Can we share our resources and experiences for trying to address this problem? I cannot understand how to get SmarterMail development team to focus on this problem. They cannot deny it is an issue -- and this has been an issue of some note for the last several releases of SM.

    Suggestions?
  8. fastc

    fastc New Member

    Hi lenjacobs,

    I totaly agree with you! So far my "solution" was to delay customer migration to SM server.

    I'm the only user on my SM server I thought this issue will be addressed soon by SmarterTools.

    To be fair I did not open any ticket because I did never find something in the log files and I'm not able to
    reproduce the problem. I'm pretty sure a simple test will not show up this problem its rather complex.

    I can only speculate about some possible issues.

    I still think its a multithreading issue but I'm not sure.

    Reducing the number of concurrent IMAP sessions in Thunderbird did help but did not resolv all issues.

    The server I'm using is not a dedicated SM server, hardware is stable I have absolutly no other problems on that server.

    There is no Antivirus Software running on that server!

    Maybe using IMAP and Webmail at the same time is a problem but I think I had this issue using IMAP only.

    I fetch mails also with my Windows Phone7 via IMAP and sometimes via ActiveSync but ActiveSync fails so often

    that I do not use it regularly.

    I'm using only Outlook 2010 (IMAP), sometimes Webmail, Thunderbird (IMAP) and IMAP on Windows Phone 7.

    I used Outlook 2007 in January
    to but I'm not sure if this did make any difference. I never used POP3.
  9. lenjacobs

    lenjacobs New Member

    What are you offering your clients as an alternative to SmarterMail?

    Have you tried connecting with any of hte SM sales or tech folks to get them to address this nagging issue? Any suggestions as to how to get their attention and focus their resources on finally fixing this bug?
  10. kurt2439

    kurt2439 Member

    We upgraded from 5->6 and then 6->7 hoping the issue would be resolved. I don't think it has been addressed because they do not know what the source of the issue is, so I don't think it is resolved in 8 unless there was a complete overhaul of the IMAP implementation. I hope we are not expected to run server releases previous to Windows 2003 to difuse the issue! We are at Windows Server 2008, no anti virus, and it is a dedicated mail server.

    People often cite running multiple IMAP connections as the source of the problem as if that is the users fault, but I have spoken to tech support at SM about this and they say running mutiple IMAP connections, for example having WebMail and Outlook open, or simultaneously checking on your iPhone, is supported -- so I don't see any reason to harp on that. They support it, so if it corrupts your e-mail then that is their problem not ours.

    I just heard back from the developer and the word is that the release will be ready by Monday. The additional logging may or may not be included in public releases. Apparently it is still being decided upon. I would definitely urge SM to include this so that getting more feedback from users is as painless as possible and I have expressed that opinion to the developer. Hope there are some more users out there who want to run this build and/or figure out some testing environments as a community.
  11. buddypepper

    buddypepper New Member

    I'm pretty glad to have come across this thread...I've been planning on switching my users from POP3 to IMAP soon, but it seems I'd better hold off just yet. One question for people with this problem - are you running the built-in SM webserver, or IIS?
  12. kurt2439

    kurt2439 Member

    We use IIS. I don't think it matters though since the corruption happens without accessing the mailbox via webmail.
  13. buddypepper

    buddypepper New Member

    Just though I'd post this link - this isn't the first IMAP corruption thread...the last one got nowhere (clearly). This problem has been around for 6 years now.....

    http://forums.smartertools.com/t/7379.aspx?PageIndex=1
  14. kurt2439

    kurt2439 Member

    I have seen that link and it's absurd ending involving the CEO closing down what had previously been a useful conversation amongst community members. But I don't want to rehash the issue of whether or not the problem exists or what the exact causes of it are.

    The point is there is attention being paid to this issue by SmarterMail developers. By getting involved and helping out by running a new build with advanced logging, which SM has said will be completed next Monday, we can help resolve the issue. Smartermail has been a great platform for us, but the IMAP issue is unacceptable. It should have been resolve by now but there is real effort being placed on solving this issue by SM developers right now. All of use that have been effected have an opportunity to resolve this now
  15. jmandala

    jmandala New Member

    Seems like a common thread here is the difficulty in reproducing the problem.

    Do you think if we were able to reproduce the problem in a safe environment that we could then help SmarterTools develop a comprehensive fix?

    If my company were to host a development SM instance, would you any of you be willing to hammer on it (forward emails, setup mail checking / downloading / other?) in order to see if we can create a reproducible state / error process?
  16. csimo

    csimo Product Expert

    Many have reported this and had trouble tickets including myself. Typically SmarterTools will see evidence in the logs and tell you that they're going to escalate this to developers and that's the last you'll hear of it.

    They ran Filemon on my server for about 2 weeks to prove that nothing was touching the files except SmarterMail.

    Here's what I can tell you:

    1. The problem can occur with any IMAP client, but is worst with Thumderbird for some reason. Once a message is corrupt you cannot recover it with webmail or POP.

    2. If you have Archiving enabled for the domain the message will be intact in the archive even after SmarterMail IMAP has corrupted the message in the users mailbox.

    3. The problem has nothing to do with high or low volumes of activity. It will happen even if the server is at very low loads. It's not a load related issue.

    4. The problem happens with every version of SmarterMail I've used. I think I stared on v3 and it still happens on v8.

    5. I don't think there is anything you as a system administrator can do to keep the problem from happening other than not allowing IMAP with SmarterMail.

    -Joe
  17. lenjacobs

    lenjacobs New Member

    Joe --

    Are you hosting SM[78]? And are you saying that none of your users use IMAP, or they do at their own risk? Do your clients/users accept that limitation? Have you investigated alternative to SM where this problem is known NOT to exist, like Imail for example? And are you skeptical that the developer will provide a build which will handle additional logging as a way to possibly unravel this problem?

    And finally, what do people think about hammering on a new installation of SM where we can all try to make the system lose some of our IMAP mail? Or is it so random that this will not provide real answers?
  18. buddypepper

    buddypepper New Member

    I'd certainly help out with hammering a test server with Thunderbird...I need IMAP to work, and I'm certainly not about to allow it on my production server at the moment - I'll gladly help out in any way I can.
  19. kurt2439

    kurt2439 Member

    It was unnecessarily painful to get traction on the issue, but it does seem like they are putting some resources into this now. This was escalated, I am in contact with the developer, and they are releasing some additional tools for debugging this -- supposedly on this upcoming Monday

    Doesn't that just verify that the problem is internal to Smartermail?

    If you are experiencing this issue often, seems like it would be great to get your support in running the advanced logging build and sending them more feedback on the issue. Would you be willing to do that?
  20. csimo

    csimo Product Expert

    Yes, we use SmarterMail 8, but we also maintain an Imal server for those domains that wish to use IMAP. I don't like Imail but we've used it for 15 years and have never had one single "blank" message due to the IMAP problem SmarterMail suffers from. I would love to get rid of the Imail server.

    We do have a few users on IMAP on SmarterMail but they have been told of the IMAP corruption problem and that there's nothing we can to do resolve the problem. We enable Archiving on those domains so the domain admin can recover the messages if needed. d

    Yes we still see the blank sender, blank subject messages all too regularly.

    As I said earlier, I doubt "hammering" on the server will do any good.

    I really don't want to get into this issue again... too frustrating for me. It's been going on for years. If you search the old forum messages you'll find the string you can use to search your IMAP logs and find the error.

    -Joe