Corrupt E-mail Issues, Friday, March 18 2016

Apparently there was significant file system corruption following the previous Unscheduled Emergency Maintenance for wilma.widomaker.com, Friday, March 18 2016, 8:00 PM EDT.

This issue only effects users with an e-mail address that ends with @widomaker.com (or @tni.net).

wilma.widomaker.com is the e-mail server that handles e-mail for customers with an @widomaker.com (or @tni.net) e-mail address. It is provided by a server hosting provider, and datacenter, in Raleigh, NC. They handle all physical maintenance of our servers at their facility in Raleigh. We choose this provider because of their excellent reputation, and because they are considered experts in the type of servers that we use. They have been providing several servers to us since 2013.

Late on Friday, the server provider notified us of a hardware issue that affected one of our servers…

The hypervisor that your VPS wilma.widomaker.com is running on experienced a drive failure earlier today. Normally a drive replacement is a routine event, however a hardware anomaly has led to system instability with the hypervisor.

The server hosting provider has explained to us that there were several hardware issues (failed hard drive and subsequent hypervisor malfunctions) with the equipment that our server runs on, that caused the operating systems, and disk drives to become unstable.

There are some indications that they failed to properly shutdown our server before removing power to perform the emergency maintenance to fix the malfunctions, and replace parts and equipment. The sudden loss of power, to a running server, would have likely caused additional file corruption, if the other failures and malfunctions had not already done so.

Through some combination of these hardware failures, and anomalies, some files were scrambled, some lost, and in some cases, mixed together with other files, causing some e-mails to be misdirected and appear in the wrong mailbox. That’s not something that is supposed to be possible, but it happened through some unusual combination of simultaneous hardware and software failures, the exact conditions of which would be highly unlikely to occur again.

The server provider has assured us that all of the malfunctioning equipment has been replaced and they do not expect a re-occurrence or similar failures.

The file system corruption has affected some users mailboxes, which were in use at the time, by causing corruption and possibly the loss of some mail messages. This can cause an error message when attempting to check for e-mail from your computer.

Typical error messages include…

• Unable to process From lines (envelopes), change recognition modes or check for corrupted mail drop.
• Can’t open INBOX: not in valid mailbox format

If you receive one of the errors listed above, contact our office to report the problem so the corruption can be removed from your mailbox which will allow e-mail delivery to resume.

Additionally, it appears that, in some cases, file system ownership, permissions, and names were scrambled, due to the corruption, in such a way that some mail files belonging to one user were assigned to another. This is highly unusual. We have never heard of this happening before. The experts with our server provider, and others we have checked with, have not heard of this happening before. However, severe file system corruption, that occurs when power is removed from a running server, or other hardware anomalies, can have unpredictable results.

While this event was completely out of our control, we do apologize to any affected customers.

Customers should be aware of the following…

• E-mail received and waiting on the server, since the last time you checked and downloaded mail to your computer (using a POP3 e-mail client like Outlook, Thunderbird, etc.), may have been lost, if the mailbox was in use, at the time. If you were expecting e-mail from someone, late on Friday (3/18/2016), but didn’t receive it, ask them to send it again.
• A small number of customers may have received e-mail sent to someone else. If you received an e-mail message addressed to someone else, please delete it.
• If you have noticed that a significant amount of mail was lost, and it was received prior to, 2016-Mar-18 19:09:32, it might be recoverable from a backup made by our server provider. You must let us know immediately (and within 7 days), if you would like us to attempt restoring a back-up. However, e-mail received between, 7:00PM and 10:15PM ET, on Friday, March 18th 2016, may still be lost.