Unfortunately as you are seeing this page, you have come during the time of the worst hardware failure in our company's history.
The website you are expecting, or the email account you were trying to reach went offline at roughly 4:45pm PST 11/23/2015; after we experienced a cascade hardware problem that affected our storage (disk) arrays. That destroyed or damaged about 1/2 million files as well as 50 thousand historical emails. As this was the perfect storm to sink us, we also found the "data snapshot" technology was saving partially corrupted files and our server level backups were unusable...
LATEST EDIT - 7:53pm PST 11/27/2015 (next update 12 midnight PST)
EMAIL IS UP - but *ALL* historical emails have been removed at this time, the level of corruption of basic items to make the mail cluster function was too high - and we'd already extended this outage an additional 38 hours trying to bring the email servers with historical email, instead we switched gears to restoring email service first, and data restore second, we will work in the background to restore any historical emails we can starting next week.
More websites have been restored, and the mast majority of sites will be online by midnight tonight. Several of you have called asking for status we apologize this has taken so long - but we are working as fast as possible.
DNS servers are now completely live on new servers in California, and new servers in Oregon.
CRITICAL - Email Services are back online - however, you may not be able to get to them yet - as we have to wait on local DNS changes at your ISP. While you are waiting - PLEASE check to see if you are using an Email Client such as Thunderbird, Outlook, Apple Mail, etc.. that can "see" all your Historical Emails right now - if YES - then we highly recommend that you use the option of "Local Folders", "Personal Folders", etc.. to make a copy of any EMAIL in your IMAP inbox. As mentioned above a large number of existing emails in the IMAP inboxes have been affected by the outage - we can not restore any missing email in your inbox. - If you don't do this BEFORE you connect to the Email Server whatever emails are missing on the server, will DELETE you local copy. PLEASE make a copy of your IMAP inbox - BEFORE you connect to the Server (use Offline Mode / Unplug your computer from the Internet / Disconnect your Wireless, etc... THEN open your Email Client and make a copy ).
NOTE if you use POP3 you should already be storing data locally on your machine, and no action is required.
CRITICAL - large numbers of static website files are missing or damaged, this means you will likely be missing images, you may need to re-upload your vBulletin, Wordpress, Drupal, etc.. Software.
NOTE: For those of you who have customer uploaded documents, images, etc... this is obviously more difficult, but we absolutely will be spending all our time after we get your website back online, looking for those missing images or documents if we can find them - the 1/2 million static files aren't completely lost but they are mangled, damaged, etc... at best it will take days or weeks to sort it all out... although it would obviously be much faster if we can find them from the source and just upload them again.
- Maybe you downloaded those weekly or daily backups we put into your "sitebackups" folder? If yes - you are ready we'll email you when your website is back online - then you just upload the backups you downloaded previously and voilla! you are back online.
- In theory all those files on your site got there when you uploaded them, if you still have them on your computer you are also ready!
- However, if you are like most of us - and you never downloaded those backups, and you can't find whatever images or documents you uploaded on your computers, a customer pointed out that this site - "WayBackMachine" located at - https://archive.org, should have copies of your static images, and other public content on your website
NOTE - backend data storage such as MySQL or MSSQL databases were not affected at all, so any "posts", "private messages", "orders", "order histories", etc.. all of those are sitting waiting for the WebServer w/proper software to reconnect. Also our older linux based Web Clusters and Windows Web Clusters were not affected.
Also - we absolutely know this is the worst time for any business to have a major email and website outage, not to mention everyone else who has a website not for business was looking forward to all the sales that this season brings - we are very aware this has been a huge problem for everyone, we apologize greatly and continue to work to restore service as quickly as possible.
11/23/2015 - 4:45pm PST - what started as seemingly simple hardware failure of 1 drive, quickly cascaded in a full outage caused by multiple hardware failures one after the next these failures spanned large sections of the storage arrays - causing all Name and Email Servers to go offline, also a large number of Web Sites were affected.
11/24/2015 - 2:00am PST - Update1: Accounts and most of our web and application servers are offline, database servers were not affected, but without application and web servers the content is not reachable - we are actively working to restore service, however between expiditing replacement parts and time to restore from backups we are expecting services to restore by roughly 6am PST, November 26th.
11/25/2015 - 2:00pm PST - Update2: Domain Name Server (DNS) Service restored, after multiple trips from service techs not having the all the right parts and then overnighting replacement parts - we replaced all the physical hardware late Tuesday; we finished restoring from Data Snapshots early afternoon Wednesday - only to find all the snapshots refused to create a consistent restore, as DNS information does not change that often, we were able to get our DNS servers back online at this time, which meant anyone who was hosted on our older linux or windows based web clusters - those sites came back online at this time. Email is still OFFLINE.
11/25/2015 - 6:48pm PST - Update3: A time drift/snapshot bug in the various backups has made full recovery impossible. We highly suggest if you are using an email client such as Thunderbird, Outlook, etc.. that you use the "Local Folders", "Personal Folders", etc.. to make a copy of any EMAIL in your IMAP inbox. A large number of existing emails in the IMAP inboxes have been affected by the outage, and as email is not something that we do daily backups of - we can not restore any email in your inbox. Please create a "Local or Personal Folder" and drag and drop all email from your IMAP inbox into a local folder stored on your machine. NOTE if you use POP3 you should already be storing data locally on your machine, and no action is required.
11/25/2015 - 11:30pm PST - Update4: Integrity checks of the restored storage snapshots are still running, and still finding lots of errors due to the time drift of the snapshots. We suggest you locate copies of the static files (pictures, app software, etc) as you may need to re-upload some content to the webservers. Again - the database servers are not affected at all - so all posts, messages, purchase history, etc.. are not affected, but the forum software, or uploaded images may be affected - we'll know more once the integrity checks finish.
11/26/2015 - 4:00am PST - Update5: Integrity checks finished - which ended up producing more questions as to how we arrived where we are; in short we have unusable backups, lots of filesystem corruption, and now sets of servers that even though they can now access working hardware refused to run at speeds faster than a computer from the late 1960s . As we have already been planning the next phase of our services to be built using Amazon AWS Cloud Servers as an Amazon Partner, we have spun up several servers there and are moving critical services there as quickly as possible. We hate to say it - but you likely won't have your website back in the next 2 hours - unless you already have your site back now. Next update will be at 12 noon PST.
11/26/2015 - 12:16pm PST - Update6: The time drift/snapshot bug in various backups as mentioned has resulted in all backups for the affected webservers and email servers to be completely useless. Further the fully restored hardware while functioning correctly is running gruesomely slow at the Hypervisor (Virtualization) layer - causing the restored servers to run so slowly they timeout before serving a single webpage or a single email. We are copying all Email Data to our newly provisioned AWS Cloud Servers - we will locate a DNS server there as well, to make sure that DNS has no further outages. At this time about 40% of the email data is transfered to AWS - and plan to bring email services back online by around 6pm PST.
11/26/2015 - 4:44pm PST - Update7: The time to copy all this data to the new AWS datacenter is taking a bit more time than expected, targeting closer to 7pm PST now.
11/26/2015 - 7:01pm PST - Update8: Obviously our ability to predict how fast the internet is on Thanksgiving Day while probably everyone is relaxing with family, friends, watching NetFlix, Hulu, etc... is not very accurate - we are about 90% of the data copied - things really slowed down in the past couple hours. Now shooting for 9pm PST to restore email and finishing moving backup DNS to the AWS datacenter.
11/26/2015 - 8:53pm PST - Update9: DNS servers are now completely removed from the previous hardware clusters, now located in California and Oregon datacenters. Email is still copying as are the Websites.
11/27/2015 - 12:53am PST - Update10: Email Services are live and in a new datacenter located in Oregon, however due to the address change your laptop/desktop/phone may still be going to the old server . If you can see your Server INBOX in your Email Client as offline now - create a local archive folder and copy everything from your Server INBOX (IMAP) to a local folder!
11/27/2015 - 3:53am PST - Update11: Obviously no sleep still - ok Email is NOT working in Secure mode at all. And my own DNS hasn't updated yet so I can't test with older clients that don't require secure connections. Will look at this soon - for now - please try and see if you can get to your email in - insecure mode
11/27/2015 - 10:56am PST - Update 12: Got a short nap! Sorry missed some calls, current update is the usernames/passwords for the IMAP/POP servers aren't working. However we do see new emails arriving in your Inboxes. We also restored additional websites - we'll notify who's back online as soon as we can login to the email system
11/27/2015 - 1:19pm PST - Update 13: Data recovery is stopping actual service from starting, all EMAIL before Nov 23rd at 5pm PST - has been moved to a backup folder for now, when you login you will see NONE of your previous emails. However any email delayed since Nov 23rd at 5pm PST will start showing up now. Really hope you read this site earlier and backed up your IMAP inbox.
11/27/2015 - 7:53pm PST - Update 14: Email is flowing in, the next set of challenges are the vast majority of your WebSites that previously were multi-server clusters routed behind software load-balancers - well the software load-balancers are a mess that's taken us over 20 hours trying to get working, we are giving up. We are instead reconfiguring all websites to run in Single IP mode - the majority of websites should be online by around midnight tonight
Professional Edge LLC