Jump to content

X-Pilot and X-Aviation Downtime

Cameron

By Cameron
in General Discussion

 Share

Recommended Posts

Cameron Mentor

Cameron

Posted
Posted

Hi, Folks,

 

As I'm sure many of you have noticed we were offline for almost two days. I certainly apologize for the downtime and inconvenience involved with this.

 

The short of this is our server has a RAID-10 setup (four hard drives, two of which are for mirroring/backup). In a rare event, not only did a production hard drive fail, but a backup drive failed at the same time. The RAID controller was also shot, which we believe to be the primary cause in this. As a result, our "live" backup of the websites were lost entirely. That said, we do snapshot backups of our server data every evening and place these backups on an entirely different server. Because of this, we have lost about only hours worth of posts on X-Pilot (very minimal), but that's about it since the snapshot last backed up was less than 24 hours prior to the server crash. In the grand scheme of things this is very minimal after what happened, and most of you won't notice anything missing at all....just a few posts from the day of February 23rd.

 

Because of the nature of the failure and having to transfer the backup from our offline backup server to our new server, this process took quite a while to come back to life.

 

I want to personally thank all of your for your patience as we have worked tirelessly to restore services.

 

Blue Skies!

-Cameron

  • Upvote 5
alpilotx Newbie

alpilotx

Posted
Posted

A dead (or dying) RAID controller is the admins nightmare. I have seen a few failing RAIDs in my life ... even one or two with complete disasters etc. ... so I can understand the implications quite well. Nice to have you back, and good to hear that you had a sound (and whats more important: an obviously working!!) disaster recovery plan!

JimboG Newbie

JimboG

Posted
Posted

Good work Cameron,

Glad you got back with "minimal" disruption... Time for a sleep methinks.

And yes, I couldn't help Googling also to see if everything was OK - DDoS, virus, hack etc. But no, good ol' fashioned multiple hardware failures at the same time. Nice.

Congrats for getting back, as coily and Andyrooc alluded to - I was getting antsy without "my" X-Pilot forum! LOL.

Now go ahead and release something on X-Aviation so that we can all queue up and make sure your POS system is still working, make everyone feel better?

Cheers

James

scubajuan_new Apprentice

scubajuan_new

Posted
Posted

Thanks so much for providing us (the community) with this awesome site and going out of your way to keep it up and running in the best possible way. Is good that is all sorted out, to bad that there might have been a few posts lost, sh@@ happens. 

 

I too felt naked without the forum, and all the interesting readings it provides.

  • Upvote 1
Keith Smith Rookie

Keith Smith

Posted
Posted

Nice work, Cameron. It's ironic, really, that of the 7 drive failures I've experienced in the last 20 years, 6 of them were in the same RAID system.  Suspicious, no?

 

I'm impressed that you had the offsite nightly backups. Great job.

Cameron Mentor

Cameron

Posted
  • Author
Posted

Thanks to all for your compliments!

 


I felt naked without xpilot  

 

Haha, it's one of those moments where you start to appreciate what's there when it's not.

 


Cameron had an exciting weekend to top any server backup failure.

 

Definitely bundles of excitement through and through!

 


Thanks, not only for reviving the site, but also for the information flow through other media.
A much appreciated, very professional approach

 

Most welcome! We gave out progressive notices through other media as we could. :)

 


A dead (or dying) RAID controller is the admins nightmare. I have seen a few failing RAIDs in my life ... even one or two with complete disasters etc. ... so I can understand the implications quite well. Nice to have you back, and good to hear that you had a sound (and whats more important: an obviously working!!) disaster recovery plan!

 

There was definitely some heart sinking going on, even with the knowledge of the snapshot backup. It's one of those moments where you know you've been as safe as you can, but don't know the final outcome until things are in place. 

 


Glad you got back with "minimal" disruption... Time for a sleep methinks.

 

Sleep never felt so good. :)

 


Nice work, Cameron. It's ironic, really, that of the 7 drive failures I've experienced in the last 20 years, 6 of them were in the same RAID system.  Suspicious, no?
 
I'm impressed that you had the offsite nightly backups. Great job.

 

I agree, Keith, it is ironic. RAID setups are supposed to be a signal of security, but when they burn, they burn bad.

 

Definitely thankful for the offsite backups as well. :)

 

Cheers, all!

  • Upvote 1
Colin S Newbie

Colin S

Posted
Posted

Thank you so much for clarifying this, I was scared it had all vanished and I had become homeless and I was ruined...

 

I really gotta wonder, sorry for prying, but how big does a website like this get? And what is the bandwidth that it uses?

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Unfortunately, your content contains terms that we do not allow. Please edit your content to remove the highlighted words below.
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
 Share
Go to topic listing

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...