Resolved: Our DNS provider has had a service outage
If you’ve already read all this and just want the latest updates:
#Update 20161213 09:36 (Resolved)
About the outage:
ChangeIP, our DNS provider that we use to make a component of your (and our) remote access work, has had a system outage. This is the technology that powers the whole “fillintheblank.redlanceremote.com” thing that you may use to connect. Their site and services have been completely down all day Sunday, and well into at least early Monday morning as of the writing of this post.
They appear to be posting status updates on their Facebook page, which is listed here so you can keep track of the status at the same time we get it:
https://www.facebook.com/ChangeIP/
If this turns out to be a super-extended outage, we have a backup DNS provider that we can use. We just need to install something on your network to make it work, and your remote access icon/procedure will need to be revised.
Update 2016-12-12 12:15
Update from the provider’s website:
Sunday 12/11/2016 at 1PM PT we experienced a database failure that cascaded through all of our redundant systems.
At 2PM PT we started the process of restoring data from systems backup. Unfortunately due to the size of the database
the restore process is taking much longer than anticipated. Once the database restore process is complete, systems
operations will be returned to normal.
Update 2016-12-12 17:58
Yep, system is still down. We haven’t heard any updates, other than the restore is “taking a long time”. We suspect they’re using cloud-based backups, which would explain the slow restore rate. (This is one of the reasons we recommend to NOT use cloud based backups).
So, we’re all still waiting.
If you need to get back to work right away, call or email us and we’ll install our backup DNS provider’s system so you can get back to business. Otherwise, we’re hoping this all gets cleared up soon.
Update 2012-12-12 23:03
They’re making progress… This is from their latest FB post as of 10 minutes ago:
Changeip DNS Service is now restored, please allow some time for DNS cache to propagate but all our nameservers are now responding again
Our website and Dynamic DNS services are still down as DB is not fully restored yet
The Dynamic DNS services are what we use, so just waiting for those to be recovered then we’ll be good to go…
Update 20161213 09:36 (Latest- RESOLVED)
Well it looks like we’re back up. Not 100%, but things appear to be working. The provider’s site is still down, but the services appear to be working. Sorry for the inconvenience! Hopefully they’ll strengthen their network to prevent this kind of thing from happening in the future!