Redundancy – Don’t get stuck by Provider or Geographic outages (23-Nov-2014)

22 Nov
Guides > Labs > Redundancy - Don't get stuck by Provider or Geographic outages (23-Nov-2014)

Drafted By:                            Greg Waehner (Facebook GregWaehner)
Edited By:                               Jim Sifferle
Last Updated:                      23-Nov-2014
Version:                                   1.1

Purpose:

This Nightscout Labs is intended to provide “do-it-yourself” (DIY) guidance on how to achieve redundancy – to avoid being impacted by Cloud Provider and/or Regional outages.

Disclaimer:

As “do-it-yourself,” this is intended for individuals who are

  1. Comfortable with Azure and MongoLab hosting;
  2. Willing to accept the increased configuration and maintenance complexity; and
  3. Willing to accept that the “Support Team” does not support this for the entire community.

Assumptions:

  • This document is written solely with the use of REST API for the Android Uploader; the legacy mongo upload method is not utilized.
  • This document is written primarily for users of Azure hosting, but others can still benefit from the concepts.

 Do you want redundancy?

Just because you can, doesn’t mean you should. Therefore, before embarking on the mission for redundancy, ask “why?” Here are the benefits to you help you decide:

  • If your site is down for hours because of a localized outage, are you OK waiting?
  • If your site is down for hours because of a Cloud Provider hardware outage, are you OK waiting?
  • If your site is down because you tried to upgrade it and failed, are you OK not having a site until you can get help, figure out what happened and possibly redo everything?

If you answered “no” to one or more of these, then you need “redundancy.”

  • When you upgrade your Nightscout website, do you get real nervous?
  • Do you hold off doing the upgrade to make sure others do it first?
  • Would you prefer to upgrade a separate website first to work out the kinks?

If you answered “yes” to any of these, then you would benefit from “redundancy,” but it’s not necessary.

 

Step 1: Document your “current state” – what you have now.

Before you decide what you want to do for redundancy, you need to understand what you currently have.

  1. Log in to MongoLab.com using your “username” and “password.” You should see your current deployment (see figure below).

redundancy 1-1

 

Next, you need to verify its details. Click the “arrow” on the left edge to expand the deployment information. (See figure below).

redundancy 1-2

redundancy 1-3

In this example, the “cgmitcdr” database is deployed in “Rackspace DFW”. This means that the Cloud Provider is “Rackspace” and the geographic location is “Dallas Fort Worth”.

 

 

  1. Log in to Azure using your “Microsoft email” and “password.” You should see your website(s) (see figure below).

redundancy 2-1

In this example, the website associated with the “cgmitcdr” database is “gcw-nightscout-dr” (disaster recovery) is in the “West US.” For Microsoft Azure, there is no separate “hosting provider.” Azure is the “hosting provider.”

 

So you now know your current state information as:

Nightscout Component Provider Cloud Provider Location
Website Windows Azure Windows Azure West US
Database MongoLab Rackspace Dallas Fort Worth

 

(Just in case it isn’t obvious….. write down your current state information as it will be needed later on.)

 


 

Step 2: Decide how you want to implement redundancy

For redundancy, you generally will want a completely separate Azure website(s) and Mongo database(s), but depending on your unique requirements, one answer does not fit all. There are many options to consider for redundancy:

Assuming Azure-only

  • Same geographic region, but different hosting providers for Site 1 vs. Site 2 database.
  • Same geographic region, but different hosting providers for Site 1 Azure vs. Site 1 Mongo
  • Different geographic region, but same hosting provider for Site 1 and Site 2
  • Different geographic region, and different hosting providers for Site 1 vs. Site 2
  • Different geographic region and hosting providers for Site 1 and Site 2, and for Azure vs. Mongo
  • A separate website and/or a separate database for lifecycle testing

Going Beyond Azure-only – Heroku

  • Heroku for different website geographic region
  • Heroku for different Cloud website provider
  • Heroku for different database region.

Note: This Labs article does not go into the details of Heroku as an alternative or complementary Nightscout Cloud Provider. “How to Heroku” is covered within a separate Labs article, but provides a robust solution for no “single-point-of-failure” (SPOF) by having a  separate Cloud Provider, technology stack, and geography.

 

For Mongo, based upon the “Single Node” Sandbox (a.k.a FREE) the following are your possible options:

Amazon Web Services US East (Virginia) Region (us-east-1)
EU (Ireland) Region (eu-west-1)
Google Cloud Platform Central 1 Region (us-central1)
Joyent Virginia Datacenter (us-east-1)
Las Vegas Datacenter (us-sw-1)
Rackspace Chicago Datacenter (ORD)
Dallas Datacenter (DFW)
Azure East US (VA)
North Europe (Dublin) Location (North Europe)
West US (CA) Location (West US)

 

Note: It is important to note here that Mongo as a JSON database is offered by many (5) Cloud Providers spanning 10 distinct infrastructures with overlapping geographic boundaries. Azure, is just one of the options.

 

For Windows Azure, the possible hosting locations are:

redundancy hosting locations

 

In addition to Cloud Provider and geographic location considerations, you must also think about performance. When implemented, your phone (“Android Uploader” application) will send the BG information to Site 1 (Azure) and then to Site 2 (Azure). Each of the Azure websites, running the node.js application cgm-remote-monitor, will use its own “mongo” and “mongo_collection” connection string to connect to, write to and retrieve data from the mongo database. As such, if you are in (for example) Virginia, then having the primary website be in Virginia would be good for performance for the uploader, but what if the database is in Ireland (eu-west-1) via Amazon Web Services? Pure physics, it will take longer for the database transactions if the website is in Virginia, and the database is in Europe. And, just like a highway, the longer the road, more potential for speed bumps, traffic jams and accidents — it’s possible that the route from Virginia to Ireland could have an outage since there are many more devices between the two points, but at the same time, there are also many more routes to take (dynamic routing).

So maybe it’s OK to have your primary site “Azure and Mongo” both be in Virginia since you are too. In this case, you could use Joyent (Virginia Datacenter) for the Mongo Cloud Provider to separate from Azure hosting both website and database, but keep it in Virginia which is close to where this figurative person lives. In this scenario, you might want to have your backup site be in a different “non US East” region, and maybe even a far-away Mongo location because you are not as concerned about performance of the “backup site” – you will only use it if there is an outage associated with your primary site.

With this simple scenario, the probability of individual failure points and scenarios becomes complex. If you have your primary site solely with Azure within the same location, then a single failure in either Azure hosting or MongoLab hosting by Azure could take out your primary website. But if you maintain solely Azure for your primary website/database, but in different regions, then a failure in either region could out the entire primary site (they both need to work). But now you have the backup site, which will have real-time concurrent data from the uploader. If the backup site does not share same Service Provider & Locations, then you are creating redundancy against Cloud Provider and geographic location disruptions.

And there are many hybrid scenarios such as having two websites, each in a different region, pointing to the same database. In this method, you are assuming the database will not fail, and if a website fails, you would have to reconfigure the “uploader” for the new REST API destination.

Finally, as mentioned earlier, there is an “advanced” option which introduces a completely separate Cloud Provider – Heroku. Heroku is more technically focused, is a completely separate process, is not documented with videos or step-by-step instructions, and is not the “community support” standard. It is still free, but just like Azure, you can do things that would become “charges.” Basic Heroku information is supported at this time, but it is truly “do-it-yourself.” The benefit of Heroku is that it is not Windows Azure, so the website is totally independent of Azure. Additionally, Heroku can use your existing Mongo database (possibly hosted by Azure, possibly not), or within Heroku, you can create your own “mongodb” in order to tightly couple the Heroku website (app) and add-on (mongo database).

 

Before proceeding, please use the following table to plan your redundancy:

(For example)

Provider Cloud Location
Website 1 Windows Azure Windows Azure West US
Mongo 1 MongoLab Joyent Las Vegas, NV (us-sw-1)
Website 2 Windows Azure Windows Azure East US 2
Mongo 2 MongoLab Amazon Web Services US East

 

(Use this table for your information….)

Provider Cloud Location

 

 


 

Step 3: Implement the Redundancy

The following steps assume just one additional website and database. But you may have multiple websites and databases if so desired.

 

  1. Create the new Mongo deployment(s)

redundancy 3-1

Generally speaking, the easiest method to creating a new mongo deployment is to copy it, or “clone existing” database. (See figure)

 

Click “Clone existing,” and it will bring you to a page where you will:

  • Choose / Select your existing database that you want to copy.
  • Choose Cloud Provider – follow your plan for redundancy
  • Choose Location – follow your plan for redundancy
  • Choose Plan – Ensure you have selected a Cloud Provider / Location combination that will provide you with the “single node” Sandbox option.
  • Enter new database name at the bottom

See figure below.

redundancy 3-2

You will now have a new database, already populated with the “data” from your original site. All the collections, all the data, including the “database username” and “database password” will be copied identically – so you don’t have to remember separate configurations if you do not want to. However, the “mongo connection string” will be different because the PORT and SERVER_NAME will be different.

 

  1. Create the new Azure website(s)

To create your new Azure website, it’s the same high-level processes as your first setup, but there is an important early step which is lost on many during the initial setup – creating a new Web Hosting Plan if you are to use a different geographic region.

  • Log into Azure, and choose New | Website | Quick Create. Enter the URL for your new site… and give it something meaningful so you will remember that this is the “backup site.”

redundancy 3-2-1

 

  • Create new Web Hosting Plan

Note: You will only do this step if you want to create a website in a different geographic location.

Click the drop-down arrow for “Web Hosting Plan” and choose “Create new web hosting plan.” For most people, you will have one and only one listing under your “Subscription: Pay-As-You-Go.” This is STILL free… it will not cost you additional money. (See the end for a discussion on this)

redundancy 3-2-12

 

Pick a new region, for example, Central US, and choose “Create Website”

redundancy 3-2-3

 

  • Azure will create your empty Website, in the new region, and return you to your listing of websites. (see figure below)

redundancy 3-2-4

  1. Configure the new Azure Website for your new Mongo database

This article will not detail step-by-step how to configure the website, but will provide the key tasks to ensure you have it covered:

  • Setup Source Code Deployment from GitHub to the new website;
  • Deploy “cgm-remote-monitor” from your GitHub fork (repository).
  • Configure the connection strings
    • mongo                                                mongodb://dbuser:dbpass@server:port/database
    • mongo_collection                          [Same as your primary site collection name]
    • PUSHOVER_USER_KEY         [refer to PUSHOVER setup if you want it.]
    • PUSHOVER_API_TOKEN      [refer to PUSHOVER setup if you want it.]
    • ENABLE                                             careportal
    • API_SECRET                                  [passphrase-greater-than-12-characters]

Just like your primary website, your backup website uses the “mongo database client” to connect to the Mongo database. However, the Android Uploader is sending data to the website using the “REST API,” and the critical piece of that is the “API_SECRET.”

Before proceeding, a little more information about the API_SECRET. This is simply a special passphrase that you define for each website. The API_SECRET can be different between websites, or they can be the same – it is up to you. One website knows nothing about the other. The API_SECRET is used as a “trust token” or simple “authentication / authorization” combined. What happens is that you will enter this information into the Uploader “REST API” configuration, so the uploader will send that information to your Website. The Website wants to ensure the data it just received is appropriate — meaning, coming from you, and not from some other entity. Because the Uploader and the Website both have the API_SECRET, it is the method to ensure “trust.”

 

  1. Modify the Android Uploader

So you now have a primary website/database, a backup website/database and your Android Uploader which uses the REST API to upload BG data to the primary website. To add redundancy to the uploader, you will add the “new website” REST API upload information to the same “REST API” configuration area in the Android Uploader preferences, but you will do so by adding a “space” after the current REST API information, and then entering the 2nd REST API information. This is called “space separated” multi-values (array). It looks like this:

 

Before change:

API_SECRET_ForSite1@https://SITE1.azurewebsites.net/api/v1

 

After change:

API_SECRET_SITE1@https://SITE1.azurewebsites.net/api/v1 API_SECRET_SITE2@https://SITE2.azurewebsites.net/api/v1

Note: There is a single space after “…api/v1” and before “API_SECRET2…” That is the only “space” in the entire line. On your uploader, it may “line wrap” onto the next line as well. And remember, as described in the previous step, the API_SECRET for Site 1 may be the same as Site 2, or it may be different — it is entirely up to you.

Save the information and you’re done. Restart your Android Uploader to ensure the new preferences are loaded.

 

  1. Pebble Watch

If you have a pebble watch, you have some options:

  • If the primary site goes down, just change the “Pebble Data Endpoint URL” to your “backup website” name; (e.g., https://backupsite.azurewebsites.net/pebble)
  • Have a 2nd watchface with a different AppID which is available as “Slot 2” from the Pebble Store, or you can deploy from CloudPebble.net.

Note: This Lab does not cover have to use CloudPebble. If you know how to do CloudPebble and you know about GUIDs and App IDs, then you’ll be fine. Otherwise, use the Pebble Store or Labs website download for the “slot 2” Nightscout Watchface.

 


 

Epilogue:  Azure, Redundancy and Free

For the majority of people, the Azure “free” account daily bandwidth limitation is not a problem. This Lab does not address the subset of users who do have a bandwidth limitation.

The implementation of redundancy in your Azure account should not introduce additional fees because the Azure bandwidth measurement is on “data out;” the information sent to your browser. As such, if you are not viewing the “backup site,” it should not be generating traffic against your quota (subject to people you give the URL to, possible search engines, etc.) You should review your Azure monthly traffic analysis to understand your current “usage” with just the primary site, and then monitor for any abnormal growth as a direct result of the redundancy.