Monitoring Locations Update

The current locations are VPS servers from DigitalOcean and Linode. I like to mix it a bit, so that we don’t rely on a single provider, but not too much, since the administrative overhead of managing dozens of providers is another lesson learned in 10 years of uptime monitoring.

This is our current monitoring network:

We are currently testing three new VPS providers:

• RackNerd
• OneProvider
• RansomIT

These allow us to deploy monitoring nodes in many locations at the same time, scale them if necessary, and their pricing is good.

Therefore we’ve deployed the following 22 new locations:

1. Chicago, IL (USA)
2. Ashburn, VA (USA)
3. Dallas, TX (USA)
4. Los Angeles, CA (USA)
5. New York, NY (USA)
6. San Jose, CA (USA)
7. Seattle, WA (USA)
8. Sao Paulo, Brazil
9. London, United Kingdom
10. Copenhagen, Denmark
11. Amsterdam, Netherlands
12. Madrid, Spain
13. Paris, France
14. Dusseldorf, Germany
15. Bursa, Turkey
16. Dubai, AE
17. Singapore
18. Hong Kong
19. Tokyo, Japan
20. Sydney, Australia
21. Adelaide, Australia
22. Auckland, New Zealand

We will be using these locations in our testing environment for a few days until we enable them in production.

Rollout procedure

The process of rolling them out is as follows, each step is a few days apart.

Step 1. Testing the new monitoring network.

This involves monitoring the nodes using our production environment (soon to be old nodes) and monitoring our production infrastructure using the new nodes. A few days should be more than enough.

Step 2. Enabling all the new nodes in production

This is when the monitoring network expands for all users. The new locations above will show up in the Monitoring Network list on the Help screen within the app. They will show up in the recent checks boxes, and in the custom location selection drop-down within the Edit Monitor screen.

We don’t need to announce the new IPs to the users just yet because only the customers that have custom node locations selected will be impacted by the update and since the old locations are still running, there’s nothing they need to do for the service to work normally.

Step 3. Announcing the retirement of the old nodes

After a week went by and all the new nodes are performing correctly in production, it’s time to announce which of the old monitoring nodes will be retired. And when that is scheduled. This is useful especially for users that have set custom locations that would be retired soon so that they can change it to another location.

Step 4. Removing old nodes from the monitoring network.

The old nodes will stop being available. At this moment the key is to ensure that none of our customers have the old nodes selected, as they will stop monitoring when this happens. I envision there will be some that will need to be manually notified. We are all busy, and human, which means a kind reminder goes much further than a “we told you so”.

Node application

Each location runs a Monitive Wopher application, which is a small, custom application that does the actual checks. It’s written in Go as we need blazing fast performance. And PHP is not the proper platform to handle that.

Each node can check thousands of sites and service every minute with just under 100 MB of RAM and 1-2 CPUs. Compared to what we had in Monitive classic, the Wopher app can monitor all our customers’ websites from just one small VPS.

I’ll tell you more about the Wopher app in another post.

Final note on latency

Global response times for all monitored services might slightly change after this update, since the monitoring network has changed, making some requests faster and others slower.

For example, monitoring a UK website might show an increase in latency since there are more monitoring locations in the Pacific than before.

I don’t know how we can go around this. I believe announcing and acknowledging this possibility is the best approach.

Hosting in the Pacific?

And last, but not least, I wish to thank Oliver Ransom from RansomIT for providing the monitoring nodes in Adelaide, Auckland, and Hong Kong.

If you need hosting in the Pacific, definitely check out RansomIT, as they can provide high quality services in 9 locations all around Australia and East Asia.