An interesting facet of the development and systems administration business is the number of 80% projects that build up over time. An 80% project is that awesome library, script, rewrite, new system, or what have you that’s cooling on your back burner. It’s almost done but it’s missing the finishing touches. Maybe it needs a few code tweaks. Maybe it needs a little more documentation. Maybe you’re still finalizing settings and playing with patches. Don’t lie; we know you’ve got these projects hanging around. I’ve got a list of 80% projects as long as my arm.

It’s time to check something off my 80% projects list. I’ve finally finished documenting and am happy to release the SoftLayer API Perl client library! This module will make Perl API hackers’ lives a whole lot easier. Previously you had to build SOAP API calls manually using the SOAP::Lite module and parse the response into something easier to handle. Now you can accomplish the same thing with a series of easy to use helper methods. Functionality is very similar to our PHP client but with a Perl twist. For instance, you can do clever one liners!

use SoftLayer::API::SOAP;

# Grab my account information.
my $account = SoftLayer::API::SOAP->new(
    'SoftLayer_Account',
    undef,
    'my API username',
    'my API key')->getObject();

Check out our README for many and more comprehensive examples. Download the library from our github page at:

 

http://github.com/softlayer/softlayer-api-perl-client

 

As with all of our projects we’re very open to feedback, so please comment or post on our forums and let us know what you think. I can’t wait knock a few more 80% projects off the ol’ list. You’re going to love them. See y’all next time!

This is my second year attending SCaLE. I went last year as a part of an open source project I was working with and had a pretty good time. I jumped at the chance to go again when I found SoftLayer had a booth at and was sponsoring SCaLE 8x.

I’m really glad I went; I dig these community-focused events. What strikes me about these events, and SCaLE in particular, was the general optimism of the staff, attendees, speakers, and exhibitors. This show is on a weekend, and people came because they genuinely care about FOSS, cool new (and sometimes old) tech, and community advocacy. Most attendees fit the classic nerd stereotype. We got to meet our fair share of skinny pale guys, scruffy and ponytailed sysadmin types, and bearded folks with wallet chains and Star Trek t-shirts.

When not working our booth I was able to duck out and attend some of the talks at the expo. Aside from being informative (I picked up a lot about technical writing that I can’t wait to throw into our API documentation), every single talk I attended was completely packed to the point where there was standing room only. It’s refreshing to see speakers talk about things other than the company they work for. Instead, they focused on cool tech or how to be more productive in your community or with a specific technology. I took pages and pages of notes this weekend, it felt like being in school again.

I think SoftLayer did well at SCaLE 8x. We got our name out like we usually do at these events, and got to introduce a new group of people to SoftLayer and our ever-changing and sometimes hidden industry. I hope we’re sponsoring SCaLE 9x. Even if we don’t you’ll still find me there. See you next year!

Object Masks, Filters, and Other V3 Black Magic

Everyone has heard the age old saying for any given job you need to have the right tool. Just as most of us have tried to use the flat rounded edge of a butter knife a time or two when what we desperately needed was a screw driver. Does that mean you weren’t able to open that little compartment on whatever gizmo and replace the batteries? Probably not. In most cases it is possible to use a butter knife when a screw driver is the tool of choice; it’s just more painful and a lot less effective.

The same can said of the SoftLayer API (SLAPI). It’s a toolset. A very flexible set of tools allowing a developer to manage every aspect of dedicated hosting from accounting and billing to physical status of remote hardware. And yet there are so many tools in the V3 API toolbox, a number of them only subtlety different from their binary brethren ( at least on the surface), it’s tempting just to reach your hand into the bag, find the first thing that resembles a screw driver, and begin turning.

I know. I’m speaking from my own experiences. As a developer who largely works on SoftLayer’s back end systems, somewhere between the bottom of TCP/IP stack and the top edge of the kernel, recently getting to do production user portal code was a new experience for me. Sure I wrote some demos, dabbled a little here and there, but when I started doing my first “real” V3/SLAPI intensive project I realized my prior attempts had entirely missed the true power and elegance of SLAPI. The magic if you will. A little something called ORM.

Those of you who spend your days toiling in the world or relational databases are probably fairly familiar with the term ORM. But for someone like me who usually comes no closer to a database than reading an I/O address from the Windows registry, I was only vaguely aware of what the acronym even stood for. I turned to Webopedia. There I discovered the following. “Short for object role modeling, ORM is a conceptual database design methodology that allows the user to express information as an object and explore how it relates to other information objects”.

So there we have it. Database. Objects. Relations. I learn hands on—so none of that amounts to a hill of beans without some real code I can see and type and run for myself. So rather than regurgitate the SLDN documentation, I will just share a simple yet real life example. Then, in the second part of the article, we can expand that example to show some of the more powerful and less documented features of the V3 SLAPI.

The code that follows is written in Microsoft C Sharp using Visual Studio 2008 Professional Edition. I am not going to step through the basics of connecting a WSDL and generating a SOAP wrapper in this article. If you need help with that, there is an SLDN blog I did a while back which covers those steps entitled, “Dot Net? You Bet!”. It is still available under the implementations section of the SLDN website. True to my MO, I am not a big GUI guy so the code I am presenting runs as a Windows console application.

For the sake of making the example clear, I am going to simplify my task. In the example in both this article, and the next in the series, we will be playing the role of a developer who needs to count how many of his or her servers are running the Microsoft Windows operating system, as opposed to one of the many Linux variants SoftLayer also offers its customers. For our first code sample, we require two WSDLs: the SoftLayer_Account service as well as the SoftLayer_Hardware_Server service. The console program below will get us connected to the SoftLayer application servers, as well as provide us some timing metrics.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace SLDN_Magic
{
   class Program
   {
       static void Main(string[] args)
       {
           //global timing vars
           DateTime stopwatch;
           TimeSpan elapsed;

           //replace with your username and api key
           string user_name = "Replace With Your User Name";
           string api_key = "Replace With Your API Key";

           Console.Write("Establishing connection to SLDN service...");

           //time it
           stopwatch = DateTime.Now;
           //declare the services
           SLDN_ACCT.SoftLayer_AccountService acct = new
               SLDN_ACCT.SoftLayer_AccountService();
           SLDN_SVR.SoftLayer_Hardware_ServerService svr = new
               SLDN_SVR.SoftLayer_Hardware_ServerService(); 

           //create an authentification object for each
           SLDN_ACCT.authenticate credentials_a = new
               SLDN_ACCT.authenticate();
           SLDN_SVR.authenticate credentials_b = new
               SLDN_SVR.authenticate();

           //assign credentials
           credentials_a.username = credentials_b.username = user_name;
           credentials_a.apiKey = credentials_b.apiKey = api_key;

           //authenticate
           acct.authenticateValue = credentials_a;
           svr.authenticateValue = credentials_b;

           elapsed = DateTime.Now.Subtract(stopwatch);
           Console.WriteLine("done (" +  elapsed.TotalSeconds.ToString() + " seconds)");

           Console.WriteLine("\nPress  to exit.");
           Console.ReadLine();
       }
   }
}

At this point, we can go ahead and run our code. It doesn’t really do anything all that useful. But never the less you should get an output similar to this.

While writing this article I connected to the SoftLayer API servers numerous times from my home. My connection times were pretty consistent. It took somewhere in the neighborhood of 20 seconds to get everything set up. That seems like a lot. But keep in mind that you only incur the overhead of connecting your services one time. Plus as we get a little further along in this article I will show you how we can use ORM to get rid of one of the references entirely. For now though, let’s move on.

For someone of my background and mindset, what seemed the most straight-forward and correct way to find out which servers were running MS Windows was to access the public method “isWindowsServer()“. This is a method off the server class. That’s why we needed to import the SoftLayer_Hardware_Server service. But we don’t want to check the OS on a single server. We want to recurse through all the servers for an account. Which is why we brought in the SoftLayer_Account service and its attractive public offering “getAllHardware()“.

Keeping this plan in mind, let’s go ahead and implement it in our console application.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace SLDN_Magic
{
   class Program
   {
       static void Main(string[] args)
       {
           //global timing vars
           DateTime stopwatch;
           TimeSpan elapsed;

           //global container
           SLDN_ACCT.SoftLayer_Hardware[] hw;

           //replace with your username and api key
           string user_name = "Replace With Your User Name";
           string api_key = "Replace With Your API Key";

           Console.Write("Establishing connection to SLDN service...");

           //time it
           stopwatch = DateTime.Now;

           //declare the services
           SLDN_ACCT.SoftLayer_AccountService acct = new
               SLDN_ACCT.SoftLayer_AccountService();
           SLDN_SVR.SoftLayer_Hardware_ServerService svr = new
               SLDN_SVR.SoftLayer_Hardware_ServerService(); 

           //create an authentification object for each
           SLDN_ACCT.authenticate credentials_a = new
               SLDN_ACCT.authenticate();
           SLDN_SVR.authenticate credentials_b = new
               SLDN_SVR.authenticate();

           //assign credentials
           credentials_a.username = credentials_b.username = user_name;
           credentials_a.apiKey = credentials_b.apiKey = api_key;

           //authenticate
           acct.authenticateValue = credentials_a;
           svr.authenticateValue = credentials_b;

           elapsed = DateTime.Now.Subtract(stopwatch);
           Console.WriteLine("done (" + elapsed.TotalSeconds.ToString() + " seconds)");

           //butter knife method

           Console.Write("Retrieving hardware using method 1...");

           //get time stamp
           stopwatch = DateTime.Now;

           hw = null;
           try
           {
               hw = acct.getHardware();
           }
           catch (Exception e)
           {
               Console.WriteLine("Exception encountered [" + e.Message + "]");
               hw = null;
           }

           int cnt = 0;

           foreach (SLDN_ACCT.SoftLayer_Hardware server in hw)
           {
               try
               {
                   SLDN_SVR.SoftLayer_Hardware_ServerInitParameters box = new
                       SLDN_SVR.SoftLayer_Hardware_ServerInitParameters();
                   box.id = (int)server.id;
                   svr.SoftLayer_Hardware_ServerInitParametersValue = box;
                   if (svr.isWindowsServer())
                   {
                       cnt++;
                   }
               }
               catch (NullReferenceException)
               {
                   //ignore...this server has not had the
                   //OS loaded on it yet!
               }
           }

           elapsed = DateTime.Now.Subtract(stopwatch);

           Console.WriteLine("done (" + elapsed.TotalSeconds.ToString() + " seconds)");
           Console.WriteLine("counted " + cnt.ToString() + " MS Windows licenses");
           Console.WriteLine("\nPress  to exit.");
           Console.ReadLine();
       }
   }
}

That’s it. Pretty straight forward stuff possibly with the exception of the way SLAPI allows you to reinitialize the server (or any object) on the fly by use of:”SoftLayer_Hardware_ServerInitParameters”. If this confuses you, again I’ll refer you to the blog “Dot Net? You Bet!”. At this point, I think we are ready for another test run.

Once again we find ourselves in the twenty second range both for connecting the services and counting the hardware. What you can’t tell from looking at this output is that the account I was using for testing had about 100 servers on it. So basically we are talking 1/5 of a second per server. It’s certainly doable with a handful of servers, but this would obviously never work if you were trying to present this information real time to users if you managed 500 or 5,000 or 50,000 servers. So you are probably asking yourself the same thing I did. What gives? If SoftLayer wishes its customers success on an enterprise level, why create an API that comes to its knees when you start trying to manage more than a few hundred servers?

Luckily, the architects of SLAPI were a lot more web / database savvy than me. The above implementation, while correct syntactically, is a gross misuse of the SLAPI. It’s the butter knife, when what we really need is the screwdriver. What we need are object masks. But exactly what are object masks and how do they relate to ORM?

The best way I have found to understand ORM and object masks, is to think of the SLAPI data objects, as a self supporting entities. Each object provides its own methods and exposes some properties specific to that object. Yet thanks to ORM, most objects can actually get to properties in related objects, through a process called tapping. You simply tap each object down the chain until you find the property or properties you are interested in, prior to retrieving an instance or instances of the object. Then the set of objects returned will expose any relevant properties in the same manner you tapped them.

For example in our case the SLDN architecture relates a server to an operating system in the following manner.

The diagram shows us that essentially, as long as we can get a hardware object, we can tap all the way down to the software description — which as the SLDN documentation states has a “name” property. There by we eliminate two of our most time consuming tasks from our original application. First off we no longer need to instantiate the SoftLayer_Hardware_Server service, since we can get to server from hardware and hardware can be retrieved via the account class. Secondly, if when we return the hardware it already contains the name of the operating system, we no longer have a need to call the “isWindows()” method. Take a look.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace SLDN_Magic
{
   class Program
   {
       static void Main(string[] args)
       {
           //global timing vars
           DateTime stopwatch;
           TimeSpan elapsed;

           //global container
           SLDN_ACCT.SoftLayer_Hardware[] hw;

           //replace with your username and api key
           string user_name = "Your User Name Here";
           string api_key = "Your Api Key Here";

           Console.Write("Establishing connection to SLDN service...");

           //time it
           stopwatch = DateTime.Now;

           //declare the services
           SLDN_ACCT.SoftLayer_AccountService acct = new
               SLDN_ACCT.SoftLayer_AccountService();

           //create an authentification object
           SLDN_ACCT.authenticate credentials_a = new
               SLDN_ACCT.authenticate();

           //assign credentials
           credentials_a.username = user_name;
           credentials_a.apiKey = api_key;

           //authenticate
           acct.authenticateValue = credentials_a;

           elapsed = DateTime.Now.Subtract(stopwatch);
           Console.WriteLine("done (" + elapsed.TotalSeconds.ToString() + " seconds)");

           //method 2
           Console.Write("Retrieving hardware using method 2...");

           //get time stamp
           stopwatch = DateTime.Now;

           hw = null;
           //attempt to pull a hardware list for this user
           try
           {
               acct.SoftLayer_AccountObjectMaskValue = new
                   SLDN_ACCT.SoftLayer_AccountObjectMask();
               acct.SoftLayer_AccountObjectMaskValue.mask = new
                   SLDN_ACCT.SoftLayer_Account();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware = new
                   SLDN_ACCT.SoftLayer_Hardware_Server[1];
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0] = new
                   SLDN_ACCT.SoftLayer_Hardware_Server();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0].operatingSystem = new
                   SLDN_ACCT.SoftLayer_Software_Component();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0].operatingSystem.softwareLicense = new
                   SLDN_ACCT.SoftLayer_Software_License();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0].operatingSystem.softwareLicense.softwareDescription = new
                   SLDN_ACCT.SoftLayer_Software_Description();
               hw = acct.getHardware();
           }
           catch (Exception e)
           {
               Console.WriteLine("Exception encountered [" + e.Message + "]");
               hw = null;
           }

           cnt = 0;

           foreach (SLDN_ACCT.SoftLayer_Hardware server in hw)
           {
               try
               {
                   if (server.operatingSystem.softwareLicense.
                                   softwareDescription.name.ToLower().
                                   Contains("windows"))
                   {
                       cnt++;
                   }
               }
               catch (NullReferenceException)
               {
                   //ignore...this server has not
                   //had the OS loaded on it yet!!!
               }
           }

           elapsed = DateTime.Now.Subtract(stopwatch);

           Console.WriteLine("done ("+ elapsed.TotalSeconds.ToString()+" seconds)");
           Console.WriteLine("counted " + cnt.ToString() + " MS Windows licenses");

           Console.WriteLine("\nPress  to exit.");
           Console.ReadLine();
       }
   }
}

You should notice right away all the references to SoftLayer_AccountObjectMaskValue prior to calling the “getHardware()” method. This is the object mask. Essentially we must create a new instance of each entity we want to include down the chain. Then when the target object is retrieved, in our case the hardware, all related objects which we have made room for will get created for that specific instance of hardware, assuming of course a record exists. You must instantiate each object down the chain. If you skip any link your result set will not conatain the property or method you were trying to get to. Some less structured langagues, like PHP, do not have this requirement. But with V3 and dot NET there is no getting around it. You’re probably thinking this version of the code looks far more cluttered and is not as straight-forward to read. You’re right. But I contend this is the electric screwdriver in the SLDN toolbox. See for yourself.

As you can see the connection overhead dropped in half, which is to be expected since we are only authenticating to half the number of services. But take a look at the second number, the number of seconds it takes to count the servers with Windows installed. It dropped from 20 seconds, to under 2 seconds. That’s a 10 times speed gain. And wait there’s more– because we are only making one call to the SLDN application servers to retrieve those records what you see is what you get. Meaning you should not expect that time to increase noticeably whether you have a hundred servers or a hundred thousand servers! That my friend is the magic of V3. The power of ORM.

Following this article I will include the entire code base, in a combined application that lets you run the tests sequentially so you can see the amazing difference object masks make for yourself. In the second part to this article, we’ll continue with the sample so if you download it keep it handy. In part two I’ll discuss the next best thing to object masks—object filters. With object filters we’ll be able to streamline this code even more. Until then…happy SLDNing!

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace SLDN_Magic
{
   class Program
   {
       static void Main(string[] args)
       {
           //global timing vars
           DateTime stopwatch;
           TimeSpan elapsed;

           //global container
           SLDN_ACCT.SoftLayer_Hardware[] hw;

           //replace with your username and api key
           string user_name = "";
           string api_key = "";

           Console.Write("Establishing connection to SLDN service...");

           //time it
           stopwatch = DateTime.Now;

           //declare the services
           SLDN_ACCT.SoftLayer_AccountService acct = new SLDN_ACCT.SoftLayer_AccountService();
           SLDN_SVR.SoftLayer_Hardware_ServerService svr = new SLDN_SVR.SoftLayer_Hardware_ServerService(); 

           //create an authentification object for each
           SLDN_ACCT.authenticate credentials_a = new SLDN_ACCT.authenticate();
           SLDN_SVR.authenticate credentials_b = new SLDN_SVR.authenticate();

           //assign credentials
           credentials_a.username = credentials_b.username = user_name;
           credentials_a.apiKey = credentials_b.apiKey = api_key;

           //authenticate
           acct.authenticateValue = credentials_a;
           svr.authenticateValue = credentials_b;

           elapsed = DateTime.Now.Subtract(stopwatch);
           Console.WriteLine("done (" + elapsed.TotalSeconds.ToString() + " seconds)");

           //method 1

           Console.Write("Retrieving hardware using method 1...");

           //get time stamp
           stopwatch = DateTime.Now;

           hw = null;
           try
           {
               hw = acct.getHardware();
           }
           catch (Exception e)
           {
               Console.WriteLine("Exception encountered [" + e.Message + "]");
               hw = null;
           }

           int cnt = 0;

           foreach (SLDN_ACCT.SoftLayer_Hardware server in hw)
           {
               try
               {
                   SLDN_SVR.SoftLayer_Hardware_ServerInitParameters box = new SLDN_SVR.SoftLayer_Hardware_ServerInitParameters();
                   box.id = (int)server.id;
                   svr.SoftLayer_Hardware_ServerInitParametersValue = box;
                   if (svr.isWindowsServer())
                   {
                       cnt++;
                   }
               }
               catch (NullReferenceException)
               {
                   //ignore...this server has not had the OS loaded on it yet
               }
           }

           elapsed = DateTime.Now.Subtract(stopwatch);

           Console.WriteLine("done (" + elapsed.TotalSeconds.ToString() + " seconds)");
           Console.WriteLine("counted " + cnt.ToString() + " MS Windows licenses");

           //method 2
           Console.Write("Retrieving hardware using method 2...");

           //get time stamp
           stopwatch = DateTime.Now;

           hw = null;
           //attempt to pull a hardware list for this user
           try
           {
               acct.SoftLayer_AccountObjectMaskValue = new SLDN_ACCT.SoftLayer_AccountObjectMask();
               acct.SoftLayer_AccountObjectMaskValue.mask = new SLDN_ACCT.SoftLayer_Account();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware = new SLDN_ACCT.SoftLayer_Hardware_Server[1];
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0] = new SLDN_ACCT.SoftLayer_Hardware_Server();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0].operatingSystem = new SLDN_ACCT.SoftLayer_Software_Component();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0].operatingSystem.softwareLicense = new SLDN_ACCT.SoftLayer_Software_License();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0].operatingSystem.softwareLicense.softwareDescription = new SLDN_ACCT.SoftLayer_Software_Description();
               hw = acct.getHardware();
           }
           catch (Exception e)
           {
               Console.WriteLine("Exception encountered [" + e.Message + "]");
               hw = null;
           }

           cnt = 0;

           foreach (SLDN_ACCT.SoftLayer_Hardware server in hw)
           {
               try
               {
                   if (server.operatingSystem.softwareLicense.softwareDescription.name.ToLower().Contains("windows"))
                   {
                       cnt++;
                   }
               }
               catch (NullReferenceException)
               {
                   //ignore...this server has not had the OS loaded on it yet
               }
           }

           elapsed = DateTime.Now.Subtract(stopwatch);

           Console.WriteLine("done ("+elapsed.TotalSeconds.ToString()+" seconds)");
           Console.WriteLine("counted " + cnt.ToString() + " MS Windows licenses");

           Console.WriteLine("\nPress  to exit.");
           Console.ReadLine();
       }
   }
}

SoftLayer has 20,000+ servers. Each of those servers is on 2 networks, the public network open to the Internet, and the private network that only our customers can use. Each of those networks exists on a 3 to 4-level network hierarchy. This gives us more than 50,000 switch ports, but we’ll use 50,000 to make the math easier. Each switch port has bandwidth in and bandwidth out, as well as packets in and packets out. That gives us 200,000 data points to poll. Bandwidth is polled every 5 minutes, giving us 57,600,000 data points per day, or 1,728,000,000 per month. Given that bandwidth data points are all 64 bit numbers, and we also have to track the server ID (32 bit), the network (32 bit), and the datetime (32 bit), that makes a month’s worth of raw data (excluding any overhead from storage engines) 34.56GB. Now, the data is to be stored redundantly, so double that. Also, we have to track bandwidth for racks, virtual racks, and private racks, so add another 50% onto the data. That gives us around 90GB per month of data.

This doesn’t seem like a lot of data at first, but we need to generate custom bandwidth graphs for use on the web. Since it’s on the web, loading times above 2 seconds are unacceptable. Also, these are not enormous files with small keys (we’ll spend more time on that later) so 45GB of bandwidth data is a whole lot different than 45GB of movie files or MP3s.

To accomplish this, we decided that we needed a data warehouse. After numerous false starts and blind alleys, we decided to make our own system from scratch using MySQL. We considered commercial products, and pre-built open source solutions, but they just didn’t seem to fit our needs properly. The data warehouse project commenced with phase 1.

Phase 1: MySQL Cluster with Read Shards, Large Tables
Our first implementation was relatively simple. We planned to have a MySQL cluster for writing all the data, with the data split into 100 tables. The ID of the hardware mod 100 would determine the table name that we would write to. Then we’d have between 5 and 20 read databases, each replicating a different table, for load reasons. Then all we have to do is index the data properly, and add code to our applications to pull data from the correct read database node, and we’ll be fine.

Bad news: MySQL cluster isn’t designed for data with huge numbers of large keys. MySQL cluster stores all indexes and keys in memory on the cluster controller multiple times. As mentioned before, our data had 12 bytes of key, and 8 bytes of data. This means that we could only get about 2 million rows into the data warehouse before MySQL would lock up and quit working with the ever helpful error message “Unknown error: 157.” Even more disturbing, deleting items from MySQL cluster didn’t free up memory, as the indexes had to be rebuilt before that would happen. We upgraded everything to MySQL 5.1.6, but per the MySQL manual, “Beginning with MySQL 5.1.6, it is possible to store the non-indexed columns of NDB tables on disk, rather than in RAM as with previous versions of MySQL Cluster.” Unfortunately, it’s the indexes that are causing us problems, so the upgrade didn’t help.

Phase 2: MySQL Cluster with Read Shards, Small Tables
Since MySQL cluster couldn’t handle a small number of really big tables, how about a really big number of small tables? We tried making hundreds of tables per day, and removing the indexes on tables older than a certain time, but that quickly ran up against operating system limitations on the number of files in a folder. When we switched operating systems to alleviate this problem, we ran into MySQL’s limitation of roughly 21,000 tables per database. Nothing could get past that, so we had to move away from the cluster entirely.

Phase 3: Large MySQL box with Read Shards, Large Tables
We then moved on to one single MySQL box with enormous hard drives and multiple read shards. This looked promising at first, but the box simply couldn’t handle the amount of inserts and updates, and the slave servers were locking up too often. We thought that if we partitioned the tables, MySQL could handle the inserts better. This, naturally, broke replication in a different way. This was mainly because we had too many tables now that each partition was getting its own file, so MySQL would be constantly opening all these table files to perform the updates, and would quickly run out of memory and begin to swap. We just had to decrease the number of tables per server. We decided to abandon the centralized master server idea, and built out 5 pairs of master/slave servers.

Phase 4: 5 Pairs of Master/Slave Servers, Large Grouped Tables
This plan really seemed like it was going to work. We actually had a working data warehouse for almost 2 weeks without any errors. We were close to breaking out the champagne, but there was one feature we still hadn’t implemented.

The time came to add the tracking of bandwidth for virtual dedicated racks as well. To accomplish this, we changed the MySQL INSERT statement to INSERT... ON DUPLICATE KEY UPDATE. For those that don’t immediately recognize the terminology, the ON DUPLICATE KEY UPDATE syntax is designed so that if a particular database key already exists in the database, simply change the INSERT statement into an UPDATE statement. The UPDATE portion is defined at the end of the clause. Since the key to our data is the large “object ID, date time, data type” combination, adding the duplicate syntax allowed us to issue multiple INSERT statements for the same data, and have it continually update with new values. This was especially handy for virtual racks with thousands of servers.

MySQL threw us another curve ball when we tried this. There was a known issue with the ON DUPLICATE UPDATE syntax and replication using binary logs. Namely, the UPDATE would remain an INSERT in the log somehow, so we’d get duplicate key errors thrown on the slave, when the master was still working fine. Each time this happened, it would require stopping both servers, re-synching the tables, clearing the logs, restarting replication, and restarting the data transfer processes. This was unacceptable, so we had to move once more.

Phase 5: 10 Individual Machines
Since we already had 10 identical database machines, we decided to make them all independent and ignorant of each other. We stayed with the item group theory, leaving multiple bandwidth items per table. With 25 different items per table, we were only going to have 1,000,000 rows per table per month, which still maintains our speed. However, the index size problem bit us again. These tables were simply too large to have 3/4 of their columns indexed.

Finally we thought we had the answer. We would split up each of these tables by month. That way, each table would max out at 1,000,000 rows, so the indexes wouldn’t be unmanageable. Remember that we’ve already split the connection points from one, to five, to ten. We’ve also broken the tables up from one large table, to one per object, and now one per group of objects. In order to cut down on the complexity of the application layer code, we decided to use merge tables to keep the table name consistent. That way we can simply select from “the table” rather than “the table from March.” When we did this, we almost immediately began getting mysterious deadlocks on the servers. The INSERT statements would conflict with any SELECT statements using the same table, and the two threads would just hang indefinitely. Oddly, the table was never actually locked, it seemed that there was some sort of “offsetting lock” happening, where both threads were deadlocked in a race condition. The table could still be used, but the application we had inserting the data would be hung, so it was causing unacceptable delays in data inserts.

Phase 6: 10 Individual Machines, Small Individual Tables
Finally we decided that enough was enough, we were going to roll our own scaled storage solution. Clustering didn’t work, throwing hardware at the problem unfortunately didn’t work, and even the fancy new features for merged tables and partitioned tables didn’t work. We simply decided that we were going to keep the old “grouped tables” model, as well as creating a new table for every month without merge tables. This way, we keep the number of tables relatively low, and the number of rows per table low. Plus, as a bonus, by controlling the table names ourselves we could ensure that MySQL wouldn’t open too many files. All inserts went into this month’s table, and all reads would come out of whatever specific month they needed. A cron job was set up to periodically issue FLUSH TABLES on all 10 data warehouse nodes, and we had our completed product! It’s been months now since we had a major database failure, we can generate any bandwidth graphs you want in less than a second, and other developers are starting to create their own table groups.

Each object’s data resides completely on two of our data warehouse nodes, so the data is redundantly stored in the event of a node going down. The application code has been written with extensive factory patterns and ORM so that all a developer has to do is create a new group type, a new data tracking object, and add data to it. The code automatically selects the applicable nodes to write the data to, creates the tables if they don’t exist, and writes the data to the tables. Similarly, a “get data” command will randomly select one location for the data, and retrieve the proper data, using only the tables that are necessary.

But Wait, There’s More!
All 6 phases above were happening simultaneously with other systems. The raw data itself needed to be stored, buffered, transferred, and translated into a format suitable for the data warehouse. Since almost a quarter of our servers are in other parts of the country, we had to have redundant data storage and transfer solutions to make sure the raw data got to where it needed to be. For this data, the rules were different.

First of all, there are two layers of raw data. Each city has a local bandwidth polling database. We use rtgpoll to poll the bandwidth data. Since rtgpoll is designed to write to a different table for each data type, we kept the data like that. For ease of data management, we created a script that would keep a rotating two-day cycle of tables, one for today and one for yesterday, with a merge table that encompassed them both. We could get away with the merge table on this layer because there are far fewer tables, far fewer rows, and different indexes. Since the interface isn’t important at this layer, we could index only the date time and get the performance we wanted by making our transfer scripts to the global buffer date-based.

The global buffer server is the same box we attempted to use in phase 3 and 4, above. It has the exact same table structure as the datacenter buffers, with the rotating tables living under a merge table. This data is replicated out to a slave server, to prevent read/write contention. At this layer, we have no ON DUPLICATE KEY statements being executed, and no partitions, plus our merge tables are much smaller, so everything works out. These tables act as a permanent raw data archive in the event of a system failure or a bandwidth dispute with a customer.

The scripts that pull data out of the datacenter buffers also inserts data into a queue for each of the data warehouse nodes. We store a lookup table in memcache that will translate a raw interface ID into the data warehouse nodes that interface’s data needs to be inserted into. That raw row is then inserted into the queues for the nodes it belongs to.

Finally, a set of scripts runs on the data warehouse nodes, constantly pulling any new data out of their queues on the global buffer slave. The data is translated from raw interface data to match up to our customer accounts, then inserted into the local data warehouse database, ready to be selected out to make graphs, reports, or billing.

All Powered by Tracking Object
The entire system is interfaced from our application code using the tracking object system. The tracking objects are a series of PHP classes that link a particular object in our existing production database to that object’s various data points in the data warehouse. Using ORM and factory patterns, we were able to abstract tracking objects to the point where any object in our database could have an associated “trackingObject” member variable. Servers, Virtual Dedicated Racks, Cloud Computing Instances, and other systems can simply call the getBandwidthData() method on their tracking object, and the tracking object system will automatically select the correct database, select the correct table, and pull the correct fields, formatting them as a collection of generic “bandwidth data” objects. Other metrics, like CPU and memory usage for servers and Cloud Computing Instances, can just as easily be retrieved.

Similarly, most of our back-end systems use the tracking objects to add data to the data warehouse. The developers don’t touch the warehouse directly, they simply load whatever object they have new data for, and pass an array of raw data objects to our internal addData() function, which automatically determines database node, write table, and data structure. The tracking object system is completely transparent to the other developers, and it means new tracking objects or data warehouse nodes can be created seamlessly without changing existing code.

By centralizing the reading and writing into these classes, the data warehouse can be extended infinitely. A new type of data can be added as easily as adding a new data warehouse data type class to the file system, as well as a row to the database. As long as that class has the data structure properly defined, new tracking objects can be created for that data type, and data can begin being recorded immediately. Creating a new tracking object will automatically choose two or more database nodes to store the data on. Creating new nodes takes the current tracking object count into consideration, so the nodes stay balanced.

So far the system has 33,115,715,147 rows in 683,460 tables spread over 10 databases. We have hundreds of customers who view their bandwidth graphs every day, and a handful that systematically pull the graphs every hour. Load tests suggest that performance doesn’t degrade until we hit 500 simultaneous graph requests, and even then we still come in under 2 seconds per graph. With the scaling potential and the single point of access for developers, we should be able to use this system indefinitely.

We’ve got instructions conveniently hosted on our cloud systems:

CloudLayer™ Storage API v.1 Documentation (1.3MB)

This is a bit of a departure from our standard API. It only covers manipulation of CloudLayer Storage accounts, and is accessed via REST-full URLs. Head dev honcho Nathan gave us a rundown of how to access it in curl a while back. It’s incredibly powerful and gives you complete control over what you host and share on the cloud. Give it a try and let us know what you think!

This year I was asked the same thing I’m asked every year, but this time it was suggested that I could probably tag along with the cool kids if I gave a talk or ran a session. That’s a pretty good idea. I pitched it to my boss, and to my surprise he agreed. That’s great! Now all I have to do is schedule a presentation, and I’m set! A month or so full of research and agonizing over minutia has led to a talk I’ll be giving with Ilya Baimetov from Parallels about providing services with APIs. Our slot is at 3:00PM on Wednesday, August 12th. Stop on by if you’re in the Washington DC area, want to see a giant collection of businesses in the hosting industry, and want to visit the guys who run your favorite dedicated and cloud hosting company. When not speaking you’ll find me at the SoftLayer booth giving API demos and free hugs.

SoftLayer is well represented in the HostingCon event schedule. Our CFO, Mike Jones, is headlining Monday’s keynote “Surviving the Financial Crisis”, and CTO Nathan Day is participating in Tuesday’s general discussion “Offering Cloud Services to Your Customers”. The event runs form Monday, August 10 to Wednesday, August 12 at the Gaylord National Resort and Convention Center in National Harbor, Maryland. We’d love to meet you. Drop on by and say hi!

One easy way to automate access to CloudLayer Storage is through curl. Curl is available as a command-line tool in most every operating system and is typically used for transferring files. In this post I’ll show some examples on how to use curl to add, get, delete, or otherwise manipulate files in CloudLayer Storage. Note that this isn’t using the SoftLayer API, but instead interfaces directly with CloudLayer Storage.

Upload a file named “DSC1012.jpg” to an account owned by username “user@example.com” with a password of “PaSsWoRd”:

# curl –F filename=@DSC1012.jpg –u user@example.com:PaSsWoRd \
https://storage.cloudlayer.com/v1/files/

The command will return some XML tags. The items of interest are “FileID” and “lockID”. These values are important for future operations on the file.


<fileID>102C9C28-65C3-11DE-1234-2BE68BA216C2</fileID>
<lockID>6CDCEEB2-6B38-11DE-A510-123F439A2728</lockID>
<lockDuration>120</lockDuration>


The lock is to protect a file form reading or being manipulated during the upload process. The lock will expire in “lockDuration” seconds or the user can disable the lock manually.

Here is how to disable the lock using the lockID and the fileID generated from the upload operation:

# curl –d \
'action=unlock&lockid=6CDCEEB2-6B38-11DE-A510-123F439A2728' \
–u user@example.com:PaSsWoRd \
https://storage.cloudlayer.com/v1/files/102C9C28-65C3-11DE-1234-2BE68BA216C2/lock

If you ever lose track of the FileID, you can use this command to retrieve a listing of the files and containers (directories) in an account along with the FileIDs which are listed as an “oid” XML tag.

# curl –u user@example.com:PaSsWoRd \
https://storage.cloudlayer.com/v1/files/list

To get the list of files in a container, just append the container oid to the URL.

# curl –u user@example.com:PaSsWoRd \
https://storage.cloudlayer.com/v1/files/list?oid=37D0F2AC-08FC-11DE-1234-3FA3A91CD1B4

To retrieve the file from CloudLayer Storage, use the FileID to retrieve it.

# curl -u user@example.com:PaSsWoRd \
https://storage.cloudlayer.com/v1/files/37D0F2AC-08FC-11DE-1234-3FA3A91CD1B4/ -o outputfilename

Alternatively, you could use “wget” to retrieve the file

# wget –http-user=user@example.com -–http-password=PaSsWoRd \
https://storage.cloudlayer.com/v1/files/37D0F2AC-08FC-11DE-1234-3FA3A91CD1B4/ -O outputfilename

To delete a file just add the POST form variable “action” with the value “delete”.

# curl –d 'action=delete' –u user@example.com:PaSsWoRd \
https://storage.cloudlayer.com/v1/files/37D0F2AC-08FC-11DE-1234-3FA3A91CD1B4/

Each of the commands listed above return data in XML format. If you would prefer json format, add a query parameter “output=json” to the query string.

# curl –u user@example.com:PaSsWoRd \
https://storage.cloudlayer.com/v1/files/list?output=json

In order to create a public URL for a file, just send a POST variable of “action=create” to the “token” endpoint.

# curl -d 'action=create' -u user@example.com:PaSsWoRd \
https://storage.cloudlayer.com/v1/files/37D0F2AC-08FC-11DE-1234-3FA3A91CD1B4/token/

The long string “37D0F2…” is the oid (a.k.a FileID) of the file that you can get from the XML returned when the file was uploaded, or retrived using the file listing example above.

In the XML (or JSON) data that is returned, there will be a “token”.

<token>B2891F7B054EF2DF764801E1CFF0079057291234</token>

That token can be combined with the oid to create a URL that anyone can use to retrieve the file.

The URL looks like this:
https://storage.cloudlayer.com/v1/public/{oid}/{token}

In our example it would be:
https://storage.cloudlayer.com/v1/public/37D0F2AC-08FC-11DE-1234-3FA3A91CD1B4/B2891F7B054EF2DF764801E1CFF0079057291234

If you are accessing CloudLayer Storage from inside a SoftLayer datacenter, you can access the storage over the SoftLayer private network (no bandwidth fees!). Just use “scs.service.softlayer.com” instead of “storage.cloudlayer.com”.

You can use the information above in conjunction with the curl libraries in PHP, C++, or one of many other programming languages with curl bindings.

We plan to turn the old API off on August 1, 2009. Please post to our forums if you need help migrating or open a support ticket if you’d like to work one-on-one. Thanks, everyone!

To that end I’ve written a new PHP API client. This small library will let you take real advantage of the features our API has to offer (like object masks, result limits, proper exception catching, and the like) in both SOAP and XML-RPC. Please download it from our new presence on the github social code hosting site. Come check us out at:

http://github.com/softlayer/

Right now all that’s up there is our PHP API client. We’ll put more client libraries and projects up as we make them. We’ve got a a few in the works right now. If you’ve already got a github account come find us! We love to meet those who want to use and modify our work. If you have a favorite language that you want to see a client library for please let us know and we’ll see what we can do. The next language on my hit list is Python 2.6. See y’all next time!

Many developers, even experienced ones, are confused by the way PHP handles arrays in foreach loops. In the standard foreach loop, PHP makes a copy of the array that is used in the loop. The copy is discarded immediately after the loop finishes. This is transparent in the operation of a simple foreach loop. For example:

$set = array("apple", "banana", "coconut");
foreach ( $set AS $item ) {
    echo "{$item}\n";
}


This outputs:

apple
banana
coconut

Even though the copy is created, the developer doesn’t notice, because the original array isn’t referenced within the loop or after the loop finishes. However, when you attempt to modify the items in a loop, you find that they are unmodified when you finish:

$set = array("apple", "banana", "coconut");
foreach ( $set AS $item ) {
    $item = strrev ($item);
}
print_r($set);


This outputs:

Array
(
    [0] => apple
    [1] => banana
    [2] => coconut
)

There are no changes from the original, even though you clearly assigned a value to $item. This is because you are operating on $item as it appears in the copy of $set being worked on. You can override this by grabbing $item by reference, like so:

$set = array("apple", "banana", "coconut");
foreach ( $set AS &$item ) {
    $item = strrev($item);
}
print_r($set);


This outputs:

Array
(
    [0] => elppa
    [1] => ananab
    [2] => tunococ
)

As you can see, when $item is operated on by-reference, the changes made to $item are made to the members of the original $set. Using $item by reference also prevents PHP from creating the array copy. To test this, first we’ll show a quick script demonstrating the copy:

$set = array("apple", "banana", "coconut");
foreach ( $set AS $item ) {
    $set[] = ucfirst($item);
}
print_r($set);


This outputs:

Array
(
    [0] => apple
    [1] => banana
    [2] => coconut
    [3] => Apple
    [4] => Banana
    [5] => Coconut
)

In this example, PHP copied $set and used it to loop over, but when $set was used inside the loop, PHP added the variables to the original array, not the copied array. Basically, PHP is only using the copied array for the execution of the loop and the assignment of $item. Because of this, the loop above only executes 3 times, and each time it appends another value to the end of the original $set, leaving the original $set with 6 elements, but never entering an infinite loop.

However, what if we had used $item by reference, as I mentioned before? A single character added to the above test:

$set = array("apple", "banana", "coconut");
foreach ( $set AS &$item ) {
    $set[] = ucfirst($item);
}
print_r($set);


Results in an infinite loop. Note this actually is an infinite loop, you’ll have to either kill the script yourself or wait for your OS to run out of memory. I added the following line to my script so PHP would run out of memory very quickly, I suggest you do the same if you’re going to be running these infinite loop tests:

ini_set("memory_limit","1M");


So in this previous example with the infinite loop, we see the reason why PHP was written to create a copy of the array to loop over. When a copy is created and used only by the structure of the loop construct itself, the array stays static throughout the execution of the loop, so you’ll never run into issues.

But wait, there’s more. PHP fails to create a copy of the array if a reference is used at all. We know that referencing $item will cause the infinite loop scenario above, but if $set is referenced anywhere else in the script, even the non-referencing foreach format will break:

$set = array("apple", "banana", "coconut");
$a = &$set;
foreach ( $set AS $item ) {
    $set[] = ucfirst($item);
}


Results in an infinite loop, even though $item isn’t by reference. Using $a instead of $set gives identical results.

This is not to say that $item is implicitly used by reference if $set is referenced. See this example:

$set = array("apple", "banana", "coconut");
$a = &$set;
foreach ( $a AS $item ) {
    $item = ucfirst($item);
}
print_r($set);


This outputs:

Array
(
    [0] => apple
    [1] => banana
    [2] => coconut
)

$set is unchanged from the original values, because even though $set is referenced by $a, and $set has not been copied, $item is still given only lexical scope in relation to the loop, and will not pass modifications back to $set. You will still have to assign it by reference to make changes to the original array:

$set = array("apple", "banana", "coconut");
$a = &$set;
foreach ( $a AS &$item ) {
    $item = strrev($item);
}
print_r($set);


This outputs:

Array
(
    [0] => elppa
    [1] => ananab
    [2] => tunococ
)

All of these examples also work in associative arrays using the foreach ( $set AS $key => $item ) syntax. $key can never be used by-reference it always comes from the array the loop construct is using, and cannot be modified. So the tricks used to modify array items in-position won’t work for modifying the keys. You can create new keys in the array, however, and unset the existing ones, like so:

$set = array("apple"=>"red","banana"=>"yellow","coconut"=>"brown");
foreach ( $set AS $key => $item ) {
    $set[ucfirst($key)] = $item;
    unset($set[$key]);
}
print_r($set);


This outputs:

Array
(
    [Apple] => red
    [Banana] => yellow
    [Coconut] => brown
)

However, as you may have already noticed, this array was copied before the loop began. If you were using the array in a situation where it couldn’t be copied, you will run into errors:

$set = array("apple"=>"red","banana"=>"yellow","coconut"=>"brown");
$a = &$set;
foreach ( $set AS $key => $item ) {
    $set[ucfirst($key)] = $item;
    unset($set[$key]);
}
print_r($set);


This outputs:

Array
(
)

Because the array was referenced and not copied, you get vastly unpredictable results when attempting to alter the physical structure of the array, especially using unset(). Without the unset() call in this example, you operate on the original array and loop through the original array, so you get the same infinite-loop generating code as before, but since we’re specifying the key for $set it doesn’t continue forever:

$set = array("apple"=>"red","banana"=>"yellow","coconut"=>"brown");
$a = &$set;
foreach ( $set AS $key => $item ) {
    $set[ucfirst($key)] = $item;
}
print_r($set);


This outputs:

Array
(
    [apple] => red
    [banana] => yellow
    [coconut] => brown
    [Apple] => red
    [Banana] => yellow
    [Coconut] => brown
)

You can prove that it’s still possible to enter an infinite loop by adding a $set[] inside your loop:

$set = array("apple"=>"red","banana"=>"yellow","coconut"=>"brown");
$a = &$set;
foreach ( $set AS $key => $item ) {
    $set[ucfirst($key)] = $item;
    $set[] = $item;
}
print_r($set);


This results in an infinite loop.

One interesting thing you can do with the $key => $item syntax when the array is copied is modify the original array structure without fear of causing loop issues:

<?php
$set = array("apple"=>"red","banana"=>"yellow","coconut"=>"brown");
foreach ( $set AS $key => $item ) {
    $set[] = ucfirst($item);
    unset($set[$key]);
}
print_r($set);


This outputs:

Array
(
    [0] => Red
    [1] => Yellow
    [2] => Brown
)

As you can see from this example, the array was copied for use in the loop construct. References to $set within the loop still refer to the outer version of $set, so the unset() call and the $set[] addition work on the original, leaving us with a nicely upper-cased version of the original, without keys.

This knowledge is useful for developers who are trying to plug memory holes in PHP applications. If you foreach through an array of objects that can be 50MB in size, you create an entire copy of the structure in memory for no reason other than to power the loop. If your loop doesn’t modify the structure of the array or add to it at all, it would be vastly more efficient to add the “cheat” of $a = &$array; right before your array to prevent PHP from making a copy.

This knowledge is also hopefully useful for programmers who cannot figure out why arrays are behaving like they are. Basically, if you don’t use references, the loop executes once for each member in the original array, regardless of what you do to the original.

NOTE: These tests were performed on PHP version 5.2.5. 5.2.0 and earlier perform differently. Run these tests yourself under controlled circumstances before relying on PHP to behave in any particular way.