In our last post we mentioned that our API now supports a REST interface. It’s really true, and it’s really here! Quoth our new, fancy, manual page:

REST API URLs are structured to easily traverse SoftLayer’s object hierarchy. A basic REST request is structured as follows:

https://<username>:<apiKey>@api.[service.]softlayer.com/rest/v3/
<serviceName>[/initializationParameter].<returnDatatype>

• All REST requests, even private network requests, must be passed through HTTP SSL.
• Use your API username and key to authenticate your request through HTTP authentication.
• The base hostname and folder name for a REST request is either api.softlayer.com/rest/v3/ or api.service.softlayer.com/rest/v3/. Use api.service.softlayer.com/rest/v3/ to access the REST API over SoftLayer’s private network. It’s a more secure way to communicate with SoftLayer, but the system making API calls must be on SoftLayer’s private network (either purchased from SoftLayer of logged into SoftLayer’s private network VPN).
• Follow up the base URL with the name of the API service you wish to call, for instance “SoftLayer_Account” or “SoftLayer_Hardware_Server”.
• If your API request requires an initialization parameter then add a slash followed by your init parameter id to your URL.
• The SoftLayer REST API can return either XML or JSON formatted output. Add “.xml” or “.json” to the end of your request to specify which format you’d like to receive data in.

Now that the boring stuff is out of the way here are a few sample calls:

The most basic call in the SoftLayer API is the getObject() method in the SoftLayer_Account API service. It also has the simplest URL in our REST API:

https://username:apiKey@api.softlayer.com/rest/v3/SoftLayer_Account.json

If you need to retrieve a particular object then append its id number to the end of the URL. This URL retrieves information about the SoftLayer_Hardware_Server record 1234:

https://username:apiKey@api.softlayer.com/rest/v3/
SoftLayer_Hardware_Server/1234.json

One of the coolest things in our API is how all of our objects relate to each other. Append a relational property to the end of your URL to retrieve that object’s related objects. This URL retrieves server 1234’s network component records:

https://username:apiKey@api.softlayer.com/rest/v3/
SoftLayer_Hardware_Server/1234/NetworkComponents.json

Chain these relational properties as far as you need to go. You can also specify single relational properties by adding its id to the end of the URL. This URL retrieves network component record 5678 from server record 1234:

https://username:apiKey@api.softlayer.com/rest/v3/
SoftLayer_Hardware_Server/1234/NetworkComponents/5678.json

Since hardware records relate back to the SoftLayer_Account service, your URL can also relate back to it. Your SoftLayer_Account record has properties for nearly every service available to you, and is handy for providing a truly RESTful interface.

https://username:apiKey@api.softlayer.com/rest/v3/SoftLayer_Account/
Hardware/1234/NetworkComponents/5678.json

I’m absolutely loving how well this works. Being able to test API functionality right in my web browser has made troubleshooting a lot easier on me. Our REST interface also supports object creation, edit, and deletion and SoftLayer API specific options like object masks and result limits. Check out our manual page for the scoop. Please give it a whirl and tell us what you think. We’ve got more good stuff coming your way soon!

Some of you have noticed that we mentioned our new mobile clients are based on our developer API but don’t require a VPN connection to our private network. Your observations are astute and indeed correct. Our iPhone and Android applications can be accessed from anywhere on the Internet, and now so can the SoftLayer API!

We’ve rolled out API endpoints on the public Internet at https://api.softlayer.com. Our public API uses SSL to keep your data transmission secure, but you’re still free to use non-SSL HTTP on our private network. Here’s a rundown of our API’s endpoint locations:

SOAP:

• https://api.softlayer.com/soap/v3: public network
• https://api.service.softlayer.com/soap/v3: private network

XML-RPC:

• https://api.softlayer.com/xmlrpc/v3: public network
• https://api.service.softlayer.com/xmlrpc/v3: private network

REST: (That’s a new one. More on RESTful services soon!)

• https://api.softlayer.com/rest/v3: public network
• https://api.service.softlayer.com/rest/v3: private network

The private network endpoints at api.service.softlayer.com are still up and aren’t going anywhere. In fact, we still recommend using the private network to send API calls from your servers and cloud instances in our datacenters. But those of you developing desktop and mobile clients have a whole new world of access. Our public API clients will soon be updated to use our public endpoints. Try these out, log into our forums, and tell us what you think!

Much of the development work that goes on here centers on the SoftLayer Customer Portal. The Customer Portal is a traditional Web application that links your browser to the powerful back-end systems that our engineers have crafted over the course of many years. Most of the engineers around here are first rate hands at web application technologies like PHP, JavaScript, and HTML.

I don’t know much HTML. I know even less PHP. I spent many years working on desktop Macintosh applications in C and C++. With the introduction of the iPhone SDK, a new job opportunity made me an iPhone developer. The leap from the Cocoa framework, on the desktop Mac OS, to the Cocoa Touch framework of the iPhone was not difficult at all. The two frameworks share a lot in common and the parts that are different still share a lot of design patterns. All in all, making the transition to a company whose primary focus is web development was a lot more intimidating than picking up the iPhone API.

I joined SoftLayer specifically to work on the Mobile Client. The goal of the SoftLayer Mobile Client is pretty straightforward, put the essential parts of the SoftLayer Customer Portal in the palm of your hand, on your favorite mobile device. We wanted to move beyond the mobile web applications the team had already created, and craft an application with a look and feel only a native applications can provide, a first rate user experience.

At the same time, all the attention to detail you can possibly give the front end doesn’t matter unless you have some way to communicate with the server. That’s where the SoftLayer v3 API comes into play.

The smart folks on the the web application development team work behind the scenes on the server, enjoying full access to the complex machine behind the SoftLayer Customer Portal. Their target has been a traditional web browser, running on a computer with Gigabytes of available RAM, Gigahertz of processing power, and several Megabits per second of network connectivity.

Comparatively, on the Mobile Applications development team, we live across the wire on tiny computers with small pipes. Mobile devices have limited memory constraints, run on a “slow” 3G networks, and have processors running at less than 1/3 of the speed of their desktop counterparts.

With such vast differences between the environment of a desktop computer’s browser and an application running on a mobile device, SoftLayer could have crafted a custom API to give our mobile applications access tailored access to the machine behind the SoftLayer Customer Portal.

But we didn’t want to do that.

With great pride of principle, our mobile applications team makes it a point to use the same API that SoftLayer presents to you, the third party developers of the SLDN network. We don’t want to use back doors, and we don’t want to use secret calls. Consuming our own API, “eating our own dog food” as the saying goes, gives us a unique vantage point which we hope to use to improve the API for all SLDN developers.

In the course of developing the Mobile Client on a variety of platforms, the mobile team has found that most aspects of the SoftLayer API work very well! We’ve also found some that presented challenges to our mobile devices. Lucky for you, however, we’re working very closely with the web development team to remove those challenges.

We plan to bring new functionality that results from our experience directly to you. In the course of the next few months, look for some changes (large and small) to the services we provide. We hope you will find them a benefit to your application development regardless of where your application runs.

Lately I’ve seen a lot of people around me writing and hacking in Python. Way back when API v1 was out we noted that Python has built-in XML-RPC support. Built-in XML-RPC support is great. It makes calling our API, especially our latest API, a snap. Some of y’all on our forums have been doing very cool things with our API in Python.

Python hackers and users, we’ve just made your lives easier. I’m pleased to present our latest language tool for your API toolbox, a Python library for the SoftLayer API. This library functions very similarly to our PHP and Perl clients. Before you had to juggle API call headers to pass along things like initialization parameters and object masks to your API calls. Now you all you have to do is declare an API client object and run methods on it. It’s much easier and saves a lot of coding on your part.

I’ve been talking to a lot of people who are building dynamic Cloud Computing Instances using our API. I wrote a simple example for one of our forum users a while back to accomplish this task in Python using it’s built-in XML-RPC support:

Here’s that same functionality using our new Python library:

This saved 20 lines of code and is much easier to read. Our client is supported in both Python 2 and Python 3 and has a handy installation script. Please download it, go nuts, and speak up on our forums, IRC channel, or a support ticket if you have any questions. As always, we’re happy to hear your feedback. Thanks, everyone!

http://github.com/softlayer/softlayer-api-python-client/

As a developer, I have a good relationship with the PHP community. Many of my personal friends are involved in large PHP projects all over the world. One friend in particular is the lead developer of The Easy API. It’s an API wrapper that does the “hard parts” for you. Some companies release “APIs” that are confusing hodgepodges of unrelated functionality. Many times the API in question is simply a web form that developers are expected to POST to and parse poorly-formatted output.

The Easy API was designed specifically to take these poorly written APIs and wrap them up in a good PHP API Interface, with real functions and objects so that you can utilize a remote, web-driven API just like you would a native set of objects, or a database wrapper class.

I was discussing this with the lead developer of the project, and we had the following conversation:

That was a great thing to hear, as a SoftLayer Developer. The reason we wrote the API the way we did was because we were all tired of companies calling something an “API” when it was really a URL that would spit out a CSV file, or a ridiculously strict XML engine that would complain about a single space out of place. In fact, I once worked with an API that would throw “not valid XML” errors on perfectly valid XML. The most ironic part was that the “not valid XML” error itself was not valid XML.

As developers who spend much of our time integrating third party products, APIs, and services, we know how hard it is to work with a poorly documented, poorly implemented interface. That’s why part of our standard release procedure is having our API Evangelist review our method names, variable names, class names, and all related documentation to make sure they’re not only easy to read, but follow the pattern that the rest of the API follows. That’s why you always see “hardware” keys on our objects: we’re simply not allowed to call something “servers,” the code cannot be released until the API-exposed functionality is ready for public consumption.

We’ve all worked very hard on the API, because the API is what drives the portal, and the portal is what drives our customers. We’re happy to see everyone using the portal, but what really excites us is when customers use the API directly to form their own custom tools. The portal is a wonderful, powerful tool, but we understand that not every customer is happy with using the same thing. That’s why we exposed the API to our customers: so you could ALL write your own custom API-enabled objects. If you do, please share them, if not with the community at large, than at least with us directly. We’d love to see how customers are using the API, and if you share with us your most difficult API tasks, we’ll work to make them better. Even though the current SoftLayer API makes API-wrapper authors say “wow,” we want to make it even better.

An interesting facet of the development and systems administration business is the number of 80% projects that build up over time. An 80% project is that awesome library, script, rewrite, new system, or what have you that’s cooling on your back burner. It’s almost done but it’s missing the finishing touches. Maybe it needs a few code tweaks. Maybe it needs a little more documentation. Maybe you’re still finalizing settings and playing with patches. Don’t lie; we know you’ve got these projects hanging around. I’ve got a list of 80% projects as long as my arm.

It’s time to check something off my 80% projects list. I’ve finally finished documenting and am happy to release the SoftLayer API Perl client library! This module will make Perl API hackers’ lives a whole lot easier. Previously you had to build SOAP API calls manually using the SOAP::Lite module and parse the response into something easier to handle. Now you can accomplish the same thing with a series of easy to use helper methods. Functionality is very similar to our PHP client but with a Perl twist. For instance, you can do clever one liners!

use SoftLayer::API::SOAP;

# Grab my account information.
my $account = SoftLayer::API::SOAP->new(
    'SoftLayer_Account',
    undef,
    'my API username',
    'my API key')->getObject();

Check out our README for many and more comprehensive examples. Download the library from our github page at:

 

http://github.com/softlayer/softlayer-api-perl-client

 

As with all of our projects we’re very open to feedback, so please comment or post on our forums and let us know what you think. I can’t wait knock a few more 80% projects off the ol’ list. You’re going to love them. See y’all next time!

After a three hour flight that took five hours from LA to Dallas I’m finally home from the Southern California Linux Expo. This was SCaLE’s 8th show, an evolution that started as a meeting of Linux User Groups from the Los Angeles area that grew into one of the largest Linux, open technology, and community focused events in the region.

This is my second year attending SCaLE. I went last year as a part of an open source project I was working with and had a pretty good time. I jumped at the chance to go again when I found SoftLayer had a booth at and was sponsoring SCaLE 8x.

I’m really glad I went; I dig these community-focused events. What strikes me about these events, and SCaLE in particular, was the general optimism of the staff, attendees, speakers, and exhibitors. This show is on a weekend, and people came because they genuinely care about FOSS, cool new (and sometimes old) tech, and community advocacy. Most attendees fit the classic nerd stereotype. We got to meet our fair share of skinny pale guys, scruffy and ponytailed sysadmin types, and bearded folks with wallet chains and Star Trek t-shirts.

When not working our booth I was able to duck out and attend some of the talks at the expo. Aside from being informative (I picked up a lot about technical writing that I can’t wait to throw into our API documentation), every single talk I attended was completely packed to the point where there was standing room only. It’s refreshing to see speakers talk about things other than the company they work for. Instead, they focused on cool tech or how to be more productive in your community or with a specific technology. I took pages and pages of notes this weekend, it felt like being in school again.

I think SoftLayer did well at SCaLE 8x. We got our name out like we usually do at these events, and got to introduce a new group of people to SoftLayer and our ever-changing and sometimes hidden industry. I hope we’re sponsoring SCaLE 9x. Even if we don’t you’ll still find me there. See you next year!

Object Masks, Filters, and Other V3 Black Magic

Everyone has heard the age old saying for any given job you need to have the right tool. Just as most of us have tried to use the flat rounded edge of a butter knife a time or two when what we desperately needed was a screw driver. Does that mean you weren’t able to open that little compartment on whatever gizmo and replace the batteries? Probably not. In most cases it is possible to use a butter knife when a screw driver is the tool of choice; it’s just more painful and a lot less effective.

The same can said of the SoftLayer API (SLAPI). It’s a toolset. A very flexible set of tools allowing a developer to manage every aspect of dedicated hosting from accounting and billing to physical status of remote hardware. And yet there are so many tools in the V3 API toolbox, a number of them only subtlety different from their binary brethren ( at least on the surface), it’s tempting just to reach your hand into the bag, find the first thing that resembles a screw driver, and begin turning.

I know. I’m speaking from my own experiences. As a developer who largely works on SoftLayer’s back end systems, somewhere between the bottom of TCP/IP stack and the top edge of the kernel, recently getting to do production user portal code was a new experience for me. Sure I wrote some demos, dabbled a little here and there, but when I started doing my first “real” V3/SLAPI intensive project I realized my prior attempts had entirely missed the true power and elegance of SLAPI. The magic if you will. A little something called ORM.

Those of you who spend your days toiling in the world or relational databases are probably fairly familiar with the term ORM. But for someone like me who usually comes no closer to a database than reading an I/O address from the Windows registry, I was only vaguely aware of what the acronym even stood for. I turned to Webopedia. There I discovered the following. “Short for object role modeling, ORM is a conceptual database design methodology that allows the user to express information as an object and explore how it relates to other information objects”.

So there we have it. Database. Objects. Relations. I learn hands on—so none of that amounts to a hill of beans without some real code I can see and type and run for myself. So rather than regurgitate the SLDN documentation, I will just share a simple yet real life example. Then, in the second part of the article, we can expand that example to show some of the more powerful and less documented features of the V3 SLAPI.

The code that follows is written in Microsoft C Sharp using Visual Studio 2008 Professional Edition. I am not going to step through the basics of connecting a WSDL and generating a SOAP wrapper in this article. If you need help with that, there is an SLDN blog I did a while back which covers those steps entitled, “Dot Net? You Bet!”. It is still available under the implementations section of the SLDN website. True to my MO, I am not a big GUI guy so the code I am presenting runs as a Windows console application.

For the sake of making the example clear, I am going to simplify my task. In the example in both this article, and the next in the series, we will be playing the role of a developer who needs to count how many of his or her servers are running the Microsoft Windows operating system, as opposed to one of the many Linux variants SoftLayer also offers its customers. For our first code sample, we require two WSDLs: the SoftLayer_Account service as well as the SoftLayer_Hardware_Server service. The console program below will get us connected to the SoftLayer application servers, as well as provide us some timing metrics.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace SLDN_Magic
{
   class Program
   {
       static void Main(string[] args)
       {
           //global timing vars
           DateTime stopwatch;
           TimeSpan elapsed;

           //replace with your username and api key
           string user_name = "Replace With Your User Name";
           string api_key = "Replace With Your API Key";

           Console.Write("Establishing connection to SLDN service...");

           //time it
           stopwatch = DateTime.Now;
           //declare the services
           SLDN_ACCT.SoftLayer_AccountService acct = new
               SLDN_ACCT.SoftLayer_AccountService();
           SLDN_SVR.SoftLayer_Hardware_ServerService svr = new
               SLDN_SVR.SoftLayer_Hardware_ServerService(); 

           //create an authentification object for each
           SLDN_ACCT.authenticate credentials_a = new
               SLDN_ACCT.authenticate();
           SLDN_SVR.authenticate credentials_b = new
               SLDN_SVR.authenticate();

           //assign credentials
           credentials_a.username = credentials_b.username = user_name;
           credentials_a.apiKey = credentials_b.apiKey = api_key;

           //authenticate
           acct.authenticateValue = credentials_a;
           svr.authenticateValue = credentials_b;

           elapsed = DateTime.Now.Subtract(stopwatch);
           Console.WriteLine("done (" +  elapsed.TotalSeconds.ToString() + " seconds)");

           Console.WriteLine("\nPress  to exit.");
           Console.ReadLine();
       }
   }
}

At this point, we can go ahead and run our code. It doesn’t really do anything all that useful. But never the less you should get an output similar to this.

While writing this article I connected to the SoftLayer API servers numerous times from my home. My connection times were pretty consistent. It took somewhere in the neighborhood of 20 seconds to get everything set up. That seems like a lot. But keep in mind that you only incur the overhead of connecting your services one time. Plus as we get a little further along in this article I will show you how we can use ORM to get rid of one of the references entirely. For now though, let’s move on.

For someone of my background and mindset, what seemed the most straight-forward and correct way to find out which servers were running MS Windows was to access the public method “isWindowsServer()“. This is a method off the server class. That’s why we needed to import the SoftLayer_Hardware_Server service. But we don’t want to check the OS on a single server. We want to recurse through all the servers for an account. Which is why we brought in the SoftLayer_Account service and its attractive public offering “getAllHardware()“.

Keeping this plan in mind, let’s go ahead and implement it in our console application.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace SLDN_Magic
{
   class Program
   {
       static void Main(string[] args)
       {
           //global timing vars
           DateTime stopwatch;
           TimeSpan elapsed;

           //global container
           SLDN_ACCT.SoftLayer_Hardware[] hw;

           //replace with your username and api key
           string user_name = "Replace With Your User Name";
           string api_key = "Replace With Your API Key";

           Console.Write("Establishing connection to SLDN service...");

           //time it
           stopwatch = DateTime.Now;

           //declare the services
           SLDN_ACCT.SoftLayer_AccountService acct = new
               SLDN_ACCT.SoftLayer_AccountService();
           SLDN_SVR.SoftLayer_Hardware_ServerService svr = new
               SLDN_SVR.SoftLayer_Hardware_ServerService(); 

           //create an authentification object for each
           SLDN_ACCT.authenticate credentials_a = new
               SLDN_ACCT.authenticate();
           SLDN_SVR.authenticate credentials_b = new
               SLDN_SVR.authenticate();

           //assign credentials
           credentials_a.username = credentials_b.username = user_name;
           credentials_a.apiKey = credentials_b.apiKey = api_key;

           //authenticate
           acct.authenticateValue = credentials_a;
           svr.authenticateValue = credentials_b;

           elapsed = DateTime.Now.Subtract(stopwatch);
           Console.WriteLine("done (" + elapsed.TotalSeconds.ToString() + " seconds)");

           //butter knife method

           Console.Write("Retrieving hardware using method 1...");

           //get time stamp
           stopwatch = DateTime.Now;

           hw = null;
           try
           {
               hw = acct.getHardware();
           }
           catch (Exception e)
           {
               Console.WriteLine("Exception encountered [" + e.Message + "]");
               hw = null;
           }

           int cnt = 0;

           foreach (SLDN_ACCT.SoftLayer_Hardware server in hw)
           {
               try
               {
                   SLDN_SVR.SoftLayer_Hardware_ServerInitParameters box = new
                       SLDN_SVR.SoftLayer_Hardware_ServerInitParameters();
                   box.id = (int)server.id;
                   svr.SoftLayer_Hardware_ServerInitParametersValue = box;
                   if (svr.isWindowsServer())
                   {
                       cnt++;
                   }
               }
               catch (NullReferenceException)
               {
                   //ignore...this server has not had the
                   //OS loaded on it yet!
               }
           }

           elapsed = DateTime.Now.Subtract(stopwatch);

           Console.WriteLine("done (" + elapsed.TotalSeconds.ToString() + " seconds)");
           Console.WriteLine("counted " + cnt.ToString() + " MS Windows licenses");
           Console.WriteLine("\nPress  to exit.");
           Console.ReadLine();
       }
   }
}

That’s it. Pretty straight forward stuff possibly with the exception of the way SLAPI allows you to reinitialize the server (or any object) on the fly by use of:”SoftLayer_Hardware_ServerInitParameters”. If this confuses you, again I’ll refer you to the blog “Dot Net? You Bet!”. At this point, I think we are ready for another test run.

Once again we find ourselves in the twenty second range both for connecting the services and counting the hardware. What you can’t tell from looking at this output is that the account I was using for testing had about 100 servers on it. So basically we are talking 1/5 of a second per server. It’s certainly doable with a handful of servers, but this would obviously never work if you were trying to present this information real time to users if you managed 500 or 5,000 or 50,000 servers. So you are probably asking yourself the same thing I did. What gives? If SoftLayer wishes its customers success on an enterprise level, why create an API that comes to its knees when you start trying to manage more than a few hundred servers?

Luckily, the architects of SLAPI were a lot more web / database savvy than me. The above implementation, while correct syntactically, is a gross misuse of the SLAPI. It’s the butter knife, when what we really need is the screwdriver. What we need are object masks. But exactly what are object masks and how do they relate to ORM?

The best way I have found to understand ORM and object masks, is to think of the SLAPI data objects, as a self supporting entities. Each object provides its own methods and exposes some properties specific to that object. Yet thanks to ORM, most objects can actually get to properties in related objects, through a process called tapping. You simply tap each object down the chain until you find the property or properties you are interested in, prior to retrieving an instance or instances of the object. Then the set of objects returned will expose any relevant properties in the same manner you tapped them.

For example in our case the SLDN architecture relates a server to an operating system in the following manner.

The diagram shows us that essentially, as long as we can get a hardware object, we can tap all the way down to the software description — which as the SLDN documentation states has a “name” property. There by we eliminate two of our most time consuming tasks from our original application. First off we no longer need to instantiate the SoftLayer_Hardware_Server service, since we can get to server from hardware and hardware can be retrieved via the account class. Secondly, if when we return the hardware it already contains the name of the operating system, we no longer have a need to call the “isWindows()” method. Take a look.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace SLDN_Magic
{
   class Program
   {
       static void Main(string[] args)
       {
           //global timing vars
           DateTime stopwatch;
           TimeSpan elapsed;

           //global container
           SLDN_ACCT.SoftLayer_Hardware[] hw;

           //replace with your username and api key
           string user_name = "Your User Name Here";
           string api_key = "Your Api Key Here";

           Console.Write("Establishing connection to SLDN service...");

           //time it
           stopwatch = DateTime.Now;

           //declare the services
           SLDN_ACCT.SoftLayer_AccountService acct = new
               SLDN_ACCT.SoftLayer_AccountService();

           //create an authentification object
           SLDN_ACCT.authenticate credentials_a = new
               SLDN_ACCT.authenticate();

           //assign credentials
           credentials_a.username = user_name;
           credentials_a.apiKey = api_key;

           //authenticate
           acct.authenticateValue = credentials_a;

           elapsed = DateTime.Now.Subtract(stopwatch);
           Console.WriteLine("done (" + elapsed.TotalSeconds.ToString() + " seconds)");

           //method 2
           Console.Write("Retrieving hardware using method 2...");

           //get time stamp
           stopwatch = DateTime.Now;

           hw = null;
           //attempt to pull a hardware list for this user
           try
           {
               acct.SoftLayer_AccountObjectMaskValue = new
                   SLDN_ACCT.SoftLayer_AccountObjectMask();
               acct.SoftLayer_AccountObjectMaskValue.mask = new
                   SLDN_ACCT.SoftLayer_Account();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware = new
                   SLDN_ACCT.SoftLayer_Hardware_Server[1];
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0] = new
                   SLDN_ACCT.SoftLayer_Hardware_Server();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0].operatingSystem = new
                   SLDN_ACCT.SoftLayer_Software_Component();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0].operatingSystem.softwareLicense = new
                   SLDN_ACCT.SoftLayer_Software_License();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0].operatingSystem.softwareLicense.softwareDescription = new
                   SLDN_ACCT.SoftLayer_Software_Description();
               hw = acct.getHardware();
           }
           catch (Exception e)
           {
               Console.WriteLine("Exception encountered [" + e.Message + "]");
               hw = null;
           }

           cnt = 0;

           foreach (SLDN_ACCT.SoftLayer_Hardware server in hw)
           {
               try
               {
                   if (server.operatingSystem.softwareLicense.
                                   softwareDescription.name.ToLower().
                                   Contains("windows"))
                   {
                       cnt++;
                   }
               }
               catch (NullReferenceException)
               {
                   //ignore...this server has not
                   //had the OS loaded on it yet!!!
               }
           }

           elapsed = DateTime.Now.Subtract(stopwatch);

           Console.WriteLine("done ("+ elapsed.TotalSeconds.ToString()+" seconds)");
           Console.WriteLine("counted " + cnt.ToString() + " MS Windows licenses");

           Console.WriteLine("\nPress  to exit.");
           Console.ReadLine();
       }
   }
}

You should notice right away all the references to SoftLayer_AccountObjectMaskValue prior to calling the “getHardware()” method. This is the object mask. Essentially we must create a new instance of each entity we want to include down the chain. Then when the target object is retrieved, in our case the hardware, all related objects which we have made room for will get created for that specific instance of hardware, assuming of course a record exists. You must instantiate each object down the chain. If you skip any link your result set will not conatain the property or method you were trying to get to. Some less structured langagues, like PHP, do not have this requirement. But with V3 and dot NET there is no getting around it. You’re probably thinking this version of the code looks far more cluttered and is not as straight-forward to read. You’re right. But I contend this is the electric screwdriver in the SLDN toolbox. See for yourself.

As you can see the connection overhead dropped in half, which is to be expected since we are only authenticating to half the number of services. But take a look at the second number, the number of seconds it takes to count the servers with Windows installed. It dropped from 20 seconds, to under 2 seconds. That’s a 10 times speed gain. And wait there’s more– because we are only making one call to the SLDN application servers to retrieve those records what you see is what you get. Meaning you should not expect that time to increase noticeably whether you have a hundred servers or a hundred thousand servers! That my friend is the magic of V3. The power of ORM.

Following this article I will include the entire code base, in a combined application that lets you run the tests sequentially so you can see the amazing difference object masks make for yourself. In the second part to this article, we’ll continue with the sample so if you download it keep it handy. In part two I’ll discuss the next best thing to object masks—object filters. With object filters we’ll be able to streamline this code even more. Until then…happy SLDNing!

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace SLDN_Magic
{
   class Program
   {
       static void Main(string[] args)
       {
           //global timing vars
           DateTime stopwatch;
           TimeSpan elapsed;

           //global container
           SLDN_ACCT.SoftLayer_Hardware[] hw;

           //replace with your username and api key
           string user_name = "";
           string api_key = "";

           Console.Write("Establishing connection to SLDN service...");

           //time it
           stopwatch = DateTime.Now;

           //declare the services
           SLDN_ACCT.SoftLayer_AccountService acct = new SLDN_ACCT.SoftLayer_AccountService();
           SLDN_SVR.SoftLayer_Hardware_ServerService svr = new SLDN_SVR.SoftLayer_Hardware_ServerService(); 

           //create an authentification object for each
           SLDN_ACCT.authenticate credentials_a = new SLDN_ACCT.authenticate();
           SLDN_SVR.authenticate credentials_b = new SLDN_SVR.authenticate();

           //assign credentials
           credentials_a.username = credentials_b.username = user_name;
           credentials_a.apiKey = credentials_b.apiKey = api_key;

           //authenticate
           acct.authenticateValue = credentials_a;
           svr.authenticateValue = credentials_b;

           elapsed = DateTime.Now.Subtract(stopwatch);
           Console.WriteLine("done (" + elapsed.TotalSeconds.ToString() + " seconds)");

           //method 1

           Console.Write("Retrieving hardware using method 1...");

           //get time stamp
           stopwatch = DateTime.Now;

           hw = null;
           try
           {
               hw = acct.getHardware();
           }
           catch (Exception e)
           {
               Console.WriteLine("Exception encountered [" + e.Message + "]");
               hw = null;
           }

           int cnt = 0;

           foreach (SLDN_ACCT.SoftLayer_Hardware server in hw)
           {
               try
               {
                   SLDN_SVR.SoftLayer_Hardware_ServerInitParameters box = new SLDN_SVR.SoftLayer_Hardware_ServerInitParameters();
                   box.id = (int)server.id;
                   svr.SoftLayer_Hardware_ServerInitParametersValue = box;
                   if (svr.isWindowsServer())
                   {
                       cnt++;
                   }
               }
               catch (NullReferenceException)
               {
                   //ignore...this server has not had the OS loaded on it yet
               }
           }

           elapsed = DateTime.Now.Subtract(stopwatch);

           Console.WriteLine("done (" + elapsed.TotalSeconds.ToString() + " seconds)");
           Console.WriteLine("counted " + cnt.ToString() + " MS Windows licenses");

           //method 2
           Console.Write("Retrieving hardware using method 2...");

           //get time stamp
           stopwatch = DateTime.Now;

           hw = null;
           //attempt to pull a hardware list for this user
           try
           {
               acct.SoftLayer_AccountObjectMaskValue = new SLDN_ACCT.SoftLayer_AccountObjectMask();
               acct.SoftLayer_AccountObjectMaskValue.mask = new SLDN_ACCT.SoftLayer_Account();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware = new SLDN_ACCT.SoftLayer_Hardware_Server[1];
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0] = new SLDN_ACCT.SoftLayer_Hardware_Server();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0].operatingSystem = new SLDN_ACCT.SoftLayer_Software_Component();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0].operatingSystem.softwareLicense = new SLDN_ACCT.SoftLayer_Software_License();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0].operatingSystem.softwareLicense.softwareDescription = new SLDN_ACCT.SoftLayer_Software_Description();
               hw = acct.getHardware();
           }
           catch (Exception e)
           {
               Console.WriteLine("Exception encountered [" + e.Message + "]");
               hw = null;
           }

           cnt = 0;

           foreach (SLDN_ACCT.SoftLayer_Hardware server in hw)
           {
               try
               {
                   if (server.operatingSystem.softwareLicense.softwareDescription.name.ToLower().Contains("windows"))
                   {
                       cnt++;
                   }
               }
               catch (NullReferenceException)
               {
                   //ignore...this server has not had the OS loaded on it yet
               }
           }

           elapsed = DateTime.Now.Subtract(stopwatch);

           Console.WriteLine("done ("+elapsed.TotalSeconds.ToString()+" seconds)");
           Console.WriteLine("counted " + cnt.ToString() + " MS Windows licenses");

           Console.WriteLine("\nPress  to exit.");
           Console.ReadLine();
       }
   }
}

Here at SoftLayer, we have a lot of things that we need to keep track of. It’s not just payments, servers, rack slots, network ports, processors, hard drives, RAM sticks, and operating systems, it’s also bandwidth, monitoring, network intrusions, firewall logs, VPN access logs, API access, user history, and a whole host more. Last year, I was tapped to completely overhaul the existing bandwidth system. The old system was starting to show its age, and with our phenomenal growth it just hasn’t been able to keep up.

SoftLayer has 20,000+ servers. Each of those servers is on 2 networks, the public network open to the Internet, and the private network that only our customers can use. Each of those networks exists on a 3 to 4-level network hierarchy. This gives us more than 50,000 switch ports, but we’ll use 50,000 to make the math easier. Each switch port has bandwidth in and bandwidth out, as well as packets in and packets out. That gives us 200,000 data points to poll. Bandwidth is polled every 5 minutes, giving us 57,600,000 data points per day, or 1,728,000,000 per month. Given that bandwidth data points are all 64 bit numbers, and we also have to track the server ID (32 bit), the network (32 bit), and the datetime (32 bit), that makes a month’s worth of raw data (excluding any overhead from storage engines) 34.56GB. Now, the data is to be stored redundantly, so double that. Also, we have to track bandwidth for racks, virtual racks, and private racks, so add another 50% onto the data. That gives us around 90GB per month of data.

This doesn’t seem like a lot of data at first, but we need to generate custom bandwidth graphs for use on the web. Since it’s on the web, loading times above 2 seconds are unacceptable. Also, these are not enormous files with small keys (we’ll spend more time on that later) so 45GB of bandwidth data is a whole lot different than 45GB of movie files or MP3s.

To accomplish this, we decided that we needed a data warehouse. After numerous false starts and blind alleys, we decided to make our own system from scratch using MySQL. We considered commercial products, and pre-built open source solutions, but they just didn’t seem to fit our needs properly. The data warehouse project commenced with phase 1.

Phase 1: MySQL Cluster with Read Shards, Large Tables
Our first implementation was relatively simple. We planned to have a MySQL cluster for writing all the data, with the data split into 100 tables. The ID of the hardware mod 100 would determine the table name that we would write to. Then we’d have between 5 and 20 read databases, each replicating a different table, for load reasons. Then all we have to do is index the data properly, and add code to our applications to pull data from the correct read database node, and we’ll be fine.

Bad news: MySQL cluster isn’t designed for data with huge numbers of large keys. MySQL cluster stores all indexes and keys in memory on the cluster controller multiple times. As mentioned before, our data had 12 bytes of key, and 8 bytes of data. This means that we could only get about 2 million rows into the data warehouse before MySQL would lock up and quit working with the ever helpful error message “Unknown error: 157.” Even more disturbing, deleting items from MySQL cluster didn’t free up memory, as the indexes had to be rebuilt before that would happen. We upgraded everything to MySQL 5.1.6, but per the MySQL manual, “Beginning with MySQL 5.1.6, it is possible to store the non-indexed columns of NDB tables on disk, rather than in RAM as with previous versions of MySQL Cluster.” Unfortunately, it’s the indexes that are causing us problems, so the upgrade didn’t help.

Phase 2: MySQL Cluster with Read Shards, Small Tables
Since MySQL cluster couldn’t handle a small number of really big tables, how about a really big number of small tables? We tried making hundreds of tables per day, and removing the indexes on tables older than a certain time, but that quickly ran up against operating system limitations on the number of files in a folder. When we switched operating systems to alleviate this problem, we ran into MySQL’s limitation of roughly 21,000 tables per database. Nothing could get past that, so we had to move away from the cluster entirely.

Phase 3: Large MySQL box with Read Shards, Large Tables
We then moved on to one single MySQL box with enormous hard drives and multiple read shards. This looked promising at first, but the box simply couldn’t handle the amount of inserts and updates, and the slave servers were locking up too often. We thought that if we partitioned the tables, MySQL could handle the inserts better. This, naturally, broke replication in a different way. This was mainly because we had too many tables now that each partition was getting its own file, so MySQL would be constantly opening all these table files to perform the updates, and would quickly run out of memory and begin to swap. We just had to decrease the number of tables per server. We decided to abandon the centralized master server idea, and built out 5 pairs of master/slave servers.

Phase 4: 5 Pairs of Master/Slave Servers, Large Grouped Tables
This plan really seemed like it was going to work. We actually had a working data warehouse for almost 2 weeks without any errors. We were close to breaking out the champagne, but there was one feature we still hadn’t implemented.

The time came to add the tracking of bandwidth for virtual dedicated racks as well. To accomplish this, we changed the MySQL INSERT statement to INSERT... ON DUPLICATE KEY UPDATE. For those that don’t immediately recognize the terminology, the ON DUPLICATE KEY UPDATE syntax is designed so that if a particular database key already exists in the database, simply change the INSERT statement into an UPDATE statement. The UPDATE portion is defined at the end of the clause. Since the key to our data is the large “object ID, date time, data type” combination, adding the duplicate syntax allowed us to issue multiple INSERT statements for the same data, and have it continually update with new values. This was especially handy for virtual racks with thousands of servers.

MySQL threw us another curve ball when we tried this. There was a known issue with the ON DUPLICATE UPDATE syntax and replication using binary logs. Namely, the UPDATE would remain an INSERT in the log somehow, so we’d get duplicate key errors thrown on the slave, when the master was still working fine. Each time this happened, it would require stopping both servers, re-synching the tables, clearing the logs, restarting replication, and restarting the data transfer processes. This was unacceptable, so we had to move once more.

Phase 5: 10 Individual Machines
Since we already had 10 identical database machines, we decided to make them all independent and ignorant of each other. We stayed with the item group theory, leaving multiple bandwidth items per table. With 25 different items per table, we were only going to have 1,000,000 rows per table per month, which still maintains our speed. However, the index size problem bit us again. These tables were simply too large to have 3/4 of their columns indexed.

Finally we thought we had the answer. We would split up each of these tables by month. That way, each table would max out at 1,000,000 rows, so the indexes wouldn’t be unmanageable. Remember that we’ve already split the connection points from one, to five, to ten. We’ve also broken the tables up from one large table, to one per object, and now one per group of objects. In order to cut down on the complexity of the application layer code, we decided to use merge tables to keep the table name consistent. That way we can simply select from “the table” rather than “the table from March.” When we did this, we almost immediately began getting mysterious deadlocks on the servers. The INSERT statements would conflict with any SELECT statements using the same table, and the two threads would just hang indefinitely. Oddly, the table was never actually locked, it seemed that there was some sort of “offsetting lock” happening, where both threads were deadlocked in a race condition. The table could still be used, but the application we had inserting the data would be hung, so it was causing unacceptable delays in data inserts.

Phase 6: 10 Individual Machines, Small Individual Tables
Finally we decided that enough was enough, we were going to roll our own scaled storage solution. Clustering didn’t work, throwing hardware at the problem unfortunately didn’t work, and even the fancy new features for merged tables and partitioned tables didn’t work. We simply decided that we were going to keep the old “grouped tables” model, as well as creating a new table for every month without merge tables. This way, we keep the number of tables relatively low, and the number of rows per table low. Plus, as a bonus, by controlling the table names ourselves we could ensure that MySQL wouldn’t open too many files. All inserts went into this month’s table, and all reads would come out of whatever specific month they needed. A cron job was set up to periodically issue FLUSH TABLES on all 10 data warehouse nodes, and we had our completed product! It’s been months now since we had a major database failure, we can generate any bandwidth graphs you want in less than a second, and other developers are starting to create their own table groups.

Each object’s data resides completely on two of our data warehouse nodes, so the data is redundantly stored in the event of a node going down. The application code has been written with extensive factory patterns and ORM so that all a developer has to do is create a new group type, a new data tracking object, and add data to it. The code automatically selects the applicable nodes to write the data to, creates the tables if they don’t exist, and writes the data to the tables. Similarly, a “get data” command will randomly select one location for the data, and retrieve the proper data, using only the tables that are necessary.

But Wait, There’s More!
All 6 phases above were happening simultaneously with other systems. The raw data itself needed to be stored, buffered, transferred, and translated into a format suitable for the data warehouse. Since almost a quarter of our servers are in other parts of the country, we had to have redundant data storage and transfer solutions to make sure the raw data got to where it needed to be. For this data, the rules were different.

First of all, there are two layers of raw data. Each city has a local bandwidth polling database. We use rtgpoll to poll the bandwidth data. Since rtgpoll is designed to write to a different table for each data type, we kept the data like that. For ease of data management, we created a script that would keep a rotating two-day cycle of tables, one for today and one for yesterday, with a merge table that encompassed them both. We could get away with the merge table on this layer because there are far fewer tables, far fewer rows, and different indexes. Since the interface isn’t important at this layer, we could index only the date time and get the performance we wanted by making our transfer scripts to the global buffer date-based.

The global buffer server is the same box we attempted to use in phase 3 and 4, above. It has the exact same table structure as the datacenter buffers, with the rotating tables living under a merge table. This data is replicated out to a slave server, to prevent read/write contention. At this layer, we have no ON DUPLICATE KEY statements being executed, and no partitions, plus our merge tables are much smaller, so everything works out. These tables act as a permanent raw data archive in the event of a system failure or a bandwidth dispute with a customer.

The scripts that pull data out of the datacenter buffers also inserts data into a queue for each of the data warehouse nodes. We store a lookup table in memcache that will translate a raw interface ID into the data warehouse nodes that interface’s data needs to be inserted into. That raw row is then inserted into the queues for the nodes it belongs to.

Finally, a set of scripts runs on the data warehouse nodes, constantly pulling any new data out of their queues on the global buffer slave. The data is translated from raw interface data to match up to our customer accounts, then inserted into the local data warehouse database, ready to be selected out to make graphs, reports, or billing.

All Powered by Tracking Object
The entire system is interfaced from our application code using the tracking object system. The tracking objects are a series of PHP classes that link a particular object in our existing production database to that object’s various data points in the data warehouse. Using ORM and factory patterns, we were able to abstract tracking objects to the point where any object in our database could have an associated “trackingObject” member variable. Servers, Virtual Dedicated Racks, Cloud Computing Instances, and other systems can simply call the getBandwidthData() method on their tracking object, and the tracking object system will automatically select the correct database, select the correct table, and pull the correct fields, formatting them as a collection of generic “bandwidth data” objects. Other metrics, like CPU and memory usage for servers and Cloud Computing Instances, can just as easily be retrieved.

Similarly, most of our back-end systems use the tracking objects to add data to the data warehouse. The developers don’t touch the warehouse directly, they simply load whatever object they have new data for, and pass an array of raw data objects to our internal addData() function, which automatically determines database node, write table, and data structure. The tracking object system is completely transparent to the other developers, and it means new tracking objects or data warehouse nodes can be created seamlessly without changing existing code.

By centralizing the reading and writing into these classes, the data warehouse can be extended infinitely. A new type of data can be added as easily as adding a new data warehouse data type class to the file system, as well as a row to the database. As long as that class has the data structure properly defined, new tracking objects can be created for that data type, and data can begin being recorded immediately. Creating a new tracking object will automatically choose two or more database nodes to store the data on. Creating new nodes takes the current tracking object count into consideration, so the nodes stay balanced.

So far the system has 33,115,715,147 rows in 683,460 tables spread over 10 databases. We have hundreds of customers who view their bandwidth graphs every day, and a handful that systematically pull the graphs every hour. Load tests suggest that performance doesn’t degrade until we hit 500 simultaneous graph requests, and even then we still come in under 2 seconds per graph. With the scaling potential and the single point of access for developers, we should be able to use this system indefinitely.