Object Masks and Filters in C Sharp

Object Masks, Filters, and Other V3 Black Magic Everyone has heard the age old saying for an

Object Masks, Filters, and Other V3 Black Magic

Everyone has heard the age old saying for any given job you need to have the right tool. Just as most of us have tried to use the flat rounded edge of a butter knife a time or two when what we desperately needed was a screw driver. Does that mean you weren't able to open that little compartment on whatever gizmo and replace the batteries? Probably not. In most cases it is possible to use a butter knife when a screw driver is the tool of choice; it's just more painful and a lot less effective.

The same can said of the SoftLayer API (SLAPI). It's a toolset. A very flexible set of tools allowing a developer to manage every aspect of dedicated hosting from accounting and billing to physical status of remote hardware. And yet there are so many tools in the V3 API toolbox, a number of them only subtlety different from their binary brethren ( at least on the surface), it's tempting just to reach your hand into the bag, find the first thing that resembles a screw driver, and begin turning.

I know. I'm speaking from my own experiences. As a developer who largely works on SoftLayer's back end systems, somewhere between the bottom of TCP/IP stack and the top edge of the kernel, recently getting to do production user portal code was a new experience for me. Sure I wrote some demos, dabbled a little here and there, but when I started doing my first "real" V3/SLAPI intensive project I realized my prior attempts had entirely missed the true power and elegance of SLAPI. The magic if you will. A little something called ORM.

Those of you who spend your days toiling in the world or relational databases are probably fairly familiar with the term ORM. But for someone like me who usually comes no closer to a database than reading an I/O address from the Windows registry, I was only vaguely aware of what the acronym even stood for. I turned to Webopedia. There I discovered the following. "Short for object role modeling, ORM is a conceptual database design methodology that allows the user to express information as an object and explore how it relates to other information objects".

So there we have it. Database. Objects. Relations. I learn hands on—so none of that amounts to a hill of beans without some real code I can see and type and run for myself. So rather than regurgitate the SLDN documentation, I will just share a simple yet real life example. Then, in the second part of the article, we can expand that example to show some of the more powerful and less documented features of the V3 SLAPI.

The code that follows is written in Microsoft C Sharp using Visual Studio 2008 Professional Edition. I am not going to step through the basics of connecting a WSDL and generating a SOAP wrapper in this article. If you need help with that, there is an SLDN blog I did a while back which covers those steps entitled, "Dot Net? You Bet!". It is still available under the implementations section of the SLDN website. True to my MO, I am not a big GUI guy so the code I am presenting runs as a Windows console application.

For the sake of making the example clear, I am going to simplify my task. In the example in both this article, and the next in the series, we will be playing the role of a developer who needs to count how many of his or her servers are running the Microsoft Windows operating system, as opposed to one of the many Linux variants SoftLayer also offers its customers. For our first code sample, we require two WSDLs: the SoftLayer_Account service as well as the SoftLayer_Hardware_Server service. The console program below will get us connected to the SoftLayer application servers, as well as provide us some timing metrics.

using System;
using System.Collections.Generic;
using System.Linq; 
using System.Text;
 
namespace SLDN_Magic
{
   class Program
   {
       static void Main(string[] args)
       {
           //global timing vars
           DateTime stopwatch;
           TimeSpan elapsed;
 
           //replace with your username and api key
           string user_name = "Replace With Your User Name";
           string api_key = "Replace With Your API Key";
 
           Console.Write("Establishing connection to SLDN service...");
 
           //time it
           stopwatch = DateTime.Now;
           //declare the services
           SLDN_ACCT.SoftLayer_AccountService acct = new SLDN_ACCT.SoftLayer_AccountService();
           SLDN_SVR.SoftLayer_Hardware_ServerService svr = new SLDN_SVR.SoftLayer_Hardware_ServerService(); 
 
           //create an authentification object for each
           SLDN_ACCT.authenticate credentials_a = new SLDN_ACCT.authenticate();
           SLDN_SVR.authenticate credentials_b = new SLDN_SVR.authenticate();
 
           //assign credentials
           credentials_a.username = credentials_b.username = user_name;
           credentials_a.apiKey = credentials_b.apiKey = api_key;
 
           //authenticate
           acct.authenticateValue = credentials_a;
           svr.authenticateValue = credentials_b;
 
           elapsed = DateTime.Now.Subtract(stopwatch);
           Console.WriteLine("done (" +  elapsed.TotalSeconds.ToString() + " seconds)");
 
           Console.WriteLine("\
Press  to exit.");
           Console.ReadLine();
       }
   }
}

At this point, we can go ahead and run our code. It doesn't really do anything all that useful. But never the less you should get an output similar to this.

While writing this article I connected to the SoftLayer API servers numerous times from my home. My connection times were pretty consistent. It took somewhere in the neighborhood of 20 seconds to get everything set up. That seems like a lot. But keep in mind that you only incur the overhead of connecting your services one time. Plus as we get a little further along in this article I will show you how we can use ORM to get rid of one of the references entirely. For now though, let's move on.

For someone of my background and mindset, what seemed the most straight-forward and correct way to find out which servers were running MS Windows was to access the public method "isWindowsServer()". This is a method off the server class. That's why we needed to import the SoftLayer_Hardware_Server service. But we don't want to check the OS on a single server. We want to recurse through all the servers for an account. Which is why we brought in the SoftLayer_Account service and its attractive public offering "getAllHardware()".

Keeping this plan in mind, let's go ahead and implement it in our console application.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
 
namespace SLDN_Magic
{
   class Program
   {
       static void Main(string[] args)
       {
           //global timing vars
           DateTime stopwatch;
           TimeSpan elapsed;
 
           //global container
           SLDN_ACCT.SoftLayer_Hardware[] hw;
 
           //replace with your username and api key
           string user_name = "Replace With Your User Name";
           string api_key = "Replace With Your API Key";
 
           Console.Write("Establishing connection to SLDN service...");
 
           //time it
           stopwatch = DateTime.Now;
 
           //declare the services
           SLDN_ACCT.SoftLayer_AccountService acct = new SLDN_ACCT.SoftLayer_AccountService();
           SLDN_SVR.SoftLayer_Hardware_ServerService svr = new SLDN_SVR.SoftLayer_Hardware_ServerService(); 
 
           //create an authentification object for each
           SLDN_ACCT.authenticate credentials_a = new SLDN_ACCT.authenticate();
           SLDN_SVR.authenticate credentials_b = new SLDN_SVR.authenticate();
 
           //assign credentials
           credentials_a.username = credentials_b.username = user_name;
           credentials_a.apiKey = credentials_b.apiKey = api_key;
 
           //authenticate
           acct.authenticateValue = credentials_a;
           svr.authenticateValue = credentials_b;
 
           elapsed = DateTime.Now.Subtract(stopwatch);
           Console.WriteLine("done (" + elapsed.TotalSeconds.ToString() + " seconds)");
 
           //butter knife method
 
           Console.Write("Retrieving hardware using method 1...");
 
           //get time stamp
           stopwatch = DateTime.Now;
 
           hw = null;
           try
           {
               hw = acct.getHardware();
           }
           catch (Exception e)
           {
               Console.WriteLine("Exception encountered [" + e.Message + "]");
               hw = null;
           }
 
           int cnt = 0;
 
           foreach (SLDN_ACCT.SoftLayer_Hardware server in hw)
           {
               try
               {
                   SLDN_SVR.SoftLayer_Hardware_ServerInitParameters box = new SLDN_SVR.SoftLayer_Hardware_ServerInitParameters();
                   box.id = (int)server.id;
                   svr.SoftLayer_Hardware_ServerInitParametersValue = box;
                   if (svr.isWindowsServer())
                   {
                       cnt++;
                   }
               }
               catch (NullReferenceException)
               {
                   //ignore...this server has not had the 
                   //OS loaded on it yet!
               }
           }
 
           elapsed = DateTime.Now.Subtract(stopwatch);
 
           Console.WriteLine("done (" + elapsed.TotalSeconds.ToString() + " seconds)");
           Console.WriteLine("counted " + cnt.ToString() + " MS Windows licenses");
           Console.WriteLine("\
Press  to exit.");
           Console.ReadLine();
       }
   }
}

That's it. Pretty straight forward stuff possibly with the exception of the way SLAPI allows you to reinitialize the server (or any object) on the fly by use of:"SoftLayer_Hardware_ServerInitParameters". If this confuses you, again I'll refer you to the blog "Dot Net? You Bet!". At this point, I think we are ready for another test run.

Once again we find ourselves in the twenty second range both for connecting the services and counting the hardware. What you can't tell from looking at this output is that the account I was using for testing had about 100 servers on it. So basically we are talking 1/5 of a second per server. It's certainly doable with a handful of servers, but this would obviously never work if you were trying to present this information real time to users if you managed 500 or 5,000 or 50,000 servers. So you are probably asking yourself the same thing I did. What gives? If SoftLayer wishes its customers success on an enterprise level, why create an API that comes to its knees when you start trying to manage more than a few hundred servers?

Luckily, the architects of SLAPI were a lot more web / database savvy than me. The above implementation, while correct syntactically, is a gross misuse of the SLAPI. It's the butter knife, when what we really need is the screwdriver. What we need are object masks. But exactly what are object masks and how do they relate to ORM?

The best way I have found to understand ORM and object masks, is to think of the SLAPI data objects, as a self supporting entities. Each object provides its own methods and exposes some properties specific to that object. Yet thanks to ORM, most objects can actually get to properties in related objects, through a process called tapping. You simply tap each object down the chain until you find the property or properties you are interested in, prior to retrieving an instance or instances of the object. Then the set of objects returned will expose any relevant properties in the same manner you tapped them.

For example in our case the SLDN architecture relates a server to an operating system in the following manner.

The diagram shows us that essentially, as long as we can get a hardware object, we can tap all the way down to the software description — which as the SLDN documentation states has a "name" property. There by we eliminate two of our most time consuming tasks from our original application. First off we no longer need to instantiate the SoftLayer_Hardware_Server service, since we can get to server from hardware and hardware can be retrieved via the account class. Secondly, if when we return the hardware it already contains the name of the operating system, we no longer have a need to call the "isWindows()" method. Take a look.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
 
namespace SLDN_Magic
{
   class Program
   {
       static void Main(string[] args)
       {
           //global timing vars
           DateTime stopwatch;
           TimeSpan elapsed;
 
           //global container
           SLDN_ACCT.SoftLayer_Hardware[] hw;
 
           //replace with your username and api key
           string user_name = "Your User Name Here";
           string api_key = "Your Api Key Here";
 
           Console.Write("Establishing connection to SLDN service...");
 
           //time it
           stopwatch = DateTime.Now;
 
           //declare the services
           SLDN_ACCT.SoftLayer_AccountService acct = new SLDN_ACCT.SoftLayer_AccountService();
 
           //create an authentification object
           SLDN_ACCT.authenticate credentials_a = new SLDN_ACCT.authenticate();
 
           //assign credentials
           credentials_a.username = user_name;
           credentials_a.apiKey = api_key;
 
           //authenticate
           acct.authenticateValue = credentials_a;
 
           elapsed = DateTime.Now.Subtract(stopwatch);
           Console.WriteLine("done (" + elapsed.TotalSeconds.ToString() + " seconds)");
 
           //method 2
           Console.Write("Retrieving hardware using method 2...");
 
           //get time stamp
           stopwatch = DateTime.Now;
 
           hw = null;
           //attempt to pull a hardware list for this user
           try
           {
               acct.SoftLayer_AccountObjectMaskValue = new SLDN_ACCT.SoftLayer_AccountObjectMask();
               acct.SoftLayer_AccountObjectMaskValue.mask = new SLDN_ACCT.SoftLayer_Account();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware = new SLDN_ACCT.SoftLayer_Hardware_Server[1];
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0] = new SLDN_ACCT.SoftLayer_Hardware_Server();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0].operatingSystem = new SLDN_ACCT.SoftLayer_Software_Component();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0].operatingSystem.softwareLicense = new SLDN_ACCT.SoftLayer_Software_License();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0].operatingSystem.softwareLicense.softwareDescription = new SLDN_ACCT.SoftLayer_Software_Description();
               hw = acct.getHardware();
           }
           catch (Exception e)
           {
               Console.WriteLine("Exception encountered [" + e.Message + "]");
               hw = null;
           }
 
           cnt = 0;
 
           foreach (SLDN_ACCT.SoftLayer_Hardware server in hw)
           {
               try
               {
                   if (server.operatingSystem.softwareLicense.softwareDescription.name.ToLower().Contains("windows"))
                   {
                       cnt++;
                   }
               }
               catch (NullReferenceException)
               {
                   //ignore...this server has not 
                   //had the OS loaded on it yet!
               }
           }
 
           elapsed = DateTime.Now.Subtract(stopwatch);
 
           Console.WriteLine("done ("+ elapsed.TotalSeconds.ToString()+" seconds)");
           Console.WriteLine("counted " + cnt.ToString() + " MS Windows licenses");
 
           Console.WriteLine("\
Press  to exit.");
           Console.ReadLine();
       }
   }
}

You should notice right away all the references to SoftLayer_AccountObjectMaskValue prior to calling the "getHardware()" method. This is the object mask. Essentially we must create a new instance of each entity we want to include down the chain. Then when the target object is retrieved, in our case the hardware, all related objects which we have made room for will get created for that specific instance of hardware, assuming of course a record exists. You must instantiate each object down the chain. If you skip any link your result set will not conatain the property or method you were trying to get to. Some less structured langagues, like PHP, do not have this requirement. But with V3 and dot NET there is no getting around it. You're probably thinking this version of the code looks far more cluttered and is not as straight-forward to read. You're right. But I contend this is the electric screwdriver in the SLDN toolbox. See for yourself.

As you can see the connection overhead dropped in half, which is to be expected since we are only authenticating to half the number of services. But take a look at the second number, the number of seconds it takes to count the servers with Windows installed. It dropped from 20 seconds, to under 2 seconds. That's a 10 times speed gain. And wait there's more-- because we are only making one call to the SLDN application servers to retrieve those records what you see is what you get. Meaning you should not expect that time to increase noticeably whether you have a hundred servers or a hundred thousand servers! That my friend is the magic of V3. The power of ORM.

Following this article I will include the entire code base, in a combined application that lets you run the tests sequentially so you can see the amazing difference object masks make for yourself. In the second part to this article, we'll continue with the sample so if you download it keep it handy. In part two I'll discuss the next best thing to object masks—object filters. With object filters we'll be able to streamline this code even more. Until then…happy SLDNing!

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
 
namespace SLDN_Magic
{
   class Program
   {
       static void Main(string[] args)
       {
           //global timing vars
           DateTime stopwatch;
           TimeSpan elapsed;
 
           //global container
           SLDN_ACCT.SoftLayer_Hardware[] hw;
 
           //replace with your username and api key
           string user_name = "";
           string api_key = "";
 
           Console.Write("Establishing connection to SLDN service...");
 
           //time it
           stopwatch = DateTime.Now;
 
           //declare the services
           SLDN_ACCT.SoftLayer_AccountService acct = new SLDN_ACCT.SoftLayer_AccountService();
           SLDN_SVR.SoftLayer_Hardware_ServerService svr = new SLDN_SVR.SoftLayer_Hardware_ServerService(); 
 
           //create an authentification object for each
           SLDN_ACCT.authenticate credentials_a = new SLDN_ACCT.authenticate();
           SLDN_SVR.authenticate credentials_b = new SLDN_SVR.authenticate();
 
           //assign credentials
           credentials_a.username = credentials_b.username = user_name;
           credentials_a.apiKey = credentials_b.apiKey = api_key;
 
           //authenticate
           acct.authenticateValue = credentials_a;
           svr.authenticateValue = credentials_b;
 
           elapsed = DateTime.Now.Subtract(stopwatch);
           Console.WriteLine("done (" + elapsed.TotalSeconds.ToString() + " seconds)");
 
           //method 1
 
           Console.Write("Retrieving hardware using method 1...");
 
           //get time stamp
           stopwatch = DateTime.Now;
 
           hw = null;
           try
           {
               hw = acct.getHardware();
           }
           catch (Exception e)
           {
               Console.WriteLine("Exception encountered [" + e.Message + "]");
               hw = null;
           }
 
           int cnt = 0;
 
           foreach (SLDN_ACCT.SoftLayer_Hardware server in hw)
           {
               try
               {
                   SLDN_SVR.SoftLayer_Hardware_ServerInitParameters box = new SLDN_SVR.SoftLayer_Hardware_ServerInitParameters();
                   box.id = (int)server.id;
                   svr.SoftLayer_Hardware_ServerInitParametersValue = box;
                   if (svr.isWindowsServer())
                   {
                       cnt++;
                   }
               }
               catch (NullReferenceException)
               {
                   //ignore...this server has not had the OS loaded on it yet
               }
           }
 
           elapsed = DateTime.Now.Subtract(stopwatch);
 
           Console.WriteLine("done (" + elapsed.TotalSeconds.ToString() + " seconds)");
           Console.WriteLine("counted " + cnt.ToString() + " MS Windows licenses");
 
           //method 2
           Console.Write("Retrieving hardware using method 2...");
 
           //get time stamp
           stopwatch = DateTime.Now;
 
           hw = null;
           //attempt to pull a hardware list for this user
           try
           {
               acct.SoftLayer_AccountObjectMaskValue = new SLDN_ACCT.SoftLayer_AccountObjectMask();
               acct.SoftLayer_AccountObjectMaskValue.mask = new SLDN_ACCT.SoftLayer_Account();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware = new SLDN_ACCT.SoftLayer_Hardware_Server[1];
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0] = new SLDN_ACCT.SoftLayer_Hardware_Server();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0].operatingSystem = new SLDN_ACCT.SoftLayer_Software_Component();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0].operatingSystem.softwareLicense = new SLDN_ACCT.SoftLayer_Software_License();
               acct.SoftLayer_AccountObjectMaskValue.mask.hardware[0].operatingSystem.softwareLicense.softwareDescription = new SLDN_ACCT.SoftLayer_Software_Description();
               hw = acct.getHardware();
           }
           catch (Exception e)
           {
               Console.WriteLine("Exception encountered [" + e.Message + "]");
               hw = null;
           }
 
           cnt = 0;
 
           foreach (SLDN_ACCT.SoftLayer_Hardware server in hw)
           {
               try
               {
                   if (server.operatingSystem.softwareLicense.softwareDescription.name.ToLower().Contains("windows"))
                   {
                       cnt++;
                   }
               }
               catch (NullReferenceException)
               {
                   //ignore...this server has not had the OS loaded on it yet
               }
           }
 
           elapsed = DateTime.Now.Subtract(stopwatch);
 
           Console.WriteLine("done ("+elapsed.TotalSeconds.ToString()+" seconds)");
           Console.WriteLine("counted " + cnt.ToString() + " MS Windows licenses");
 
           Console.WriteLine("\
Press  to exit.");
           Console.ReadLine();
       }
   }
}

February 19, 2010

Object Masks and Filters in C Sharp