A Story about Walt and The Vblock Revolution

Thu, Jul 15, 2010

General

I submit that CIOs today are looking for return on invested capital (ROIC) with an ever increasing “impatient” pace and I agree. Return on Invested Capital is a financial measure that calculates how well a company generates cash flow relative to the capital it has invested in the business.

When referencing ROIC you are in essence measuring how well a company is using it’s money to generate returns. I say why can’t or wouldn’t we use that as a measurement in our calculation of the success of a investment being made in the data center. I ask because we have all seen organizations invest significant capital in infrastructure only to see that equipment sit in boxes or in a lab doing burn in or waiting for some other integration component to be certified/tested for production.

The time this investment sits is time that it is not producing a return for the business. The objective should be to compress this timeframe so that the projects a CIO and their staff have planned can provide the business function they intended to serve. This is why I have taken the ROIC measurement and said, lets apply it to technology, specifically the data center and use it as a factor of determining the standard for success.

To articulate my point relating it back to an environment you may be familiar with, I need to introduce my long time friend Walt. I have worked with Walt for many years. Walt and I have probably worked together on 25 million dollars worth of IT projects over the last decade. Walt is a brilliant IT administrator with skills in networking, compute, applications, virtualization, storage and security. Walt knows his infrastructure well and is one of a handful of guys that his CIO goes to for projects where new services are being rolled out to his organization.

Walt is highly experienced in his trade and genuinely knowledgeable about how each of the technologies, interact. In the early years Walt and I had had some very spirited discussions and I cautioned myself from loosing my mind and telling him exactly how frustrated I was with his insistence that nothing my company made or any other for that matter could satisfy his requirements. Walt’s position was that the infrastructure he had was good enough because he and his staff understood how to support it. Walt wasn’t against change, he was against blind rapid change with no experience or trust that there was sustainability and support. He was also constantly implementing new projects because of the organizations rapid growth and lifecycle changes on products that were already in production but he was cautious to introduce technologies outside of a standard.

In the first years of our relationship, Walt regularly had equipment that was End of Life and End of Support (EoL and EoS) status. He was only replacing that gear months after EoS. That practice ended quickly after an outage of a EoS piece of equipment caused an extended outage where leadership said, we won’t do that again. From 2002-2006 Walt actually purchased new technologies 6-12 months following initial release, depending on the needs of his organization. After purchase, implementation would lag 3-6 months and in some cases greater. In the last 4 years virtualization has become a mission critical component of Walt’s data center and purchases of infrastructure have become synchronized and in some cases technologies are being purchased only months after initial release. Those purchases include, network, storage, compute and virtualization releases.

Prior to virtualization being a mission critical component to Walt’s data center, purchases were staggered based on lifecycle of the individual product. Today Walt’s organization makes investments based on the lifecycle of the entire virtualized solution. Walt’s challenge is manpower and time against the backdrop of the thoroughness needed to establish specific organizational standards for the solution being implemented. Solutions which are implemented must be classified as production and configurations must be standardized to enable mission critical operation. This production “mission critical” classification is beyond any individual product, it is a solutions classification that encompasses all aspects of function from operation to change/modification to troubleshooting and ultimately system recovery.

This is the lifecycle of operation that Walt is concerned with. Showing a demo of how something works never was good enough for Walt. Walt wanted to see it break because he and his staff needed to know how to fix it when it did break. Walt was reluctant to introduce new Vendors into production workflows without some experience with how well support and parts replacement processes would function.

Virtualization changed Walt’s world, in many cases it enabled him to do more things quickly. In other cases it created a situation where his ability to realize a quick return on the invested capital was impossible. Once the infrastructure was built, Walt could respond quickly, but it took significantly greater time to production classify the new platform because of so many new components coming online at once.

Now, Walt wasn’t focused on ROIC and his CIO wasn’t either, at-least in a form they realized. I remember numerous meetings where we would meet with the CIO and he would ask when the new virtualization project was going to be operational. Lets define what their most recent virtualization project was to get an idea of the scope.

Virtualization Project

  • Compute – New Generation Servers
  • Virtualization – vSphere 4 (current production is ESX 3.5)
  • Network – New 10GbE Top of Rack Switches (current production is End of Row 10GbE)
  • Storage – New Generation Storage Processor, New Disk Types and Size
  • Application – Exchange 2010, Small SQL 2008, Virtual Desktops

In the past Walt would introduce new servers or switches to address smaller tactical needs in the data center with a server here or a switch there. The point is that the implementation of new technologies in the past was staggered versus all at once. This gave Walt and his team time to dedicate brain cells to operationalizing the new component.

Virtualization synchronized the technology spend and now Walt is building platforms inclusive of Compute, Network, Storage & Applications. He must learn and operationalize all new systems, applications and equipment while delivering the same or greater quality that he did when he was just focused on a single new technology.

Organizations which have dove into virtualization all see their technology investments cycles synchronizing because the solution is a platform which is dependent on other components to function more so now than ever before. If your organizations technology investment life-cycles have synchronized and you are implementing these solutions it is taking you more time to operationalize the entire solution because of the interdependencies of the solution as a whole.

This brings us back to Return on Invested Capital. When Walt’s CIO was asking him when the new virtualization solution was going to be operational, he was asking when the investment he approved was going to provide a production service back to the business so that he could communicate back to his leadership that the capital was at work.

A CEO in many cases doesn’t draw a correlation between the $150K firewall, IT invested in versus the $500K VMware project. The firewall was purchased and installed in a 1 month period with no professional services. The VMware project has $100K of professional services and it is apparently going to be difficult to complete in 6 months. In many executive minds the time to operate versus investment is out of sync for these projects. I submit that they are right.

They are right because IT staffs and integration partners are focused on the wrong thing. They are focused on the operational integration of numerous components of a solution that often have too many knobs to tune. Engineers circle on the best options and time circling is directly proportional to the time it takes to implement the solution. That time is exaggerated exponentially when technologies are either new to the market or new to the engineers working on the project. In many cases people are learning as they go, as Walt has done many times over the last decade. What paralyzes us all is that engineers (like myself) often teeter between choosing options that provide the highest performance versus stability. I was a fan of Home Improvement (TV Show), I believe in “more power”, faster is better. However, we all know that faster sometimes isn’t the most stable.

This is not to say that it is any IT staffs or integration partners fault. They are simply doing what is required to do in order to get the job done. In many cases they do it well but there are elements of the implementation timeframe which simply cannot be compressed because there are far too many unknowns. This is the manufacture of the components fault not the customer or partner, plain and simply put. After all don’t we as manufactures always push customers to buy the latest and greatest?

Enter the Vblock Revolution. Vblock is a pioneer in the technology industry where 3 industry leading companies (Cisco, EMC & VMware) have came together to make Walt’s and partner integrators lives easier. We do the integration and testing work, determine firmware, configurations and operating systems (on the infrastructure components) and sell all components as a single integrated solution. It is not a model airplane were assembly is required. Vblock is a fighter aircraft with your own custom cockpit and choice of engines and weaponry.


The intent is to quickly enable the deployment of that virtualization platform so Walt and his partner integrator can immediately go to operationalizing and building out Virtual Machines to support the business. Oh and I forgot to say that VCE has staffed the support organization with experts from all three principal companies, providing a single support number for Walt or the partner to call.

Walt and his partner now spend time on automation, provisioning and security policy, not integration and infrastructure deployment customization. Walt was previously spending time testing and building the infrastructure, playing custom integrator. In some cases decisions Walt or a partner were required to make for certain elements of Walt’s environment went against the best practices of one manufacture or another. This put the organization in the unenviable position of having a unique configuration with potentially limited support by one or all manufactures.

The Vblock architecture has made this dilemma go away for Walt. He knows that ordering a Vblock equals pre-determined configurations based on best practices which all three manufactures have jointly tested and support. Walt is not spending endless hours opening 4 different support cases with various support engineers making ready, shoot, AIM troubleshooting suggestions. I am not taking shots at any independent support organization because they all have the same issue, if they don’t have visibility into the entire system, they can only suggest configuration alternatives to solve the problem the system is experiencing based on their view of that problem. This is the revolutionary aspect to Vblock, it is viewed, supported, roadmaped and exists as a integrated product with certain options for configuration.

Now every criticism I ever had for Vblock and those I have ever heard is that its sizing guidelines or form factor doesn’t fit every customer situation. After my own education and understanding of the solution I have a new opinion. It is that Vblock simply can and does fit most any situation you can throw at it. There are always corner cases to the extreme but I am comfortable in stating this because of the following logic.

When you buy just about any solution, outside of the technology industry you might have the option to customize it but you are selecting one model versus another based on price and features to generally accomplish a required function specific to your need.

For example, I just bought a new coffee maker. It has 24 hour advance programing, I would have preferred a weekly schedule but 24 hour will do. I did like the thought of the shower head feature which evenly distributes water over grounds but couldn’t really tell the difference between coffee made with that feature versus one that didn’t. I assume that if I were to take the technology logic I would have asked for that feature to be removed because I didn’t really need it and it would save me some amount of money. In general coffee makers or products that fall into this sort of investment category provide a specific set of features, driven by a particular market segment and there are limited options when it comes to customization. There are customization options, there just aren’t a great number of them. This type of analogy is synchronous with the logic of Vblock type 0, it is designed to not be tweaked and tuned but more specifically fit a segment of the market with smaller requirements and ultimately fewer customization options. It definitely has options, just fewer of them.

Vblock type 1 and 2 are designed for larger infrastructures and lend themselves more to broader customization. To emphasize this example, about 8 years ago my wife and I built our dream house. That house was a custom home and more on the order of magnitude of what smaller companies may invest in their data center.

The custom home building process was an exciting one, decisions to make everywhere, yet all decisions were derived from standards the builder had established. Any deviation from the standard was okay but we had to go through a process of making sure it worked for the design of the house. I want to be clear, we chose the floor plan, the size of the house and what we wanted the style of the outside of the house to look like. The standards were components of the build from types of flooring, molding, lights, ceiling height, outside stairway to basement, spiral staircase etc..

Many items we accepted the standards, other items were logical upgrades “easy decisions”. Each Vblock has these sort of minor upgrades which are functions of additions or subtractions of certain components. In the case of subtractions our builder would not let us build a house without a roof, we won’t let you build a Vblock without one either.

At one point in our building process we decided we needed a three car garage and the decision to have that size of garage dictated a slight change to the roofline of the home. A little later in the process we decided that the powder room on the first floor was a little small and it needed to be bumped out. This change required all sorts of interesting changes to the house, foundation and roof. All in all the house was a perfect example of the solutions we are trying to build today in many data centers.

Data Centers and the infrastructure placed in them must be built proportional. You want a bigger garage, we need to expand the roof line. You want a bigger room on the first floor, we need to expand the foundation to support that room. If you didn’t take this approach you would have custom homes built that literally would fall over because of rooms that weren’t proportional to foundations and roofs. Now you can take this example to an extreme and tell me that there have been houses built on the edge of cliffs with balsa wood and they worked fine. I am sure they did but they weren’t built in mass, and if the wind blows too hard one day, that house is gone.

In the world of our data center the example may be a few hundred users and the apps they are accessing have some sort of event which spikes utilization and if the infrastructure is not designed proportionally to handle the spike, you could tip the entire thing over. The Vblock design principles are to simply avoid that scenario.

So for lack of a better description Vblock is a custom home and the choices you make for your environment are variables between the minimum and maximum configs within each Vblock. You may choose to upgrade from a standard, and doing so may dictate a change required elsewhere. This is to ensure architectural stability and guys like myself from Cisco (Solution Architects) and vSpecialists from EMC play the role of architect to make sure that your data center is architecturally sound.

It is my belief that we have entered a new day in consumption of data center architectures. That consumption is driven by the business necessity to establish a RAPID Return on Invested Capital (ROIC). Vblock is the architectural answer to the integrated virtualized data center problem. IT Administrators and the CIOs they work for understand this and welcome the approach. I for one am excited to be apart of a new conversation that spends less time talking about what and if and goes directly to how and why.

More thoughts and ramblings soon,
Trey


, , , ,

2 Responses to “A Story about Walt and The Vblock Revolution”

  1. Todd Williams Says:

    Interesting use of ROIC, I hadn’t heard the term but it definitely makes sense. Quicker to production is an absolute must today…..

  2. Jay Cuthrell Says:

    Taking the construction metaphor a bit further…

    My first thought reading through this was the market for prestressed concrete forms; specifically, prestressed concrete beams

    a) predetermined stresses
    b) quantified services load capability
    c) engineered composites of standard materials
    d) mass production consistency
    e) ubiquitous use

Leave a Reply