vSphere Sets the Bar Really High
One of the things that blew me away when I joined this company and started the Customer Journey project, was the incredible good will we have within our customers.
They really love our platform and what this company has done for them todate. Kudos to the engineers who build this great technology. The reason I bring this up is that today I was at the Burton Catalyst conference in San Diego and I attended John Hillmer’s (Northwestern Mutual)
presentation.
Northwestern Mutual is a good customer ouf ours and John talked about his experience with ESX, vSphere and View. One thing he said really stuck with me and I quote:
“We upgraded our farm to vSphere 4.0 wihtout even telling anybody. We did it as part of our montly patch management process”
Now, think about it… he upgraded a critical component of his production infrastructure from 3.5 to 4.0 (a major release) as part of their routine path management. This was not a service path, it was a major product upgrade.
And I quote again:
“We vMotioned all the VMs to a different cluster, upgraded to vShere 4.0, vMotioned all the VMs back. Everything worked perfectly”
It is stories like this that make me honored to work at Vmware.
vSphere sets the bar for the industry and the rest of us building products at this great company
Vittorio
Virtualization Journey on Forbes
Forbes just published my second article on Cloud Computing. They are forcing me to 700 words which is hard for me but it is a good forcing function to get to the point quickly 
I will still use this blog for more details
Vittorio
Virtualizing Oracle Databases – Customer Best Practices
In this previous post I talked like there are some customers that do start their virtualization journey virtualizing mission critical databases. They are the exceptions, not the rule but that post raised a lot of interest as many customers are looking at virtualizing mission critical application and databases as their next step in the virtualization journey.
So, I went back to one of these customers and I did an in depth interview with Douglas Babb, Chief IT Systems Architect, System Implementers, Inc, to get more information about his experience. It is a bit long but there is a lot of good stuff in here. If you go back to the key drivers of the virtualization journey, Doug is a good example of an individual that embodies two or the critical virtualization adoption elements: Confidence and Sponsorship as he is both very technical and experienced (confidence) as well as very senior and respected (sponsorship).
Q: Doug, a lot of customers are concerned about virtualizing production databases. What is your experience? Why virtualize databases?
Doug: We look at application candidates holistically from the app logic all the way to the databases, from a performance perspective as well as quality of service requirements.
Also, we make sure we deploy our databases in the virtual environment across the development lifecycle so that we have the same architecture and quality or service across all phases: development, test and production.
Q: When did you start virtualizing databases in your journey?
Doug: The first step was to ensure we had the framework for virtualization. We did not have any Enterprise Storage network, we had no fabric, we consolidated 6 data centers into two. We focused on building the framework, which was network, data fabric, storage and power distribution.
If we had started virtualization without have a stable data center, stable enterprise storage, and fabrics we would have failed.
Q: So, when did you start with the actual virtualization effort?
Doug: In the first phase of the project we moved our databases that were running on proprietary systems such as Unix, Open VMS and Mainframes to the x86 architecture.
Q: What was the technical reason why databases were a good candidate for virtualization?
Doug: Prior to starting this effort, we had been down 8 times in 3 months and the cost of just one of the application was 1 million per hour.
We were shocked by the cost of downtime. Just with the cost of salary alone, we paid for the virtualization effort in the first year.
I found that ERP techniques work well in large virtualization projects, and we took that Oracle Application Methodology (AIM) with short cycles step-wise refinements in order to perfect the process, so we used those type of ERP methodologies for consolidation efforts.
Q: If you had to help a customer virtualize databases based on your experience, what would the recipe be?
Doug: The first thing is to understand the performance of the application before virtualizing, so our concept is to create a jail that is a confined environment where we get quantifiable data as to how the application performs.
The next thing is to determine if the application/DB is a fit.
Q: do you have examples of configurations that you stayed away from?
Doug: We have not stayed away from anything so far other than this one application that does 3.5 trillion I/Os in 2.5 hours and we know that with VMware currently we can do 100,000 I/Os per sec per VM so that one was not a good fit.
Then we profile the application after virtualization. The thing that we are most concerned about is the user experience and so we have used multiple tools to measure it including an open source one called grinder http://grinder.sourceforge.net/
or Mercury test professional
http://en.wikipedia.org/wiki/HP_QuickTest_Professional or loadrunner http://en.wikipedia.org/wiki/LoadRunner
That is the metric that really matters.
We also capture the standard CPU/Network and I/O memory metrics.
When we go into production, we also keep the original application running and if the users say “it did not use to work that way”, we can point them to a working instance prior to virtualization and show them the actual data. Quantifiable data is hard to argue with.
Q: So, you are saying that applications and databases run better in the virtual environment from a technical perspective, tell me from a business perspective, what is the business justification for
Doug: In most cases we are 2% to 8% of the previous cost of hosting.
And even if you look at our virtualization journey, we have an application that we just refreshed and it is going to be 40% less even after saving 90%.
Q:Where does the cost saving come from?
Doug: All of the above: fewer physical servers, most application did not use de-duplication of backups, they did not use de-duplication of storage, we also use information lifecycle management where we only use high performance storage when required.
Q: So, when you move to the virtual environment you also take the chance to upgrade all the related components.
And we also remove things where possible. We consolidate, standardize, automate and repeat. And we don’t try to get it right the first time, we try to get it close, and the we iterate. Sometime we iterate 10-12 times before we go into production.
Q: oh yes, I am a big proponent of Agile and Scrum methodologies.
Doug: We really like design thinking and Agile processes. A lot of people think that they are only applicable to development but they work just as well for operations
We are always trying to use lean methodology http://www.lean.org/WhatsLean/
which came fro Toyota: Plan, do check act.
Q: Did you have to deal with concerns from DBAs?
Doug: Always.
Q: How did you address them?
Doug: Two nights ago a virtualized application started to double its response time. So the DBA sends out an email saying that something is going on with the virtual environment.
What we found by correlating the performance events from the physical server, from the storage, the operating system from Virtual Center, and then from Oracle itself we were able to determine that they just crossed the boundary and by changing to minor parameters in Oracle the system went back to normal. It had nothing to do with virtualization. It had to do with the fact that the cost-space optimizer crossed the boundary whereby a query just did not fit into memory anymore.
We constantly fight with this attitude that virtualization is evil because of all the mis-information that is out there.
A lot of it is also because of over-subscription, people who are reducing cost sometimes oversubscribe their environment to the point where performance suffers.
So one of the things that we do is we never over-subscribe RAM, the RAM is so cheap we chose not to over-subscribe it. We always buy servers with maximum RAM.
We are bringing our servers in with ½ a terabyte of RAM.
There is so much value brought by the VMware product ecosystem with being able to standardize your images and use Site Recovery manager and other flexible features… why over-subscribe something that is so inexpensive.
Q: Do you see DBAs embracing virtualization?
Doug: No. They understand why we do it but they feel like they are losing something. There is always the fear of loss, whenever something is taken away from you… if you can build a kingdom, if you can be in an isolated area you are more survivable and the economy, the way everything is working I think people feel better to have ownership of their entire environment.
Q: hug their servers…
Doug: Yeah. They want to hug it for sure. It is something especially for the DBAs, and especially for the FUD that has been spread in the industry about virtualizing databases.
Q: do you virtualize both Oracle and SWLServer?
Doug: We do
Q: How did you get over the concern about the Oracle support concern?
Doug: That’s a huge issue. For us, we have built into our ELA with both Microsoft and Oracle that if they want to continue working with us they have to support VMWare. That was the first thing we did. We reached out to our ELA team to ensure that there was verbiage in there, including RAC, that specified that in our environment we want to be able to run on VMware.
Since then, there has been a huge change with Oracle. With the exception of Oracle RAC everything is supported on VMware now. I think that a lot of this changed when they acquired BEA.
Q: What about other ISVs?
Doug: We had an instance with Cognos where they were going to deploy a new version (version 
and they said they would not support us on VMware. I excused myself from the room, I called their technical account manager who called the team at VMware and within 5 minutes there was an email to the Cognos sales person with the support statement.
Oracle, with the exception RAC we have no issues, with Microsoft we have no issues as far as supporting us on VMware.
Microsoft Exchange is the only Microsoft application that we have not virtualized, but we don’t have control over that environment currently.
Q: so, to recap, we said
1 – Measure everything before and after virtualization. Constantly measure.
2- Keep existing physical environment around for a while
3- Use HA expensively. We are also very interested in FT when it supports multiple CPUs
Thank you Doug!!
Virtualization Journey on Forbes
I have been quiet on this blog lately as I got busy with my new position at VMware. I also took a quick break over the 4th of July week. The pipeline of articles is in good shape though. I am very excited about finalizing an interview with a customer on how to virtualize Oracle databases. Stay tuned, I am just waiting for their approval.
In the meantime here is the first of a series of articles on the Journey that I am writing for Forbes
Enjoy
Vittorio
Product Design Rants – Airports
I have been on the road for 10 days across Europe and the East coast. Maybe it is because I am getting increasingly cranky, but these product design issues started getting to me… I have written about product management in the past and I wonder what the product managers behind these products/solutions were thinking.
here they go in order:
Check-in Kiosk – What’s your Reservation Code???
Here is a the welcome screen of a check-in kiosk in Europe
Really? You really want me to have that handy? When I travel, I know my last name and at best my destination… the reservation code????? Really?
The funny thing is that there was a clerk walking around and helping people to facilitate adoption of these automatic check-in kiosks and he told me “See, the Reservation Code is very important” at which I wanted to respond “Maybe to you and your developers, not to us users of this system”. I refused to look up my reservation code and went for a manned check-in station
Departure Time Table
The airports in Europe sort the departing time table by time of departure, not by destination like they do here in the US. It is annoying.
I do know where I am going, and I look at the departure table to figure out WHEN I am going. I don’t really care the the next flight to board is London if I am going to Paris. I just want to look for Pars and see when is the next flight and from which gate.
Is it just me??? You should see the amount of people staring at these screens desperately looking for their flights… They should just try both type of sorting, measure the average staring time and figure out which one is better. My money is on destination city sorting.
The question to me is not “When and where is everybody going and in which order?” but instead “When am I going?”
Installer-Centered Elevator Design
Here is the buttons of an elevator in the Frankfurt airport
I understand that when the builder installed this elevator the fact that it was servings floor 3 and 5 was very important. I do. But as a user the only thing i care is where is the launge and where is the exit. That’s all I want to know. The fact that they happen to be on the 5th and 3rd floor respectively, not very interesting or useful if those are my only two choice.
Ah… cranky traveling
Vittorio
I bought it, I *might need it*, why should I share it?
Last week I met with around 25 customers at a Cisco, NetApp, VMware event. A couple of highlights
Virtualization of Mission Critical Applications
When I asked the audience how many of them are virtualizing mission critical applications or databases 90% of them raised their hand. In fact a lot of the questions during the A&A session where around people and processes, cloud computing best practices, and other more in depth technical questions that told me I was looking at a sample of pretty advanced customers. This was refreshing.
As I often talked about on this blog, virtualizing mission critical applications is the second major step in the customer virtualization journey and although I still run into customers who are not there yet, the number of the ones who are squarely in this phase increases every day as more and more customers are getting into the resistance is futile stage
The second highlight was this quote from a customer who presented their experience. When talking about the resistance from the line of business about sharing infrastructure with other applications/LOBs the customer said that the typical attitude is
“I bought it, I *might need it*, why should I share it?”
Where do I begin???
This is a super common scenario in both private and public cloud computing environments and it points to both the need to consolidate more to achieve better utilization and lowering costs, as well as the ability to pool resources and allocate them dynamically to different applications based on load and service level agreements. This is all possible today with vSphere and partner solutions such as secure multi-tenant from Cisco, NetApp and VMware but the reality is that there is still a lot of education that needs to happen out there to get people informed about what is possible and letting them go of hugging their servers 
As I type this post, I am sitting in a room with one of our professional service gurus and he is telling me a story about this one company that is going un-named here which has a “Shared infrastructure Group”. What this group really does is to consolidate all their servers into the same “shared’ room… not quite the shared infrastructure required for cloud computing. It is more like share space and air conditioning.
Again, more education needed
Vittorio
Screwdrivers don’t Scale
I was on a panel at Interop with Randy Rowland and he told me this phrase that stuck with me
“Screwdrivers don’t scale”
He was referring to the fact that in cloud computing you cannot afford to have people in the middle of the provisioning, management and operation processes. Infrastructure and applications must be provisioned on the spot and the whole thing has to be able to scale up and down to keep up with demand. In other words, the level of automation required in a cloud computing environment (whether private or public) is at a completely different scale than in the past.
As I dive deeper in the Desktop Virtualization market and use cases, this phrase is even more applicable. Unlike the data center where you can at least walk around in a confined space with your screwdriver to rack servers, fix breakage and so on, with desktop and laptop computers the problem it is not the same. Offices are distributed and people travel, when something break it is always mission critical event and the end point not always easily reachable.
This is why thousands of organization are looking at desktop virtualization to address this problem and create a more centralized desktop infrastructure that offers better security, manageability and compliance while still giving end-users a great desktop experience. I will post more about these type of topics going forward.
I just wanted to share this phrase I learned from Randy and is now part of my standard cloud pitch
Screwdrivers don’t scale!!!
thanks Randy!
Vittorio
Cloud Panel at Interop
Last week, I participated in a cloud panel at Interop in Las Vegas with Simon Crosby (Citrix) and Randy Rowland (Terremark)
I mostly talked about the Virtualization Journey as a necessary condition to build a private cloud.
Based on what I leanred from customers, I positioned the journey as the mechanism to provide existing applications with the core characteristic of cloud computing, that is
- Abstraction
- Pooling
- Elasticity
- Management Automation
As I discussed in multiple posts in the past, from a business perspective the journey starts with consolidation and savings, it evolves around business continuity and quality of services and it lands with business agility. From a cloud readiness perspective, it starts with abstration and pooling, then moves to elasticity and management automation.
Here is the video, the panel starts around minute 9
Vittorio
Business Production Drill Down – Part 2 – Key Elements
In previous posts, I covered the 3 main stages of virtualization adoption, I drilled down into the IT Production Phase and gave an overview of the Business Production phase.
The transition from IT production to Business Production is the most critical part of the virtualization journey and many customers are dealing with it right now. Let’s dive deeper into it.
Business Production Key Adoption Elements
First, lt’s take a look at the characteristics of the key journey adoption elements in this phase
Confidence
Confidence refers to the level of competence and experience that the IT team has around virtualization technology. While in the IT Production phase we describe confidence as reactive, in the Business production phase confidence is better described as selective. Wha I mean is that the team is not yet ready to virtualize everything that comes their way but they are getting there. They selectively apply virtualization to increasingly more challenging workloads.
In IT production there is a big inflection point when the team learns how to use vMotion as it unlocks their ability to perform maintenance without downtime and start demonstrating one of the most tangible and visible advantages of virtualization: abstraction and workload mobility.
In Business production there are multiple confidence inflection point as the team learns how to deploy products such as SRM for disaster recovery, HA and FT for business continuity and quality of service, DRS for automatic resource management, and VMWare view for Virtual Desktop deployment. These are the products that build out the platform needed to properly support the virtualization of business applications and databases.
Sponsorship
This is the stage of the journey where the sponsorship level for the virtualization initiative is most critical to adoption and success. As I discussed in this other post, there are different stakeholders at the table during this phase such as Application Owners and Database Administrators. To properly win their concerns and get to the resistance is futile stage, IT teams have to get application owners comfortable with the technology. This is typically achieved in one of 2 ways
- Address their Technical Concerns
Show them their apps/DBs running virtual, set up a demo of vMotion, HA, FT and so on. Show them that apps run better in the virtual environment. - Go over their Head
Seek a higher level of sponsorship in the organization, ideally the CIO.
Point 1 above is the slower path. You look for a suitable application, you buddy up with the application owner, get him/her comfortable with demo and POCs, you go ahead and virtualize the app, measure the results and evangelize the success. Ideally, turn the app owner into a virtualization evangelist. We have seen this over and over again.
But… if the IT operation team is very comfortable with virtualization technology and with the management products mentioned above (HA, FT, DRS etc), the best and fastest way to achieve both 1 and 2 above it to lead the process from a business perspective, turn it into a business case. Depending on the business application at stake and the requirements and constraints around it, the business case can be made around business continuity, better SLA, faster provisioning (or even self service where applicable), faster time to market for new versions combined with the usual cost saving from hardware consolidation.
This is how you get the attention of the senior management team which in turn will give you air cover to win over the application owners and database administrators.
Value
Business value is always a key to drive any sort of transformation and technology adoption. As mentioned above the key in this phase is to switch the value proposition from consolidation and cost saving to business continuity and better service level for business applications. We have seen this in every customer that is successfully going through the Business Product phase. Cost saving for consolidation is “last decade” value prop to them. It is taken for granted. It becomes all about building a better environment to run Virtual Desktops, Business Applications and Databases.
I always use this example that stuck with me as I was interviewing our customers on their journey last year. After meeting with 4 customers that were squarely in the IT Production phase, I met with this IT professional who told me, and I quote: “I am virtualizing a mission critical application that manages 40 billion dollars in revenue every year and I am doing it because as long as that application runs physical, I don’t sleep tight”.
Back then, I did not realize exactly what I was looking at. Now, after more than 50 in-depth customer interviews I know. I was looking at a customer in the middle of the business production phase with a very high level of confidence and clearly the right level of sponsorhip. He was even willing to go down to a 1 to 1 consolidation ratio (in certain cases) to get the benefits of the virtualization platform. He was clearly past the consolidation stage.
Do you see this?
If you are a customer of ours and you are not yet thinking this way, it means that you are not there yet and you should look at the key elements above (confidence, sponsorship and Value) and see which one you should work on to move forward.
Do you have the right experience but you are getting push back from the app owners? From the Top? Do you have the right mandate but you don’t have the confidence to tackle business applications? Do you have a formal way of tracking the business value you are deriving from your virtualization projects? Is it just around cost savings? Are you reporting it routinely to your management team?
We are rolling out tools to help our field and our customers answering these questions and in the next few posts I will share some of the business production obstacles and best practices that we learned form our own customers
Ciao
Vittorio
Don’t Quit in the Dip!!!
This blog is about the virtualization journey but I sometimes write about company culture, hiring, teams and things that work and don’t work for me on the job…
…like in this post.
I have been reading Seth Godin’s books lately… I really like his stuff. An inspiring mix of common sense and outrageous out of the box thinking.
The other day I was playing piano with my 6-year-old son and he was about to quit in the middle of learning a new song, so I told him about The Dip. You know you are facing a good idea worth spreading when my 6 years old turns around and explains it back to me like this:
I think it stuck with him…
…and me
Ciao
Vittorio
