If you are reading this article, chances are that you are one of the 50% of x86 architectures that (according to Gartner) will be running in virtual machines by 2012. Organizations of all sizes are exploring virtualization because it enables them to increase server utilization and reduce hardware costs and complexity, as well as the overall total cost of ownership. Here at Ipswitch we haven't purchased a single server in almost three years now, so we are going to walk you through the steps that we have taken to build and manage our internal cloud. As a full disclosure, we are the makers of WhatsUp Gold, a suite of IT management solutions, but the checklist and tips that we are sharing in this article are vendor-neutral.

Prep Word: Know your infrastructure, so you can identify what to virtualize
Proper planning is an essential part of any virtualization deployment. It's critical to get a good understanding on not only how your infrastructure is physically setup but logically as well. For example, are you vlaning? Are there security concerns? Will you be isolating a management network, backup network, etc?

DO #1: Discover, map and create an inventory of your network resources
A layer 2/3 discovery tool will identify not only the physical server that you have deployed in your environment, but how everything is connected down to the individual port. This process is actually an eye opener for many organizations, because you can discover pieces of unaccounted hardware, or even inter-device connections that you didn't even know they were there.

Figure 1: Use layer2/3 discovery tools to document your network

Once you have a hardware inventory in place, you can use it to document your network for auditing purposes or simply to try to reduce costs by repurposing or moving to the cloud underutilized resources. And since networks are "living entities", don't forget to re-run your discovery on a periodic basis (more frequently if you are part of a larger organizations, or if you have distributed network locations) and export this information into other tools such as Visio or Excel for continuous use throughout your organization. Next step in the process: monitor key performance metrics across all your hardware.

DO #2: Consider moving the servers that are clearly underutilized to the cloud
When exploring the possibility of virtualization or planning a virtualization pilot, the first thing to decide is which servers should be virtualized. Typically, supplementary servers like DNS, domain controllers (in small or medium size Active Directory environments), DHCP, and file and print servers are great candidates. These types of servers generally do not take full advantage of the hardware on which they run, so they are perfect targets for virtualization. Web servers, mail servers, or small database servers can also be very good targets. More complex servers such as application servers or database servers can also be virtualized, but this scenario requires very careful planning. Remember to baseline key servers that will be moving to your cloud environment. But also take into consideration peak times of performance. A good approach is to measure processor utilization, memory utilization, storage, network usage and disk I/O individually on potential virtualization targets for benchmarking. If a server is clearly "overloaded" to begin with, it shouldn't be virtualized.

Figure 2: If a server is clearly overloaded, it shouldn't be virtualized

DO #3: Select your vendor- yes- there are other options other than VMware
Current top virtualization players - the ones that provide their own hypervisor - are EMC/VMware, Microsoft, Citrix/XenSource, and Virtual Iron. According to IDC, EMC/VMware held 65.6 percent of the $1.045 billion worldwide virtual machine software (VMS) market in 2006, so if you are looking into virtualization now, chances are that you are exploring VMware. VMware products - from a free VMware Server to VMware Infrastructure 3 enterprise solution - provide full/native virtualization by inserting a thin layer of software directly on the computer hardware or within the hosted operating system. In our local cloud we chose to go with VMware for our virtual environment, but there are plenty of options out there, and Microsoft Hyper-V seems to be rapidly picking up market share. Through our deployments we chose to keep a consistent naming convention on all aspects ranging from vlans to LUN and volume names.

DO #4: Review your virtualization requirements- get all the stakeholders together
Virtualization is a disruptive technology driving the transformation of IT, which means that there are some risks associated with it. After all, adopting a virtual model is something that impacts not only IT, but the business owners of the applications that become part of this virtual environment. And depending on the size of your organization, a virtualization effort could involve larger teams across multiple business units! Therefore, it is important to prepare a Virtualization Overview document containing: goals and objectives - in terms of business needs and deliverables - success criteria, and listing constraints and high-level risks.

This document should be reviewed and agreed upon with all key stakeholders (sponsoring, business and technical) during a virtualization requirements review meeting. Couple of quick pointers to cut-down the number of required follow-up meetings: Make sure business owners are clear on what the meeting will entail, so they can involve team members as needed, and keep meetings brief and to the point. You should also discuss with key business owners future needs and long-term expansion goals as well. For example, some simple questions that worked well for us were:

• How many people are currently using the system
• What kind of growth do you see in the next 6 months, year, 3 years?
• Will there be additional add-ons or features which will have to be installed?
• Will these add-ons require additional connectivity to external applications, file shares, databases, etc?

Once the Virtualization Overview document is approved by all stakeholders, you can move forward with confidence that everyone has had their expectations set relative to the project's purpose, parameters and what defines project success.

DO#5: Move to the cloud
Once you have identified which resources will be virtualized, you should update firmware on all of your hardware prior to the deployment, including switches, storage devices, servers, etc. Also, keep in mind that applications that regularly talk to each other and are on the same subnet can be configured to run on the same host through VMware DRS rules.

Some quick pointers that worked well for us:

A good practice to follow is to confirm that your processor settings are identical on all host servers. If not you could run into issues later on when trying to move VMs from one host to another via VMware vMotion. For our storage needs we chose to go with ISCSI because we felt we got more bang for our buck. However, in doing so keep in mind that you may want to isolate your normal ISCSI traffic from your vMotion traffic.

Before formatting your vmfs Disk/LUNs, take into consideration how large the drives on your servers will be. By default VMware formats in a 1 MB block size which only allows for up to 256G drives. We ended up bumping ours up to 8MB which allows for up to 2TB drives. Not that we necessarily need them that large, but at least we won't have to worry about it in the future.

Don't forget to utilize resource pools especially if you have highly critical systems running with less critical systems. You want to ensure your most critical systems have priority to hardware resources.

DO #6: For a more accurate measurement of the impact of virtualization in your environment, look beyond virtualization vendor reports
Now that you have virtualized the right resources in a pre-production environment, you should start monitoring your infrastructure - network devices, servers, applications and virtual resources - right away to detect any performance degradation or any other type of problem. Remember, virtualization vendors have done a good job in terms of measuring the additional overhead that virtualization introduces to individual physical servers for key performance metrics such as CPU usage, memory, network, or timing. While these metrics are a good reference to start your planning efforts, they don't provide a complete and comprehensive indicator of the overall impact that a virtualized underlying infrastructure will have on the end-to-end performance of your application.

Performance monitoring should be a part of every server virtualization project, both before virtualization takes place - when you have your native servers in place so that you can create a baseline against which to measure later - and after virtualization to help you ensure that you did your job right.

Build your infrastructure management strategies
Once you move into production, you need to define the virtualization monitoring strategy that is appropriate for your organization. This is a key area to minimize risks and ensure overall project success. Bear in mind that as mentioned earlier, apart from its multiple benefits, the dynamic nature of virtualization introduces a new set of management challenges such as how to:

• Discover and document both physical and virtualized resources
• Maintain an accurate physical to VM mapping at all times
• Reduce the risk of virtual sprawl, or the rapid proliferation of virtual servers
• Ensure that applications perform at optimal pre-virtualization levels
• Integrate alerting, escalation and response procedures across your entire infrastructure and VMware and non-VMware administrators
• Rapidly locate the true root cause of a performance problem

To address these challenges, and efficiently manage dynamic virtualized infrastructures, you need to discover, map, monitor and control your network devices, server, applications, virtual resources, network traffic and events files from a SINGLE console. After all, if you are using multiple management consoles you will have to examine multiple reports and interfaces to correlate information across hundreds of virtual machines and physical servers. And having to perform manual tracking tasks can severely hinder troubleshooting efforts and increase MTTR (mean time to resolution), making your job much more difficult and time consuming.

Here at Ipswitch, we are using WhatsUp Gold and WhatsUp Virtual to manage our entire environment from a single console. As you can see here, we monitor those key parameters that you would normally monitor like CPU, Interface, Memory, and disk utilization on the VM and host level.

Figure 3: Look for ways to monitor physical and virtual resources from a single console

The great thing about monitoring disk utilization on the host level is that you can effectively protect yourself against snapshots growing to the limit of your volume. In addition to alerts at the physical layer, we are alerted on specific VMware problems like variety of migration errors, clusters being overcommitted, insufficient failover resources, when a general VM error, or host warnings/errors are triggered.

Figure 4: On-going discovery & mapping is key to prevent virtual sprawl

That way, we can reduce the risk of virtual sprawl, since we can quickly access the number of VMs in our environment, understand physical to VM mapping, and be able to track changes to those configurations on an ongoing basis.

If you choose to use snapshots keep a close eye on them. Don't let them hang out there for weeks on end. Otherwise you may find yourself coming in one day finding that you've ran out of space on your datastore. Since the datastores total usage can't be monitored through a VM, you should configure your monitoring solution via the VMware API or SNMP to monitor its capacity.  We typically configure our monitors to alert us on 80-90% utilization.

Compliance concerns and how to address them
As mentioned earlier, there are some risks associated with virtualization, such as management complexity, security (new threats and penetrations, lack of controls); and virtual machine proliferation. Achieving compliance in a virtual world will also be very different if you using virtualization in a testing/pre-preproduction environment, or if you are moving mission-critical applications to your internal cloud. In those environments where the back-up data center used for disaster recovery and business continuity purposes is being moved to the cloud, it is very likely that some customer data, including potential data that falls under compliance regulations, is stored in virtual test environments. One of the first questions to ponder is, how will you audit for compliance if you don't know where all your production VMs resides, or even how many you have? As IT becomes more and more dependent on virtual environments, it is becoming increasingly more critical to be able to quickly access the number of VMs in your environment, understand how they are configured, and be able to track changes to those configurations on an ongoing basis. You should follow the motto that "a virtual server is still a server", with all the policies and security and management concerns of a physical box

Lessons learned and DON'T's
Virtualization is no longer a cutting-edge technology; in fact, it is rapidly becoming mainstream worldwide. While virtualization provides tremendous advantages and is a technology that you should investigate and adopt, it can have an impact on the overall performance of your mission-critical applications. Therefore, when working with virtualized infrastructures or when testing applications deployed across virtualized infrastructures, don't forget this checklist that we have used here at Ipswitch:

1. Properly scope your virtualization project, objectives and goals, as well as success metrics
2. Get buy-in from all key stakeholders (sponsoring, business and technical)
3. Put a policy in place for how physical to virtual migrations are to be executed
4. Put strict controls in place to allow for VM creation
5. Closely monitor physical resources, cpu, interface, memory, and disk utilization.
6. Use Resource pools allowing production servers to take precedence over hardware.
7. Look for a management tool that offers single console monitoring across network devices, servers, applications, virtual resources, network traffic and events and log data
8. Inventory your physical and virtualized environments on an on-going basis
9. Overlay compliance and data security policies and organizational and management requirements
10. Don't forget to monitor, alerts and report on critical vCenter alerts, there are actually 300+ vCenter events

Good luck in your virtualization endeavors!