ESX Partitioning

Reference

• http://www.vmware.com/pdf/vsphere4/r40_u1/vsp_40_u1_esx_vc_installation_guide.pdf
• http://vmetc.com/2009/07/22/best-practices-for-vsphere-esx-4-service-console-partitions/
• http://www.yellow-bricks.com/2009/05/27/partitioning-your-esx-host-part-ii/
• vSphere documentation with all the kickstart params and definitions: http://pubs.vmware.com/vsp40/wwhelp/wwhimpl/common/html/wwhelp.htm#href=install/c_scripted_installation_of_esx.html&single=true
• You’ll probably be most interested in this page: http://pubs.vmware.com/vsp40/wwhelp/wwhimpl/common/html/wwhelp.htm#href=install/r_ks_commands.html#1_16_9_17_1&single=true
• Setting up a VMKcore partition to capture purple screen output:

  http://kb.vmware.com/kb/1000328

• Also, Mike La Spina has a great post for kickstart sample for vSphere ESX 4: http://blog.laspina.ca/ubiquitous/automating-vsphere-esx4-host-installations
• http://blog.colovirt.com/2009/08/06/vmware-vsphere-esx-4-server-partitioning/

Kickstart Example:

#System bootloader configuration

#bootloader –driveorder=/dev/sda –location=mbr

#Disk partitioning information

part /boot –fstype=ext3 –size=250 –ondisk=/dev/sda

part :storage1 –fstype=vmfs3 –size=10000 –grow –ondisk=/dev/sda

part none –fstype=vmkcore –size=110 –ondisk=/dev/sda

# Create the .vmdk for the cos on the vmfs partition.

virtualdisk esxconsole –size=7712 –onvmfs=$host:storage1

# Partitioning the cos virtual disk.

part swap –fstype=swap –size=800 –onvirtualdisk=esxconsole

part /var/log –fstype=ext3 –size=2048 –onvirtualdisk=esxconsole

part / –fstype=ext3 –size=3030 –grow –onvirtualdisk=esxconsole

Run Windows 7 Desktop from your iPad

Introduction

Ingredients:

• 1 iPad (either wifi or 3G will be fine)
• 1 Bluetooth Keyboard
• Wyse PocketCloud App
• Wyse PocketCloud Windows Companion application
• Cisco PCF file (needed for Secure Enterprise VPN View use)

Directions:

WARNING: Be sure to read entire recipe prior to beginning work. Failure to follow directions could cause your expense report to be rejected by the finance department.
NOTE: We take a bottom up approach. This is less about instant gratification but ensures your first experience is successful.

1. Configure your Windows 7 Desktop to be accessed via RDP from the Internet.
2. This can be done at home on a physical machine.
3. Most businesses will choose to do this with Virtual Machines hosted on VMware ESX and brokered by VMware View 4 Manager for controlled access from the Internet with RSA 2 Factor Authentication One Time Password Tokens. Be sure the ESX hosts are protected with a tool like HyTrust.

1. Use the iPad as a remote thin terminal screen.
2. Use the bluetooth keyboard to give you a real computer like experience.
3. No mouse? No Problem – just use your fingers to click, double-click, right click, select, copy and paste, etc.

References:

• Get the Wyse PocketCloud App from the Apple App Store. Many companies will simply purchase a $50 Apple iTunes Gift card and have their employees expense it. Learn more from this this web site:
  http://www.wyse.com/products/software/pocketcloud/index.asp
  http://itunes.apple.com/app/wyse-pocketcloud-remote-desktop/id326512817?mt=8
  Updated: May 13, 2010
  Current Version: 1.3.78
  Size: 4.2 MB
  Full screen mode supported for both iPhone and iPad (removal of iPhone/iPad status bar).
  Application fully supports all device orientations.
  When connecting at iPad native resolution, screen auto-locks for ease of use.

• Wyse PocketCloud Windows Companion:

  http://www.wyse.com/supportdownload/PocketCloud/PocketCloud%20Windows%20Companion.exe

• Bluetooth keyboards are pretty cheap now. Models are available from Apple, Logitech, and Micro$oft:
  http://www.engadget.com/2009/12/16/microsoft-bluetooth-mobile-keyboard-6000-the-perfect-travel-key/
  http://www.sonyinsider.com/2009/10/25/vaio-bluetooth-keyboard-vgp-bkb1/
  http://www.logitech.com/en-us/keyboards/keyboard/devices/3848
• News reports on iPad and Windows 7:

  http://www.crn.com/mobile/224201173

http://www.vmware.com/pdf/vsphere4/r40_u1/vsp_40_u1_availability.pdf

Network Isolation Addresses

A network isolation address is an IP address that is pinged to determine if a host is isolated from the network. This address is pinged only when a host has stopped receiving heartbeats from all other hosts in the cluster. If a host can ping its network isolation address, the host is not network isolated, and the other hosts in the cluster have failed. However, if the host cannot ping its isolation address, it is likely that the host has become isolated from the network and no failover action is taken.

By default, the network isolation address is the default gateway for the host. There is only one default gateway specified, regardless of how many service console networks have been defined, so you should use the das.isolationaddress[...] advanced attribute to add isolation addresses for additional networks. For example,  das.isolationAddress2 to add an isolation address for your second network, das.isolationAddress3 for the third, up to a maximum of das.isolationAddress9 for the ninth.

When you specify additional isolation address, VMware recommends that you increase the setting for the das.failuredetectiontime advanced attribute to 20000 milliseconds (20 seconds) or greater. A node that is isolated from the network needs time to release its virtual machine’s VMFS locks if the host isolation response is to fail over the virtual machines (not to leave them powered on.) This must happen before the other nodes declare the node as failed, so that they can power on the virtual machines, without getting an error that the virtual machines are still locked by the isolated node.

For more information on VMware HA advanced attributes, see “Customizing VMware HA Behavior,” on page 26.

das.isolationaddress
Sets the address to ping to determine if a host is isolated from the network. This address is pinged only when heartbeats are not received from any other host in the cluster. If not specified, the default gateway of the console network is used. This default gateway has to be a reliable address that is available, so that the host can determine if it is isolated from the network. You can specify multiple isolation addresses (up to 10) for the cluster: das.isolationaddressX, where X = 1-10. Typically you should specify one per service console. Specifying too many addresses makes isolation detection take too long and can affect VMware HA behavior.

das.usedefaultisolationaddress
By default, VMware HA uses the default gateway of the console network as an isolation address. This attribute specifies whether or not this default is used (true|false).

Are certain appliacations running slowly occasionaly? Sometimes things are superfast and then they slow to a crawl. What’s going on?

Any tool that uses SNMP to gather performance metrics can be used to baseline and stress test infrastructure and determine where the bottle necks are.

Basic methodology could go something like this…

1 – identify end to end system components from end user terminal through network to virtual machines, esx hosts, and storage.

2 – configure SNMP for all devices

3 – verify use patterns and confirm data collection over 1 week. Tune alerts for normal use.

4 – schedule stress test for each component to determine performance ceiling and baseline throughput capacity.

5 – make changes as needed to improve end user experience.

6 – verify changes had desired effect.

Performance Troubleshooting for VMware vSphere

vsphere4-performance-troubleshooting.pdf (2.1 MB)

http://communities.vmware.com/docs/DOC-10352

Possible tools that could be used to poll for SNMP performance metrics include:

http://www.scriptlogic.com/Products/perspective/

http://www.vizioncore.com/products/vFoglight/features.php

http://www.whatsupgold.com/technology/network-management/monitoring-technologies/index.aspx

http://www.quest.com/Quest_Site_Assets/PDF/DSA-FoglightNetworkDevice-US-VC.pdf

http://network-optimisation.com/technology/network_monitoring/snmp_monitoring.php

http://www.microsoft.com/systemcenter/operationsmanager/en/us/default.aspx

http://www.manageengine.com/products/opmanager/index.html

http://www.managementsoftware.hp.com

http://www.solarwinds.com/products/orion/modules.aspx

http://www.veeam.com/vmware-esx-monitoring.html

http://www.monitorsnmp.com/

http://www.cisco.com/en/US/tech/tk869/tk769/technologies_white_paper09186a008011fde2.shtml

http://www.sage.org/lists/sage-members-archive/2002/msg01878.html

The SHA hash functions are a set of cryptographic hash functions designed by the National Security Agency (NSA) and published by the NIST as a U.S. Federal Information Processing Standard. SHA stands for Secure Hash Algorithm.

Vendors provide a sha-1 hash for software downloads. This enables you to verify that your downloaded files are unaltered from the original.

To confirm file integrity, use an sha-1 utility on your computer to calculate your own hash for files downloaded from the VMware web site.

If your calculated hash matches the message digest we provide, you are assured that the file was downloaded intact.

sha-1 utilities are available for Windows and Linux and Mac. Most UNIX installations provide a sha1sum command for sha-1 hashes. You may need a newer linux kernel to calculate the checksums for larger files.

The File Checksum Integrity Verifier (FCIV) can be used on Windows based products to verify sha-1 values. Please see http://support.microsoft.com/kb/841290 for details on FCIV.

Mac OS X: How to Verify a SHA-1 Digest http://support.apple.com/kb/HT1652

Instructions on checking an sha-1 checksum on a Mac:
In Finder, browse to /Applications/Utilities.
Double-click on the Terminal icon. A Terminal window will appear.
In the Terminal window, type: “openssl sha1 ” (sha1 followed by a space).
Drag the downloaded file from the Finder into the Terminal window.
Click in the Terminal window, press the Return key, and compare the checksum displayed to the screen to the one on the vendor’s download page.

From TechNet

VMware Data Recovery (CD ISO)
Released 11/19/09 | Version 1.1 | Size 418 MB | Binary (.iso)
Deploy VMware Data Recovery virtual appliance plus management components.
SHA1SUM 44dc0cd0c3e774d4912412b51dabeadf28d959b9

Background to Using Host Profiles

The vDS UI also allows a phased migration of vmnics from vSS to vDS without disruption to an operational environment. VMs can be migrated from a vSS to a vDS on the fly so long as the vDS and vSS have connectivity to the same network at the same time and the origin Port Group on the vSS and destination DV Port Group on the vDS are configured to the same VLAN.

Host Profiles provide a way to migrate multiple hosts at one time. Host Profiles use a golden profile from a migrated host to propagate a configuration to a number of other hosts.

When applying a Host Profile to a host, the host must be in Maintenance Mode. This requires VMs to be either powered down or migrated to another host.

Host Profiles are most appropriate for new installations of similarly configured hosts (i.e. same number of vmnics, same vmnic to physical switch configuration, no active VMS).

The table below summarizes the deployment situations and suggested methods for migration from vSS to vDS. Note: These are suggestions only; both methods will work within the guidelines mentioned above.

Summary of Migration Methods

Table 1 – Summary of vSS to vDS Migration Methods

Note: vDS UI = Use vDS UI; HP = use Host Profiles; vDS + HP = use vDS UI to deploy first host and Host Profiles for remaining hosts.

Applying NIC Teaming Policies to DV Port Groups With a vSS, NIC teaming policies are defined on the virtual switch with an optional override on each Port Group definition.  With vDS, NIC teaming policies are only defined on the DV Port Groups and apply to dvUplinks, not vmnics.  The vmnics are mapped to the dvUplinks on a per host basis.  This enables each host to have a different vmnic to physical host configuration and yet use the same NIC teaming policy over all hosts spanned by the vDS.

Monitoring Hash vmnic Selection in NIC Teams

The esxtop command from the ESX console can reveal the physical NIC (vmnic) used by virtual port or VM within a NIC team.

Use esxtop to see the following information:

• PORT-ID represents an internal port number on the virtual switch
• USED-BY column shows what that port number is used by (e.g. VMkernel, VM, etc)
• TEAM-PNIC column shows what physical nic (vmnic) is being used for traffic from that virtual port (the result of the hash within the NIC team)
• The remaining columns indicate the Receive and Transmit traffic rates on those ports.

To use esxtop, type esxtop from the ESX console and then type n.

A list of commands for the ESX command line interface is published in Chapter 6 of the ESX 4.0 Configuration Guide (available at http://www.vmware.com/support/pubs/). To control console output to one page at a time by adding the | more suffix to the commands. For example:
esxcfg-vswitch –l | more

 Reference: http://vmware.com/files/pdf/vsphere-vnetwork-ds-migration-configuration-wp.pdf 

(See page 8)

/var/log/vmkernel

/var/log/secure

/var/log/vmkwarning

/var/log/vmksummary

/var/log/vmksummary.txt

/var/log/messages

/var/log/vmware/*.log

/var/log/vmware/aam/*.log

/var/log/vmware/aam/*.err

/var/log/vmware/webAccess/*.log

/var/log/vmware/vpx/vpxa.log

/vmfs/volumes/*/*/*.log

Table with Explanation of files to log for VMware vSphere ESX Classic version 4

Table  – List of ESX Host Files to Log

VMware just updated their KB: Reserved or overhead ports for virtual switches (http://kb.vmware.com/kb/1008040) and we’ve run into this issue a number of times since upgrading to vSphere ESX 4. These new high memory hardware architectures allow an unprecedented number of virtual machine guests to be consolidated on a single ESX host.

By default a vswitch may not have enough ports to support the consolidation ratio your equipment can support. New ESX hosts can have 256 GB of RAM with 4 hex core processors and easily support 100 or more virtual machines. These virtual machines might have 1, 2, or more vNICs configured and each would need a port on the vswitch. One might imagine the need for 500 to 1000 ports needed per esx host. Why not just make it 2000 so we don’t have to worry about it later on?

Once you run out of vswitch ports you cannot power on any more vms on that host and even get errors about unplugged network cable.  Increasing the vSwitch port allocation seems easy enough, vmotion all workload off the host, put it in maintenance mode, change the vswitch config, reboot. Some system administrators run into this issue and decide to make the number of ports allocated to the vswitch really high to prevent this from ever being an issue. This can cause problems though.

There’s a limit of how many vswitch ports in total an ESX host has to hand out to it’s various vswitches. In addition, if security is a concern, you may start running firewall virtual appliances like vShield Zones or Catbird. WAN Accerators and Performance Monitoring tools like AppSpeed also require additional vSwitches to be created. Ports used on these vSwitches all take away from the total bucket of available ports.

Once 4096 ports are allocated to existing vSwitches you will not be able to add additional hosts to a vNetwork Distributed Switch either.

We also have the following Security Recommendation:

Only allocate vswitch ports to virtual machines on demand and as needed.

This will make it difficult if not impossible to “plug” a VM into the wrong network by accident. Testing for this can be done manually through the vSphere Client. If there are no ports available on a vSwitch then this is a positive test.

1. While connected to the vCenter Server Navigate to Home – Inventory – Networking in the vSphere Client and click on the vDS in question.
2. Click on the Ports Tab
3. If all of the ports in the list have a VM associated with it in the “connected”column then this is a positive test.

Deployment scenarios where a very large number of uplinks are teamed together on a single virtual switch might significantly impact the number of  ports on that virtual switch available for virtual machine use, and the overall size of the virtual switch might need to be adjusted accordingly.
 
The current port utilization data for virtual switches can be reviewed by using the esxcfg-vswitch –list command.
 
The current overhead utilization on a given virtual switch can be calculated by subtracting the Used Ports value for all PortGroups from the Used Ports value for that virtual switch.

Recommendation: Use VNDS vNetwork distributed Switches for all Virtual Machine traffic and limit the number of ports assigned to each standard vSwitch used for vmkernel and service console.

Standard vSwitch Procedure:

Note: A server reboot is required to apply the following configuration change.  Migrate the virtual machines off the ESX host to prevent any downtime.   On the vswitch there is an option to specify the number of ports the vswitch supports.  

To view this setting:

Reference Links

• Network cable of a virtual machine appears unplugged -
  http://kb.vmware.com/kb/1004883
• Unable to deploy configuration from -
  http://kb.vmware.com/kb/1007441
  Error: SysinfoException: Node (VSI_NODE_net_create) ; Status(bad0006)= Limit exceeded;
• Reserved or overhead ports for virtual switches -
  http://kb.vmware.com/kb/1008040

How to create a virtual appliance 

Background:

The Open Virtualization Format (OVF) specification is a standard being developed within the Distributed Management Task Force (DMTF) association to promote an open, secure, portable, efficient, and extensible format for the packaging and distribution of software to be run in virtual machines.

For use within an organization, Level 1 or Level 2 compatibility may be good enough, since the OVF package is distributed within a controlled environment where specific purchasing decisions of hardware or virtualization platforms can ensure consistency of the underlying feature set for the OVF.

Level 1. Only runs on a particular virtualization product and/or CPU architecture and/or virtual hardware selection. This would typically be due to the OVF containing suspended virtual machines or snapshots of powered on virtual machines, including the current run-time state of the CPU and real or emulated devices. Such state ties the OVF to a very specific virtualization and hardware platform.

Notes:

How to Make a Portable Virtual Appliance

You can export a virtual machine to a virtual appliance, making it available to other users to import into their inventories. The resulting virtual appliance is an OVF 1.0 appliance and contains one virtual machine. OVF Virtual Appliances contain many files that are typically compressed into an archive that can be put on removable media or downloaded from a server. This file much be decompressed prior to being imported and is more cumbersome to use. Consider using OVA for internal enterprise use.

You cannot select a virtual appliance destination for physical machine sources or virtual appliance sources.

Install Converter Standalone in Windows

Start the Wizard for a Conversion

1. Start the VMware vCenter Converter Standalone application.
2. Click Convert Machine in the application menu.

What to do next

Select a Source to Convert

• Select a VMware Infrastructure Virtual Machine Source
  You can convert a virtual machine that resides on an ESX host or ESX host that vCenter Server manages.

What to do next

You can now select the destination for your new virtual machine.

Select a Destination for the New Virtual Machine

Prerequisites