http://www.vmware.com/pdf/vsphere4/r40_u1/vsp_40_u1_availability.pdf

Network Isolation Addresses

A network isolation address is an IP address that is pinged to determine if a host is isolated from the network. This address is pinged only when a host has stopped receiving heartbeats from all other hosts in the cluster. If a host can ping its network isolation address, the host is not network isolated, and the other hosts in the cluster have failed. However, if the host cannot ping its isolation address, it is likely that the host has become isolated from the network and no failover action is taken.

By default, the network isolation address is the default gateway for the host. There is only one default gateway specified, regardless of how many service console networks have been defined, so you should use the das.isolationaddress[...] advanced attribute to add isolation addresses for additional networks. For example,  das.isolationAddress2 to add an isolation address for your second network, das.isolationAddress3 for the third, up to a maximum of das.isolationAddress9 for the ninth.

When you specify additional isolation address, VMware recommends that you increase the setting for the das.failuredetectiontime advanced attribute to 20000 milliseconds (20 seconds) or greater. A node that is isolated from the network needs time to release its virtual machine’s VMFS locks if the host isolation response is to fail over the virtual machines (not to leave them powered on.) This must happen before the other nodes declare the node as failed, so that they can power on the virtual machines, without getting an error that the virtual machines are still locked by the isolated node.

For more information on VMware HA advanced attributes, see “Customizing VMware HA Behavior,” on page 26.

das.isolationaddress
Sets the address to ping to determine if a host is isolated from the network. This address is pinged only when heartbeats are not received from any other host in the cluster. If not specified, the default gateway of the console network is used. This default gateway has to be a reliable address that is available, so that the host can determine if it is isolated from the network. You can specify multiple isolation addresses (up to 10) for the cluster: das.isolationaddressX, where X = 1-10. Typically you should specify one per service console. Specifying too many addresses makes isolation detection take too long and can affect VMware HA behavior.

das.usedefaultisolationaddress
By default, VMware HA uses the default gateway of the console network as an isolation address. This attribute specifies whether or not this default is used (true|false).

VMware just updated their KB: Reserved or overhead ports for virtual switches (http://kb.vmware.com/kb/1008040) and we’ve run into this issue a number of times since upgrading to vSphere ESX 4. These new high memory hardware architectures allow an unprecedented number of virtual machine guests to be consolidated on a single ESX host.

By default a vswitch may not have enough ports to support the consolidation ratio your equipment can support. New ESX hosts can have 256 GB of RAM with 4 hex core processors and easily support 100 or more virtual machines. These virtual machines might have 1, 2, or more vNICs configured and each would need a port on the vswitch. One might imagine the need for 500 to 1000 ports needed per esx host. Why not just make it 2000 so we don’t have to worry about it later on?

Once you run out of vswitch ports you cannot power on any more vms on that host and even get errors about unplugged network cable.  Increasing the vSwitch port allocation seems easy enough, vmotion all workload off the host, put it in maintenance mode, change the vswitch config, reboot. Some system administrators run into this issue and decide to make the number of ports allocated to the vswitch really high to prevent this from ever being an issue. This can cause problems though.

There’s a limit of how many vswitch ports in total an ESX host has to hand out to it’s various vswitches. In addition, if security is a concern, you may start running firewall virtual appliances like vShield Zones or Catbird. WAN Accerators and Performance Monitoring tools like AppSpeed also require additional vSwitches to be created. Ports used on these vSwitches all take away from the total bucket of available ports.

Once 4096 ports are allocated to existing vSwitches you will not be able to add additional hosts to a vNetwork Distributed Switch either.

We also have the following Security Recommendation:

Only allocate vswitch ports to virtual machines on demand and as needed.

This will make it difficult if not impossible to “plug” a VM into the wrong network by accident. Testing for this can be done manually through the vSphere Client. If there are no ports available on a vSwitch then this is a positive test.

1. While connected to the vCenter Server Navigate to Home – Inventory – Networking in the vSphere Client and click on the vDS in question.
2. Click on the Ports Tab
3. If all of the ports in the list have a VM associated with it in the “connected”column then this is a positive test.

Deployment scenarios where a very large number of uplinks are teamed together on a single virtual switch might significantly impact the number of  ports on that virtual switch available for virtual machine use, and the overall size of the virtual switch might need to be adjusted accordingly.
 
The current port utilization data for virtual switches can be reviewed by using the esxcfg-vswitch –list command.
 
The current overhead utilization on a given virtual switch can be calculated by subtracting the Used Ports value for all PortGroups from the Used Ports value for that virtual switch.

Recommendation: Use VNDS vNetwork distributed Switches for all Virtual Machine traffic and limit the number of ports assigned to each standard vSwitch used for vmkernel and service console.

Standard vSwitch Procedure:

Note: A server reboot is required to apply the following configuration change.  Migrate the virtual machines off the ESX host to prevent any downtime.   On the vswitch there is an option to specify the number of ports the vswitch supports.  

To view this setting:

Reference Links

• Network cable of a virtual machine appears unplugged -
  http://kb.vmware.com/kb/1004883
• Unable to deploy configuration from -
  http://kb.vmware.com/kb/1007441
  Error: SysinfoException: Node (VSI_NODE_net_create) ; Status(bad0006)= Limit exceeded;
• Reserved or overhead ports for virtual switches -
  http://kb.vmware.com/kb/1008040

SUMMARY: For best performance be sure to upgrade your VMFS Block Storage when you upgrade your ESX hosts to vSphere.

VMFS 3 versions and upgrade paths

Purpose

Resolution

Additional Information

http://kb.vmware.com/kb/1005325

http://www.vfrank.org/2010/01/31/vmfs-3-versions-maybe-you-should-upgrade-your-vmfs/

http://virtualizationreview.com/blogs/everyday-virtualization/2009/06/vstorage-vmfs-version-notes.aspx

http://communities.vmware.com/message/1071323

http://www.onlinetechblog.com/blog/index.php/2009/11/vsphere-4-0-places-service-console-in-local-vmfs-volume/

esx network ports

The amazing Dudley Smith, from VMware’s Technical Account Manager team has release a larger version of his vSphere Network Connections and Ports for ESX diagram and an accompanying excel spreadsheet listing all the TCP/IP ports for various communication purposes.

Get them directly from the VMware blog site here:

http://communities.vmware.com/blogs/dudleysmith

HyTrust Appliance 2.0 is available. Building on the successes of 2009, which included our initial product launch and numerous awards, we’re happy to see the streak continue into 2010 by delivering a major new release that will empower enterprises to capitalize on the wave of datacenter virtualization and accelerate efforts to virtualize tier-one applications. The features available in HyTrust Appliance 2.0 deliver true enterprise-class policy management and access control capabilities to virtual infrastructure. New features include the following:

* Root Password Vault: Locks down privileged host accounts and provides passwords for temporary use to enable time-limited privileged account access. Root accounts on hypervisors are extremely powerful and, as a consequence, can create a significant liability if not kept out of the wrong hands. With the aid of Root Password Vault, all root account access is attributable to an individual and every action is logged, providing far greater visibility and accountability.
* Federated Deployment: Secure distributed system architecture allows for automated replication of policies and templates across multiple HyTrust Appliances as well as geographic boundaries. For larger enterprises with multiple datacenters and collocation facilities, Federated Deployment of HyTrust Appliances ensures consistency of controls across the entire infrastructure.
* Virtual Infrastructure Search: Enables quick and easy accessibility to all virtual infrastructure objects, policies, and logs within HyTrust Appliance.
* Remote API: Interface to remotely access and automate the administration of the HyTrust Appliance. Provides the greater scalability demanded by large, enterprise-wide deployments of virtualization.
* Object Policy Labels: Creates a policy categorization structure, similar to “Web 2.0 tagging” for virtual infrastructure objects, which enables better organization and tighter, more consistent controls. Object Policy Labels enable access, network segment, and zoning policies, which allows administrators to dictate which virtual machines are allowed to connect to which network segments or hosts via RuleSets and Constraints.
* Router-Mode: a deployment option where all VMware management traffic is forced to flow through the HyTrust Appliance. HyTrust Appliance acts as a router for the “protected” management subnet and ESX/ESXi hosts and vCenter Server use HyTrust Appliance as their default gateway. This adds yet another flexible deployment option to the other existing options, ensuring the HyTrust Appliance will easily adapt to any enterprise architecture.

Along with the new capabilities delivered in 2.0, we’d like to introduce you to the new editions of HyTrust Appliance:

* Community Edition is a free version of the product that supports up to three hosts.
* Standard Edition supports an unlimited number of hosts and offers more flexible deployment options.
* Enterprise Edition supports an unlimited number of hosts, offers more flexible deployment options, supports federation of multiple HyTrust Appliances, enables privileged account management via Root Password Vault, allows two-factor authentication, and offers a remote API for additional management flexibility.

You can download the Community Edition of HyTrust Appliance at http://www.hytrust.com/community.

From http://www.vmware.com/pdf/vsphere4/r40/vsp_40_esx_server_config.pdf <— look on page 98 or the VMware ESXi Configuration Guide

Network Attached Storage

ESX supports using NAS through the NFS protocol. The NFS protocol enables communication between an NFS client and an NFS server.

The NFS client built into ESX lets you access the NFS server and use NFS volumes for storage. ESX supports only NFS Version 3 over TCP.

You use the vSphere Client to configure NFS volumes as datastores. Configured NFS datastores appear in the vSphere Client, and you can use them to store virtual disk files in the same way that you use VMFS-based datastores.

*** NOTE: ESXi does not support the delegate user functionality that enables access to NFS volumes using non- root credentials.

Also see these links for more info on read only capabilities for different licenses.

http://partnerweb.vmware.com/comp_guide/docs/vSphere_Comp_Matrix.pdf

On the ViOPs site there is a comparison matrix of ESXi/ESX in case we’re asked ‘which one should I use?’.

VMware ESX and ESXi 4.0 Comparison – http://kb.vmware.com/kb/1015000

VMware ESX and ESXi 3.5 Comparison - http://kb.vmware.com/kb/1006543

RCLI is limited to read-only access for the free version of VMware ESXi. To enable full functionality of RCLI on a VMware ESXi host, the host must be licensed with VI Foundation, VI Standard, or VI Enterprise.

http://www.vmware.com/products/vsphere/buy/editions_comparison.html

Comparison of product offerings for vSphere 4.0 and VMware Infrastructure 3.X – http://kb.vmware.com/kb/1010579

VMware makes periodic updates to the ESXi Installable version you can download. This page was created to help track and locate those.

Use these numbers to determine when a system was patched last and to make sure the VMware Infrastructure Client is the right one.

Best Practice:

ESXi: Run the VMware Infrastructure Update tool from a windows management station with the VMware Infrastructure Client every month.

ESX: Use vCenter Update Manager to scan and remediate ESX hosts when new security patches are available.

How to Check the Version Numbers:

1. Download the VMware Infrastructure Client from the Web User Interface.
   For example: https://ESX-HOST-IP-ADDRESS/client/VMware-viclient.exe
2. Start the VMware Infrastructure Client
3. Click the Help Menu
4. Select “About”
5. Note the Version and Build for both the Client and Server.
6. Compare to list below to ensure they are at same release.
7. If you update the Server you should connect to the Web User Interface and download the latest VMware Infrastructure Client.

Latest Install ISO is VMware ESXi 3.5 Installable Update 4 Build Number: 153875
Released: (2009.03.20)

ESXe350-200907401-O-SG – PATCH Build 176894 (2009.05.28) – VIC 147633 – Tools 176894 <– Latest Patch

ESXe350-200906401-O-BG – PATCH Build 169697 (2009.05.28) – VIC 147633 – Tools 169697

ESXe350-200905401-O-BG – PATCH Build 163429 (2009.05.28) – VIC 147633 – Tools 158874
ESXe350-200904401-O-SG – PATCH Build 158874 (2009.04.29) – VIC 147633 – Tools 158874
ESXe350-200904201-O-SG – PATCH Build 158869 (2009.04.10) – VIC 147633 -
ESXe350-200903201-O-UG – UPDATE Build 153875 (2009.03.30) – VIC 147633 <– Update 4
ESXe350-200903411-O-BG – PATCH Build 153840 (2009.03.20) – VIC 119801
ESXe350-200901401-O-SG – PATCH Build 143129 (2009.01.30) – VIC 143129
ESXe350-200811401-O-SG – PATCH Build 130755 (2009.12.02) – VIC 119801
ESXe350-200810401-O-UG – UPDATE Build 123629 (2008.11.17) – VIC 119801 Update 3

The typical way to apply patches to ESXi hosts is through the VMware Update Manager. For details, see the VMware Update Manager Administration Guide.

ESXi hosts can also be updated by downloading the most recent “O” (offline) patch bundle from http://support.vmware.com/selfsupport/download/ and installing the bundle using VMware Infrastructure Update or by using the vihostupdate command through the Remote Command Line Interface (RCLI). For details, see the ESX Server 3i Configuration Guide and the ESX Server 3i Embedded Setup Guide (Chapter 10, Maintaining ESX Server 3i and the VI Client) or the ESX Server 3i Installable Setup Guide (Chapter 11, Maintaining ESX Server 3i and the VI Client).

Note: ESXi hosts do not reboot automatically when you patch with the offline bundle.

Reference:

http://support.vmware.com/selfsupport/s3portal.portal?_nfpb=true&_windowLabel=SearchPatch&SearchPatch_actionOverride=%2Fportlets%2Fpatchupdate%2FfindPatchByProductVersion&_pageLabel=s3portal_pages_downloadPatch_page&version=3.5&product=ESXi%20(Embedded%20and%20Installable)

There are many customers we’re setup with virtualized active directory domain controllers. Windows 2003 at first and now Windows 2008 both work fine as Virtualized Domain Controllers.

Here are some of the links and notes that help as references…

–> http://www.vmware.com/files/pdf/Virtualizing_Windows_Active_Directory.pdf

An anti-affinity DRS rule is used when you want to keep 2 virtual machines on separate hosts when they provide a redundant service and locating them on the same host would eliminate that redundancy.

–>http://vmprofessional.com/2009/06/drs-and-anti-affinity-rules.html

The Virtual Machine on 64-Bit Windows Server

If using the x64 version of Windows Server 2003 or 2003 R2, one of the primary goals will be to contain the entire Active Directory database within the virtual machine’s RAM cache. On 64-bit Windows, employing 16 GB of RAM cache will accommodate a database of approximately 2.5 million users.
Caching the Active Directory database in 64-bit Windows will avoid performance hits related to certain disk operations. For a virtual machine that is a domain controller, adding, modifying, searching, deleting and update operations generally benefit significantly from caching. Write operations will always incur a slight penalty, regardless of whether a domain controller is running on a physical or virtual machine.
There is limited benefit for filling cache on 32-bit Windows for customers with large directories; in fact, in some cases this actually can exhaust kernel resources.

–> http:/viops.vmware.com/home/docs/DOC-1223

–> http:/xtravirt.com/xd10095
First Published: 17 June 2009
Windows 2008 Server and Windows 2008 Server R2 further refine the functionality with the service being renamed Active Directory Domain Services.

–> http://support.microsoft.com/kb/875495/
This article describes a condition that occurs when a domain controller that is running Microsoft Windows 2000 or Microsoft Windows Server 2003 starts from an Active Directory database that has been incorrectly restored or copied into place. This condition is known as an update sequence number rollback, or USN rollback. When a USN rollback occurs, modifications to objects and attributes that occur on one domain controller do not replicate to other domain controllers in the forest. Because replication partners believe that they have an up-to-date copy of the Active Directory database, monitoring and troubleshooting tools such as Repadmin.exe do not report any replication errors.

–> http://download3.vmware.com/vmworld/2006/tac9710.pdf
Here is a link to a VMworld 2006 Presentation titled TAC 9710 -
Virtualizing a Windows Active Directory Domain Infrastructure:
* Clock synchronization
* Network performance
* Multi-master replication model
* Security
* Potential single point of failure
* Disaster recovery

–> http://technet.microsoft.com/en-us/library/dd348449.aspx?ppud=4

# To help prevent a potential update sequence number (USN) rollback situation, see Appendix A: Virtualized Domain Controllers and Replication Issues.

–> http://technet.microsoft.com/en-us/library/dd348479(WS.10).aspx

– I b e n
iben.rodriguez – gmail
Follow me on http://twitter.com/iben

I recently had opportunity to install VMware ESX on some old Dell hardware. These notes helped me move through the process.

VM-Help database of links: http://www.vm-help.com/esx/esx3.5/Whiteboxes_SATA_Controllers_for_ESX_3.5_3i.htm

How to enter nocheckCPUIDLimit on ESX with screen shots – http://communities.vmware.com/message/1136449#1136449
This is a two step process. Once for initial install from cd-rom and once for rebooting off hard drive. I did have to tweak it as it did not work 100% as advertised (see BrennanB post on April 27th).

Please let me know what other links worked for you and your old hardware.  Be sure to update vm-help with any feedback too!

I b e n

VMware Converter Standalone

VMware converter is used for migrating Physical servers to Virtual Machines, Virtual Machines to Virtual Machines.

Directions for conducting a V2V or P2V for Windows Servers

1. Update or Open a tracking ticket to track progress
2. Ensure system is documented and monitored on portal
3. Notify stakeholders via DL – setup DL if needed
4. Login to the Machine to be converted
5. Run defrag and diskcheck if possible
6. Run Microsoft Update
7. Do a reboot test to ensure machine stability
8. Download VMware Converter Standalone version 4 – VMware-converter-4.0.0-146302.exe
9. Download Sysprep tools – unzip but do not run
10. Download NewSID – unzip but do not run
11. Download and run BGinfo – apply
12. Download and run treesize free and clean up unneeded files
13. Install VMware Converter
14. Copy Sysprep files to correct location – c:\documents and settings\all users
15. Launch VMware Converter
16. Import Machine
17. Select the device type; Physical Computer, Virtual Computer from ESX or VMware Workstation.
18. Enter in the remote IP address of the target:
19. Do not select Automatically uninstall the files when the import is successful
20. Select all the drives you wish to migrate to the new Virtual Machine
21. Select ESX or Virtual Center
22. Enter the Virtual Center and user credentials
23. Select the Virtual Machine name
24. Select the ESX host
25. Select the appropriate DataStore
26. Select the appropriate network
27. Check the box install vmware tools
28. If you desire to customize the settings, check the box.
29. Select Finish
30. When completed test new machine and configure.
31. Run newsid if new host names is needed… Keep in mind you cannot have two machines with same name or IP on same network.
32. Run BGinfo and apply again.
33. Verify reboot test and monitoring is functioning.
34. Verify system time.
35. Adjust services as needed.
36. Remove old hardware’s software.
37. Notify stakeholders when old machine is off and new machine is on.

See also: http://communities.vmware.com/thread/129871