I ran into a case tonight where I needed to set the default path selection policy (PSP) for an EMC NS120 array from Fixed to Round Robin. The fixed policy is set automatically by the VMW_SATP_ALUA_CX SATP (storage array type plugin) when the array targets are in failover mode 4.

This behavior is less than ideal and should be updated whenever possible.

To correct this, I executed this via the vSphere 5 CLI on a local serve:

esxcli --server 1.1.1.1 --username root --password ****** storage nmp satp set --satp VMW_SATP_ALUA_CX --default-psp VMW_PSP_RR

Note: This could also be done via the local ESXi console if needed by leaving out the server / username / password sections.

This differs slightly from what you will find on Duncan's site which lists the command for 4.1 and below.

I also changed all of the existing LUNs to Round Robin with this quick and dirty PoSh script:

Connect-VIServer localhost

foreach ($ESXhost in get-cluster "Cluster Name Here"| get-vmhost)

{Get-VMHost $ESXhost |Get-ScsiLun -LunType "disk"|where {$_.MultipathPolicy -ne "RoundRobin"}| Set-ScsiLun -MultipathPolicy "RoundRobin"}

At my previous place of employment, we ran into an issue on a project where the customer was running their View 3.x connection brokers on an unsupported platform (Windows 2003 x64). To upgrade them to View 4.0.x, the decision was made to switch them of the x64 platform and get the View Connection Brokers / Security Servers on to a VMware Supported x86 platform (See Page 10 of the View 4 Upgrade Guide)

While, yes, there is View ADAM database export scripts, it is difficult to cleanly cut over potentially large, complicated, and incredibly active environments all at one time. This led to having View 3.x and 4.x up concurrently. This presented us with a myriad challenges, primarily that of transferring the entitlements over to the new connection brokers for the desktop sources (this environment had A LOT of stand alone desktops) / pools / etc. We found it difficult, time consuming, and error prone to try to extract this information by hand in advance. Compounded by the lack of a scripting interface (besides the limited vdmadmin cli) this left all little choice but to grab the information directly from the View ADAM instance.

To ease this challenge, we wrote a script to extract entitlement information from the View 3 connection broker and output it to a tsv file, which you can open and modify with your favorite spreadsheet application. While this wasn't the full solution, at least it was able to give a very reliable spreadsheet that makes life a lot easier, providing you have patience and at least two monitors. :)

This script works against the View 4.0.x ADAM instance as well, in case you are finding yourself in this situation upgrading to View 4.5 / 4.6. This script may also be useful for compliance and auditing reasons (to show auditors / management exactly who has access to what). DR / BCP Documentation also comes to mind.

As Always, DO NOT TEST THIS IN PRODUCTION FIRST! Be sure that your ADAM config backups are current. Please familiarize yourself with VMware KB Article 1008046 before you attempt to run this script!

Download Script Here

Change the extension to .vbs for best results.

What many people may not realize, unless their environment heavily depends on it, is that, among other things, your WINS settings are lost during this upgrade (specifically the Virtual Hardware Version upgrade). Many organizations that I have worked with in the past, if they even still use WINS, usually have 2 "go to" WINS servers, or at least, a set of WINS servers that they generally set for different parts of their infrastructure with replication set up appropriately on the back end to help spread out the load.

Resetting all of these settings by hand can be a truly daughting task for the traditionally over burderned Windows / VMware admin. Though this issue has been discussed to death in many VMware Communities threads and has been properly documented by VMware KB Article 1015572, neither location gives you a concise way (besides scripting before hand with netsh) to rectify the problem.

With that in mind, I whipped up a VBScript a while back that will help the VMware / Microsoft Admins remotely set their WINS Server entries. This script reads in a list of servers from a secondary file (in this case "prodServers.txt") and uses VBS and WMI to remotely set Primary and Secondary WINS entries on ANY active NIC with an IP address. It would be VERY easy to populate this list with a list of FQDN's from your vSphere console by using the Virtual Machine view and then using the "Export List" option under the File menu. This script was primarily tested in a Windows Server 2003 environment, though I would imagine it would also work in 2008 / 2008 R2 as well. This script assumes that the run as user has appropriate levels of permissions (generally the MS / VMware Admins have Domain or Enterprise Administrator accounts, but a delegated local admin account will work just as well).

Download Script Here

In the interest of most Firewalls and Anti-Virus, I have changed the extention to .txt. Change this to .vbs, create a .txt file named "prodServers.txt" (or change the script to your bidding, which you will need to do anyway to set your WINS servers) and you will be in business! As always, TEST this in your lab or an otherwise controlled environment first. If you break all your VM's, don't come crying to me :)

The Exagrid EX5000 is a Backup to Disk with integrated DeDuplication solution for any small to medium sized business. Multiple arrays can be linked together (like EqualLogic groups, to an extent) to suit larger business as well. These arrays can be replicated between physical sites to facilitate disaster recovery / business continuity. At my previous firm, Exagrid was nice enough to loan us an EX5000 to get hands on experience for about a year ago. Below is a concise overview of my experience with this array at that time.

Photo Shoot

EX5000 Front

EX5000 Back

EX5000 Side

Printed Documentation and Cables

Heavy Ass Box 

(I found out afterwards that it had handles, this was an empty box, and as you can see, I didn’t use the handles. #Fail)

Don’t be a Noob, Call Support (Seriously!) 

Hard Drives

The Insides (Explained Below) 

First Impression

This product is REALLY HEAVY. Don’t lift it by yourself. Even with the hard drives out (don’t bother taking them out, it’s a waste of time), this thing could stop a fleet of tanks. In terms of the actual product and its usability and feature set, it seems pretty solid. This box is basically a rack mountable Linux box with front facing hot swappable hard drives. The specs of our demo unit are as follows:

• 2x Intel E5410 Quad Core @ 2.3 GHZ
• 8GB (2x4GB) Ram
• 16x750GB WD Enterprise Storage (SATA) connected to a shared backplane with 4 SATA outputs. These run back to a PCI mounted hard drive controller with an attached battery backup cache.
• The hard drive set is formatted in RAID6 with dual drive redundancy, though it is formatted into 2 5TB partitions…1 for backup storage (AKA ‘Landing’) and 1 for Archive / DeDup (AKA ‘Retention’). There are also a few hidden partitions for the Linux software (~50GB).
• N+1 Redundant PSUs (3 PSUs total)
• Keyboard Mouse Monitor output, you can get right on the console with this. If you ever need to get on to one of these (you should be calling support) but if you NEED to have root access, the user is ‘root’ and the password is ‘inflection’ (at time of demo, could have changed now). SSH from your favorite client (putty, etc) works as well if the interfaces are accessible.
• 4x 1GB interfaces (2 integrated, 2 on an Intel PCI card)

Exagrid appliances are pretty darned easy to setup, however, there are stickers places strategically on the unit, along with inserts in the product documentation, that instruct everyone to contact support to have them help you setup the array. Following along with what a customer or other customer would experience, I called Exagrid support. Everything went really well from here. I did not wait on hold to long to get to speak to an engineer.

Support

Neat thing about Exagrid support is that when you buy support for an array (or for several arrays) for a single company, you get a dedicated (named) support contact, however, I found that even if that engineer is OFF DUTY, you can still get help (HORRAY!). My engineer’s name was Bill B and he was based out of their support department in Massachusetts and I can absolutely classify their support as top notch. Bill seemed to not only know _every_ (in detail) about the Exagrid arrays but was overall a knowledgeable, personable, and very patient engineer.

Exagrid support walks you through EVERYTHING from setting up the interfaces / IP address, configuring all of the best options, and setting up your backup application to start sending jobs to the unit. They also handle all patching and firmware upgrades, so while they are showing you how to perform backups properly, they are downloading the latest firmware and helping you apply it.

Firmware Updates

Firmware Updates bring new features, bug fixes, new product documentation, performance enhancements, and just about everything else one would expect. As previously mentioned Exagrid support handles these for you by remoting in, downloading them off their FTP site, and launching the update for you. Each update takes at least 1 reboot and to get the array up to the latest rev, there may be several revisions that one would have to go through. Support engineers are supposed to contact their clients when new firmware comes out, although they may not contact you when maintenance revisions come out that are not addressed at your environment (IE: a fix for BackupExec when you run vRanger). For secure sites / sites with no external internet access, Exagrid can send you a USB drive with the firmware updates on them and the support engineer can walk you through applying the updates.

Web Admin Page

All administration is done through the Web Admin Page. Some administration can be done through SSH, though it seems like pretty much everything you need is right here. When you log on, you are presented with a summary splash page showing you all the arrays that this site is managing and their space, etc (all the stuff you would expect in a summary):

Documentation

In addition to Exagrid’s excellent support, their product documentation is top notch. For every backup application that the array supports, they have a PDF guide that walks you through how to properly set it up with the array and the guides do not seem to skimp on anything. In the context of backup exec, it shows you how to create a job properly, what options to turn on / off, how to configure tape rotation policy (the files are stored like virtual tapes), and how to configure automatic tape offload (if you want to use it). This product documentation is located in the ‘Online Library’ (Exagrid Array Home Page -> Help -> Online Library) integrated into the unit, which is updated through almost every firmware update (which seem to be quite frequent). The documentation that ships with the unit (physically) is also pretty good at walking you through the initial setup if for some reason you can’t call support (or if you name is Ranger Dan and you are too cool to call support but still need to RTFM).

Operations

Once you get the unit connected, up to date, and configured, there isn’t much to do as far as I remember on the array side. You will want to make sure you size up your environment correctly to make sure you will have enough space, however, past that, most of the operations will take place in your backup application.

Shares

Backup targets are called shares. When making a share, you need to do a few things. First, you need to specify a share type, which correlates with the backup application you will be using (IE: Backup Exec, vRanger, etc). There are a few special types (Utility and FSD-TAR), which are not deduped / compressed / replicated. Second, you have to configure an allowed IP list (for each share), which is basically just an IP ACL. Once a share is configured, it is accessible via the UNC path of the interface you would like to use IE: \\10.20.30.2\BackupExec1. Backups generally go to the Backup server first, although the array does support Open Storage Technology (much like Data Domain et al), which with certain vendors, like Backup Exec, you can have your backup application send data directly to the Exagrid array, potentially increasing performance.

Though I am thoroughly familiar with Symantec Backup Exec’s B2D Folder, Bill helped me setup 4 jobs in our lab. At the time, the lab setup wasn’t too ideal and the only spare server we had had a 100MB NIC in it, so speed was slow but I am confident this was our setup and not the EX5000

Virtual Tapes in BackupExec B2D folder

Maximums

At the time of demo, here were a few of the maximums I discovered:

• 20 Shares per server
• 80 Shares per site
• 1 Site name per location
• 1 Unique name per array

Network Interfaces

The base unit has 2 1GB interfaces built into the unit with a PCI expansion card that delivers 2 more. In our demo unit, these were 1GB but you can order this expansion card as a 10GB card. NIC 0 is exclusively used for inter-site communication (to talk to other Exagrid arrays, if you have them) as well as replication (if memory serves me correctly). This interface cannot be used for anything else and is to be left disconnected if you only have 1 array. There are special IP subnet restrictions for this interface that you should be aware of (can only be 192.168’s I believe). All of the other interfaces are used for backup traffic and are configured separately. These interfaces do not have IP address / subnet restrictions. Most communication occurs over CIFS protocol, although you can set it up as an NFS server as well.

If you have more than 1 array, the site will load balance your ‘retained’ (deduped) backups, so you can direct a lot of backups to 1 array and it should spread it across other storage.

There are a few cons to the networking:

• Any and all changes force a reboot. Takes about 5 minutes per reboot and they array is 100% unavailable during that time
• No Jumbo Frames (as of when I demoed it)
• No Link Aggregation / Group IP style targeting. While the unit supports multiple interfaces, each interface has its own IP and each interface has to be setup as a separate backup device in applications like Backup Exec. Not necessarily the biggest con per say but more of a ‘gotcha’ that you have to be aware of

Neat Features

Dial Home for Performance – Every night the unit sends backup job statistics back to Exagrid. If support engineers notice that you array is not performing well, they might contact you and try to help you crank some more juice out of your backups / DeDup. This also works for hardware failures and alerts, much like any other product with dial home.

Latest backup is not deduped – makes restores pretty fast, backup does not have to be hydrated to be restored. If you go farther back, performance decreases as data needs to be rehydrated.

Now, that all being said, this demo was about one year back, so things may have changed in terms of features, functionality, etc.