Press "Enter" to skip to content


How To: Upgrade vCSA 6.0 to vCSA 6.5

Today marks the release of vSphere 6.5 and with that, a new vCenter Server Appliance that is worth paying attention to. Beyond the traditional boost of configuration maximums and security, this version comes loaded with features that have been requested over the past few years. Some highlights include:

  • Built-in migration tool to go from vCenter Server 5.5 or 6.0 to vCSA 6.5
  • Built-in VMware Update Manager
  • Native HA support deployed in Active-Passive-Witness architecture
  • No Client Integration Browser Plug-in
  • Adobe Flex AND HTML5 web clients
  • API Explorer via vCSA portal page for your automation needs
  • Tons of other little enhancements that you can read about here
  • This post will be a guide on getting you from vCSA 6 to 6.5 with setting up vCSA HA at a later date.

    Crack open the ISO in your preferred flavor of OS and run the vCenter Server Appliance Installer. You’ll be greeted by this step1

    Hit next, accept the EULA, and fill out your environmental info. FQDN/IP of 6.0 vCSA, SSO details, and the ESXi host info that is currently housing your 6.0 vCSA. step2

    Now, if you are running VUM on a Windows server in your environment, you will see the following error: Unable to retrieve the migration assistant extension on source vCenter Server. Make sure migration assistant is running on the VUM server. Copy the ‘migration-assistant’ folder to the VUM server and run ‘VMware-Migration-Assistant.exe’, type in the password for the VUM service account and return back to the vCSA 6.5 Installer. step4

    The next few pages are choosing your cluster resources, folder organization, and general deployment information. Since this was done in my lab, I chose to stick with the ‘tiny’ vCenter deployment since I do not expect to ever need anything larger than that… hopefully. step7step8step9step10step11

    Once all that is done and dusted, you will get to the confirmation page to verify you didn’t fat finger any settings. If they all look good, click Finish. step12

    Assuming everything was chosen properly, you will see this lovely screen step16
    Congrats, you now have 2 vCSA’s running… but that’s not what we are here for. We want to decommission the 6.0 in favor of 6.5 with all of our lovely settings. So let’s get that crackin’


    Hit next and fill in your vCSA 6.0 info as well as the host that is running your 6.0 vCSA. You may get a warning about DRS being enabled on the cluster so feel free to change that setting depending on if your settings are set too aggressively.

    Next you will choose what data you wish to migrate from your old 6.0 to your new 6.5. I wanted all that lovely historical data so I went with the longer, last option. step19

    After that, you should be good to go! You will see some progress bars and then greeted with links to your shiny, new 6.5 vCSA. *Hint* It’s the same info as your 6.0, thanks migrations!


    After you login, check out your About vSphere menu and you should see vSphere 6.5 listed as current build. You will also notice that your original 6.0 VM is powered off and can be decommissioned to your liking. step23

    From there, you can hop into the Update Manager tab and upgrade your hosts to 6.5 automatically as well! Happy trails, friends and enjoy all the new awesomeness that vCSA 6.5 has dropped into your lap.

    Comments closed

    DISA Approves STIGs for VMware NSX on DoD Networks

    NSX STIG Photo credit: @_stump

    There a lot of abbreviations in this title so I will give a very brief rundown on what it all means and why some of you should care.

    In the public sector, our systems are hardened (locked down) a bit more drastically than your traditional private company might do things. Simply deploying a fresh copy of Windows from ISO is prohibited unless strictly spelled out in your lab environment. The governing body who regulates these mandatory compliance settings is known as the Defense Information Systems Agency, or DISA for short. They work closely with the product teams to ensure that when said product is deployed onto a network, it is as secure as possible while still maintaining functionality. These guides are known as STIGs or security technical implementation guides.

    With DISA approving the NSX STIGs, VMware’s NSX becomes the first software-defined network solution to do so.

    Now, as anyone who has deployed STIGs knows, sometimes the settings within them have a tendency to break previous functionality. With that said, take your time, test everything as you implement, and don’t be afraid to take note of any exemptions your project may need to adjust. Work closely with your ISSO’s and document everything up front as it will save you pain as you go along.

    Here are links to the direct zip’s for the STIGs above:

    VMware NSX STIG Overview, Version 1
    VMware NSX Manager STIG, Version 1
    VMware NSX Distributed Firewall STIG, Version 1
    VMware NSX Distributed Logical Router STIG, Version 1

    Comments closed

    VMware Cloud Foundation – At a glance

    Today was the first day of VMworld 2016 in Las Vegas and within the first of two General Sessions at the conference, VMware announced a new product dubbed Cloud Foundation. Cloud Foundation is a full stack integrated solution that will allow for seamless transition of your enterprise to utilize private and public clouds all in an automated, easily managed solution.

    Here is what you need to know about it:

    • Integrates vSphere, NSX, and VSAN into a unified stack managed by a new tool called SDDC Manager
    • SDDC Manager automates and simplifies the deployment of the entire stack
    • SDDC Manager does NOT take the place of vCenter
    • Cloud Foundation minimum requirements are 8 nodes as of writing (min. 4 nodes coming soon)
    • 1 vCenter per SDDC Manager
    • SDDC Manager can manage up to 192 nodes
    • vRA does not come with Cloud Foundation but can be used in conjunction with vCF (SDDC Manager handles the underlying infrastructure, vRA handles VM’s)
    • vCF via NSX extensibility allows for seamless migration between private and public clouds
    • As of writing, IBM Softlayer is only public cloud supported (Obviously, Azure and AWS enroute) and VCE VxRack 1000 is the only full rack supported solution
    • Cisco and Arista are your ToR and Spine supported network solutions for the time being
    • vCF supports VSAN Ready Nodes (8 of them minimum) as well
    • Licensing is per CPU
    • GA is September 1, 2016

    There ya have it, the quick and dirty rundown of what vCF brings to the table. VMware has teased this type of solution before in regards to EVO SDDC which is being retired as of September 1 in lieu of vCF. vCF is bringing more IP to the table and is what EVO SDDC should have been when first announced.

    Comments closed

    Fixing crashed alerts service in Nutanix Prism

    This week I ran across this issue randomly when I went to resolve some alerts through a STIG applied version of IE11. Prism itself was fully functional with the exception of any place where ‘Alerts’ would appear, I was met with:

    In order to properly troubleshoot, I SSH’d into a CVM, ran ‘cluster status’ to verify all components were up on each CVM… when I found they were, it was log parsing time. Thankfully Nutanix has an awesome log system in place to troubleshoot things like this. All logs are kept in /home/nutanix/data/logs and are organized in such a way that it makes it very easy to sift through what you need. Each major component has an INFO, WARNING, ERROR, and FATAL log to expedite the hunt.

    With this issue, I knew the problem was with the alert_manager since that was the component not functioning properly in Prism. I changed directors, grep’d the alert_manager files, and then tail’d the alert_manager.FATAL log.

    The FATAL log only contained 1 line:

    This verified that the alert_manager was crashed and since the last thing I did was try to clear an alert, it made sense to manually clear the alerts from SSH and restart the service.

    Close and re-open your flavor of browser and re-login to Prism and you should see that your Alerts are now empty, but displaying information properly.

    Comments closed

    Setup dual-home Nutanix cluster on Acropolis 4.6

    Note: This is not a Nutanix best practice and should not be done before discussing the caveats of this with upper management.

    This is a unique situation I came across in the most recent build of a Nutanix cluster. The project required the Nutanix cluster to support 2 distinct and separate networks. Since the NTNX nodes each have 2x 10Gbps and 2x 1Gbps, I needed to create 2 bonds… each bond having a single 10Gbps and a single 1Gbps.

    Herein lies the issue and why this is an unsupported setup. The architecture of NTNX’s clusters leverages a Controller VM that resides on each node in the cluster. These CVM’s talk to one another and are essentially the entire brains behind the operation and what makes NTNX so great. This is why the best practice is to pair your 10Gbps links in bond0 and pair your 1Gbps links in bond1. If you do a dual-homed setup and happen to lose one of your 10Gbps links, most scenarios will see you saturating that 1Gbps link and performance will be degraded. As long as all parties understand the risks associated with a dual-homed setup of this nature, let’s get cracking.

    Time to create the new bridge… SSH into one of your AHV hosts and run:

    SSH into the CVM of your choice or if you’re already SSH’d into your AHV node, just run ‘ssh nutanix@’ for internal access to the CVM.

    Take a look at your interfaces first:

    Acropolis sets up all your interfaces into a single bridge and bond by default so we will create a new bridge, new bond, and split up the interfaces into eth0/eth2 and eth1/eth3. Obviously you will have to physically run your cables to the appropriate separate networks so these interfaces might reflect differently on your end.

    Modify bridge0 and bond0 to only contain the interfaces of network 1:

    Create bridge1 and bond1 to only contain the interfaces of network2:

    Verify everything looks good:

    Now if that all looks good, you don’t want to have to do this manually on every CVM so use the built-in allssh command:

    At this point, your new bridge and bond is setup to allow for access to 2 different physical networks for dual-homed NTNX love. But now you’re seeing an alert in Prism about your CVM’s using an interface which is slower than 10 Gbps. This alert will trigger all the time in this setup regardless of if the active interface is the 10Gbps or the 1Gbps but thankfully there is a way to set which interface is active. Let’s make sure that our new bonds are using the 10 Gbps.

    Since we have mismatched interface speeds in our bonds, we need to use the active-backup bonding mode. If this was a traditional cluster, you could have the option of load balancing between 2 active links.

    From the AHV host:

    First verify that you are in active-backup mode, but if you are not, you can set it by:

    Once set, we need to see which interface holds the “active slave” parameter. In my example above, that would be eth2. The active slave mac is set to eth2, as well as under the eth2 interface information, you can see “active slave.” I will admit that calling the active interface the “active slave” seems a bit counter-intuitive but alas.

    In my case, the 10 Gbps port is the active slave so I am done. If you aren’t as lucky as me or perhaps you have a 10 Gbps link fail and have to manually set this back (it will not auto-repair back to 10 Gbps as of Acropolis 4.6), all you have to do is:

    Repeat that for your appropriate bond# and eth# on your hosts and that’s it. Long winded post but ultimately a few commands to get the job done. If you have a large amount of hosts, feel free to do these AHV host commands via for loop in bash to automate it a bit.

    Comments closed

    Citrix PVS 7.x : No servers available for disk

    This error appeared out of the blue on a vDisk maintenance patch day after the scheduled auto-update was set to start. I was given notice by a colleague that the latest vDisk had failed and thus was not updated. Upon further investigation, turns out that when the update VM was set to boot, it pulls an IP from DHCP but is never served a vDisk even though PVS creates the proper maintenance version.

    Good news! This fix is about as simple as it gets by only needing a single new key added to the PVS registry on each PVS server.

    That’s it, copy/paste that into your GPO’s or SCCM deployments and your PVS maintenance VM’s will boot without issue. No PVS server reboot required!

    Comments closed

    Homelab of Doom – Thanks to PernixData

    Labotage Labotage2

    Some of you may have seen on Twitter that I was the lucky, insanely lucky, winner of the VMworld 2015 PernixData Force Awakens Homelab. Winning it was so shocking, I stared at the Tweetdeck ‘notification’ column for a good 5 minutes in disbelief but low and behold, it was true!

    I’ve always been a huge proponent of homelabs for building, breaking, and in turn, learning. Prior to winning this, I had 2 HP Proliant tower servers running over iSCSI to my Synology NAS. I loved that lab… it was with me for my VCP-Cloud, VCP-5, and VCP-5.5 certifications and even learning Xen Server and Hyper-V. But, BRING ON THE SPEED!

    The PernixData lab is loaded with 3 SuperMicro 5018D-FN4T’s each with:

  • 8 core Intel Xeon D-1540 CPU’s
  • 64GB of DDR4 PC4-2133
  • 64GB mSATA SSD for ESXi
  • 400GB Intel SSD P3600 PCI NVMe
  • 3TB Seagate Enterprise HDD
  • 1x 1Gbps IPMI NIC
  • 2x 1Gbps NICs
  • 2x 10Gbps NICs
  • Needless to say, these hosts SCREAM. EMC ScaleIO is serving up the capacity layer via the spinning disks while PernixData FVP is giving the performance over Intel SSD’s.

    As you can see, they also supplied some nice switches to get all this connected:

  • Netgear GS728TSB 24 x 1Gbps
  • Netgear XS708E 8 x 10Gbps
  • Beyond the speed of this setup, it’s also surprisingly quiet. Lab noiseI used a decibel meter app on my Nexus 6P to gauge the sound and as you can see, it’s remarkably quiet. That is 3 servers, 2 switches, and 2 NAS’s all running.

    I cannot even begin to thank PernixData, Intel, EMC, and Micron for putting this all together.

    Comments closed

    Citrix Xen Server 6.x Error: VDI Not Available

    Enterprise team shot this issue up to me today which I hadn’t seen before. There was a hung VDI VM in our Xen Server 6.5 farm that refused to boot. Every time it would attempt to power up, it would fail with the error of “VDI Not Available.” Typically this occurs if a VM hangs and is terminated improperly or a host fails. In my case, the host wouldn’t let go of the writecache disk for the VDI VM. Thankfully the fix is relatively easy and doesn’t result in any downtime for the rest of the environment.

  • First SSH into the host holding onto the VM in question
  • Get the UUID of the VM and copy it to your clipboard

  • Check to see if the VM is properly shutdown across the cluster/site

  • Get the VDI disk UUID and copy that to your clipboard

  • Run the reset script packaged with Xen Server
  • And that’s all there is to it. Quick, simple, and effective. If only all solutions were this simple…

    Comments closed

    vSphere Thick Client? I don’t need no stinkin’ thick client!

    Quick post about an awesome, new VMware Fling that was released somewhat recently. I’m a little late to the party but I haven’t needed to deploy a new host since its release until today.

    If you haven’t heard, VMware Flings are small applications built by the VMware engineers that aren’t officially supported immediately but can still prove very powerful for your environment. Recently, they addressed something that bugged me from the day VMware announced that the web client was the future and the C# thick client was going bye-bye. Long story short, if you wanted to directly interface with an ESXi host without vCenter middle-manning, you were left with either PowerCLI, SSH, or the bloated C# client.

    This new fling is called the ESXi Embedded Host Client, a lightweight web client installed on your hosts that gives you a familiar vCenter web client experience. It takes about a minute to install via a one line esxcli command.

    SSH into your host(s) and execute this command:

    That will pull down the latest version of the Fling from VMware’s servers and auto-install. From that point on, you can use your favorite flavor of browser and point to the DNS/IP of your host and interface with it as you please.

    If you find you love this Fling and want to deploy it your datacenter, Brian Graf wrote a nice pCLI script to automate the whole ordeal which you find here.

    Comments closed

    Upgrading vCSA 6.0 to 6.0U1 without VAMI

    Quick post here on updating your vCSA 6.0 in what I believe to be the fastest way to update your vCSA installation. The VAMI that comes with vCSA is a great little tool but I find it to be hit or miss at times so I wanted to find a more reliable and visible way to upgrade. Behold the baked in software-packages tool via SSH!

    1) Go to and search for the latest VC patches and download
    2) Upload to a datastore visible by your vCSA
    3) Attach *.iso to the vCSA VM
    4) SSH into the vCSA with your root credentials
    5) Run software-packages install –iso –acceptEulas and wait for the update to finish, it should look like:

    6) Reboot vCSA via shutdown reboot -r updating and rejoice!

    Comments closed