Category: VMware


Recently Sunny Dua posted internally and on his public accounts (@Sunny_Dua on Twitter) about some new dashboards for vROPs that will show you the performance impacts related to Spectre/Meltdown.  These are very essential that I recommend you also read up about it from the VMware offical blog site.

My homelab is just a small setup consisting of three Intel NUCs Series 7 running the i3 CPUs.  Just something that gives me a chance to run a couple VMs per host and still get that hands on feeling.  I figured there wouldn’t be much to do hardware wise for now but there have been for the VMware bits.  (Some may have been pulled due to reasons that you should reach out to your account team or TAM to get the info.)

If you currently don’t have vROPs, get the trial and check this out!

 First dashboard – CPU Bug – Performance Monitoring

CPU Bug-Performance Monitoring

  • This dashboard will give you a quick glance over your vSphere environment which will come in handy if you’ve already did the hardware patches to see the impact overall.
  • You can also drill down cluster level (I only have the one) but you can see the CPU demand and CPU contention graphed in an easy to read and understand format.
  • Last is per ESXi host in the selected cluster and then it’ll list the VMs with more than 8 vCPUs and their usage of those with 5-8 vCPUs (I have none).  This will come into play after for patching.

Second dashboard – CPU Bug – vSphere Patching

CPU Bug-vSphere Patching

  • This doesn’t show much for my setup since I’ve already deployed the ESXi patches but you will notice the Lab vCenter is showing there is a patch available for it.
  • Your VM Hardware version comes into play when you install patches that require at the least VM HW version 9 to support vMotioning. As you can see I need to upgrade several VMs myself.
  • This will also show which hosts have been patched per VMware’s recommendation.

Third and last dashboard – CPU Bug – VM Patching

CPU Bug-VM Patching

  • This last dashboard will give you the performance per VMs to help you determine which server to do OS level patches.
  • Work on the idle VMs first (unless you don’t have any like I’m showing) and the heavy CPU hitters last.

Side note – I did just patch that vCenter Appliance.  The VAMI is so slick that it was done before I even finished typing this post up.

Advertisements

Career Opportunities

For the last 14 years, I have been working in higher education at two different places.  I built my IT foundation at OCCC and moved on to OUHSC to further my career opportunities.  I rarely had my hands in the firewall or networking before the move but now I can say I have experience with Juniper firewalls and Cisco Nexus switches to say the least.  But the key to changing careers is being able to learn from others and hopefully spread some of your knowledge too.

At OUHSC, we had a Top 70 list of different tasks that we did in a flat operational structure.  Being able to train others on how to maintain a VMware shop was my duty as well as providing customers a great and stable environment for their services.  Then we started this Shared Services model of bridging the three campus into one virtual data center.  I presented at VMworld ’13 as well as a local conference for higher ed institutions on what we designed and created.  It’s been awesome building this platform that stretches across boundaries of teams then teaching them how to run it.  In my six years here I have been blessed in all aspects, my peers that created strong friendships and how well the leadership listened and took care of my family while giving me the chance to keep my passion going with virtualization. (Even let me host the OKC VMUG on campus because it was a great way to reach others.)

I was introduced to the VMware TAM organization last year when I applied for a position that ended up moving to Denver.  My leadership knew that I was interviewing because I was honest with them, they treated me fairly so I felt I should too.  Well another TAM position opened but this time it was SLED (State, Local and Education) based in Oklahoma.  Right up my alley right? 

Well my last day at OUHSC is now July 9th, three days after starting there six years ago.  And Monday, July 13th, I will start at VMware as a TAM based out of OKC.  It’s been fun and I will keep on preaching what Shared Services is doing in the private cloud space for higher education.  They are leading where others are just talking.

Now how do I legally change my name to Victor Michael Ware? 😉

Monitoring VMs FTW!

Recently I had to help figure out why some customers have been getting slow performance with VMs.  Reservations were used but didn’t help.  What did I do to find the issue?  We are a VMware shop using vCloud Suite Enterprise which gives us vCenter Operations Manager (has a new name but I will always call it vCOPS) and the ability to use custom dashboards.  Sadly I did not have it setup to use LDAP nor shared dashboards.  After getting it going so our Operations staff can login and see the TOP-N graphs as well as the vCloud Director dashboards, I started seeing off the bat several high CPU Ready% VMs.  Wasn’t too long that I was able to see that the VMs were hammering the vCPUs but the default Pay-as-You-Go organization setting in vCD is limiting the vCPU speed to 1GHz.  No wonder right?  The problem is that you have to make that change in the VDC of the organization and then restart the vApp.  Not something you can do just in the middle of the day.

Like below I created a dashboard for our biggest customer so they can see how their environment is doing.

high-CPU-Ready

Notice that the top VM is at 31.5% CPU Ready.  The issue is that this VM is in vCloud Director with the vCPU limited at 1GHz.  Changed it to 4GHz but we cannot reboot this VM in the middle of the day so I removed the limit within vCenter.

high-CPU-Ready1

You cannot tell since I blurred out their VM names but the one that was being limited is now off the list.  This is actually the 2nd VM in the first picture that’s now at the top.

high-cpu-ready3

The above is right after I removed the limit.  Notice the spike to just above 3,750MHz for this 2vCPU VM.  Since they have 2vCPU they were able to hit 2000MHz (1GHz x 2vCPU) but the demand was more at this time.  They could have added more vCPU’s but then we could run into other issues of too many vCPUs per host if that was the stance.  Now I’m not saying the high CPU Ready% is always going to be this case, it could be the we need to right size VMs across the board and maxing out the hosts so there’s a CPU wait going on.  So use what you can and monitor it as often as you can if you are providing services to customers.  This is one case where I was able to find a problem before the customer reported it.  That’s a win in my books!

 

 

 

I am sitting on the flight back home from VMworld and listening to podcasts from the conference. Seems the biggest deal was EVO:RAIL/RACK coming out. At first when I heard about Nutanix two years ago, I didn’t see them being an enterprise ready but man has the area grown.

Enter in Simplivity, I met with the local reps about 4 or so months ago and got to see what they are doing in this hyper-convergence market (like Nutanix and now EVO) and got to say I was impressed. One because they used Dell servers, which is a big partner of ours, but importantly all the functionality they have built in to their product. I had Nutanix sponsor our OKC VMUG last month and next Simplivity on October 24th, so it will good to see the differences. I highly recommend you to research them both so I don’t show bias here.

But why would you go the hyper-convergence route in your data
center? For me lately, I have been tasked to architect building blocks for both server and VDI environments. I totally see these three (Nutanix/Simplivity/EVO:RACK) be the path for VDI. One, each bring their own storage solutions so I can keep the IOPs off our primary storage (cause man Oracle is crazy needy!). And two, they make it easy to say “you need 100 VDIs, you need one of these..” and grow accordingly.

This doesn’t mean they suck at server VMs but I don’t need additional storage for those right now. Most of us already have that, but if you are building new look at them. Save rack space, power, cooling, and network ports by going this way in your virtualization environment. Each have limitations and functionality that the other don’t have but don’t want to start a vendor war.

Now my question, would you switch?

This has been my 6th straight VMworld (Thank you OUHSC for sending me each time!) and following each year we hear about new products or how one product will be replaced with another as the “go to” as in vCD to vCAC movement.  This year was slightly different.  There weren’t really any “new” products but a new suite of existing products, a new way of getting a hyper-convergence from partners with VMware and finally it seems like we can finally say it was the year of EUC (not really just VDI).

I’m still trying to figure this one out.  About 2 years ago, VMware came out with the vCloud Suite that included all of their products needed to make a private cloud.  The following year they came out with a product called Log Insight that was inspired to replace Splunk or other syslog products.  But they didn’t include it in the vCloud Suite which I felt it would have made it an even better sell.  At OU Shared Services, we bought the enterprise edition so we could easily build VMs with vCD (you can see my presentation at VMworld 2013 to find out more) as the ease of per proc license instead of a-la-carte style with per block of VMs.  So where am I going, the vRealize Suite is labeled as “..cloud management platform purpose-built for the hybrid cloud. It provides a comprehensive management stack for IT services on vSphere and other hypervisors, physical infrastructure and external clouds, all with a unified management experience..” Didn’t that sound like vCloud Suite? The catch is that this suite would take the other further into the hybrid-cloud be it off-prem or non-VMware infrastructures.  To which I say “dang you VMware!”  But we have seen VMware make changes before after announcing here so I will hold my breath of frustration until after VMworld Barcelona.

It’s been joked about every year that it was the year of the VDI but it never really seemed to be.  VDI is very hard to get going with some lacking features.  In VMware’s case it was that they didn’t have a good virtual app stance for those that didn’t need full virtual desktops.  VMware Horizon 6 fixed that with RDSH support.  I’ve been playing with this in our lab since release (Thanks vExpert program for licenses!) and really love how easy it is to provision applications in the exact same interface for VDI’s.  During one session they even talked about in the next update they will have printer support that some called out in the weeks leading up to VMworld.  So VMware is definitely engaged in EUC and not playing around.  Air Watch is getting some good looks but I’m not big into the EMM/MDM/whatever they want to be called in the market so I need to follow up more on them.

The biggest announcement, I would say, is EVO: RAIL (and futures RACK).  There was a big burst on Twitter about a picture someone took inside of HQ.  It had the caption “Project Marvin” with a little Android looking bot.  This kicked off speculation on what was going on and many speculated that VMware was getting in the hardware business to attract those looking at Nutanix and Simplivity.  But it’s a half truth.  VMware just made a new product called EVO:RACK that includes the Enterprise Plus license, a vCenter license and VSAN that you buy in 4 node blocks of a 2U “Appliance” from partners like Dell/EMC(what!)Fujitsu/SuperMicro.  I am sure more will come but that’s the initial list.  Still servers but makes it easier for those to deploy a Software-Defined Data Center (SDDC) in their environment.  Heard two different reports of either VMware is the single point of contact for all problems but then heard you contact the partner support and VMware will be the Tier 3 support.  The big take off is that it only takes 13 minutes to deploy these after racking and stacking due to the new HTML interface that configures and setups everything on the nodes.

 

Today is the last day, it was great hanging out with the community and I look forward to future announcements!

Requirements:

One or more RDSH (Windows Terminal Servers) with Horizon View Agent installed.

One or more VMware Horizon View Connection Servers version 6 already setup.

Software installed (or Thinapp’d!)

rdsh-01

In your View Manager, add your Farms for the RDSH hosts:

rdsh-02

Add the installed software:

rdsh-03

rdsh-04

Add entitled users:

rdsh-05

Should look like this after you add the software:

rdsh-06

Add a manual application (if only one host has the software* or if you are pointing to a ThinApp’d streaming package).

*I say if only one host because the software I installed wasn’t on the list but I may not have given it time to find it.

rdsh-07

rdsh-08

rdsh-09

Misconfiguration with the path, Office 15 is no space and should be \Office15\

Now I get:

rdsh-10

Since I only installed Visio on one of the RDSH instances then it will tell you which one is missing the software. If I needed to reboot view-rdsh-01 then the application would not be available in a HA scenario and this is a good way to find out those cases.

Now from the same client for desktops, you can also launch applications as well as create a shortcut for a quick launch of both:

rdsh-11

Since View uses PCoIP, any device that is supported can launch these applications like the desktops with the same interface.  The most time consuming part of the whole thing is just installing the software on the RDSH servers but you don’t need to do anything special.

I had the great benefit of meeting GS last year at VMworld 2013.  Great guy and has a passion to help bring the virtualization community out of just behind our desks staring at monitors.  Being able to listen to what others are doing is an awesome way to learn and make contacts with those doing the same as you.

Episode 19 – Joey “VM” Ware – VCAC’d on what I’ve been doing here at OU|S2 for the last few years.

Check out what his Virtualization User Podcast as a Service and subscribe in iTunes to get the new episodes.

This past Sunday, my phone was blowing up with email alerts.  But what caught my eye was the alert that only happens when vCenter Service starts:

vC-vRAM-Alert

* For those that didn’t know, at one point VMware started this much hated vRAM tax on vSphere 5.0 and as you can see we still have some hosts/licenses tied to this model that they did away with in vSphere 5.1

When I finally saw this alert over and over, I knew that the vCenter Service was automatically restarting almost every 10 minutes.  So after logging in I saw the following in the Windows Event Log:

vC-SQL-Error

As you can see that it states “..database ‘vcenter’ because the ‘PRIMARY’ filegroup is full…”  so I checked the SQL server that was created just for VMware related databases was indeed out of space.  Luckily it’s a VM so I just added more space to and then restarted the VMware VirtualCenter Server service and all was fixed within 10 minutes.

One last thing to check, if you have vCloud Director as well, is to reconnect the vCenter inside of vCD or you may not be able to provision new VMs or open Console:

vCD-Reconnect

OUIT Shared Services – 2013

This year has been packed with things that we’ve been doing in S2 at University of Oklahoma.  We went production with both data centers in OKC and Norman being a stretched cluster.  Brought up vCloud Director to give our users self provisioning and better visibility into their environment.  This meant we had to get storage, both block and file, working across then we even upgraded the controllers for the Dell Compellent SAN.  We were able to do this with little to no downtime to our customers which proved what we designed it to do.

Now was it all easy and full of great smelling flowers?  No way.  I would love to say we can engineer an active/active data center with no issues.  But we have enough great engineers to quickly resolve or remedy a problem.  I’d say over all phase 1 of S2 is going pretty well.  If it wasn’t I doubt I would have been able to present over it on a vBrownBag podcast or a session at VMworld 2013 in San Francisco.  Now that was exciting, not everyone can say they spoke at a conference that is attended by 22,000+ people from around the world.

Phase 1 included:

  • Two Active/Active Datacenters (OKC and Norman with Tulsa being a separate data center for now)
  • 4 Dell Compellent SANs (2 each DC) with 800 TB RAW all together – largest environment in Oklahoma currently
  • 2 EMC Isilon NAS (1 each DC) with around 250 TB that can replicate to each other
  • 4 Cisco 7K’s (2 each DC) and multiple 5K’s and 2K’s that are linked together using either our dark fiber between us or OneNET’s path.
  • Juniper SRX firewalls
  • Palo Alto for IPS/IDS
  • 16 vSphere ESXi 5.1 general cluster nodes (8 each DC/Dell PowerEdge blades)
  • vCloud Director 5.5/vCOPS 5.7
  • View 5.2 (OKC only for now)

Phase 2 will be finishing the upgrade of our VMware environment to vCloud Suite 5.5 and seeing what vCAC 6 could bring to the table.  More training is already scheduled for our Operations and Design teams on vCloud Director, vCOPS, View and CommVault.  Simpana 10 backup is installed and configured but slowly rolling this out so we can finish up transitioning this into Shared Services as an offering to our customers.

It will be a busy year for us in 2014 but it will be a ride.. time will tell if it’s a great, good or heck of one!

Many thanks goes out to:

  • My coworkers, specifically David Stricklin (@strickfila) for backing me up when needed/keeping me in check and our VP David Horton (@hortonhearsyou) for doing all that he does for OU/OUHSC.
  • Sean O’Dell (@theseanodell) from VMware on always assisting me the OKC VMUG, keeping us up-to-date with VMW products and co-presenting at VMworld.
  • The vBrownBag community (@vbrownbag/#vbrownbag) like Jon Harris (@thevcacguy), Damian Karlson (@sixfootdad), Cody Bunch (@cody_bunch) and many others.  Check them out at http://www.professionalvmware.com
  • The VMUG community (@myvmug) as a whole and there’s not another one like it.
  • Also all past and current vExperts to which I’m thankful to say I am 1 out of 581 to be named in 2013.

If you do have Twitter, I highly recommend following these people as well as our Shared Services (@ouits2) and OKC VMUG (@OKCVMUG) pages to see what is going on for 2014.

There are several podcasts that I recommend besides just vBrownBag:

 

vExpert!

Well, today was the day that the vExperts for 2013 would be announced. I applied over a month ago basically for the conferences in OKC that I spoke at, the Shared Services initiative that I have been helping on at work and bigger reason, the OKC VMUG.

I really didn’t think I would be accepted the first year of trying, a vExpert is not something you just earn for saying you do virtualization. It’s a community based upon helping the community grow. It’s not a club that you can say I was once a member and always a member, you have to keep doing your “job” as a virtualization community member or you most likely will not get the vExpert title the following year.

So without further ado, the link to the close to 580 vExperts for 2013.

Thank you all!