Wednesday, August 20, 2014




Our whitepaper “Hybrid Cloud with NVGRE (Cloud OS)” has reached an unbelievable milestone today.
The paper has been downloaded more than 10.000 times!

I am truly humbled and honored to know that our effort by writing this whitepaper has been greatly appreciated by the community worldwide.

The Story

The initial idea back in October in 2013 was to create a comprehensive guide that would help people to implement Network Virtualization (NVGRE) using System Center Virtual Machine Manager 2012 R2 and Hyper-V in Windows Server 2012 R2.
I started the job together with Flemming Riis – which had this real-world fabric available for us to test, crash and build. Learn, apply, and repeat.

The first release was based on the Preview bits of the Cloud OS and we decided to update the content with the RTM builds as soon as possible; in order to address many questions we knew would arise in the different TechNet forums.

Our reviewers was Daniel Neumann and Stanislav Zhelyazkov.

We had now a new version.

In addition, we added a comprehensive “FAQ” chapter to the paper based on experience from early adoption, TechNet forums and feedback.
This is when we decided that we had to “hire” Stanislav Zhelyazkov. He provided us with unique details that gave a much better quality to the whitepaper, and especially around this section.

This was the third version.

Our forth version added Windows Azure Pack to the paper where we ended up putting NVGRE into context. We highlighted how to leverage the multi-tenant IaaS platform we had been building with VMM, with the service management API in Azure Pack. This was a big update, which included several other elements such as gallery items, remote console and much more.

Our fifth version is where we added a “FAQ” chapter for the Azure Pack part, and hence the “hiring” of Marc Van Eijk, which gave us a deeper insight and a better perspective based on his experience as well, and this is still the current version that we know are helping people on a day-to-day basis.

We know that this whitepaper has been greatly appreciated by Microsoft Support and is widely used by their customers when they are facing challenges regarding these technologies. That is truly a confirmation that we really did something useful this time J

My promise to you:

Instead of putting too much effort into books with all those heavy processes, I will instead continue to write fresh, up-to-date and deep technical whitepapers that can make your life easier.
That means that you can expect more to come from this side as we are seeing new releases of this stack.

I also know that my team are with me and on behalf of them; I can only say that we are very grateful and appreciate all the feedback we have received on our way.

A big thank you from me, Flemming, Stanislav, Daniel and Marc!

Monday, August 18, 2014

VM Cloud is missing in Windows Azure Pack

Recently, I’ve encountered a bug when working with WAP and VM Cloud as the resource provider.


You have connected the service management API to your SPF endpoint and added a VMM management stamp together with a Remote Desktop Gateway.

If you decide to change the FQDN of the Remote Desktop Gateway registered with your VMM management stamp, you will end with a blank VM Cloud in the admin portal.
The connection to the SPF endpoint is still present, but the VMM management stamp with its cloud is missing.

This causes also the VMs and the virtual network for the tenants to appear as missing in the tenant portal.

On the SPF server you will find the following event logged for ManagementODataServices:

On the server where the admin API is installed, you will find the following in the event viewer:

When you make changes to the FQDN of the Remote Desktop Gateway in WAP, you will have another SCSPFServer record present in SPF together with a SCSPFSetting that has the same ID as the previous records.

As you can see from the screenshot below, we have now two records of the ServerType “RDGateway”.

If we dig deeper, the following screenshot shows that we have two entries with the same ID, both registered to the VMM management stamp.

In short, the VMM management stamp is registered again, which generates a duplicate ID that results in this behavior.


In order to clean up, we have to work directly on the SPF server using the SPFAdmin module with PowerShell.

Note: when doing this correctly, you will not delete, loose or cause any harm to your production environment so pay attention.

1.       Log on to your SPF server and import the SPFAdmin module

2.       Run the following cmdlets to identify and remove your RDGateway servers! In our case, we have two records and have to remove both of them before we later add the RDGateway we want.
The reason for that is that because when you try to add the RDGateway in WAP afterwards, you will see that this column is empty although it exist in SPF. If you try to add the RDGateway again, you will end up in the exact same situation. Therefore we must remove both servers in SPF.

3.       Remove the duplicate SCSpfSetting with the following cmdlets. The SCSpfSetting on the top is the setting you want to remove with the duplicate ID.

4.       Next, we want to register the RDGateway directly to our stamp with SPF to avoid creating duplicate ID's.

Once this is done, you can perform a refresh in both the admin portal and the tenant portal, and your VMM management stamp should again be present.
Also edit the connection to verify that the RDGW is registered with the correct values.

 Please note: If you register your VM Cloud resource provider in WAP with all the settings at once, you will not run into this issue. It's only if you decide to add the RDGateway afterwards, or are making changes to the existing one.

Tuesday, August 12, 2014

Applied UR3 for VMM? Update your VMM DHCP Server Extension now!

Update your SCVMM DHCP Server Extension now!

From the KB:

“When using System Center 2012 R2 Virtual Machine Manager (VMM 2012 R2), you may discover that some virtual machines that are deployed on Hyper-V Network Virtualization networks with dynamic IP address allocation may not get an IP address for a few minutes after a reboot of the VM. Eventually the VM gets the IP address and otherwise functions normally.

The behavior can occur if the host has an older version of the VMM DHCP server extension. In order to verify this, find the version of “Microsoft System Center Virtual Machine Manager DHCP Server (x64)” installed on the host by running the following Powershell command:

Get-WmiObject –Class win32_product –filter ‘Name = “Microsoft System Center Virtual Machine Manager DHCP Server (x64)”’

The resolution is to first uninstall the old version of the DHCP extension manually, and then install the updated version from VMM installation folder\SwExtn\DHCPExtn.msi

Default path is: C:\Program Files\Microsoft System Center 2012 R2\Virtual Machine Manager\SwExtn\DHCPExtn.msi

Once this is done, the VMs should no longer experience a delay in acquiring an IP address after a reboot.


Windows Server 2012 R2 Hyper-V introduced several enhancements related to NVGRE.
One of these things are “Dynamic IP address learning”.

What is Dynamic IP address learning?

Feedback from customers told Microsoft that it was important to enable highly available services to run in a VM network. To support that, dynamic IP address learning was brought to the table.
In other words, services such as DHCP, DNS and AD is supported in a NVGRE based network on Hyper-V.
First for broadcast or multicast packets in a VM network, we will use a PA multicast IP address if configured. However, the typical data center operator does not enable multicast in their environments. As a result, when a PA multicast address is not available we use intelligent PA unicast replication. What this means is that we unicast packets only to PA addresses that are configured for the particular virtual subnet the packet is on. In addition, we only send one unicast packet per host no matter how many relevant VMs are on the host. Finally, once a host learns a new IP address it notifies SCVMM. At this point, the learned IP address becomes part of the centralized policy that SCVMM pushes out. This allows for both rapid dissemination of HNV routing policy and limits the network overhead for disseminating this HNV routing policy.”

Second, what is the SCVMM DHCP Extension?

In order to leverage NVGRE, you would have to manage your hosts entirely with powershell, if you didn't have VMM in place.

When VMM is in place (and really should be in place, when using NVGRE), VMM act as the complete management layer, also for the NVGRE part. Since NVGRE basically is a policy driven technology, VMM need to keep track of every IP address used with NVGRE. During deployment of virtual machines connected to a VM Network with NVGRE (often referred to as Customer Addresses), VMM is able to configure static IP addresses on to these VMs, using the agent you are mentioning. This was introduced in VMM 2012 SP1, and is present in the R2 Release. 
Therefore, to summarize, it is a Hyper-V Switch Extension that is required on all Windows Server with Hyper-V to have DHCP to work correctly.

Great. But can I deploy the updated agents to all my Hyper-V hosts in a single operation?

From your VMM server, you can run the following script (please let it match your computer names in the fabric before you run it):

$setup = "\\vmm01\c$\Program Files\Microsoft System Center 2012 R2\Virtual Machine Manager\SwExtn\DHCPExtn.msi"

Invoke-Command -ComputerName hv03, hv04, hv01, hv02, hvfm01, hvfm02, hvgw01, hvgw02 -Command { $setup }

Monday, August 11, 2014

Free Webinars - Azure Technologies in the Private Cloud

This is just an announcement that I will be holding a presentation related to my last whitepaper, published by Savision in the upcoming weeks.

During this session, I will walk through the importance of a private cloud and how you can make this become real with technologies from Microsoft.
Especially interesting is the focus on Windows Azure Pack that has gotten a lot of attention during the last months.

Dive in to see what Azure Pack is all about and what the benefits are.

I encourage you all to ask any questions during these webcast, as long as it is related to the content or Led Zeppelin ;-)