Reconfigure your Resource Providers for Azure Pack

Configuring your Resource Providers for Azure Pack

While deploying Windows Azure Pack, several factors plays its part when it comes to the design and layout of the solution. As you may be aware of, Windows Azure Pack contains a lot of different sites, APIs and resource providers – just so that you can enable and realize Azure technologies within your own datacenter.

It’s more than a glorified self-service portal so the requirements for the design, load and can be overwhelming for some customers.

Before I get to the big point of this blog post, I would like to put it into some context first.

Normally at customer sites, we see the following different designs when it comes to Windows Azure Pack.

Express

Organizations who want to just test and play around are deploying the single install, express setup of Windows Azure Pack. This will install all the sites and APIs onto a single virtual machine, and the organizations can easily add resource providers to start testing the powerful cloud enabled tool.

Although I have seen some examples where the Express setup has been used in production, it is far from what we recommend. The public facing parts of Azure Pack, such as the Tenant Public API, Tenant Site and eventually Tenant Authentication Site are directly exposed on the internet. Having everything on the same virtual machine will increase the attack surface as well as lead to performance, HA and scale issues.

Configuration requirements using this design:

There aren’t any hard requirements using the Express solution as we like to think that people are only using it in lab and test. However, if you want to make it available and actually use it across firewalls, you will have to perform the following:

·       Reconfigure tenant site (FQDN, certificate and port)

·       Reconfigure tenant authentication site (FQDN, certificate and port)

·       Reconfigure tenant public API (FQDN, certificate and port)

Optional:

·       Reconfigure admin site (FQDN, certificate and port)

·       Reconfigure admin authentication site (FQDN, certificate and port)

Basic

For some of the smaller customers where HA is not the most important thing, we often see a basic implementation of Windows Azure Pack. This means that we have a single virtual machine running the high-privileged services – such as the Admin API, Admin Site, Tenant API and eventually Admin Authentication site together with the default resource providers. This virtual machine is located behind the firewall and in most cases within the same Active Directory Domain with its resource providers (SCVMM+SPF, SQL, ServiceBus, WebSites etc).

For the public facing part (the parts mentioned before, directly exposed on the internet) they use another – dedicated virtual machine which might be located in DMZ and available on the internet.

Of course, both the high-privileged VM and the internet facing VM are running on a Hyper-V cluster so that the VMs themselves are highly available.

Configuration requirements using this design:

I strongly recommend using a highly available WAP design whenever you plan to put it into production. But in this design, the only presence of HA is at the hypervisor level.

You will have to perform the following using this design:

·       Reconfigure tenant site (FQDN, certificate and port)

·       Reconfigure tenant public API (FQDN, certificate and port)

·       Reconfigure tenant authentication site (FQDN, certificate and port)

o   Or

·       Integrate with Active Directory Federation Services and remove tenant authentication site

Optional:

·       Reconfigure admin site (FQDN, certificate and port)

·       Reconfigure admin authentication site (FQDN, certificate and port)

o   Or

·       Integrate with Active Directory Federation Services and remove admin authentication site

Minimal Distribution

The most common design of Windows Azure Pack and what’s normally at least what I am recommending, Is where we have at least two virtual machines for the high-privileged servers, configured as highly available behind a load balancer, and the same for the internet facing part.

This will indeed require load balancers and VIPs, but also some additional reconfiguration when it comes to the Azure Pack environment.

Configuration requirements using this desing

Having the high-privileged services as well as the internet facing parts scaled across several virtual machines, helps us to address performance, availability and scale issues.

You will have to perform the following reconfiguration to make this work:

·       Reconfigure tenat site (FQDN, certificate and port)

·       Reconfigure tenant public API (FQDN, certificate and port)

·       Reconfigure tenant authentication site (FQDN, certificate and port)

o   Or

·       Integrate with Active Directory Federation Services and remove tenant authentication site

·       Reconfigure admin site (FQDN, certificate and port)

·       Reconfigure admin authentication site (FQDN, certificate and port)

o   Or

·       Integrate with Active Directory Federation Services and remove admin authentication site

·       Reconfigure resource providers (FQDN and certificate):

o   Sqlserver

o   MySQL

o   Monitoring

o   Marketplace

o   Usageservice

o   Systemcenter

o   Webspaces

o   Servicebus

Summary

So whenever you plan to scale out and ensure HA across all sites and APIs, you have to reconfigure the components as mentioned with the Minimal Distribution design. The same rules apply if you intend to be more drastic around this, having dedicated VMs for each and every site and API. The reconfiguration is still mandatory.

Windows Azure Pack has been available for over a year now, and the majority of organizations are adopting the VM Cloud resource provider. The good thing here is that even if you have scaled out the SPF endpoint, you are simply adding the endpoint to the admin API and everything is handled.

There’s really not much reconfiguration required if you have configured SPF correctly with FQDN and certificates upfront.

What’s more of a concern is when you want to add resource providers such as SQL server(s) and/or MySQL server(s).

By default, when you install the first high-privileged server with the admin API, admin site and so on, you also get the default resource providers added, such as SQL, MySQL, Usage, Monitoring, Servicebus and Marketplace. The FQDN’s are bound to the computer name of this machine.

Once you add the second – or even third VM that should be located behind a load balancer together with the first VM, these resource providers must also be reconfigured so that you are not pointing toward an individual virtual machine, but towards a FQDN that is associated with a VIP behind the load balancer.

Reconfiguring the default Resource Providers – and why that can be a pain

In order to reconfigure the Windows Azure Pack portals, APIs and resource providers, we have to instrument the databases in a supported way. The supported way is through Powershell, and together with my good friend Flemming Riis, we have convered how to reconfigure the high-privileged services – as well as the internet facing parts in some earlier blog posts.

As a result of that, I won’t cover it over again, but rather refer to those URL’s, hoping you will notice them, read them and continue reading this blog post as I am about to reach my point.

Internet facing parts – Flemming Riis

High-privileged services – Kristian Nese

Tenant Public API – Kristian Nese

Admin API – Kristian Nese

Allright, let us continue on the resource providers.

You are probably familiar with the reconfiguration of the tenant and admin stuff by now, and understand that we have several sets of APIs and portals involved. In the end of the day, everything here should interact nicely together, being able to reach each other and expose the right set of information to both an administrator and a tenant.

If we look at the resource providers we are dealing with directly in the database, we can see that we have several endpoints to each and every resource provider.

We have an endpoint for the resource provider when it comes from the admin API, and we have an endpoint for the resource provider when coming from the tenant site and API.

In addition, each resource provider have an endpoint for usage and notification too.

Example:

The SQLserver resource provider will have the following endpoints:

AdminEndPoint.ForwardingAddress: https://FQDN:30010/

TenantEndPoint.ForwardingAddress: https://FQDN:30010/Subscriptions/

UsageEndPoint.ForwardingAddress: https://FQDN:30010/

NotificationEndPoint.ForwardingAddress: https://FQDN:30010/

So, when only reconfiguring the FQDN, certificates and ports for the high-privileged services and internet facing parts, these endpoints are left behind.

The same is applicable for the other resource providers as well, an in order to turn the in to highly available resource providers, you must perform this through powershell:

Example:

Since ADSF is added to the mix, we need to create a function that will allow us to get the token we require for accessing the AdminURI.

function Get-AdfsToken([string]$adfsAddress, [PSCredential]$credential)

{

    $clientRealm = 'http://azureservices/AdminSite'

    $allowSelfSignCertificates = $true

    Add-Type -AssemblyName 'System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'

    Add-Type -AssemblyName 'System.IdentityModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'

    $identityProviderEndpoint = New-Object -TypeName System.ServiceModel.EndpointAddress -ArgumentList ($adfsAddress + '/adfs/services/trust/13/usernamemixed')

    $identityProviderBinding = New-Object -TypeName System.ServiceModel.WS2007HttpBinding -ArgumentList ([System.ServiceModel.SecurityMode]::TransportWithMessageCredential)

    $identityProviderBinding.Security.Message.EstablishSecurityContext = $false

    $identityProviderBinding.Security.Message.ClientCredentialType = 'UserName'

    $identityProviderBinding.Security.Transport.ClientCredentialType = 'None'

    $trustChannelFactory = New-Object -TypeName System.ServiceModel.Security.WSTrustChannelFactory -ArgumentList $identityProviderBinding, $identityProviderEndpoint

    $trustChannelFactory.TrustVersion = [System.ServiceModel.Security.TrustVersion]::WSTrust13

    if ($allowSelfSignCertificates)

    {

        $certificateAuthentication = New-Object -TypeName System.ServiceModel.Security.X509ServiceCertificateAuthentication

        $certificateAuthentication.CertificateValidationMode = 'None'

        $trustChannelFactory.Credentials.ServiceCertificate.SslCertificateAuthentication = $certificateAuthentication

    }

    $ptr = [System.Runtime.InteropServices.Marshal]::SecureStringToCoTaskMemUnicode($credential.Password)

    $password = [System.Runtime.InteropServices.Marshal]::PtrToStringUni($ptr)

    [System.Runtime.InteropServices.Marshal]::ZeroFreeCoTaskMemUnicode($ptr)

    $trustChannelFactory.Credentials.SupportInteractive = $false

    $trustChannelFactory.Credentials.UserName.UserName = $credential.UserName

    $trustChannelFactory.Credentials.UserName.Password = $password #$credential.Password

    $rst = New-Object -TypeName System.IdentityModel.Protocols.WSTrust.RequestSecurityToken -ArgumentList ([System.IdentityModel.Protocols.WSTrust.RequestTypes]::Issue)

    $rst.AppliesTo = New-Object -TypeName System.IdentityModel.Protocols.WSTrust.EndpointReference -ArgumentList $clientRealm

    $rst.TokenType = 'urn:ietf:params:oauth:token-type:jwt'

    $rst.KeyType = [System.IdentityModel.Protocols.WSTrust.KeyTypes]::Bearer

    $rstr = New-Object -TypeName System.IdentityModel.Protocols.WSTrust.RequestSecurityTokenResponse

    $channel = $trustChannelFactory.CreateChannel()

    $token = $channel.Issue($rst, [ref] $rstr)

    $tokenString = ([System.IdentityModel.Tokens.GenericXmlSecurityToken]$token).TokenXml.InnerText;

    $result = [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($tokenString))

    return $result

} 

Next, we will list the resource providers we have and query for the endpoints:

### Change the variables to fit your environment

$adfsAddress = 'https://adfsserver.com

$username = 'domain\username'

$password = 'P@@$Word'

$adminuri = ‘https://adminapi.domain.com:30004’

$securePassword = ConvertTo-SecureString -String $password -AsPlainText -Force

$credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $username,$securePassword

$token = Get-AdfsToken -adfsAddress $adfsAddress -credential $credential

$token

$FQDN = ‘https://wap.domain.com’

### Get a list of all your resource providers

Get-MgmtSvcResourceProvider -IncludeSystemResourceProviders -AdminUri $adminUri -Token $token -DisableCertificateValidation | Format-List -Property "Name"

Get-MgmtSvcResourceProvider -IncludeSystemResourceProviders -AdminUri $adminuri -Token $token -DisableCertificateValidation | fl -Property "name"

# Get a list of resource providers with the current configured endpoint values

$rp = Get-MgmtSvcResourceProvider -IncludeSystemResourceProviders -AdminUri $adminUri -Token $token -DisableCertificateValidation

$rp | Select Name, @{e={$_.AdminEndPoint.ForwardingAddress}}, @{e={$_.TenantEndpoint.ForwardingAddress}}, @{e={$_.UsageEndpoint.ForwardingAddress}}, @{e={$_.healthcheckendpoint.forwardingaddress}}, @{e={$_.notificationendpoint.forwardingaddress}}

# STEP 1 - Configure new FQDN for the SQLserver resource provider

$resourceProviderName = "sqlservers"

$adminEndpoint = "https://$FQDN:30010/"

$tenantEndpoint = "https://$FQDN:30010/subscriptions"

$usageEndpoint = "https://$FQDN:30010/"

$healthCheckEndpoint = $null

$notificationEndpoint = "https://$FQDN:30010/"

# STEP 2 – Modify and update the resource providers

$rp = Get-MgmtSvcResourceProvider -Name $resourceProviderName -IncludeSystemResourceProviders -AdminUri $adminUri -Token $token -DisableCertificateValidation

# update all the endpoints using the new fqdn:

if ($rp.AdminEndpoint -and $adminEndpoint) {

# update endpoint

$rp.AdminEndpoint.ForwardingAddress = New-Object System.Uri($adminEndpoint)

}

if ($rp.TenantEndpoint -and $tenantEndpoint) {

# update endpoint

$rp.TenantEndpoint.ForwardingAddress = New-Object System.Uri($tenantEndpoint)

}

if ($rp.UsageEndpoint -and $usageEndpoint) {

# update endpoint

$rp.UsageEndpoint.ForwardingAddress = New-Object System.Uri($usageEndpoint)

}

if ($rp.HealthCheckEndpoint -and $healthCheckEndpoint) {

# update endpoint

$rp.HealthCheckEndpoint.ForwardingAddress = New-Object System.Uri($healthCheckEndpoint)

}

if ($rp.NotificationEndpoint -and $notificationEndpoint) {

# update endpoint

$rp.NotificationEndpoint.ForwardingAddress = New-Object System.Uri($notificationEndpoint)

}

Set-MgmtSvcResourceProvider -ResourceProvider $rp -AdminUri $adminUri -Token $token -DisableCertificateValidation -Force

# Repeat STEP 1 and STEP 2 on the remaining resource providers

 By following the steps in this blog post, you shouldn’t have any warnings or errors in your WAP portals.

Business Continuity with SCVMM and Azure Site Recovery

Business Continuity for the management stamp

Back in November, I wrote a blog post about the DR integration in Windows Azure Pack, where service providers can provide managed DR for their tenants - http://kristiannese.blogspot.no/2014/11/windows-azure-pack-with-dr-add-on-asr.html

I’ve been working with many service providers over the last months where both Azure Pack and Azure Site Recovery has been critical components.

However, looking at the relatively big footprint with the DR add-on in Update Rollup 4 for Windows Azure Pack, organizations has started in the other end in order to bring business continuity to their clouds.

For one of the larger service providers, we had to dive deep into the architecture of Hyper-V Replica, SCVMM and Azure Site Recovery before we knew how to design the optimal layout to ensure business continuity.

In each and every ASR design, you must look at your fabric and management stamp and start looking at the recovery design before you create the disaster design. Did I lost you there?

What I’m saying is that it’s relatively easy to perform the heavy lifting of the data, but once the shit hit the fans, you better know what to expect.

In this particular case, we had a common goal:

We want to ensure business continuity for the entire management stamp with a single click, so that tenants can create, manage and operate their workloads without interruption. This should be achieved in an efficient way with a minimal footprint.

When we first saw the release of Azure Site Recovery, it was called “Hyper-V Recovery Manager” and required two SCVMM management stamps to perform DR between sites. The feedback from potential customers were quite loud and clear: people wanted to leverage their existing SCVMM investment and perform DR operations with a single SCVMM management stamp. Microsoft listened and let us now perform DR between SCVMM Clouds, using the same SCVMM server.

Actually, it’s over a year ago since they made this available and diving into my archive I managed to find the following blog post: http://kristiannese.blogspot.no/2013/12/how-to-setup-hyper-v-recovery-manager.html

So IMHO, using a single SCVMM stamp is always preferred whenever it is possible, so that was also my recommendations when it came to the initial design for this case.

In this blog post, I will share my findings and workaround for making this possible, ensuring business continuity for the entire management stamp.

The initial configuration

The first step we had to make when designing the management stamp, was to plan and prepare for SQL AlwaysOn Availability Groups.

System Center 2012 R2 – Virtual Machine Manager, Service Manager, Operations Manager and Orchestrator does all support AlwaysOn Availability Groups.

Why plan for SQL AlwaysOn Availability Groups when we have the traditional SQL Cluster solution available for High-Availability?

This is a really good question – and also very important as this is the key for realizing the big goal here. AlwaysOn is a high-availability and disaster recovery solution that provides an enterprise-level alternative to database mirroring. The solution maximizes the availability of a set of user databases and supports a failover environment for those selected databases.

Compared to a traditional SQL Cluster – that can also use shared VHDXs, this was a no brainer. A shared VHDX would have given us a headache and increased the complexity with Hyper-V Replica.

SQL AlwaysOn Availability Groups let us use local storage for each VM within the cluster configuration, and enable synchronous replication between the nodes on the selected user databases.

Alright, the SQL discussion is now over, and we proceeded to the fabric design.

In total, we would have several Hyper-V Clusters for different kind of workload, such as:

·       Management

·       Edge

·       IaaS

·       DR

Since this was a Greenfield project, we had to deploy everything from scratch.

We started with the Hyper-V Management Cluster and from there we deployed two VM instances in a guest cluster configuration, installed with SQL Server for Always On Availability Groups. Our plan was to put the System Center databases – as well as WAP databases onto this database cluster.

Once we had deployed a Highly-Available SCVMM solution, including a HA library server, we performed the initial configuration on the management cluster nodes.

As stated earlier, this is really a chicken and egg scenario. Since we are working with a cluster here, it’s straightforward to configure the nodes – one at a time, putting one node in maintenance mode, move the workload and repeat the process on the remaining node(s). Our desired state configuration at this point is to deploy the logical switch with its profile settings to all nodes, and later provision more storage and define classifications within the fabric.

The description here is relatively high-level, but to summarize: we do the normal fabric stuff in VMM at this point, and prepare the infrastructure to deploy and configure the remaining hosts and clusters.

For more information around the details about the design, I used the following script that I have made available that turns SCVMM into a fabric controller for Windows Azure Pack and Azure Site Recovery integration:

https://gallery.technet.microsoft.com/SCVMM-Fabric-Controller-a1edf8a7

Once the initial configuration was done, we deployed the NVGRE gateway hosts, DR hosts, Iaas hosts, Windows Azure Pack and the remaining System Center components in order to provide service offerings through the tenant portal.

If you are very keen to know more about this process, I recommend to read our whitepaper which covers this end-to-end:

https://gallery.technet.microsoft.com/Hybrid-Cloud-with-NVGRE-aa6e1e9a

Here’s an overview of the design after the initial configuration:

If we look at this from a different – and perhaps a more traditional perspective, mapping the different layers with each other, we have the following architecture and design of SCVMM, Windows Azure Pack, SPF and our host groups:

So far so good. The design of the stamp was finished and we were ready to proceed with the Azure Site Recovery implementation

Integrating Azure Site Recovery

To be honest, at this point we thought the hardest part of the job was done, such as ensuring HA for all the workloads as well as integrating NVGRE to the environment, spinning up complex VM roles just to improve the tenants and so on and so forth.

We added ASR to the solution and was quite confident that this would work as a charm since we had SQL AlwaysOn as part of the solution.

We soon found out that we had to do some engineering before we could celebrate.

Here’s a description of the issue we encountered.

In the Microsoft Azure portal, you configure ASR and perform the mapping between your management servers and clouds and also the VM networks.

As I described earlier in this blog post, the initial design of Azure Site Recovery in an “Enterprise 2 Enterprise” (on-prem 2 on-prem) scenario, was to leverage two SCVMM management servers. Then the administrator had the opportunity to duplicate the network artifacts (network sites, VLAN, IP pools etc) across sites, ensuring that each VM could be brought online on the secondary site with the same IP configuration as on the primary site.

Sounds quite obvious and really something you would expect, yeah?

Moving away from that design and rather use a single SCVMM management server (a single management server, that is highly-available is not the same as two SCVMM management servers), gave us some challenges.

1)      We could (of course) not create the same networking artifacts twice within a single SCVMM management server

2)      We could not create an empty logical network and map the primary network with this one. This would throw an error

3)      We could not use the primary network as our secondary as well, as this would give the VMs a new IP address from the IP pool

4)      Although we could update IP addresses in DNS, the customer required to use the exact IP configuration on the secondary site post failover

Ok, what do we do now?

At that time it felt a bit awkward to say that we were struggling to keep the same IP configuration across sites.

After a few more cups of coffee, it was time to dive into the recovery plans in ASR to look for new opportunities.

A recovery plan groups virtual machines together for the purposes of failover and recovery, and it specifies the order in which groups of VMs should fail over. We were going to create several recovery plans, so that we could easily and logically group different kind of workloads together and perform DR in a trusted way

Here’s how the recovery plan for the entire stamp looks like:

So this recovery plan would power off the VMs in a specific order, perform the failover to the secondary site and then power on the VMs again in a certain order specified by the administrator.

What was interesting for us to see, was that we could leverage our Powershell skills as part of these steps.

Each step can have an associated script and a manual task assigned.

We found out that the first thing we had to do before even shutting down the VMs, was to run a powershell script that would verify that the VMs would be connected to the proper virtual switch in Hyper-V.

Ok, but why?

Another good question. Let me explain.

Once you are replicating a virtual machine using Hyper-V Replica, you have the option to assign an alternative IP address to the replica VM. This is very interesting when you have different networks across your sites so that the VMs can be online and available immediately after a failover.

In this specific customer case, the VLAN(s) were stretched and made available on the secondary site as well, hence the requirement to keep the exact network configuration. In addition, all of the VMs had assigned static IP addresses from the SCVMM IP Pools.

However, since we didn’t do any mapping at the end in the portal, just to avoid the errors and the wrong outcome, we decided to handle this with powershell.

When enabling replication on a virtual machine in this environment, and not mapping to a specific VM network, the replica VM would have the following configuration:

As you can see, we are connected to a certain switch, but the “Failover TCP/IP” checkbox was enabled with no info. You probably know what this means? Yes, the VM will come up with an APIPA configuration. No good.

What we did

We created a powershell script that:

a)       Detected the active Replica hosts before failover (using the Hyper-V Powershell API)

b)      Ensured that the VM(s) were connected to the right virtual switch on Hyper-V (using the Hyper-V Powershell API)

c)       Disabled the Failover TCP/IP settings on every VM

a.       Of all of the above were successful, the recovery plan could continue to perform the failover

b.       If any of the above were failing, the recovery plan was aborted

For this to work, you have to ensure that the following pre-reqs are met:

·        Ensure that you have at least one library server in your SCVMM deployment

·        If you have a HA SCVMM server deployment as we had, you also have a remote library share (example: \\fileserver.domain.local\libraryshare ). This is where you store your powershell script (nameofscript.ps1)  Then you must configure the share as follow:

a.       Open the Registry editor

b.       Navigate to HKEY_LOCAL_MACHINE_SOFTWARE\Microsoft\Microsoft System Center Virtual Machine Manager Server\DRAdaper/Registration

c.        Edit the value ScriptLibraryPath

d.       Place the value as \\fileserver.domain.local\libraryshare\. Specify the full fully qualified domain name (FQDN).

e.       Provide permission to the share location

This registry setting will replicate across your SCVMM nodes, so you only have to do this once.

Once the script has been placed in the library and the registry changes are implemented, you can associate the script with one or more tasks within a recovery plan as showed below.

Performing the recovery plan(s) now would ensure that every VM that was part of the plan, was brought up at the recovery site with the same IP configuration as on the primary site.

With this, we had a “single-button” DR solution for the entire management stamp, including Windows Azure Pack and its resource providers.

-kn

Azure Site Recovery - Survey

Happy New Year!

Now, let us get back to work.

I have made a very short survey just to get a better understanding of the potential DR scenarios with Microsoft Azure Site Recovery.

As you already know, Azure can be your DR site today, where you can have ongoing replication from your private cloud(s) to Azure, which eliminates the need for a secondary site that you have to manage and operate yourself.

However, there are some limitations when using Azure, such as lack of support for Generation 2 VMs and the advance usage of VHDX.

Please take 30 seconds to complete this short survey - and I will be very grateful.

https://no.surveymonkey.com/s/KWSCD6W

-kn