Configuring your Resource Providers for Azure Pack
While deploying Windows Azure Pack, several factors plays
its part when it comes to the design and layout of the solution. As you may be
aware of, Windows Azure Pack contains a lot of different sites, APIs and
resource providers – just so that you can enable and realize Azure technologies
within your own datacenter.
It’s more than a glorified self-service portal so the
requirements for the design, load and can be overwhelming for some customers.
Before I get to the big point of this blog post, I would
like to put it into some context first.
Normally at customer sites, we see the following
different designs when it comes to Windows Azure Pack.
Express
Organizations who want to just test and play around are
deploying the single install, express setup of Windows Azure Pack. This will
install all the sites and APIs onto a single virtual machine, and the
organizations can easily add resource providers to start testing the powerful
cloud enabled tool.
Although I have seen some examples where the Express
setup has been used in production, it is far from what we recommend. The public
facing parts of Azure Pack, such as the Tenant Public API, Tenant Site and
eventually Tenant Authentication Site
are directly exposed on the internet. Having everything on the same virtual
machine will increase the attack surface as well as lead to performance, HA and
scale issues.
Configuration
requirements using this design:
There aren’t any hard requirements using the Express
solution as we like to think that people are only using it in lab and test.
However, if you want to make it available and actually use it across firewalls,
you will have to perform the following:
·
Reconfigure tenant site (FQDN, certificate and
port)
·
Reconfigure tenant authentication site (FQDN,
certificate and port)
·
Reconfigure tenant public API (FQDN, certificate
and port)
Optional:
·
Reconfigure admin site (FQDN, certificate and
port)
·
Reconfigure admin authentication site (FQDN,
certificate and port)
Basic
For some of the smaller customers where HA is not the
most important thing, we often see a basic implementation of Windows Azure
Pack. This means that we have a single virtual machine running the high-privileged
services – such as the Admin API, Admin Site, Tenant API and eventually Admin Authentication site together with the
default resource providers. This virtual machine is located behind the firewall
and in most cases within the same Active Directory Domain with its resource
providers (SCVMM+SPF, SQL, ServiceBus, WebSites etc).
For the public facing part (the parts mentioned before,
directly exposed on the internet) they use another – dedicated virtual machine
which might be located in DMZ and available on the internet.
Of course, both the high-privileged VM and the internet
facing VM are running on a Hyper-V cluster so that the VMs themselves are
highly available.
Configuration
requirements using this design:
I strongly recommend using a highly available WAP design
whenever you plan to put it into production. But in this design, the only
presence of HA is at the hypervisor level.
You will have to perform the following using this design:
·
Reconfigure tenant site (FQDN, certificate and
port)
·
Reconfigure tenant public API (FQDN, certificate
and port)
·
Reconfigure tenant authentication site (FQDN,
certificate and port)
o Or
·
Integrate with Active Directory Federation
Services and remove tenant authentication site
Optional:
·
Reconfigure admin site (FQDN, certificate and
port)
·
Reconfigure admin authentication site (FQDN,
certificate and port)
o Or
·
Integrate with Active Directory Federation Services
and remove admin authentication site
Minimal
Distribution
The most common
design of Windows Azure Pack and what’s normally at least what I am
recommending, Is where we have at least two virtual machines for the
high-privileged servers, configured as highly available behind a load balancer,
and the same for the internet facing part.
This will indeed require load balancers and VIPs, but
also some additional reconfiguration when it comes to the Azure Pack
environment.
Configuration
requirements using this desing
Having the high-privileged services as well as the
internet facing parts scaled across several virtual machines, helps us to
address performance, availability and scale issues.
You will have to perform the following reconfiguration to
make this work:
·
Reconfigure tenat site (FQDN, certificate and
port)
·
Reconfigure tenant public API (FQDN, certificate
and port)
·
Reconfigure tenant authentication site (FQDN,
certificate and port)
o Or
·
Integrate with Active Directory Federation
Services and remove tenant authentication site
·
Reconfigure admin site (FQDN, certificate and
port)
·
Reconfigure admin authentication site (FQDN,
certificate and port)
o Or
·
Integrate with Active Directory Federation
Services and remove admin authentication site
·
Reconfigure resource providers (FQDN and
certificate):
o Sqlserver
o MySQL
o Monitoring
o Marketplace
o Usageservice
o Systemcenter
o Webspaces
o Servicebus
Summary
So whenever you plan to scale out and ensure HA across
all sites and APIs, you have to reconfigure the components as mentioned with
the Minimal Distribution design. The same rules apply if you intend to be more
drastic around this, having dedicated VMs for each and every site and API. The
reconfiguration is still mandatory.
Windows Azure Pack has been available for over a year
now, and the majority of organizations are adopting the VM Cloud resource
provider. The good thing here is that even if you have scaled out the SPF
endpoint, you are simply adding the endpoint to the admin API and everything is
handled.
There’s really not much reconfiguration required if you
have configured SPF correctly with FQDN and certificates upfront.
What’s more of a concern is when you want to add resource
providers such as SQL server(s) and/or MySQL server(s).
By default, when you install the first high-privileged
server with the admin API, admin site and so on, you also get the default
resource providers added, such as SQL, MySQL, Usage, Monitoring, Servicebus and
Marketplace. The FQDN’s are bound to the computer name of this machine.
Once you add the second – or even third VM that should be
located behind a load balancer together with the first VM, these resource
providers must also be reconfigured so that you are not pointing toward an
individual virtual machine, but towards a FQDN that is associated with a VIP
behind the load balancer.
Reconfiguring the
default Resource Providers – and why that can be a pain
In order to reconfigure the Windows Azure Pack portals,
APIs and resource providers, we have to instrument the databases in a supported
way. The supported way is through Powershell, and together with my good friend
Flemming Riis, we have convered how to reconfigure the high-privileged services
– as well as the internet facing parts in some earlier blog posts.
As a result of that, I won’t cover it over again, but rather
refer to those URL’s, hoping you will notice them, read them and continue
reading this blog post as I am about to reach my point.
Allright, let
us continue on the resource providers.
You are probably familiar with the reconfiguration of the
tenant and admin stuff by now, and understand that we have several sets of APIs
and portals involved. In the end of the day, everything here should interact
nicely together, being able to reach each other and expose the right set of
information to both an administrator and a tenant.
If we look at the resource providers we are dealing with
directly in the database, we can see that we have several endpoints to each and every resource provider.
We have an endpoint for the resource provider when it
comes from the admin API, and we have an endpoint for the resource provider
when coming from the tenant site and API.
In addition, each resource provider have an endpoint for
usage and notification too.
Example:
The SQLserver resource provider will have the following
endpoints:
So, when only reconfiguring the FQDN, certificates and
ports for the high-privileged services and internet facing parts, these
endpoints are left behind.
The same is applicable for the other resource providers
as well, an in order to turn the in to highly available resource providers, you
must perform this through powershell:
Example:
Since ADSF is added to the mix, we need to create a
function that will allow us to get the token we require for accessing the
AdminURI.
function Get-AdfsToken([string]$adfsAddress, [PSCredential]$credential)
{
$clientRealm
= 'http://azureservices/AdminSite'
$allowSelfSignCertificates
= $true
Add-Type -AssemblyName 'System.ServiceModel,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'
Add-Type -AssemblyName 'System.IdentityModel,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'
$identityProviderEndpoint
= New-Object
-TypeName System.ServiceModel.EndpointAddress
-ArgumentList ($adfsAddress
+ '/adfs/services/trust/13/usernamemixed')
$identityProviderBinding
= New-Object
-TypeName System.ServiceModel.WS2007HttpBinding
-ArgumentList ([System.ServiceModel.SecurityMode]::TransportWithMessageCredential)
$identityProviderBinding.Security.Message.EstablishSecurityContext = $false
$identityProviderBinding.Security.Message.ClientCredentialType =
'UserName'
$identityProviderBinding.Security.Transport.ClientCredentialType =
'None'
$trustChannelFactory
= New-Object
-TypeName System.ServiceModel.Security.WSTrustChannelFactory
-ArgumentList $identityProviderBinding, $identityProviderEndpoint
$trustChannelFactory.TrustVersion =
[System.ServiceModel.Security.TrustVersion]::WSTrust13
if ($allowSelfSignCertificates)
{
$certificateAuthentication
= New-Object
-TypeName System.ServiceModel.Security.X509ServiceCertificateAuthentication
$certificateAuthentication.CertificateValidationMode = 'None'
$trustChannelFactory.Credentials.ServiceCertificate.SslCertificateAuthentication = $certificateAuthentication
}
$ptr = [System.Runtime.InteropServices.Marshal]::SecureStringToCoTaskMemUnicode($credential.Password)
$password
= [System.Runtime.InteropServices.Marshal]::PtrToStringUni($ptr)
[System.Runtime.InteropServices.Marshal]::ZeroFreeCoTaskMemUnicode($ptr)
$trustChannelFactory.Credentials.SupportInteractive
= $false
$trustChannelFactory.Credentials.UserName.UserName = $credential.UserName
$trustChannelFactory.Credentials.UserName.Password = $password #$credential.Password
$rst = New-Object -TypeName System.IdentityModel.Protocols.WSTrust.RequestSecurityToken
-ArgumentList ([System.IdentityModel.Protocols.WSTrust.RequestTypes]::Issue)
$rst.AppliesTo = New-Object -TypeName System.IdentityModel.Protocols.WSTrust.EndpointReference
-ArgumentList $clientRealm
$rst.TokenType = 'urn:ietf:params:oauth:token-type:jwt'
$rst.KeyType = [System.IdentityModel.Protocols.WSTrust.KeyTypes]::Bearer
$rstr = New-Object -TypeName System.IdentityModel.Protocols.WSTrust.RequestSecurityTokenResponse
$channel
= $trustChannelFactory.CreateChannel()
$token
= $channel.Issue($rst, [ref] $rstr)
$tokenString = ([System.IdentityModel.Tokens.GenericXmlSecurityToken]$token).TokenXml.InnerText;
$result =
[System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($tokenString))
return
$result
}
Next, we will list the resource providers we have and
query for the endpoints:
### Change the variables to fit your environment
$username = 'domain\username'
$password = 'P@@$Word'
$adminuri
= ‘https://adminapi.domain.com:30004’
$securePassword = ConvertTo-SecureString
-String $password
-AsPlainText -Force
$credential = New-Object -TypeName System.Management.Automation.PSCredential
-ArgumentList $username,$securePassword
$token = Get-AdfsToken -adfsAddress
$adfsAddress -credential
$credential
$token
$FQDN
= ‘https://wap.domain.com’
### Get a list of all your resource providers
Get-MgmtSvcResourceProvider -IncludeSystemResourceProviders -AdminUri $adminUri
-Token $token
-DisableCertificateValidation | Format-List -Property "Name"
Get-MgmtSvcResourceProvider -IncludeSystemResourceProviders -AdminUri $adminuri
-Token $token
-DisableCertificateValidation | fl -Property "name"
# Get a list of resource providers with the current configured
endpoint values
$rp = Get-MgmtSvcResourceProvider -IncludeSystemResourceProviders
-AdminUri $adminUri
-Token $token
-DisableCertificateValidation
$rp | Select Name, @{e={$_.AdminEndPoint.ForwardingAddress}},
@{e={$_.TenantEndpoint.ForwardingAddress}}, @{e={$_.UsageEndpoint.ForwardingAddress}},
@{e={$_.healthcheckendpoint.forwardingaddress}}, @{e={$_.notificationendpoint.forwardingaddress}}
# STEP 1 - Configure new FQDN for the SQLserver resource
provider
$resourceProviderName = "sqlservers"
$adminEndpoint = "https://$FQDN:30010/"
$tenantEndpoint = "https://$FQDN:30010/subscriptions"
$usageEndpoint = "https://$FQDN:30010/"
$healthCheckEndpoint = $null
$notificationEndpoint = "https://$FQDN:30010/"
$rp = Get-MgmtSvcResourceProvider -Name
$resourceProviderName -IncludeSystemResourceProviders -AdminUri $adminUri
-Token $token
-DisableCertificateValidation
# update all the endpoints using the new fqdn:
if ($rp.AdminEndpoint -and
$adminEndpoint) {
# update endpoint
$rp.AdminEndpoint.ForwardingAddress =
New-Object System.Uri($adminEndpoint)
}
if ($rp.TenantEndpoint -and
$tenantEndpoint) {
# update endpoint
$rp.TenantEndpoint.ForwardingAddress =
New-Object System.Uri($tenantEndpoint)
}
if ($rp.UsageEndpoint -and
$usageEndpoint) {
# update endpoint
$rp.UsageEndpoint.ForwardingAddress =
New-Object System.Uri($usageEndpoint)
}
if ($rp.HealthCheckEndpoint -and
$healthCheckEndpoint) {
# update endpoint
$rp.HealthCheckEndpoint.ForwardingAddress =
New-Object System.Uri($healthCheckEndpoint)
}
if ($rp.NotificationEndpoint -and
$notificationEndpoint) {
# update endpoint
$rp.NotificationEndpoint.ForwardingAddress =
New-Object System.Uri($notificationEndpoint)
}
Set-MgmtSvcResourceProvider -ResourceProvider $rp
-AdminUri $adminUri
-Token $token
-DisableCertificateValidation -Force
# Repeat STEP 1 and STEP 2 on the remaining resource providers
By following the steps in this blog post, you shouldn’t have any warnings or errors in your WAP portals.
