Troubleshooting SCVMM crashes (collect traces)

I am currently testing the SCVMM 2012 BETA, and I want to share how you collect SCVMM traces.

(This applies to both SCVMM 2008 R2 and SCVMM 2012)

I had an issue that when I added a WSUS server to the Fabric, the VMMservice crashed. It only crashed when it was properly configured. That means, if I configured the wrong TCP-port for the WSUS connection, I got a correct error message. But when I hit the high notes, it crashed with no mercy.

The recipe that Carmen Summers (the Program Manager of SCVMM) has made available:

From what computer should I collect the traces:

If it`s a console crash issue.

·         Please collect the traces from both the computer where you run admin console, and your VMM Server.

If it’s an “Add Hosts” issue,

·         Please collect the traces from the VMM server;

If it’s a host status (Needs Attention) or VM issue,

·         Please collect the traces from both the VMM server and the host in question.

If it's self-service portal issue,

·         Please collect the traces from the Web server and the VMM server

What are the steps to collect traces?

Install DebugView from http://www.microsoft.com/technet/sysinternals/utilities/debugview.mspx on your VMM server, your host in question and your Web server (if it's a self-service portal issue).

Save the following code into a text file and name it as "odsflags.cmd":

@echo off
echo ODS control flags - only trace with set flags will go to ODS

if (%1)==() goto :HELP
if (%1)==(-?) goto :HELP
if (%1)==(/?) goto :HELP

echo Setting flag to %1...
reg ADD "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Tracing\Microsoft\Carmine" /v ODSFLAGS /t REG_DWORD /d %1 /f
echo Done.
goto :EXIT

:HELP
echo Usage: odsflags [flag], where flag is
echo TRACE_ERROR = 0x2,
echo TRACE_DBG_NORMAL = 0x4,
echo TRACE_DBG_VERBOSE = 0x8,
echo TRACE_PERF = 0x10,
echo TRACE_TEST_INFO = 0x20,
echo TRACE_TEST_WARNING = 0x40,
echo TRACE_TEST_ERROR = 0x80,

:EXIT

·         Save the following code into a text file and name it as "odson.reg":

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Tracing\Microsoft\Carmine]
"ODS"=dword:00000001

·         Save the following code into a text file and name it as "odsoff.reg":

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Tracing\Microsoft\Carmine]
"ODS"=dword:00000000

·         Copy the above three files onto your VMM server, your host in question and your Web server (if it's a self-service portal issue).

·         In a command window on the machine that you want to capture VMM tracing, run “odson.reg” and “odsflags.cmd 255”. (If you need to collect traces for both VMM Server and the host or the Web server, make sure to run these commands on all computers.)

·         Open DebugView and run it as administrator, make sure that in its Capture menu, you have both "Capture Win32" and "Capture Global Win32" checked. You should be able to see tracing from the VMM components showing up in the DebugView. (If you need to collect traces for both VMM Server and the host, make sure to do these steps on all computers.)

·         Restart vmmservice on VMM server with “net stop vmmservice” and “net start vmmservice”.

·         Restart the agent service on the host with “net stop vmmagent” and “net start vmmagent”.

·         Restart the IIS service on the Web server with "iisreset".

·         Reproduce the issue that you found.

·         Save the output from the DebugView to a text file and email it to the people who can help you diagnose the issue.

·         Don't forget to turn off the tracing after you are done collecting by running "odsoff.reg" on the machine

EXAMPLE:

In my case, where the VMMservice crashed when I added the WSUS server, I was able to locate the following in the VMM.LOG afterwards:

00004729             77.38172150       [5092] 13E4.0868::07/27-21:26:31.413#04:UpdateServer.cs(265): Adding Update Server to Pangaea, ServerName - VMM.lab.local, Port - 8530, SSLEnabled - False             

00004730             77.38208008       [3668] 0E54.0AD4::07/27-21:26:31.411#21:Callback.cs(53): Client uuid:a3258eb4-18cf-4f58-811f-0692c049677e;id=1 - events processed       

00004731             77.48566437       [3668] 0E54.0AD0::07/27-21:26:31.523#24:ConsoleViewModel.cs(294): UI Load: ConsoleViewModel completed AddPage for Jobs - 00:00:00.1359864    

00004732             77.48571014       [3668] 0E54.0AD0::07/27-21:26:31.523#24:ConsoleViewModel.cs(303): ConsoleViewModel begin OnClientCacheInitialized

00004733             77.66319275       [3668] 0E54.0AD0::07/27-21:26:31.700#24:ConsoleViewModel.cs(329): UI Load: ConsoleViewModel completed OnClientCacheInitialized - 00:00:00.1769823      

00004734             77.94854736       [432]    

00004735             77.94854736       [432] *** HR originated: -2147024774  

00004736             77.94854736       [432] ***   Source File: d:\iso_whid\amd64fre\base\isolation\com\copyout.cpp, line 1302               

00004737             77.94854736       [432]    

00004738             77.94861603       [432]    

00004739             77.94861603       [432] *** HR propagated: -2147024774              

00004740             77.94861603       [432] ***   Source File: d:\iso_whid\amd64fre\base\isolation\com\enumidentityattribute.cpp, line 144           

00004741             77.94861603       [432]    

00004742             77.94880676       [432]    

00004743             77.94880676       [432] *** HR originated: -2147024774  

00004744             77.94880676       [432] ***   Source File: d:\iso_whid\amd64fre\base\isolation\com\copyout.cpp, line 1302               

00004745             77.94880676       [432]    

00004746             77.94882202       [432]    

00004747             77.94882202       [432] *** HR propagated: -2147024774              

00004748             77.94882202       [432] ***   Source File: d:\iso_whid\amd64fre\base\isolation\com\enumidentityattribute.cpp, line 144           

00004749             77.94882202       [432]    

00004750             77.94924927       [432]    

00004751             77.94924927       [432] *** HR originated: -2147024774  

00004752             77.94924927       [432] ***   Source File: d:\iso_whid\amd64fre\base\isolation\com\copyout.cpp, line 1302               

00004753             77.94924927       [432]    

00004754             77.94928741       [432]    

00004755             77.94928741       [432] *** HR propagated: -2147024774              

00004756             77.94928741       [432] ***   Source File: d:\iso_whid\amd64fre\base\isolation\com\enumidentityattribute.cpp, line 144           

00004757             77.94928741       [432]    

00004758             82.27567291       [5092] 13E4.0868::07/27-21:26:36.311#04:WatsonExceptionReport.cs(756): Unhandled exception caught.          

00004759             82.27619934       [5092] 13E4.0868::07/27-21:26:36.312#04:WatsonExceptionReport.cs(757): Unhandled exception.         

00004760             82.27857208       [5092] 13E4.0868::07/27-21:26:36.314#04:WatsonExceptionReport.cs(757): System.ArgumentOutOfRangeException: An attempt was made to access an invalid or unsupported language.                

The last line indicates that this is an issue caused by my regional settings on my servers. Since this is a Beta, there is no support for non US regional settings.

SCVMM 2012 - the main tool for you infrastructure and clouds

One of the most viewed blog post I`ve written, is about Failover Cluster Manager, SCVMM, and Hyper-V Manager – which tool you should use to manage your virtual infrastructure.

SCVMM 2008 R2 had a lack of features when it came to managing your cluster, and you would normally rely on the information you got from Failover Cluster Manager, and not the SCVMM 2008 R2 console.

It would be moderate to say that this has changed a bit with SCVMM 2012.

You are not only able to manage your Hyper-V Cluster with SCVMM 2012, you can also create Hyper-V Clusters.

A Hyper-V Cluster is a major part of the private cloud, and is combination of the physical resources configured in the Fabric. As I have written several times before, SCVMM 2012 gives you the possibility to manage your entire virtual datacenter with a single console. This includes deep storage integration through the SMI-S protocol, out of band management so that you can deploy bare metal Hyper-V servers, and to use them in conjunction with Power Optimization and Dynamic Optimization (Dynamic Optimization does not require out of band management).

In a nutshell, you are able to do the following to get your private cloud ready:

1)      Deploy SCVMM 2012

2)      Deploy bare metal Hyper-V servers

3)      Deploy Hyper-V Clusters

4)      Assign LUN`s, CSV, and networking

5)      Take portions of the physical resources in the Fabric, and assign it to a private cloud that you define

6)      Assign the cloud to your users. (Normally after you have configured some available and usuable resources in the library)

Another thing to note, is that SCVMM 2008 R2 did not let you create stand-alone VM`s on the cluster, but you`re able to do this in SCVMM 2012. This is ideal when it comes to the domain controller-dilemma  - since you should not locate your DC`s on a CSV only, but have them outside the cluster, so that the cluster would come online again after a complete shutdown. In other words, you can put them on dedicated LUN`s, or on DAS. For more information, see the Failover Clustering and Domain Requirements (by example) article.

Another feature that is worth mentioning, is the ability to patch your cluster with a dedicated WSUS server. This one will save you some serious amount of time, since there is no need for manual input.

Again, in a nutshell, this is what you would do:

1)      Define a baseline of patches that WSUS should deploy

2)      All the servers in the Fabric would be scanned to determine which hosts are not compliant

3)      The non-compliant hosts are identified

4)      An orchestrated patching workload will place a node into “maintenance mode”, live migrate the VMs to the other nodes in the cluster, patch the node, remote it from “maintenance mode”, and move the VM`s back

So, with SCVMM 2012 on its way, we can look forward to a more simplified day-to-day job.

From now on, this will be the tool, not just to manage your virtual infrastructure and clouds, but also the starting point when you are dealing with deployment of the various servers, storage, and networks you are in control of.

Cheers,

Load Balancing and Cloud Computing (SCVMM 2012)

One characteristic of cloud computing is virtualized network access to your applications/services. The point is that no matter where you access the service, you are automatically directed to the available resources. We are using load balancing to distribute service requests to these resources.

This is normally implemented using hardware load balancers, like F5 Big IP servers.

What can be load balanced?

·         Access to application instances

·         Services such as DNS, FTP, HTTP, and HTTPS

·         Storage resources

·         Connections through intelligent switches

If you ask me, cloud computing would be difficult to “achieve” without load balancing. It provides redundancy through redirection, and fault tolerance when coupled with a failover mechanism.

So, this brings us to the Load Balancing feature in System Center Virtual Machine Manager 2012.

First, download the load balancer from Citrix (NetScaler VPX) which you can run as a virtual machine. (Yeah, I know I said hardware load balancer, but this will simulate their load balancer and is ideal for testing).

Here`s the link by the way: Download NetScaler VPX on Hyper-V. You have to register to Citrix to get a license, and you can do it here.

1)      After the download, extract the content, and import the VM into Hyper-V. (Important: the VM needs 2 vCPU, and at least 4GB RAM. Also connect it to your virtual network you intend to use)

2)      Configure IP, subnet, and gateway

3)      Access the NetScaler VPX through a web browser, with the IP address you assigned

4)      Activate the product… (I was not sure how to obtain a license key on this one. After a long registration process, and some cups of coffee, I decided to test the twitter power. Luckily, there was one Citrix-guru who responded in about 60 seconds (thx to @Niraj_S_Patel). He gave me a tip that to obtain the license, you had to provide Citrix with the MAC-address of you NetScaler VM. This was actually confusing, since Citrix called it “hostname”)

5)      Download the license, and upload it to the NetScaler through the web-interface.

6)      Activate the features in NetScaler

Once all this is done, we can move over to the Fabric in VMM 2012.

First thing first: Download and install Citrix Netscaler LB Provider into VMM 2012. (You may need a reboot before VMM recognizes the provider)

1)      Go to Networking

2)      Click Load Balancers

3)      Right click on Load Balancers, and select “Add Load Balancer”

4)      On the Specify Run As account page, find your Run As account with permission to the load balancer. If you`re using the NetScaler VM for Hyper-V, the username and password is nsroot.

5)      Select the check box next to each host group where the load balancer should be available

6)      On the Manufacturer and Model page, select the load balancer manufacturer and model. (Citrix Systems, Inc. and Citrix NetScaler)

7)      On the Address page, specify IP, FQDN, or NetBIOS name of the load balancer. In the port number box, enter the port number that you use to connect to for management of the load balancer.

8)      On the Provider page, click an available provider to use for load balancer configuration.

9)      Validate, and test the load balancer.

10)   You`re now done, and have hopefully added your NetScaler load balancer.

We have now added a load balancer to our Fabric, and to take the next step, we`ll create a VIP Template. This is a template that contains a load balancer, and is related to specific type of network traffic. You can have many templates, one for HTTP, HTTPS, yeah, you got it.

This is ideal when you want to load balance a service tier. (You can add load balancer into the service designer when you are designing your services).

Since we`re still in the Fabric, navigate to:

1)      Networking, and click VIP Templates

2)      Right click VIP Templates, and click Create VIP Template

3)      Give it a name, description, and the virtual IP port to use. The virtual IP port is the port that is used for the type of network traffic that you want to load balance.

4)      On the Type page, select either Generic or Specific. (Generic can be used on any supported load balancer, and Specific applies to specific load balancers (model).

5)      On the Protocol page, select the protocol you intend to use. (HTTP, HTTPS, or Custom)

6)      On the Persistence page, you can select Enable persistence to enable session persistence (affinity).

7)      On the Load balancing method page, you can choose between the fast connections, round robin, or the least connections.

8)      On the Healt Monitors page, specify a request that will occur at regular intervals against the load balancer to verify that a load balancer is available.

9)      You`re done! Click finish, and you have the first VIP Template created, and ready for action.

Cheers,

SCVMM 2012 – What does it mean to your business?

I`ve read in several places, that there is no such thing, as a private cloud.

Ok. What do you mean?

“To have an elasticity that could be used in a cloud-scenario, our organization would have to invest in some serious hardware”

Yes – it`s all a matter of cost, right? (Maybe I should have use the correct terms here. CapEx and OpEx)

The thing about cloud, is that you should start small, and think big.

SCVMM 2012 fits in right here.

·         Bare-Metal deployment of your Hyper-V servers, as well as creating Hyper-V servers using your existing Windows 2008 R2 servers (everyone is using virtualization today. If not, become the hero of the day and start on Monday)

·         Create Hyper-V clusters

·         Extended library – VM templates, ISO files, scripts, VHD`s + +

·         Create logical and virtual networks

·         Support of more hypervisors – Hyper-V (Windows Server 2008 R2 with Hyper-V role enabled, Server Core with Hyper-V enabled, and Hyper-V 2008 R2), XenServer (version 5.6 with Feature Pack 1), VMware (vSphere through vCenter (4.1) and hosts with ESXi/ESX 4.1 and 3.5) Microsoft Virtual Server 2005 R2 is no longer supported in VMM 2012

·         You can optimize your hosts and clusters

·         You can create a Private Cloud . This is the underlying physical resources (infrastructure) that is pooled together, available in a self-service manner for your users.

·         Deploy services to your private cloud, with the ability to scale out and in (elasticity)

·         Orchestrate updates of your Hyper-V cluster nodes (using WSUS dedicated for your cloud, configured in the Fabric)

·         Deep storage integration (see hyper-v.nu for some details around this. Hans Vredevoort is lucky enough to have a SAN that support the SMI-S provider, that is required for this scenario J )

·         Self-service portal – let the guys on accounting test a new CRM system, a new idea – just when they want no, with no interference from IT.

If you have the NIST definition of cloud computing in mind when you`re reading the list above, you`ll see that how it fits into the big cloud thing.

So if you do not have a so called cloud today, what can SCVMM 2012 do for you?

You could:

Start to deploy some virtualization in your environment, convert your physical workloads to virtual, virtual to virtual (from VMware to Hyper-V, or just add the VMware host into SCVMM 2012).

SCVMM 2012 is not only a rich management tool for your enterprise datacenter, it`s also the first stepping stone towards your private cloud. With a focus on resource pooling and resource optimization, it suddenly starts to sound economic as well.

A private cloud facilitates not only for better server utilization, but also a better exploitation of your IT-department

I will fill in a FAQ on SCVMM 2012 on this post next week, so it will be updated on a regular basis from now on.

Cheers,

Understanding the Service-concept in VMM 2012

For those of you that have tested and played around with VMM 2012, may be aware of the new concept of services. VMM 2012 inherited the great possibility to create VM templates from VMM 2008 R2, and has taken this a bit further in 2012.

When you create a VM template in VMM 2012, you can define the following configuration:

·         Hardware Configuration (HW Profile)

·         OS Configuration (OS Profile)

·         Application Configuration (Application Profile)

·         SQL Server Configuration (SQL Profile)

It`s worth to pay attention to that if you`re touching the Roles/Features in the OS profile, or/and the Application Profile/SQL Profile, you will get a message when you deploy your VM, that it need to be deployed as a service, since it contains additional configuration (not only the HW profile with a reference to a vhd, networking, CPU`s, RAM, and the OS profile with admin password, domain information, time zone, and license key).

If you choose to continue, these options will be ignored.

So, let`s move over to the Service Templates.

In VMM 2012, you can create a service, which is a logical grouping of VMs that you can manage as a single entity.

You can either create a single-tier service, or a multi-tier service.

Explanations:

A single-tier service contains a VM with ‘additional’ configuration. Like Application configuration, OS roles and features, SQL deployment/DACPAC`s, web apps, and so on. You can create a single-tier service, and scale it out. We will come back to the scaling later in this article.

A multi-tier service contains more than one VM. For example, you can deploy one SQL server tier, one Server App-V tier, and two Web App tier. These VMs will together define the service, and be their own little ecosystem.

It`s worth to mention that some people think that the service deployment is suited for other types of deployments. You can, technically deploy your Domain Controller, Remote Desktop Server, VPN server, and whatever server you`d like through this feature, but that is not what it`s meant for, unless it`s all a part of an application on the top, which you intend to scale out. Let me try to explain.

With VMM 2012, you`re able to manage the lifecycle of your applications. But remember that each and every application has some sort of software dependencies, some sort of manual input and requirements. So the service deployment and ‘scale-out’ is not suited for every application.

You can add, sort, and manage the underlying hardware in the Fabric in VMM 2012. In these days, the word Fabric is the new term of ‘infrastructure’. VMM 2012 gives you the opportunity to create a Private Cloud, which is primarily an Infrastructure – as a Service offering. With the Service deployment-feature, you`re actually able to provide your users with Software as a Service. But I will stress again, the scale-out purpose is meant for stateless application. Talking about stateless applications, this is particularly well for applications distributed with Server App-V. You can save the state of the application and reapply that state to the “new” VM. The new drag-and-drop Service Template Designer makes it easy to visually position VMs that comprise a service. (BTW: When you scale-in a service in VMM 2012, you`re in fact – deleting the VM).

Next blog: Example of a scale-out scenario.

Microsoft Most Valuable Professional – SCVMM

Dear Kristian Nese,

Congratulations! We are pleased to present you with the 2011 Microsoft® MVP Award! This award is given to exceptional technical community leaders who actively share their high quality, real world expertise with others. We appreciate your outstanding contributions in System Center Virtual Machine Manager technical communities during the past year.

Today, I was awarded with the MVP Award.

I just want to thank some important persons for me.

First of all, my brother Lars Ørjan Nese (he`s a great developer by the way) - who ‘fixed’ me into the business in 2006.

Ben Magne Stokland – you got me started, and encouraged me to learn whatever I was capable of.

Morten Pettersen – thank you for all our discussions, and the knowledge you share with me on a day to day basis.

Hans Jørgen and Jannike Neset – Thank you for listening to all of my ideas, and also to let me put them into life (and our infrastructure).

The great guys at Microsoft Norway – this would not be possible without your recommendations and guidance. You know who you are J

The community: Fawzi, Finn, Carvalho, Vredevoort, Rachfal ++

And last, my lovely son; Lukas. Thank you for letting me spends so much time with this, and thank you for being my biggest inspiration.

I am really looking forward to share my ideas with the Product Team, and hopefully be able to make an impact. There is a lot of interesting things going on in VMM 2012, in conjunction with Windows Azure, Hyper-V, and Codename Concero. It`s all about cloud computing, and to optimize your infrastructure.

(I will post a FAQ on VMM 2012 in July, so stay tuned.)

Cheers,