When do we need a Failover Cluster solution ?

An interesting question was raised in the forum this weekend. “When is it overkill with Failover Cluster?”

We should think of failover clustering as a solution, and not a product. We have to examine our needs, pros and cons, and the cost. My reply to this thread can be red in its entirety here:

You ask a very interesting question: “When is it overkill with Failover Clustering”

It is tempting to say: never, but as with many other aspects in the IT infrastructure,- it depends.

It seems to me that you already have the hardware/software required for running Failover Clustering.

And since you are also running 2008 R2, you have the ability to enable CSV, and later support dynamic memory with the release of Service Pack 1.

Are you considering an Active-Active cluster, or Actice-Passive cluster? To get the most out of your HW, it`s recommended to use active-active, and as mentioned earlier, with SP1 this will allow you to over commit your hosts and ‘decrease’ the requirements for additional HW.

When it comes to administration and management overhead of the cluster, you really have to look at the benefits to be able to weigh that against each other.

It gives you redundance, easier maintenance on your physical nodes in the cluster, simplified administration of your VMs, and much more.   In addition, you can also create guest clusters within the VMs, for your SQL server etc. The key word here is ‘highly available’.

As you may know, Failover Clustering requires an Active Directory domain. It is recommended to place this role outside your cluster. I have written some posts about this topic on my blog, check it out here:  http://kristiannese.blogspot.com/2010/11/failover-clustering-and-domain.html and here: http://kristiannese.blogspot.com/2010/11/failover-cluster-and-domain.html

Since you are planning to have another pair of blade servers and SAN at another physical location, you should also consider Multi-Site clustering, that will extend the cluster from being only Highly Available but also a Disaster recovery solution. Check it out here: http://www.microsoft.com/windowsserver2008/en/us/failover-clustering-multisite.aspx

To answer your question, it really depends on your needs. In comparison, we are 50 employees in our company, and we`re running Failover Clustering with 4 nodes at our primary location, and have a disaster recovery site as well. We develop and deliver software, services, and need to be available to our customers  24/7. We`re also running some services in the ‘public cloud’ that is synced with our servers on-premise.

Technically you have the foundation for a Failover Clustering solution, but you should examine your needs. 

App-V 4.6 SP1

SP 1 for App-V 4.6 is around the corner, and has some significant improvements.

Check it out: http://www.msteched.com/2010/Europe/VIR304

App-V is not only suited for avoiding application conflicts, but also helps you to deploy applications faster to your  users, and is also a must for developers and testers, for testing the same application but with different versions on the same machine.

I`m looking forward to test SP1, and give you some more details about it.

Remember that though the sequencer in SP1 has many improvements, it is still important to documenting the process. I have experienced that I often have to do the sequencing over and over before I`m happy with the result, and if the application requires lots of testing before deploying it, be sure to keep your notes and also share them among the rest of the sequencing-personal.

Refreshers in SCVMM

As I have mentioned in some of my previously posts – the most accurate information about your Hyper-V hosts and Failover Clusters will be presented in Hyper-V Manager and Failover Cluster Manager.

I have often seen that the information that SCVMM presents, is misleading and not trustworthy.

So I will explain the refreshers in SCVMM in this post.

SCVMM periodically collects info from your hosts and library servers and compares tem with knowledge that already exists in the database.

The changes that are detected will be updated, and an audit log is created in the job view in SCVMM console.

The refresher times are customizable but the info about ‘how-to’ is not made public since the impact on performance to the virtualized environment is considerably.

You can off course manually refresh the status of an object.

VM Properties Refresher

It runs every 2 minutes on every host and does the following:

1)      Checks the host for successful connections through WinRM

2)      Checks the status of all the VMs residing on that host

3)      Marks a VM as missing if it no longer exists on the host

4)      Imports newly discovered VMs from the host if they do not exist in VMM

VM Refresher

This is the ‘heavy’ refresher, and runs every 30 minutes on every host and does the following:

1)      Refreshes all the virtualization information for all VMs on the hosts. (Settings, disks drives, storage,DVD, Network, clustering (HA)

2)      Refreshes FC, ISCSI for every VM

3)      Refreshes snapshots information and differencing disk information for each VM

Host Refresher

This refresher also runs every 30 minutes on every host and does the following:

1)      Updates host properties and status

2)      Updates physical disks and SAN information

3)      Updates networking (physical NIC, and vNICS)

(This refresher will not update any information for hosts that are in maintenance mode in VMM)

Library refresher

Runs default every 1 hour, but a user can configure this to run max 336 hours. This refresher can also be turned off completely. It does the following:

1)      Updates library shares that are under management in VMM

2)      Detects new library objects, changes, and marks objects as missing if they can no longer be found in the library

3)      Finds and imports any offline or stored VM in the library that were not already under management

Cluster refresher

Runs every 30 minutes and does the following:

1)      Refreshes all cluster related properties that is displayed in VMM, also available storage for creating new HA VMs.

2)      Adds newly added cluster nodes

3)      Removes removed cluster nodes

Performance refresher

Runs every 9 minutes on every hosts or whenever there is any state changing operation on the VM.

It collects counters information for both hosts and VMs on them.

Tip: Do not always rely on the information presented in SCVMM, but double check with your Hyper-V Manager and Failover Cluster Manager. If there is always misleading information in SCVMM, it may be a reason for troubleshooting this.

Upgrade your databases to SQL 2008 R2 (script)

I think I have upgraded SQL servers and databases more often than I have eaten hot dinners.

So I have created a SQL script that will do the following:

1.       Set Auto_close to off (makes your DBs faster)

2.       Set Auto_shrink to off (the Auto_shrink feature gives your SQL server a performance overhead, and the DBs will grow to the same size anyhow afterwards. Use db_shrink for shrinking your DBs)

3.       Set Parameterization to forced (optimizing queries)

4.       Set Auto_create_statistics to on (helps your SQL server to query more intelligent)

5.       Set Page_verify to checksum (gives you the best security for dirty pages and corruption)

6.       Set Compatibility_level to 100 (this is the compatibility level for SQL 2008 (80 is MSDE, 90 is SQL 2005, so when you upgrade to SQL 2008/R2, also upgrade the compatibility level))

In general, you can use this stored procedure for setting dboptions on your databases:

----------------------

EXECUTE sp_MSforeachdb

'

IF (''?'' NOT IN (''master'', ''tempdb'', ''msdb'', ''model''))

   EXECUTE (''ALTER DATABASE [?] SET AUTO_CLOSE OFF WITH NO_WAIT'') '

---------------------

In my script, I will not touch the recovery model for the databases, since this is not me to decide, but the the organizations backup policy.

I also have to mention that you should set the options properly on the ‘Model’ database as well, since the databases you create will inherit the settings from here.

The script will run in SSMS 2008 (just copy the content to notepad, and save the as sqlscript.sql

--------------------------------------------------------------------------------------------------------------------------------------

--SQL script from Virtualization and some coffee

--Auto_close, auto_shrink = off, Parameterization = forced, Auto_create_statistics = on, Page_verify = Checksum, Auto_update_statistics = on

--Sets compability level to 100 - SQL 2008

EXECUTE sp_MSforeachdb

'

IF (''?'' NOT IN (''master'', ''tempdb'', ''msdb'', ''model''))

  

EXECUTE (''ALTER DATABASE [?] SET AUTO_CLOSE OFF WITH NO_WAIT'')

'

EXECUTE sp_MSforeachdb

'

IF (''?'' NOT IN (''MASTER'', ''TEMPDB'', ''MSDB'', ''MODEL''))

EXECUTE (''ALTER DATABASE [?] SET AUTO_SHRINK OFF WITH NO_WAIT'')

'

EXECUTE sp_MSforeachdb

'

IF (''?'' NOT IN (''MASTER'', ''TEMPDB'', ''MSDB'', ''MODEL''))

EXECUTE (''ALTER DATABASE [?] SET PARAMETERIZATION FORCED WITH NO_WAIT'')

'

EXECUTE sp_MSforeachdb

'

IF (''?'' NOT IN (''MASTER'', ''TEMPDB'', ''MSDB'', ''MODEL''))

EXECUTE (''ALTER DATABASE [?] SET PAGE_VERIFY CHECKSUM WITH NO_WAIT'')

'

EXECUTE sp_MSforeachdb

'

IF (''?'' NOT IN (''MASTER'', ''TEMPDB'', ''MSDB'', ''MODEL''))

EXECUTE (''ALTER DATABASE [?] SET AUTO_UPDATE_STATISTICS ON WITH NO_WAIT'')

'

EXECUTE sp_MSforeachdb

'

IF (''?'' NOT IN (''MASTER'', ''TEMPDB'', ''MSDB'', ''MODEL''))

EXECUTE (''ALTER DATABASE [?] SET AUTO_CREATE_STATISTICS ON WITH NO_WAIT'')

'

DECLARE GET_DATABASES CURSOR 

READ_ONLY  

FOR SELECT NAME FROM SYS.DATABASES WHERE COMPATIBILITY_LEVEL != '100' 

 

DECLARE @DATABASENAME NVARCHAR(255)  

DECLARE @COUNTER INT 

 

SET @COUNTER = 1  

 

OPEN GET_DATABASES  

FETCH NEXT FROM GET_DATABASES INTO @DATABASENAME  

WHILE (@@fetch_status <> -1)  

BEGIN 

IF (@@fetch_status <> -2)  

BEGIN 

-- Endrer db-compatibility  

EXECUTE sp_dbcmptlevel @DATABASENAME , '100' 

 

PRINT  @DATABASENAME + ' changed' 

 

SET @COUNTER = @COUNTER + 1  

END 

 

FETCH NEXT FROM GET_DATABASES INTO @DATABASENAME  

END 

 

CLOSE GET_DATABASES  

DEALLOCATE GET_DATABASES  

 

GO 

How to shrink your SCVMM database (Transactional Log)

(Generally speaking, this is not a ‘SCVMM-problem’. It`s linked to SQL, and the way the SQL server act when it comes to I/O, Memory, Data, - and log files. But since this is a virtualization-blog, I want to angle this toward the SCVMM server since it`s also relevant here)

Ok, you have had your SCVMM server running for a while now, all roles and services installed on a single server. Remember that the SCVMM 2008 R2 server installs a SQL 2005 Express if you don’t specify another instance for this purpose.

The storage limit for a database in SQL 2005 Express is 4GB, so what do you do when the VirtualManagerDB_log starts to grow out of control ?

First, we have to locate our databases.

SCVMMàAdministrationàGeneralàDatabase Connection

Here we`ll find the information we need

We`ll see the Database Server Name, Instance, and the name of our database.

We should also locate the file location for the databases.

Startàcmd

SQLCMD –S .\Microsoft$VMM$  (Log on to your SQL instance with trusted connection)

1.       Select * from sysdatabases where name = ‘VirtualManagerDB’

2.       Go

Now we`ve located the database, and we can see that the log file is growing (the reason for this, is that the VirtualManagerDB has the ‘Recovery Model’ set to full, which allows the log file to grow since you are able to run log-backups, and can restore the database to any point in time).

We can actually shrink this log file in two ways.

1)      In a heavy way (Fullbackup->log backup->shrink log file (log backup and shrink log file x2)

2)      In a awesome way (sp_detach_db, rename log file, sp_attach_single_file_db)

I`ll go for number 2.

(Be careful if you are planning to do this in production, and off course have a solid backup and make a copy of the log file before editing).

This procedure will cause the SCVMM server to be offline, since we are detaching the database from our SQL server.

Log on to your SQL server instance with SQLCMD –S .\Microsoft$VMM$

1.       Sp_detach_db virtualmanagerdb

2.       Go

3.       Rename the VirtualManagerDB_log to OLDVirtualManagerDB_log

4.       Sp_attach_single_file_db virtualmanagerdb, ‘C:\Program Files (x86)\Microsoft SQL Server\MSSQL.1\MSSQL\Data\virtualmanagerdb.mdf’

5.       Go

6.       In the data folder for your SQL server, you will now have a newly created log file for VirtualManagerDB at the size of 512KB.

7.       Alter database VirtualManagerDB set recovery simple with no_wait

8.       Go

The database is now in simple recovery, meaning that the log file will not grow out of control again but on a regularly basis create a checkpoint to the data file. (The way SQL uses its log files does not changes, whether you run full, bulk, or simple).

Snapshots in Hyper-V - Ground Rules

“Snapshots are read-only “point-in-time” images of a VM. You can capture state and configuration of a VM at any point in time, and return the VM to that state with minimal interruption”

-          This is fantastic, but will this replace the need for a backup of our VMs ?

No, absolutely not. And I`ll stress in this post why you should not use Snapshots in production, and what the consequences will be.

Instead of just point at the reasons for why we should not use snapshots, I will also try to explain how it works, when you should consider a snapshot, and some common pitfalls.

Virtual Machine Snapshots – How it works

As stated earlier in this post, you can capture state and configuration of a VM at any point in time.

You can also create multiple snapshots, delete snapshots, and apply snapshots.

In Hyper-V manager, you can right-click a VM, and take a snapshot when the current state of the VM is:

·         On

·         Off

·         Saved

Let`s take a look what happens when you initiate a snapshots of a running VM:

1.       The VM pauses

2.       An AVHD is created for the snapshot

3.       The VM is configured

4.       The VM is pointed to the newly created AVHD

5.       The VM resumes (Point 1 and 5: This would not affect the end user)

6.       While the VM is running, the contents of the VM memory are saved to disk. (If a guest OS attempts to modify memory that has not yet been copied, the write attempt is intercepted and copied to the original memory contents)

7.       Now that the Snapshot is completed, the VM configuration, saved states files, and AVHD are stored in a folder under the VM snapshots directory.

 1. You initiate a snapshot In Hyper-V Manager. Right click a VM, and select ‘Snapshot’

2. The process starts




3. The snapshot is now completed

4. If we take a look at the settings of the VM virtual disk, we see that the newly created AVHD is the current disk for this vm. Also note the type “Differencing virtual hard disk”.









If we take another snapshot now, we will have another AVHD file that contains the changes of the VM since the last snapshot (AVHD).

If we browse explorer to find the Virtual Hard Disks - folder for this VM, we see the following:

What happens when you apply a Virtual Machine Snapshot?

When you apply a snapshot, you literally copy the entire VM state from the selected snapshot to the active VM. This will return your current working state to the previous snapshot state. In other words, you will lose every changes, configuration, data, and so on if you don’t take a new snapshot of the current state before applying the selected snapshot.

The process:

1.       The VM saved state files (.bin, .vsv) are copied

2.       A new AVHD is created and then linked to the parent AVHD

Applying a previous snapshot creates another snapshot to your hierarchy, starting at the applied snapshot

And last, what happens when you delete a Virtual Machine Snapshot?

When you delete a snapshot, you also delete all the saved state files (.bin and .vsv)

The process:

1.       The copy of the VM configuration taken during the snapshot process is removed

2.       The copy of the VM memory taken during the snapshot process is removed

3.       When the VM is powered off, the contents of any deleted AVHDs are merged with its parent (VHD)



Ok, so now we some basic facts about the Snapshot process.

What is best practice, and my personally recommendations ?

Don’t use snapshot in production.

The root reason for this is the downtime required for merging the snapshots.

If you create a snapshot of a VM with the default IDE size 127GB disk, you can consume additional 127GB of your storage. And this contiunes for the next snapshot, and so on.. Snapshots requires storage, and if you don’t have the free storage required and available when merging your snapshots, this will cause the merging process to fail. (In Hyper-V 2008 R2, you can take a new snapshot, export that snapshot, and import it as a VM – that will in some situations save you, or at least shorten the merge process)

So the keyword here is: Storage, and downtime.

The merge process is writing the VMs history back to the VHD file, and that can take hours. You don’t want to find yourself in that situation (believe me J ).

Snapshots are great for testing, and give you the ability to test and devastate your VMs.

But also for testing, I would not recommend using snapshots on a DC, even it`s in a test environment.

That may lead to inconsistency of the Active Directory Database, and as you know – every part of an Active Directory Domain would be affected.

Another tip is to try to name your snapshots with some useful names, so that when you have a snapshot-hierarchy you are able to sort them to find out what is what.

And last, let the Hyper-V manager take care of your snapshots. Don’t rename, edit, move, replace, or delete the associated snapshot files.

Drive carefully when considering snapshots, and have fun.

Failover Clustering and Domain Requirements (by example)

In my last post about this subject, I simply referred to things that might happens if you don’t place a DC outside your cluster. Today, we will take a look at it in more detail. 

My lab:

·         2 identical nodes with the Hyper-V role enabled

·         Both nodes are member servers of my domain

·         Cluster configured with CSV

·         ISCSI with MPIO enabled

·         Quorum: File Share Witness

·         One Domain Controller (as a VM on the cluster J )

I`ve simulated the following scenario:

1.       The entire cluster shuts down

2.       Both nodes comes online again

3.       Now what ?

(Ok, I have to admit, that I have cheated a bit so I could demonstrate the stage AFTER you are able to log on to your hosts. Since the DC was powered off, both nodes hade some troubles to login. And if you speculate what opportunities you have if you log on locally, well, here is the answer):

Anyhow, we are now logged in to both nodes, and the cluster service is in the state of ‘stopped’.

Let’s try to start it on both nodes:

Ok ! So far, so good.

Now, let’s try to start up the Failover Cluster Manager Console:

The console shows us that it`s empty. No cluster to manage, so we have to try to add our cluster.

As the error message indicate, we have a DNS lookup problem. That makes sense, since the only DC is powered off.

If we run the cmd ‘cluster node’ on both hosts, you can see that they indicate that everything is fine as far they concern, but don’t know that the other node is ‘joining’ as well.

(When you tell the Cluster Service to start in Windows 2008 R2 Failover Cluster, it just immediately starts. Then it sends out notifications to the other nods that it wants to join the Cluster. It is also calculating the number of ‘votes’ needed to achieve ‘quorum’.  Since there is no DNS connection between the nodes in this example, both nodes will be in a ‘joining’ type mode. They just wait for each other. If both nodes in this example and the witness could come online, the cluster would achieve quorum and go on its way).

Ok, so we have a DNS issue.

Since I know the IP address of my cluster, nodes, and also the witness share, and know that the first thing the DNS client does, is that it checks the local hosts file (c:\windows\system32\drivers\etc\hosts), I will add the DNS names of the involved servers here, and hopefully get the quorum.

(COLD and STONE are nodes in the cluster, CLASH is the cluster name, and SCVMM has the File Share Witness)

Now, I`m able to ping the servers by name, and let`s try to run the cluster node command again:

OK, looking good.

Let us try to add the cluster in Failover Cluster Manager again:

Are we saved ?

- What happens if we try to bring one of our VM online ?

Nothing, you cant.

If we take a look at the event log on one of our nodes, it shows some important information right here:

So, after all this struggle you are still unable to start you VMs. Moral ?

Please plan your cluster configuration carefully, and where you want to place your Domain Controller. This would easily be solved if we had a Domain Controller outside the cluster. And since I already have that, I would like to show what happens after I boot this machine.

My VMs comes online again, ready to play.