Backup and Restore Exchange on VMware using NetBackup with GRT – Part1

Backup and Restore Exchange on VMware using NetBackup – Part1

In this blogging series we will explore how we can protect MS applications installed on VMware using NetBackup, this part will be the first and will explore how to configure NetBackup to protect Exchange 2013 installed on VMware ESXi 5.5

Although it might look straight forward, it is not that straight forward, you must understand some requirements and prerequisites in order to protect MS applications installed inside VMware VMs using Symantec NetBackup.

This lab will assume that you have:

  • 1 Domain Controller installed.
  • 1 Server running ESXi 5.5
  • 1 Server running Exchange 2013 installed on a VM on the ESXi host.
  • 1 Server running NetBackup Software for Windows (the configuration should be different on Linux installation).

 

So let us get started:

Introduction:

Symantec Netbackup 7.6.x can protect VMs and perform VM level backup using offloaded backups to VMware backup host, this backup method can accelerate backup and offload the backup load from the server.

Backups that are performed at the VM level are quiesced for VM consistent backup using VSS, additionally you can perform backup while the machine is running using VMware Snapshot technology.

Often times, MS gurus mix with using snapshots for VM protection and using Snapshots to perform backups, I believe that no one can explain it better than AbdulRasheed , he wrote a great article about it here http://www.symantec.com/connect/blogs/nuts-and-bolts-netbackup-vmware-virtual-machine-snapshots-backing-business-critical-applicatio

 So in summary, using snapshots for backups is not the same as using Snapshots to protect Exchange VM.

Now, although we can perform VM level backup at the host level without an agent installed on the ESXi or the VM by connecting vCenter directly, this backup method doesn’t support file level recovery for applications or GRT (you can perform regular file restore but not mailbox or database restore for SQL for example), so you will need to install Symantec NetBackup client on the Exchange or SQL VM in order to perform application aware backup.

Note: up to the date of publishing this article 13/4/2015, Symantec doesn’t support GRT for Exchange 2013, but GRT is supported for earlier versions either using VM policies or Exchange policies.

A word about SAN transport:

One element is to be aware of is the SAN transport option, traditionally if you backup a VM using the backup agent, you will transport the data over the IP network, but what if you have large data set…very large ones.

Then you can use the FC or SAN transport, where you can back up the data directly to the SAN over the SAN network (either to SAN storage or Tape Library).

In VMware, you can perform VM agentless or agent assisted backups and transport the data over FC which can give you increased speed up to 4 times, all what you need is to present the LUNs to the VMware backup host as offline LUNs and configure the policy to use SAN transport, nice haaa

Prerequisites:

  • Install the NetBackup Agent inside the VM
  • Install Symantec VSS provider
  • Install and configure NFS to browse backup images for GRT (for Exchange 2007/2010).

You can Refer to the documentation on how to perform the above steps.

NetBackup Configuration:

Assuming that you have everything configured including installing the Netbackup agent on the Exchange VM, you can start connecting to the vCenter:

08-04-2015 08-03-07
Add VMware Virtual Machine Server

 

Enter the FQDN of the vCenter server

 

Add the vCenter information and account with credentials to connect, and specify the backup host, in my case it is the master server:

08-04-2015 08-03-35

08-04-2015 08-04-05

one note: make sure to add the account using domain\username, because the GUI doesn’t accept user@domain.com

 

Now you can proceed with configuring a policy, so you can launch the policy configuration wizard:

08-04-2015 08-06-06

Specify a policy name

08-04-2015 08-06-18

In the Policy Storage select your storage destination, unless you have many of them and you want to load balance the backup jobs:

08-04-2015 08-06-38

In the virtual machine options, make sure to specify the VMware backup host, and enable the Exchange recovery.

08-04-2015 08-07-40

Note: in the primary VM identifier, you can select the VM host name, this requires the VM tools to be installed and DNS lookups (forward and reverse) are working, for the simplicity I like to choose VM Display name.

08-04-2015 08-18-38

In order to select a VM for application protection you must use query based to select the VM or you will get the error “Application Protection options for VMware policies are only valid when using the query option for virtual machine selection in the clients tab”, so you need to create a query in the VM selection to include the required VM

08-04-2015 09-25-31

In the backup selection, you must select full backups, you have to note that you can’t perform application level backups using incremental backups, backups of applications at the VM level must utilize full backups.

08-04-2015 09-24-46

Then specify the desired internal and retention and schedule.

08-04-2015 09-25-01

 

08-04-2015 09-25-13

If everything is configured correctly, you should start seeing the policy kicking in and snapshot is being taken and backups are being performed

08-04-2015 09-31-36

08-04-2015 09-32-33

08-04-2015 09-43-07

08-04-2015 09-43-35

Note regarding policy schedule:
if you right click on the policy and chose to run manual backup, the policy will kick in but the backup job will be equivalent to copy backups, meaning that no Exchange logs will be flushed, in order to perform application level backup, you will have to wait for the policy to kick in.

Policy Schedule and Policy Window

Another point that many NBU admins struggle with is the policy frequency, Window and retention, so let me elaborate on this:

Policy Frequency: is how often the policy will kick in, so in the above screenshot, the policy will start every week and this is equivalent to weekly backups.

Policy Window: is the window allowed for the policy to start, so depending on the configured policies, running policies and queued policies, a policy will start when the window comes, or wait until additional resources are freed if all the resources are not available (like a free tape for example).

If the window ended without available resources, the policy will not start and you will miss the backup window, once the policy start, it can exceed the policy window safely, policy window affects the policy start and will not end the policy.

In part 2 of this series we will see how we can perform a restore from the taken backup.

 

Unified Boxes, The Sum of all fears

Correction: By mistake I included SQL in the supportability statement, apparently I was speaking about the stack as hall including backup, sorry for that.

Hi there, earlier this week, fellow MVP Michel Di Rooij published a blog post http://eightwone.com/2014/07/02/exchange-and-nfs-a-rollup/ speaking about NFS/Exchange support “Again”, the post motivated me to delve into the pool and add my experience.

There was some hesitation in the MVP community about if we should blog/speak about it or not, Michel was so brave to jump and speak about the topic, and after exchanging some emails, we (including Fellow MVP Dave Stork) agreed that this blog is critical and we created it.

IF you want to read more, check Tony Redmond’s article http://windowsitpro.com/blog/raging-debate-around-lack-nfs-support-exchange 

So, from where the story begins ???!!!

I am currently working for a major data center provider. In my current role we try to find new ways, innovate and find new technologies that will save us time, effort and money and my team was working on investigating the unified boxes option.

But before delving into the technical part, let me give you a brief background from where I am coming, my position as an architect in a service provide is an awkward position, I am a customer, partner and a service provide, so I don’t innovate only, I don’t design only, I don’t implement only, I don’t support only and I don’t operate only, I do all of that, and that makes me keen investigating how every piece of new innovations will be designed, implemented, supported and operated.

Now speaking about the unified boxes, I was blown away with their capabilities. The capabilities of saving space, time and effort using these boxes are massive, but there is a catch, they use NFS, the source of all evil.

NFS has been used for years by VMware to provide “cost effective” shared storage option, a lot of customer adopted NFS over FC because of the claimed money saving and complexity, but NFS has its own issues (we will see that later).

I was a fan of the technology, and created a suggestion on ideascale.com to bring the issue to the PG attention, we did our best but Microsoft came back and informed us that NFS won’t be supported, they have their own justifications, we are not here to speak about it because we can’t judge Microsoft, but the bottom line, NFS is not supported as storage connectivity protocol for Exchange.

Now the reason of this post is to highlight to the community 2 things:

  • NFS is not supported by Microsoft for Exchange (any version), there is no other workaround this.
  • Choosing a unified box as a solution has its own ramifications that you must be aware about.

I am not here to say nutanix/simplivity/VMware VSAN..etc are good or bad, I am highlighting the issues associated with them to you, and the final decision will be yours, totally yours.

I was fortunate to try all of the above, got some boxes to play with and tested them to the bone, the testing revealed some issues, they might not to you, but they are from my point of view:

  • Supportability: Microsoft doesn’t support placing Exchange on NFS, with the recent concerns about the value of Exchange virtualization (see a blog post from fellow MVP Devin Ganger http://www.devinonearth.com/2014/07/virtualization-still-isnt-mature/) using these boxes and these set of technologies might not the best way for those specific products, you might want to choose going with physical servers or other options for Exchange/SQL rather than going with non-supported configurations, although that vendors might push you to go for their boxes and blinding you with how great and shiny these solutions are. The bottom line, they are not supported by Microsoft and they won’t in the near future.
  • Some of the above uses thin provisioned disks, meaning that disks are not provisioned ahead for Exchange which is the only supported configuration for virtual harddisks for Exchange. Disks are thinly provisioned meaning they are dynamically expanded on the fly as storage consumed which is another not supported configuration.
  • The above boxes have no extensibility to FC, also you are limited to a max of 2 * 10 GbE connections (I don’t know if some have 4 but I don’t think so) meaning that you have no option to do FC backups, all the backups will have to go through Gbe Network, we can spend years discussing which is faster or slower, in my environment I run TBs if not PBs of backups and they were always slow on GbE networks, all of our backups as to be done over FC.
  • The above means you will run backup, operations, production and management traffic on single team on shared networks, maybe 2 teams or will run it over 1 GbE, this might be fine with you, but for larger environments, it is not.
  • The above limitations limits you to a max number of network connection, a single team with 2 NICs might be sufficient to your requirements, 2 teams maybe, but some of my customers have different networking requirements and this will not fit them.
  • Some of the above boxes does caching for reads/writes, I have some customers ran into issues when running Exchange jetstress and high IO applications, the only solution as provided by the vendor’s support is to restart the servers to flush the cache drives.
  • Some of the vendors running compression/deduplication in software and this requires a virtual machine of 32 GB or larger to start utilizing deduplication.
  • All of the above uses NFS, meaning you will lose VAAI, VAAI is very critical as it accelerates storage operations by offloading those tasks directly to the array, you can use VAAI with NFS with virtual machines that has snapshots or running virtual machines, meaning that you rely on the cache or you must shutdown the virtual machines to use VAAI, VAAI is very important and critical element, so you must understand what are the effects of losing it.
  • Those boxes don’t provide tiering, tiering is another important if you are running your own private cloud, by allowing you to provision different storage grades to different workloads, also it is important if you want to move hot data to faster tiers and cold data to slower tiers. Tiering touches the heart and soul every cloud (private or public) and you must understand how this will affect your business, operations, charging and business model.
  • From support/operations and compliance point of view, you still running unsupported configuration from disk provisioning and storage backend, again it is your call to decide.

I am not saying that unified boxes are bad, they are a great solution for VDI, Big Data, branch offices, web servers and applications servers and maybe databases that support this sort of configuration, but certainly not for Exchange.

We can spend years and ages discussing if the above is correct or not, valid or not and logic or not, but certainly they are concerns that might ring some bills at your end, also it is certain that the above configurations are not supported by Microsoft, and unless Microsoft changes its stance, we can do nothing about it.

We, as MVPs, have done our duty and raised this as a suggestion to Microsoft, but the decision was made not support it, and it is up to you to decide if you want to abide to this or not, we can’t enforce you but it is our duty to highlight this risk and bring it to your attention. And as MVPs and independent experts, we are not attracted to the light like butterflies, it is our duty to look deeper and further beyond the flashlights of the brightest and greatest and understand/explain the implications and consequences of going this route so you can come up with the best technical architecture for your company.

ERROR: MsiGetActiveDatabase() failed. Trying MsiOpenDatabase(). while installing VMware SRM 5.5 and installation fails

You receive error message : ERROR: MsiGetActiveDatabase() failed. Trying MsiOpenDatabase(). while installing VMware SRM 5.5 and installation fails

If you are installing VMware SRM 5.5 you might get the following error message in installer:

Can not start VMware Site Recovery Manager Service.

when digging in the installation logs you will find the following error message:

ERROR: MsiGetActiveDatabase() failed. Trying MsiOpenDatabase().

To fix this issue, change the logon on the VMware Site Recovery Manager service from Local System to an Account with the DB Owner on the database.