Unified Boxes, The Sum of all fears

Correction: By mistake I included SQL in the supportability statement, apparently I was speaking about the stack as hall including backup, sorry for that.

Hi there, earlier this week, fellow MVP Michel Di Rooij published a blog post http://eightwone.com/2014/07/02/exchange-and-nfs-a-rollup/ speaking about NFS/Exchange support “Again”, the post motivated me to delve into the pool and add my experience.

There was some hesitation in the MVP community about if we should blog/speak about it or not, Michel was so brave to jump and speak about the topic, and after exchanging some emails, we (including Fellow MVP Dave Stork) agreed that this blog is critical and we created it.

IF you want to read more, check Tony Redmond’s article http://windowsitpro.com/blog/raging-debate-around-lack-nfs-support-exchange 

So, from where the story begins ???!!!

I am currently working for a major data center provider. In my current role we try to find new ways, innovate and find new technologies that will save us time, effort and money and my team was working on investigating the unified boxes option.

But before delving into the technical part, let me give you a brief background from where I am coming, my position as an architect in a service provide is an awkward position, I am a customer, partner and a service provide, so I don’t innovate only, I don’t design only, I don’t implement only, I don’t support only and I don’t operate only, I do all of that, and that makes me keen investigating how every piece of new innovations will be designed, implemented, supported and operated.

Now speaking about the unified boxes, I was blown away with their capabilities. The capabilities of saving space, time and effort using these boxes are massive, but there is a catch, they use NFS, the source of all evil.

NFS has been used for years by VMware to provide “cost effective” shared storage option, a lot of customer adopted NFS over FC because of the claimed money saving and complexity, but NFS has its own issues (we will see that later).

I was a fan of the technology, and created a suggestion on ideascale.com to bring the issue to the PG attention, we did our best but Microsoft came back and informed us that NFS won’t be supported, they have their own justifications, we are not here to speak about it because we can’t judge Microsoft, but the bottom line, NFS is not supported as storage connectivity protocol for Exchange.

Now the reason of this post is to highlight to the community 2 things:

  • NFS is not supported by Microsoft for Exchange (any version), there is no other workaround this.
  • Choosing a unified box as a solution has its own ramifications that you must be aware about.

I am not here to say nutanix/simplivity/VMware VSAN..etc are good or bad, I am highlighting the issues associated with them to you, and the final decision will be yours, totally yours.

I was fortunate to try all of the above, got some boxes to play with and tested them to the bone, the testing revealed some issues, they might not to you, but they are from my point of view:

  • Supportability: Microsoft doesn’t support placing Exchange on NFS, with the recent concerns about the value of Exchange virtualization (see a blog post from fellow MVP Devin Ganger http://www.devinonearth.com/2014/07/virtualization-still-isnt-mature/) using these boxes and these set of technologies might not the best way for those specific products, you might want to choose going with physical servers or other options for Exchange/SQL rather than going with non-supported configurations, although that vendors might push you to go for their boxes and blinding you with how great and shiny these solutions are. The bottom line, they are not supported by Microsoft and they won’t in the near future.
  • Some of the above uses thin provisioned disks, meaning that disks are not provisioned ahead for Exchange which is the only supported configuration for virtual harddisks for Exchange. Disks are thinly provisioned meaning they are dynamically expanded on the fly as storage consumed which is another not supported configuration.
  • The above boxes have no extensibility to FC, also you are limited to a max of 2 * 10 GbE connections (I don’t know if some have 4 but I don’t think so) meaning that you have no option to do FC backups, all the backups will have to go through Gbe Network, we can spend years discussing which is faster or slower, in my environment I run TBs if not PBs of backups and they were always slow on GbE networks, all of our backups as to be done over FC.
  • The above means you will run backup, operations, production and management traffic on single team on shared networks, maybe 2 teams or will run it over 1 GbE, this might be fine with you, but for larger environments, it is not.
  • The above limitations limits you to a max number of network connection, a single team with 2 NICs might be sufficient to your requirements, 2 teams maybe, but some of my customers have different networking requirements and this will not fit them.
  • Some of the above boxes does caching for reads/writes, I have some customers ran into issues when running Exchange jetstress and high IO applications, the only solution as provided by the vendor’s support is to restart the servers to flush the cache drives.
  • Some of the vendors running compression/deduplication in software and this requires a virtual machine of 32 GB or larger to start utilizing deduplication.
  • All of the above uses NFS, meaning you will lose VAAI, VAAI is very critical as it accelerates storage operations by offloading those tasks directly to the array, you can use VAAI with NFS with virtual machines that has snapshots or running virtual machines, meaning that you rely on the cache or you must shutdown the virtual machines to use VAAI, VAAI is very important and critical element, so you must understand what are the effects of losing it.
  • Those boxes don’t provide tiering, tiering is another important if you are running your own private cloud, by allowing you to provision different storage grades to different workloads, also it is important if you want to move hot data to faster tiers and cold data to slower tiers. Tiering touches the heart and soul every cloud (private or public) and you must understand how this will affect your business, operations, charging and business model.
  • From support/operations and compliance point of view, you still running unsupported configuration from disk provisioning and storage backend, again it is your call to decide.

I am not saying that unified boxes are bad, they are a great solution for VDI, Big Data, branch offices, web servers and applications servers and maybe databases that support this sort of configuration, but certainly not for Exchange.

We can spend years and ages discussing if the above is correct or not, valid or not and logic or not, but certainly they are concerns that might ring some bills at your end, also it is certain that the above configurations are not supported by Microsoft, and unless Microsoft changes its stance, we can do nothing about it.

We, as MVPs, have done our duty and raised this as a suggestion to Microsoft, but the decision was made not support it, and it is up to you to decide if you want to abide to this or not, we can’t enforce you but it is our duty to highlight this risk and bring it to your attention. And as MVPs and independent experts, we are not attracted to the light like butterflies, it is our duty to look deeper and further beyond the flashlights of the brightest and greatest and understand/explain the implications and consequences of going this route so you can come up with the best technical architecture for your company.

Backup&Restore Exchange 2010 mailbox database or mailbox item using ARCserve R16 #msexchange #arcserve

In my ultimate Journey discovering how to backup and restore Exchange 2010 by every single application on our universe, I blog today about how to do that using CA’s ARCserve r16 SP1.

We will continue using my single Exchange server hen installing ARCserver r16 SP1 and then discovering how to make a backup job to backup Exchange and Restore from our backup.

Installing ARCserve r16 SP1:

There is nothing genius about installing the ARCserve, you possible want to plan ahead for the following:

  • SQL Database location.
  • Configuring Windows authentication instead of the ARCserve authentication.
  • If you will configure windows authentication later, you need to remember the password you used for the default admin account “caroot” because you will use it to login.

    other than that, the installation itself is no brainer, next, next and ok Smile

    Configuring ARCserve r16 Devices:

    Once you are finished installing and opening the ARCserve console “Manage”, you will be prompt with a very nice tutorial that walks you through the basic configuration of your ARCserve.

    In this step we will configure “Disk device” that we will use for our backup to disk, so from Devices choose launch device configuration:

    image

    In the Login Server screen, enter your credentials to login to the server:

    image

    In the Login Server  choose your login server:

    image

    In the Device Configuration screen, choose Windows File System Devices to configure a backup folder (the de-duplication device is a folder that could configured to store multiple backups, the ARCserve then divide the backup to small chunks that is compared and de-duplicated using the proprietary ARCserve algorithm) then click add:

    image

    image

    image

    and if you somehow missed the wizard, you can do the same using the device wizard from the administration menu:

    image

    Once the Device is configured, we can deploy the Agent and start protecting our Exchange server, you can do that from the administration, and then go to Agent Deployment :

    Note: In Order to backup the Exchange server using ARCserve you must installing MAPI CDO, this is a must because unlike Symantec which uses EWS to restore emails, ARCserve using MAPI CDO to backup and restore individual email, also note that MAPI CDO must be installed before installing the ARCserve if you don’t you will get the following error message:

    “The request is denied by the agent. The requested agent is not installed.”

    image

    When you deploy the agents for the first time, you must specify the ARCserve source to copy the agents from it, once copied you won’t need to do that again and you will be able to proceed with the deployment:

    image

    Once copied, you will proceed with the agent deployment, so specify the Login Server:

    image

    In the agent installation option and normally you will get the automatic, you might want to choose custom to fine tune the installation options:

    image

    In the agent select the agents that needs to be deployed:

    image

    In the host selection, you have a nice option here to discover the Exchange servers and deploy the agent to them automatically:

    image

    to discover the Exchange infrastructure, Just specify you Domain Controller and credentials and the ARCserve will discover the Exchange server for you, nice!!!:

    image

    image

    image

    image

    Backup Exchange 2010 Mailbox Database and Mailboxes using ARCserve:

    To Create a backup job, it is so easy, from the Protection & Recovery menu choose Backup:

    image

    From the Job Setup Menu select your Job Setup Type:

    image

    In the Source, select the Mailbox Database, if you want to recover specific mailboxes or mailox items you must configure the Document Level Type backup, unlike Symantec which uses 1 type of backups to either restore Mailbox Database or Mailbox or Mailbox item, ARCserve uses 2 types of backup (mailbox database backup for mailbox level and Mailbox Document level for Mailboxes and Mailbox items):

    image

    In the Schedule, select your scheduling:

    image

    In the Destination, select your destination, in my case I will use the folder I already configured previously:

    image

    Once all set, click the Submit button to submit the job for run.

    Restore the Exchange Mailbox Database or Mailbox items from the ARCserve Backup:

    Now you can restore either the Mailbox Database or the Mailbox items, you can go to the Restore section, explore the Exchange infrastructure and either select the Mailbox Database or the Mailbox Items:

    image

    Conclusion:

    In this Article we have explored the basic ARCserve configuration and how to backup and restore Exchange 2010 Mailbox and Mailboxes using ARCserve. it was easy and sweet although I don’t understand why in ARCserve I have to create 2 jobs and duplicates to backup Mailbox Database and Mailboxes (Document level).

    So what is the next product, I don’t know I will be waiting for your suggestions Open-mouthed smile, so let me know so I can blog it.

  • Restoring Entire Mailbox Exchange 2010 Database using Backup Exec 2012 #Symantec #backupexec #msexchange

    In previous posts we have seen how to backup Mailbox database and restore single item from the backup.

    In this post we will explore how to restore the entire database to its original location, although you might ask why would I do that when I can restore the item that I want directly from my backup set, Well there might be some scenarios where you want to restore an entire database:

    – Database corruption either physically or logically.

    – reseed operation.

    – restoring to restore database for finer search and extraction.

    we will use the same backup we did last time to restore the entire database, let us start:

    User one received 2 emails (Diff 1 and Diff2):

    [sociallocker id=”1891″]

    image

    It looks that those emails some how caused a Database corruption, and the database is dismounted and can’t be mounted again (this simulates a logical or physical corruption at the database level):

    image

    If I try to mount it I get the error:

    image

    Also there is an error in the event viewer:

    image

    Now I need to restore the entire database, from the Backup Exec management console Select the Exchange server and click restore, in the restore type, select Microsoft Exchange databases or storage groups:

    image

    In the Resource view, select the backup job you want to restore:

    image

    In the restore location, I will choose the original location since I want to restore it on top of the current one since the current one is corrupted, you might want to restore it to another location or the recovery database or to another server in case of dial-tone recovery.

    image

    In the overwrite page, I will choose to overwrite existing DB and logs, if you trust that logs are ok and your DB is having troubles due to a corrupted harddisk for example you can restore the database set and keep exiting logs and when the replay starts it will restore the database into the most recent status, however in my case there is a logical corruption caused by bad emails thus bad logs, so I don’t want these and I will overwrite them:

    image

    In the Temporary location, I will chose the default location, but you need to make sure that the selected location has enough space to hold the restored data:

    image

    In the next screen, you have the option to wait to start mounting the database, if you are restoring from differential backup or you want to run eseutil before mounting the database for example you might want not to mount the database otherwise, the backup exec will mount the database and start playing the logs directly, in my case I will choose to mount the database:

    image

    In the job name and schedule, set your options and click next:

    image

    on done, go to the Job list, select the restore job and click run now, the job will start restoring your database:

    image

    after the restore completes, the DB is mounted and everything is back to track :

    image

    User1 can login now to his mailbox, but you will note that Diff1 and Diff2 emails (the problematic ones) are not restored since they are weren’t backed up:

    image

     

    In the next post we will see how to restore differential backup, we have been talking about the full backups and we will see how to configure and restore differential backups.

    [/sociallocker]