Avamar GLR with ItemPoint for Exchange

I have implemented Avamar for a couple of customers recently that were running Exchange 2010 and Exchange 2013 in house.  Avamar versions were 7.4.1-32 (there is a hotfix for Exchange that has been released for this version hotfix 285198) and 7.5.0-183.

When installing the Avamar agent and the Exchange VSS plugin (including the EMC ItemPoint software) at each customer both installed and registered as expected.  After installing and rebooting the Exchange server we attempted a GLR from Avamar (steps for this can be found in the Exchange VSS guide located under documents on the Avamar Documents and Downloads page).

The mounts of the .edb file from the Avamar/DD were successful and displayed the path of the .edb and the log file as designed.  When we opened ItemPoint and selected the location of the .edb and the log file (selection of the log file is not a requirement) it displayed the restored .edb in the top pane of Avamar and the live mailboxes in the bottom pane.

When trying to browse the live mailboxes in the bottom pane we received a permissions error.  Upon opening a ticket with Dell EMC (they had to get Kroll, the developers of the ItemPoint software installed), we were able to resolve the issue.

After running into this at the first customer I made sure to document the fix.  I hit this same issue at the second customer and was able to resolve this myself.  I have pasted the link to the fix as well as the command.  Sometimes copy/paste (even from notepad) doesn’t copy correctly inside of the Exchange Powershell so if you get errors with the command try typing it out.

Use Example 4 from the link below:

https://technet.microsoft.com/en-us/library/bb124097

This is the command you will need to run (after the -User flag insert whatever user you are logging into the server with Kroll installed trying to perform GLR):

Get-Mailbox -ResultSize unlimited -Filter {(RecipientTypeDetails -eq ‘UserMailbox’) -and (Alias -ne ‘Admin’)} | Add-MailboxPermission -User avamarbackupuser -AccessRights FullAccess -InheritanceType all

Advertisements

Dell EMC World 2017

This year I was able to attend Dell EMC World.  I attended many sessions of which most I was pleasantly surprised with the content.  I tried to get notes and slide pictures from all of the sessions I attended.

The notes cover the products listed below:

  • Nutanix
  • Cloud Data Protection
  • VMAX
  • Data Domain
  • RecoverPoint
  • RecoverPoint For Virtual Machines (RP4VMs)
  • Data Protection Suite
  • Unity

You can view all my notes at the OneDrive link below.  Enjoy!

https://1drv.ms/o/s!Ak4awN7U0kM4hZVv1T1ezyqmvTcBxA

Avamar Proxy Deployment Manager ‘Error Deploying Proxy’

Working with a customer this week and we were adding a new vCenter into their Avamar environment.  With this we needed to deploy some new proxies.  The proxy deployment manager makes this process very easy and allows you to deploy multiple proxies in a short amount of time.

After getting all the proxies configured I tried to apply the settings (this is the point that it goes out to vCenter and tries to deploy the proxies in the VMware environment) which failed.  There were no good errors in the GUI of Avamar or VMware so I dug a little further into Avamar and found the issue.

After examining the log (/usr/local/avamar/var/log/vcs/deploymentmanager.log) I found an error referencing an unsupported curve issue.  After some searching on support I found the fix.  Outlined below are the steps to remediate this issue.

  • Login to the Avamar server as admin and run the following commands:
    • su – root
    • cd /tmp
    • mkdir javafix
    • cd javafix
    • ftp://avamar_ftp:anonymous@ftp.avamar.com/software/jsafe-6.2_v0002.tgz
    • tar -xzvf jsafe-6.2_v0002.tgz
    • emwebapp.sh –stop
    • mv *jsafe* /usr/local/avamar-tomcat/lib/
    • cd /usr/local/avamar-tomcat/lib/
    • ./replace_jsafe_libraries.pl
    • emwebapp.sh –start

The above is a condensed list of all the steps.  The detailed issue is outlined in KB# 000489358.  After making the above changes I was able to successfully use the proxy deployment manager to deploy proxies.

Avamar Instant Access Restore Fails

Instant Access is a restore method in Avamar that allows you to mount a NFS export directly from the Data Domain to a VMware host in your environment.  This is a great method when you need a restore done quickly (1-2 minutes) rather than having to wait for it to copy all the blocks to a datastore already present in your VMware environment.  I usually have great success with this restore method but ran into an issue yesterday.  I got the following error on the Avamar job log after the restore failed:

2016-08-08 16:43:19 vmir Error <0000>: Create NFS datastore failed to start with error: Failed to add NFS directory on Data Domain system.
2016-08-08 16:43:19 vmir Error <0000>: Failed to create NFS datastore

After doing some troubleshooting I found out that the ddboost user that the Avamar was using to integrate with the Data Domain was locked by running the ‘user show list’ command.

Capture1

I figured out that the default password aging policy for users created is 90 days so all of my DDBoost users that were older than 90 days were locked.  I ran the ‘user enable ddboost-user‘ command to enable the user and then I ran the ‘user password aging set ddboost-user max-days-between-change 99999′.  After doing this you can run ‘user password aging show’ command to see that the users password aging policy has been changed.

Capture2

At this point I was able to run the instant access restore successfully.

Avamar Replication

I have been troubleshooting some Avamar replication integrated with Data Domain for over a week now and have finally got to a point where I am making some progress.  While troubleshooting with EMC and my own research I have learned a lot about how Avamar replicates and what the processes under the covers are.

So now for the background of my issue, I have a customer that had one particular job that hasn’t replicated in over a month.  We looked at the network and found some issues which were fixed but still did not allow the replication to finish.  I opened a ticket with EMC Data Domain to see if there were any settings on the Data Domain that would be limiting the bandwidth used for replication (I check all the normal settings on Avamar and Data Domain pertaining to throttling but wasn’t sure if there were others that could be set that I was missing).  EMC got back with me and we determined that the Data Domain was working as expected so they recommended moving my ticket to the Avamar integration team.

Once I started working with that team they gave me some tweaks to try and speed things up.  The first thing was allowing more streams per process on the Avamar side.  So in the GUI based replication job on the options page you can change the maximum concurrent processes from 1-8 (1 is the default and 8 is the max).  This will allow up to 8 clients to run a replication job at the same time.  What I did not realize is there are streams that can be specified to allow more streams per process giving you better replication performance.  By default the stream count per process is 6.  To give an example if you have 8 processes setup on the Avamar replication it will run 48 streams (8 processes multiplied by 6 streams).  Below is the process to modify this:

  1. ***DATA DOMAIN HAS REPLICATION STREAM LIMITS, MAKE SURE YOU DO NOT EXCEED THE STREAM COUNT ALLOWED BY YOUR DATA DOMAIN MODEL***
  2. Edit the replication job that you want to modify
  3. Go to the Overview screen
  4. Click More Options
  5. Click More at the bottom right hand side of the window
  6. Input the following:
    1. [avtar]ddr-repl-max-parallel-streams    =     # of streams to be allocated
    2. Capture1
  7. Make sure to click the + sign to add the command

After making the above change I could see under the replication window on Data Domain for DDBoost that the files that were replicating at one time did increase to the number I set but it still didn’t allow the job to finish and it always appeared to get hung while replicating.  The EMC tech gave me one more thing to change which is what helped the most.

I had no idea there were underlying replication types of Avamar replication.  The EMC tech recommended that I forced this job to use AMS/NCR rather than VSR.  Before I forced anything I wanted to know more about these replication types which was not real easy to find.  I found an EMC KB (000479374) which explains the different types of replication.  AMS/NCR stands for Automated Multi-Streaming/Ninja Chopping Replication, yes that’s right I said Ninja Chopping Replication which is a pretty awesome name :-).  VSR stands for Virtual Synthetic Replication which is not nearly as cool.  Below are some of the things I pulled from the KB listed above that explains the two.

Automated Multi-Streaming (AMS)/Ninja Chopping Replication (NCR)

Automated Multi-streaming replication, also known as Ninja Chopping Replication, works by inflating a backup on the Data Domain and walking that backup using multiple streams to “chop” the backup up before sending it to the target Data Domain. AMS/NCR replication is supported in Avamar 7.0 and up.

Key Features of AMS/NCR:

  • Leverages multiple parallel streams to process and send backups to the replication target.
  • Introduced in Avamar v7.0
  • By default the Avamar uses 6 streams per concurrent process, but this value can be changed using the proper flags.
  • NCR must scan the entire backup rather than just the changes, so it often has to more data to scan than if VSR were used.
  • NCR typically has better performance with high change rate clients when the bandwidth between the sites is large.
  • NCR can cause space differences between the source and target Data Domains. This is discussed further in KB 335802.
  • No restrictions on client types for this type of replication.

Example of Avamar choosing NCR Replication:

2015-11-22 10:05:55 avtar Info <6654>: Replicating backup 367, 528.7 GB
2015-11-22 10:05:56 avtar Info <40047>: id:1 VSR selected because file
size is too small for NCR (container.1.cdsf)
2015-11-22 10:05:56 avtar Info <40050>: id:2 NCR selected because it could
not be determined if VSR is possible (container.2.cdsf)

Virtual Synthetic Replication (VSR)

Virtual Synthetic replication is a single stream replication process that leverages the previously replicated backup so that only the changes are required to be scanned and sent to the target Data Domain. Once those changes have been sent, the target Data Domain is able to virtually synthesize a full backup, using the backup from the previous replication as a base file, stitching it together with the newly replicated changes. This is in contrast to the NCR/AMS process which requires the full backup be scanned. VSR is supported in Avamar 7.1/DDOS 5.5 and up.

Key Features of VSR:

  • Single Data Domain stream consumed per concurrent Avamar process.
  • Introduced in Avamar 7.1/DDOS 5.5
  • By default Avamar will attempt to user VSR whenever possible.
  • Replicates only the changed data.
  • Highly network efficient.
  • Ideal for low change rate clients.
  • Only compatible with NDMP, VMware Image, and File system backups.

Requirements for VSR:

  • All of these conditions must be met, otherwise replication will default back to NCR:
    • Both the replication source and the replication destination Avamar servers must be at AV7.1 or
      later.
    • Both the replication source and the replication destination Data Domain systems must be at DD OS
      5.5 or later.
    • Backup dataset must be compatible with VSR; i.e., backup dataset must be VMware image, file
      system, or NDMP backups to AV/DD.
    • “Base” backup, which is the immediate previous backup of that dataset, must exist on replication
      destination.
    • Replication method must be set to default or to force VSR.
    • If the backup leverages the snapview process then all of the partials must also be replicated in the
      correct order for VSR to work correctly. Please see the additional notes on VSR section for further details.
      When attempting to replicate a backup, the initial replication will leverage NCR to seed the base file on to the replication target. Once the base files are located on the target Data Domain, VSR will be used assuming all VSR requirements are met. As long as VSR continues to work correctly the base file is then updated every time the replication completes.

Example of Avamar choosing VSR replication:

2016-03-15 07:37:17 avtar Info <6654>: Replicating backup 471, 100.0 GB
2016-03-15 07:37:17 avtar Info <40198>: id:19 VSR selected because file
size is too small for NCR (16912D3D130B98D763593BD4DCD0BC586766FE46)
2016-03-15 07:37:17 avtar Info <40198>: id:20 VSR selected because file
size is too small for NCR (04C560CC90AC7A626F4E7866573CF93D7D32EFE6)
2016-03-15 07:37:17 avtar Info <40198>: id:21 VSR selected because file
size is too small for NCR (9D78E4B1CD2DF728D7A45B2E374D6D28A853ED2C)
2016-03-15 07:37:17 avtar Info <40198>: id:22 VSR selected because file
size is too small for NCR (41A1BDB73F3BB615E03A95FC3758CDE1A2725E6F)
2016-03-15 07:37:17 avtar Info <40198>: id:23 VSR selected because file
size is too small for NCR (1D6784C9E27240132EB7FBFEE3C0D4136D81AA8D)
2016-03-15 07:37:17 avtar Info <40198>: id:24 VSR selected because file
size is too small for NCR (FC295BC80B73D67BFE437291253D46FAD42650C2)
2016-03-15 07:37:17 avtar Info <40198>: id:25 VSR selected because file
size is too small for NCR (93B06F7711E1F64111BCAD4FBE966E051655EB94)
2016-03-15 07:37:17 avtar Info <40198>: id:26 VSR selected because file
size is too small for NCR (55A58F9CDCB82A6756853E3D84A3D2A6993F2C3D)
2016-03-15 07:37:17 avtar Info <40200>: id:27 VSR selected because all
base files are available (6122D9EF93CB95529D5E59327A3FF4789B0D0911)

Additional Notes on VSR
In some cases where Avamar leverages partial backups VSR will not always be a viable solution. By design, Avamar expires partial backups every 7 days. If a backup is not replicated with all of its partials within that 7 day time limit Avamar removes them. This typically affects large backup clients like NDMP.

For additional information on this issue please see one of the following KBs:

  • For Avamar 7.1.X KB 463951
  • For Avamar 7.2.X KB 463040

So now for the instructions of how to force the replication type to AMS/NCR:

  1. Edit the replication job that you want to modify
  2. Go to the Overview screen
  3. Click More Options
  4. Click More at the bottom right hand side of the window
  5. Input the following:
    1. [avtar]ddr-repl-method-control=N
      Where N is the desired value.
      Setting this value to 0 forces AMS/NCR
      Setting this value to 10 forces VSR
      By default this value is set to 5
    2. .Capture2
  6. Make sure to click the + sign to add the command

Avamar 7.3 with Data Domain

I upgraded the Avamar in my lab yesterday to the GA release of Avamar 7.3.  After the upgrade finished I had several errors pertaining to the integration with my Data Domain.  Errors on the Avamar were related to the certificate exchange between Avamar and Data Domain and also that a password was not specified.  In Data Domain System Manager there is an option under Administration->Access->Administrator Access that allows you to set a passphrase.  I set this passphrase on the Data Domain and went back to Avamar and generated a new SSH-Key and everything started reporting correctly again.  The specific error in Avamar was ‘System does not have any passphrase.  System passphrase needs to be set to proceed.’

DD Passphrase

Steps for editing Avamar are below (perform these steps after you set the passphrase on the Data Domain):

  1. Once logged into Avamar Administrator go to Server
  2. Go to Server Management tab
  3. Highlight the Data Domain and click the icon with the magnifying glass at the top to edit
  4. Click Ok to re-establish connection with Data Domain now that the passphrase has been set
  5. If you have an SSH key error after this you can regenerate the SSH key from the same screen by editing the Data Domain again

Screenshots of errors below:

Avamar Error 1

Avamar Error 2

 

 

Cannot browse CIFS share presented on Data Domain

I have had this issue a couple times recently and want to put the resolution out in case someone encounters this same issue.  SMB signing is disabled by default on Data Domain so if you if your organization requires SMB signing this will need to be configured on the Data Domain.

  1. Login to the DD as sysadmin
  2. Run the following command
    • cifs option set “server signing” auto

A couple of things to note from the DD documentation with SMB signing enabled:

  • SMB signing is supported from DDOS 5.2.4.
  • SMB signing is disabled by default for performance reasons. When enabled, this feature can cause
    throughput performance drops of from 29 percent (reads) to 50 percent (writes). Individual system
    performance will vary.

Hope this helps!