Dell EMC World 2017

This year I was able to attend Dell EMC World.  I attended many sessions of which most I was pleasantly surprised with the content.  I tried to get notes and slide pictures from all of the sessions I attended.

The notes cover the products listed below:

  • Nutanix
  • Cloud Data Protection
  • VMAX
  • Data Domain
  • RecoverPoint
  • RecoverPoint For Virtual Machines (RP4VMs)
  • Data Protection Suite
  • Unity

You can view all my notes at the OneDrive link below.  Enjoy!!Ak4awN7U0kM4hZVv1T1ezyqmvTcBxA

Avamar Proxy Deployment Manager ‘Error Deploying Proxy’

Working with a customer this week and we were adding a new vCenter into their Avamar environment.  With this we needed to deploy some new proxies.  The proxy deployment manager makes this process very easy and allows you to deploy multiple proxies in a short amount of time.

After getting all the proxies configured I tried to apply the settings (this is the point that it goes out to vCenter and tries to deploy the proxies in the VMware environment) which failed.  There were no good errors in the GUI of Avamar or VMware so I dug a little further into Avamar and found the issue.

After examining the log (/usr/local/avamar/var/log/vcs/deploymentmanager.log) I found an error referencing an unsupported curve issue.  After some searching on support I found the fix.  Outlined below are the steps to remediate this issue.

  • Login to the Avamar server as admin and run the following commands:
    • su – root
    • cd /tmp
    • mkdir javafix
    • cd javafix
    • tar -xzvf jsafe-6.2_v0002.tgz
    • –stop
    • mv *jsafe* /usr/local/avamar-tomcat/lib/
    • cd /usr/local/avamar-tomcat/lib/
    • ./
    • –start

The above is a condensed list of all the steps.  The detailed issue is outlined in KB# 000489358.  After making the above changes I was able to successfully use the proxy deployment manager to deploy proxies.

Avamar Instant Access Restore Fails

Instant Access is a restore method in Avamar that allows you to mount a NFS export directly from the Data Domain to a VMware host in your environment.  This is a great method when you need a restore done quickly (1-2 minutes) rather than having to wait for it to copy all the blocks to a datastore already present in your VMware environment.  I usually have great success with this restore method but ran into an issue yesterday.  I got the following error on the Avamar job log after the restore failed:

2016-08-08 16:43:19 vmir Error <0000>: Create NFS datastore failed to start with error: Failed to add NFS directory on Data Domain system.
2016-08-08 16:43:19 vmir Error <0000>: Failed to create NFS datastore

After doing some troubleshooting I found out that the ddboost user that the Avamar was using to integrate with the Data Domain was locked by running the ‘user show list’ command.


I figured out that the default password aging policy for users created is 90 days so all of my DDBoost users that were older than 90 days were locked.  I ran the ‘user enable ddboost-user‘ command to enable the user and then I ran the ‘user password aging set ddboost-user max-days-between-change 99999′.  After doing this you can run ‘user password aging show’ command to see that the users password aging policy has been changed.


At this point I was able to run the instant access restore successfully.

Avamar Replication

I have been troubleshooting some Avamar replication integrated with Data Domain for over a week now and have finally got to a point where I am making some progress.  While troubleshooting with EMC and my own research I have learned a lot about how Avamar replicates and what the processes under the covers are.

So now for the background of my issue, I have a customer that had one particular job that hasn’t replicated in over a month.  We looked at the network and found some issues which were fixed but still did not allow the replication to finish.  I opened a ticket with EMC Data Domain to see if there were any settings on the Data Domain that would be limiting the bandwidth used for replication (I check all the normal settings on Avamar and Data Domain pertaining to throttling but wasn’t sure if there were others that could be set that I was missing).  EMC got back with me and we determined that the Data Domain was working as expected so they recommended moving my ticket to the Avamar integration team.

Once I started working with that team they gave me some tweaks to try and speed things up.  The first thing was allowing more streams per process on the Avamar side.  So in the GUI based replication job on the options page you can change the maximum concurrent processes from 1-8 (1 is the default and 8 is the max).  This will allow up to 8 clients to run a replication job at the same time.  What I did not realize is there are streams that can be specified to allow more streams per process giving you better replication performance.  By default the stream count per process is 6.  To give an example if you have 8 processes setup on the Avamar replication it will run 48 streams (8 processes multiplied by 6 streams).  Below is the process to modify this:

  2. Edit the replication job that you want to modify
  3. Go to the Overview screen
  4. Click More Options
  5. Click More at the bottom right hand side of the window
  6. Input the following:
    1. [avtar]ddr-repl-max-parallel-streams    =     # of streams to be allocated
    2. Capture1
  7. Make sure to click the + sign to add the command

After making the above change I could see under the replication window on Data Domain for DDBoost that the files that were replicating at one time did increase to the number I set but it still didn’t allow the job to finish and it always appeared to get hung while replicating.  The EMC tech gave me one more thing to change which is what helped the most.

I had no idea there were underlying replication types of Avamar replication.  The EMC tech recommended that I forced this job to use AMS/NCR rather than VSR.  Before I forced anything I wanted to know more about these replication types which was not real easy to find.  I found an EMC KB (000479374) which explains the different types of replication.  AMS/NCR stands for Automated Multi-Streaming/Ninja Chopping Replication, yes that’s right I said Ninja Chopping Replication which is a pretty awesome name :-).  VSR stands for Virtual Synthetic Replication which is not nearly as cool.  Below are some of the things I pulled from the KB listed above that explains the two.

Automated Multi-Streaming (AMS)/Ninja Chopping Replication (NCR)

Automated Multi-streaming replication, also known as Ninja Chopping Replication, works by inflating a backup on the Data Domain and walking that backup using multiple streams to “chop” the backup up before sending it to the target Data Domain. AMS/NCR replication is supported in Avamar 7.0 and up.

Key Features of AMS/NCR:

  • Leverages multiple parallel streams to process and send backups to the replication target.
  • Introduced in Avamar v7.0
  • By default the Avamar uses 6 streams per concurrent process, but this value can be changed using the proper flags.
  • NCR must scan the entire backup rather than just the changes, so it often has to more data to scan than if VSR were used.
  • NCR typically has better performance with high change rate clients when the bandwidth between the sites is large.
  • NCR can cause space differences between the source and target Data Domains. This is discussed further in KB 335802.
  • No restrictions on client types for this type of replication.

Example of Avamar choosing NCR Replication:

2015-11-22 10:05:55 avtar Info <6654>: Replicating backup 367, 528.7 GB
2015-11-22 10:05:56 avtar Info <40047>: id:1 VSR selected because file
size is too small for NCR (container.1.cdsf)
2015-11-22 10:05:56 avtar Info <40050>: id:2 NCR selected because it could
not be determined if VSR is possible (container.2.cdsf)

Virtual Synthetic Replication (VSR)

Virtual Synthetic replication is a single stream replication process that leverages the previously replicated backup so that only the changes are required to be scanned and sent to the target Data Domain. Once those changes have been sent, the target Data Domain is able to virtually synthesize a full backup, using the backup from the previous replication as a base file, stitching it together with the newly replicated changes. This is in contrast to the NCR/AMS process which requires the full backup be scanned. VSR is supported in Avamar 7.1/DDOS 5.5 and up.

Key Features of VSR:

  • Single Data Domain stream consumed per concurrent Avamar process.
  • Introduced in Avamar 7.1/DDOS 5.5
  • By default Avamar will attempt to user VSR whenever possible.
  • Replicates only the changed data.
  • Highly network efficient.
  • Ideal for low change rate clients.
  • Only compatible with NDMP, VMware Image, and File system backups.

Requirements for VSR:

  • All of these conditions must be met, otherwise replication will default back to NCR:
    • Both the replication source and the replication destination Avamar servers must be at AV7.1 or
    • Both the replication source and the replication destination Data Domain systems must be at DD OS
      5.5 or later.
    • Backup dataset must be compatible with VSR; i.e., backup dataset must be VMware image, file
      system, or NDMP backups to AV/DD.
    • “Base” backup, which is the immediate previous backup of that dataset, must exist on replication
    • Replication method must be set to default or to force VSR.
    • If the backup leverages the snapview process then all of the partials must also be replicated in the
      correct order for VSR to work correctly. Please see the additional notes on VSR section for further details.
      When attempting to replicate a backup, the initial replication will leverage NCR to seed the base file on to the replication target. Once the base files are located on the target Data Domain, VSR will be used assuming all VSR requirements are met. As long as VSR continues to work correctly the base file is then updated every time the replication completes.

Example of Avamar choosing VSR replication:

2016-03-15 07:37:17 avtar Info <6654>: Replicating backup 471, 100.0 GB
2016-03-15 07:37:17 avtar Info <40198>: id:19 VSR selected because file
size is too small for NCR (16912D3D130B98D763593BD4DCD0BC586766FE46)
2016-03-15 07:37:17 avtar Info <40198>: id:20 VSR selected because file
size is too small for NCR (04C560CC90AC7A626F4E7866573CF93D7D32EFE6)
2016-03-15 07:37:17 avtar Info <40198>: id:21 VSR selected because file
size is too small for NCR (9D78E4B1CD2DF728D7A45B2E374D6D28A853ED2C)
2016-03-15 07:37:17 avtar Info <40198>: id:22 VSR selected because file
size is too small for NCR (41A1BDB73F3BB615E03A95FC3758CDE1A2725E6F)
2016-03-15 07:37:17 avtar Info <40198>: id:23 VSR selected because file
size is too small for NCR (1D6784C9E27240132EB7FBFEE3C0D4136D81AA8D)
2016-03-15 07:37:17 avtar Info <40198>: id:24 VSR selected because file
size is too small for NCR (FC295BC80B73D67BFE437291253D46FAD42650C2)
2016-03-15 07:37:17 avtar Info <40198>: id:25 VSR selected because file
size is too small for NCR (93B06F7711E1F64111BCAD4FBE966E051655EB94)
2016-03-15 07:37:17 avtar Info <40198>: id:26 VSR selected because file
size is too small for NCR (55A58F9CDCB82A6756853E3D84A3D2A6993F2C3D)
2016-03-15 07:37:17 avtar Info <40200>: id:27 VSR selected because all
base files are available (6122D9EF93CB95529D5E59327A3FF4789B0D0911)

Additional Notes on VSR
In some cases where Avamar leverages partial backups VSR will not always be a viable solution. By design, Avamar expires partial backups every 7 days. If a backup is not replicated with all of its partials within that 7 day time limit Avamar removes them. This typically affects large backup clients like NDMP.

For additional information on this issue please see one of the following KBs:

  • For Avamar 7.1.X KB 463951
  • For Avamar 7.2.X KB 463040

So now for the instructions of how to force the replication type to AMS/NCR:

  1. Edit the replication job that you want to modify
  2. Go to the Overview screen
  3. Click More Options
  4. Click More at the bottom right hand side of the window
  5. Input the following:
    1. [avtar]ddr-repl-method-control=N
      Where N is the desired value.
      Setting this value to 0 forces AMS/NCR
      Setting this value to 10 forces VSR
      By default this value is set to 5
    2. .Capture2
  6. Make sure to click the + sign to add the command

Avamar 7.3 with Data Domain

I upgraded the Avamar in my lab yesterday to the GA release of Avamar 7.3.  After the upgrade finished I had several errors pertaining to the integration with my Data Domain.  Errors on the Avamar were related to the certificate exchange between Avamar and Data Domain and also that a password was not specified.  In Data Domain System Manager there is an option under Administration->Access->Administrator Access that allows you to set a passphrase.  I set this passphrase on the Data Domain and went back to Avamar and generated a new SSH-Key and everything started reporting correctly again.  The specific error in Avamar was ‘System does not have any passphrase.  System passphrase needs to be set to proceed.’

DD Passphrase

Steps for editing Avamar are below (perform these steps after you set the passphrase on the Data Domain):

  1. Once logged into Avamar Administrator go to Server
  2. Go to Server Management tab
  3. Highlight the Data Domain and click the icon with the magnifying glass at the top to edit
  4. Click Ok to re-establish connection with Data Domain now that the passphrase has been set
  5. If you have an SSH key error after this you can regenerate the SSH key from the same screen by editing the Data Domain again

Screenshots of errors below:

Avamar Error 1

Avamar Error 2



Cannot browse CIFS share presented on Data Domain

I have had this issue a couple times recently and want to put the resolution out in case someone encounters this same issue.  SMB signing is disabled by default on Data Domain so if you if your organization requires SMB signing this will need to be configured on the Data Domain.

  1. Login to the DD as sysadmin
  2. Run the following command
    • cifs option set “server signing” auto

A couple of things to note from the DD documentation with SMB signing enabled:

  • SMB signing is supported from DDOS 5.2.4.
  • SMB signing is disabled by default for performance reasons. When enabled, this feature can cause
    throughput performance drops of from 29 percent (reads) to 50 percent (writes). Individual system
    performance will vary.

Hope this helps!

DDBoost for Microsoft Applications (SQL)

I got a chance to play around in the lab today using the new DDBoost for Microsoft Applications. This is a feature that gives the SQL DBAs the ability to use SQL Server Management Studio to perform their backups while still being able to use DDBoost. Prior to this being an option, if the company was using Data Domain to store their SQL backups using SQL jobs they had to use a CIFS share on the Data Domain. Using CIFS from a Data Domain is a perfect way to accomplish backups if the DBs want to control them; however, there is no dedup at the client side so the whole DB backup has to go over the network to the Data Domain before the Data Domain can dedup and compress. With DDBoost that dedup is offloaded to the server and only sends changed blocks across the network drastically decreasing network congestion. I have screenshots below of the setup procedure.

1. Download the Data Domain Boost for Microsoft Applications (either 32-bit or 64-bit) from

2. A password is required for extracting the zipped folder after download. This password would have been received with the software license purchased.

3. Once extracted, install the EMCDDBMA64.msi or EMCDDBMA.msi

4. After the install completes and you open SQL Server Management Studio you will click on the EMC DataDomain Boost for Microsoft Applications:

5. Once opened you will see where you can choose the server instance, database, recovery model, and backup type. Select the DB you want to backup and give it a name and description as well as an expire after date.

6. You will now setup the Data Domain with a Storage Unit and a user for that Storage Unit.

7. Now, back in SQL Server Management Studio, you will click the add button under Destination and fill in the appropriate information.

8. That’s it, you can now run a manual backup by selecting the Data Domain system you just added and clicking run.

9. You will get a completed job message once the job finishes.

10. Click on the restore tab and click the browse button to choose the Data Domain Server, then SQL Server Instance, the the Database to make sure you see the completed backup. This is also the same place you would come to do a restore of the Database.

11. If you want to schedule a job in SQL Server Management Studio to run automatically you would navigate to SQL Server Agent-Jobs, right click on Jobs and select new job. You would then create the job with a step to backup the Database. I have shown through screenshot and pasted the command needed to make this work, replace the appropriate sections with your SQL DB and Data Domain information.

Make sure the below command is all on one line in the job:
“c:\Program Files\EMC DD Boost Modules\DDBMA\bin\ddbmsqlsv.exe” -c SQLAO1.labclt.local -l full -a “NSR_DFA_SI=TRUE” -a “NSR_DFA_SI_USE_DD=TRUE” -a “NSR_DFA_SI_DD_HOST=” -a “NSR_DFA_SI_DD_USER=sql” -a “NSR_DFA_SI_DEVICE_PATH=/sqlao1” “MSSQL:master”

Once this job runs you will be able to see the backup by referring to step 10 above.