Data Domain Account Locked

Ran into an issue this week where a customer didn’t know their ddboost password and subsequently locked their account out.  Since this was the DDBoost user we were able to login with the sysadmin user and reset the password but the account was still locked out.  After searching on support I was able to figure out how to unlock the account and successfully login with the new password.  In this example I used the sysadmin account to login since the customer was able to provide that password.

  1. Log-in to DD with user having admin role.
  2. Go to Bash mode
    • log into system
    • system show serialno (paste this when it asks for password)
    • priv set se
    • ctrl-c three times
    • uname
    • fi st
    • df
    • ctrl-c three times
    • shell-escape
  3. Reset the tally account for the specific user as below:
    • pam_tally –user <username>
    • pam_tally –reset
    • pam_tally –user <username>

Unexpected (mkdir) system problem; command aborted

I wrote a blog a few years ago about Avamar integration failing because it failed to authenticate with the SSH key file.  I still run into this issue from time to time but this particular time I received another error when trying to add the ssh key to Data Domain.

The error I encountered basically was saying it didn’t have permissions to a directory or in my case didn’t have a directory at all to add the ssh key.  The error I got was ‘Unexpected (mkdir) system problem; command aborted’.  After searching on support I found the solution and thought I would share here in case anyone else runs into it.

  1. Elevate to the bash mode on the DD
    • log into system with the sysadmin account
    • system show serialno (paste this when it asks for password)
    • priv set se
    • ctrl-c three times
    • uname
    • fi st
    • df
    • ctrl-c three times
    • shell-escape
  2. cd /ddr/var/home/
  3. ls -hal
  4. if the ddboost user doesn’t exist:
    • mkdir ddboost
    • chown boost:admin ddboost
    • chmod 750 ddboost
  5. ls -hal
  6. Proceed with adding the Data Domain to the Avamar

After running through these steps you should be able to successfully add the ssh key to the Data Domain.

Avamar GLR with ItemPoint for Exchange

I have implemented Avamar for a couple of customers recently that were running Exchange 2010 and Exchange 2013 in house.  Avamar versions were 7.4.1-32 (there is a hotfix for Exchange that has been released for this version hotfix 285198) and 7.5.0-183.

When installing the Avamar agent and the Exchange VSS plugin (including the EMC ItemPoint software) at each customer both installed and registered as expected.  After installing and rebooting the Exchange server we attempted a GLR from Avamar (steps for this can be found in the Exchange VSS guide located under documents on the Avamar Documents and Downloads page).

The mounts of the .edb file from the Avamar/DD were successful and displayed the path of the .edb and the log file as designed.  When we opened ItemPoint and selected the location of the .edb and the log file (selection of the log file is not a requirement) it displayed the restored .edb in the top pane of Avamar and the live mailboxes in the bottom pane.

When trying to browse the live mailboxes in the bottom pane we received a permissions error.  Upon opening a ticket with Dell EMC (they had to get Kroll, the developers of the ItemPoint software installed), we were able to resolve the issue.

After running into this at the first customer I made sure to document the fix.  I hit this same issue at the second customer and was able to resolve this myself.  I have pasted the link to the fix as well as the command.  Sometimes copy/paste (even from notepad) doesn’t copy correctly inside of the Exchange Powershell so if you get errors with the command try typing it out.

Use Example 4 from the link below:

This is the command you will need to run (after the -User flag insert whatever user you are logging into the server with Kroll installed trying to perform GLR):

Get-Mailbox -ResultSize unlimited -Filter {(RecipientTypeDetails -eq ‘UserMailbox’) -and (Alias -ne ‘Admin’)} | Add-MailboxPermission -User avamarbackupuser -AccessRights FullAccess -InheritanceType all

Dell EMC World 2017

This year I was able to attend Dell EMC World.  I attended many sessions of which most I was pleasantly surprised with the content.  I tried to get notes and slide pictures from all of the sessions I attended.

The notes cover the products listed below:

  • Nutanix
  • Cloud Data Protection
  • VMAX
  • Data Domain
  • RecoverPoint
  • RecoverPoint For Virtual Machines (RP4VMs)
  • Data Protection Suite
  • Unity

You can view all my notes at the OneDrive link below.  Enjoy!!Ak4awN7U0kM4hZVv1T1ezyqmvTcBxA

Avamar Proxy Deployment Manager ‘Error Deploying Proxy’

Working with a customer this week and we were adding a new vCenter into their Avamar environment.  With this we needed to deploy some new proxies.  The proxy deployment manager makes this process very easy and allows you to deploy multiple proxies in a short amount of time.

After getting all the proxies configured I tried to apply the settings (this is the point that it goes out to vCenter and tries to deploy the proxies in the VMware environment) which failed.  There were no good errors in the GUI of Avamar or VMware so I dug a little further into Avamar and found the issue.

After examining the log (/usr/local/avamar/var/log/vcs/deploymentmanager.log) I found an error referencing an unsupported curve issue.  After some searching on support I found the fix.  Outlined below are the steps to remediate this issue.

  • Login to the Avamar server as admin and run the following commands:
    • su – root
    • cd /tmp
    • mkdir javafix
    • cd javafix
    • tar -xzvf jsafe-6.2_v0002.tgz
    • –stop
    • mv *jsafe* /usr/local/avamar-tomcat/lib/
    • cd /usr/local/avamar-tomcat/lib/
    • ./
    • –start

The above is a condensed list of all the steps.  The detailed issue is outlined in KB# 000489358.  After making the above changes I was able to successfully use the proxy deployment manager to deploy proxies.

Avamar Instant Access Restore Fails

Instant Access is a restore method in Avamar that allows you to mount a NFS export directly from the Data Domain to a VMware host in your environment.  This is a great method when you need a restore done quickly (1-2 minutes) rather than having to wait for it to copy all the blocks to a datastore already present in your VMware environment.  I usually have great success with this restore method but ran into an issue yesterday.  I got the following error on the Avamar job log after the restore failed:

2016-08-08 16:43:19 vmir Error <0000>: Create NFS datastore failed to start with error: Failed to add NFS directory on Data Domain system.
2016-08-08 16:43:19 vmir Error <0000>: Failed to create NFS datastore

After doing some troubleshooting I found out that the ddboost user that the Avamar was using to integrate with the Data Domain was locked by running the ‘user show list’ command.


I figured out that the default password aging policy for users created is 90 days so all of my DDBoost users that were older than 90 days were locked.  I ran the ‘user enable ddboost-user‘ command to enable the user and then I ran the ‘user password aging set ddboost-user max-days-between-change 99999′.  After doing this you can run ‘user password aging show’ command to see that the users password aging policy has been changed.


At this point I was able to run the instant access restore successfully.

Avamar Replication

I have been troubleshooting some Avamar replication integrated with Data Domain for over a week now and have finally got to a point where I am making some progress.  While troubleshooting with EMC and my own research I have learned a lot about how Avamar replicates and what the processes under the covers are.

So now for the background of my issue, I have a customer that had one particular job that hasn’t replicated in over a month.  We looked at the network and found some issues which were fixed but still did not allow the replication to finish.  I opened a ticket with EMC Data Domain to see if there were any settings on the Data Domain that would be limiting the bandwidth used for replication (I check all the normal settings on Avamar and Data Domain pertaining to throttling but wasn’t sure if there were others that could be set that I was missing).  EMC got back with me and we determined that the Data Domain was working as expected so they recommended moving my ticket to the Avamar integration team.

Once I started working with that team they gave me some tweaks to try and speed things up.  The first thing was allowing more streams per process on the Avamar side.  So in the GUI based replication job on the options page you can change the maximum concurrent processes from 1-8 (1 is the default and 8 is the max).  This will allow up to 8 clients to run a replication job at the same time.  What I did not realize is there are streams that can be specified to allow more streams per process giving you better replication performance.  By default the stream count per process is 6.  To give an example if you have 8 processes setup on the Avamar replication it will run 48 streams (8 processes multiplied by 6 streams).  Below is the process to modify this:

  2. Edit the replication job that you want to modify
  3. Go to the Overview screen
  4. Click More Options
  5. Click More at the bottom right hand side of the window
  6. Input the following:
    1. [avtar]ddr-repl-max-parallel-streams    =     # of streams to be allocated
    2. Capture1
  7. Make sure to click the + sign to add the command

After making the above change I could see under the replication window on Data Domain for DDBoost that the files that were replicating at one time did increase to the number I set but it still didn’t allow the job to finish and it always appeared to get hung while replicating.  The EMC tech gave me one more thing to change which is what helped the most.

I had no idea there were underlying replication types of Avamar replication.  The EMC tech recommended that I forced this job to use AMS/NCR rather than VSR.  Before I forced anything I wanted to know more about these replication types which was not real easy to find.  I found an EMC KB (000479374) which explains the different types of replication.  AMS/NCR stands for Automated Multi-Streaming/Ninja Chopping Replication, yes that’s right I said Ninja Chopping Replication which is a pretty awesome name :-).  VSR stands for Virtual Synthetic Replication which is not nearly as cool.  Below are some of the things I pulled from the KB listed above that explains the two.

Automated Multi-Streaming (AMS)/Ninja Chopping Replication (NCR)

Automated Multi-streaming replication, also known as Ninja Chopping Replication, works by inflating a backup on the Data Domain and walking that backup using multiple streams to “chop” the backup up before sending it to the target Data Domain. AMS/NCR replication is supported in Avamar 7.0 and up.

Key Features of AMS/NCR:

  • Leverages multiple parallel streams to process and send backups to the replication target.
  • Introduced in Avamar v7.0
  • By default the Avamar uses 6 streams per concurrent process, but this value can be changed using the proper flags.
  • NCR must scan the entire backup rather than just the changes, so it often has to more data to scan than if VSR were used.
  • NCR typically has better performance with high change rate clients when the bandwidth between the sites is large.
  • NCR can cause space differences between the source and target Data Domains. This is discussed further in KB 335802.
  • No restrictions on client types for this type of replication.

Example of Avamar choosing NCR Replication:

2015-11-22 10:05:55 avtar Info <6654>: Replicating backup 367, 528.7 GB
2015-11-22 10:05:56 avtar Info <40047>: id:1 VSR selected because file
size is too small for NCR (container.1.cdsf)
2015-11-22 10:05:56 avtar Info <40050>: id:2 NCR selected because it could
not be determined if VSR is possible (container.2.cdsf)

Virtual Synthetic Replication (VSR)

Virtual Synthetic replication is a single stream replication process that leverages the previously replicated backup so that only the changes are required to be scanned and sent to the target Data Domain. Once those changes have been sent, the target Data Domain is able to virtually synthesize a full backup, using the backup from the previous replication as a base file, stitching it together with the newly replicated changes. This is in contrast to the NCR/AMS process which requires the full backup be scanned. VSR is supported in Avamar 7.1/DDOS 5.5 and up.

Key Features of VSR:

  • Single Data Domain stream consumed per concurrent Avamar process.
  • Introduced in Avamar 7.1/DDOS 5.5
  • By default Avamar will attempt to user VSR whenever possible.
  • Replicates only the changed data.
  • Highly network efficient.
  • Ideal for low change rate clients.
  • Only compatible with NDMP, VMware Image, and File system backups.

Requirements for VSR:

  • All of these conditions must be met, otherwise replication will default back to NCR:
    • Both the replication source and the replication destination Avamar servers must be at AV7.1 or
    • Both the replication source and the replication destination Data Domain systems must be at DD OS
      5.5 or later.
    • Backup dataset must be compatible with VSR; i.e., backup dataset must be VMware image, file
      system, or NDMP backups to AV/DD.
    • “Base” backup, which is the immediate previous backup of that dataset, must exist on replication
    • Replication method must be set to default or to force VSR.
    • If the backup leverages the snapview process then all of the partials must also be replicated in the
      correct order for VSR to work correctly. Please see the additional notes on VSR section for further details.
      When attempting to replicate a backup, the initial replication will leverage NCR to seed the base file on to the replication target. Once the base files are located on the target Data Domain, VSR will be used assuming all VSR requirements are met. As long as VSR continues to work correctly the base file is then updated every time the replication completes.

Example of Avamar choosing VSR replication:

2016-03-15 07:37:17 avtar Info <6654>: Replicating backup 471, 100.0 GB
2016-03-15 07:37:17 avtar Info <40198>: id:19 VSR selected because file
size is too small for NCR (16912D3D130B98D763593BD4DCD0BC586766FE46)
2016-03-15 07:37:17 avtar Info <40198>: id:20 VSR selected because file
size is too small for NCR (04C560CC90AC7A626F4E7866573CF93D7D32EFE6)
2016-03-15 07:37:17 avtar Info <40198>: id:21 VSR selected because file
size is too small for NCR (9D78E4B1CD2DF728D7A45B2E374D6D28A853ED2C)
2016-03-15 07:37:17 avtar Info <40198>: id:22 VSR selected because file
size is too small for NCR (41A1BDB73F3BB615E03A95FC3758CDE1A2725E6F)
2016-03-15 07:37:17 avtar Info <40198>: id:23 VSR selected because file
size is too small for NCR (1D6784C9E27240132EB7FBFEE3C0D4136D81AA8D)
2016-03-15 07:37:17 avtar Info <40198>: id:24 VSR selected because file
size is too small for NCR (FC295BC80B73D67BFE437291253D46FAD42650C2)
2016-03-15 07:37:17 avtar Info <40198>: id:25 VSR selected because file
size is too small for NCR (93B06F7711E1F64111BCAD4FBE966E051655EB94)
2016-03-15 07:37:17 avtar Info <40198>: id:26 VSR selected because file
size is too small for NCR (55A58F9CDCB82A6756853E3D84A3D2A6993F2C3D)
2016-03-15 07:37:17 avtar Info <40200>: id:27 VSR selected because all
base files are available (6122D9EF93CB95529D5E59327A3FF4789B0D0911)

Additional Notes on VSR
In some cases where Avamar leverages partial backups VSR will not always be a viable solution. By design, Avamar expires partial backups every 7 days. If a backup is not replicated with all of its partials within that 7 day time limit Avamar removes them. This typically affects large backup clients like NDMP.

For additional information on this issue please see one of the following KBs:

  • For Avamar 7.1.X KB 463951
  • For Avamar 7.2.X KB 463040

So now for the instructions of how to force the replication type to AMS/NCR:

  1. Edit the replication job that you want to modify
  2. Go to the Overview screen
  3. Click More Options
  4. Click More at the bottom right hand side of the window
  5. Input the following:
    1. [avtar]ddr-repl-method-control=N
      Where N is the desired value.
      Setting this value to 0 forces AMS/NCR
      Setting this value to 10 forces VSR
      By default this value is set to 5
    2. .Capture2
  6. Make sure to click the + sign to add the command