Avamar Instant Access Restore Fails

Instant Access is a restore method in Avamar that allows you to mount a NFS export directly from the Data Domain to a VMware host in your environment.  This is a great method when you need a restore done quickly (1-2 minutes) rather than having to wait for it to copy all the blocks to a datastore already present in your VMware environment.  I usually have great success with this restore method but ran into an issue yesterday.  I got the following error on the Avamar job log after the restore failed:

2016-08-08 16:43:19 vmir Error <0000>: Create NFS datastore failed to start with error: Failed to add NFS directory on Data Domain system.
2016-08-08 16:43:19 vmir Error <0000>: Failed to create NFS datastore

After doing some troubleshooting I found out that the ddboost user that the Avamar was using to integrate with the Data Domain was locked by running the ‘user show list’ command.


I figured out that the default password aging policy for users created is 90 days so all of my DDBoost users that were older than 90 days were locked.  I ran the ‘user enable ddboost-user‘ command to enable the user and then I ran the ‘user password aging set ddboost-user max-days-between-change 99999′.  After doing this you can run ‘user password aging show’ command to see that the users password aging policy has been changed.


At this point I was able to run the instant access restore successfully.

Avamar Replication

I have been troubleshooting some Avamar replication integrated with Data Domain for over a week now and have finally got to a point where I am making some progress.  While troubleshooting with EMC and my own research I have learned a lot about how Avamar replicates and what the processes under the covers are.

So now for the background of my issue, I have a customer that had one particular job that hasn’t replicated in over a month.  We looked at the network and found some issues which were fixed but still did not allow the replication to finish.  I opened a ticket with EMC Data Domain to see if there were any settings on the Data Domain that would be limiting the bandwidth used for replication (I check all the normal settings on Avamar and Data Domain pertaining to throttling but wasn’t sure if there were others that could be set that I was missing).  EMC got back with me and we determined that the Data Domain was working as expected so they recommended moving my ticket to the Avamar integration team.

Once I started working with that team they gave me some tweaks to try and speed things up.  The first thing was allowing more streams per process on the Avamar side.  So in the GUI based replication job on the options page you can change the maximum concurrent processes from 1-8 (1 is the default and 8 is the max).  This will allow up to 8 clients to run a replication job at the same time.  What I did not realize is there are streams that can be specified to allow more streams per process giving you better replication performance.  By default the stream count per process is 6.  To give an example if you have 8 processes setup on the Avamar replication it will run 48 streams (8 processes multiplied by 6 streams).  Below is the process to modify this:

  2. Edit the replication job that you want to modify
  3. Go to the Overview screen
  4. Click More Options
  5. Click More at the bottom right hand side of the window
  6. Input the following:
    1. [avtar]ddr-repl-max-parallel-streams    =     # of streams to be allocated
    2. Capture1
  7. Make sure to click the + sign to add the command

After making the above change I could see under the replication window on Data Domain for DDBoost that the files that were replicating at one time did increase to the number I set but it still didn’t allow the job to finish and it always appeared to get hung while replicating.  The EMC tech gave me one more thing to change which is what helped the most.

I had no idea there were underlying replication types of Avamar replication.  The EMC tech recommended that I forced this job to use AMS/NCR rather than VSR.  Before I forced anything I wanted to know more about these replication types which was not real easy to find.  I found an EMC KB (000479374) which explains the different types of replication.  AMS/NCR stands for Automated Multi-Streaming/Ninja Chopping Replication, yes that’s right I said Ninja Chopping Replication which is a pretty awesome name🙂.  VSR stands for Virtual Synthetic Replication which is not nearly as cool.  Below are some of the things I pulled from the KB listed above that explains the two.

Automated Multi-Streaming (AMS)/Ninja Chopping Replication (NCR)

Automated Multi-streaming replication, also known as Ninja Chopping Replication, works by inflating a backup on the Data Domain and walking that backup using multiple streams to “chop” the backup up before sending it to the target Data Domain. AMS/NCR replication is supported in Avamar 7.0 and up.

Key Features of AMS/NCR:

  • Leverages multiple parallel streams to process and send backups to the replication target.
  • Introduced in Avamar v7.0
  • By default the Avamar uses 6 streams per concurrent process, but this value can be changed using the proper flags.
  • NCR must scan the entire backup rather than just the changes, so it often has to more data to scan than if VSR were used.
  • NCR typically has better performance with high change rate clients when the bandwidth between the sites is large.
  • NCR can cause space differences between the source and target Data Domains. This is discussed further in KB 335802.
  • No restrictions on client types for this type of replication.

Example of Avamar choosing NCR Replication:

2015-11-22 10:05:55 avtar Info <6654>: Replicating backup 367, 528.7 GB
2015-11-22 10:05:56 avtar Info <40047>: id:1 VSR selected because file
size is too small for NCR (container.1.cdsf)
2015-11-22 10:05:56 avtar Info <40050>: id:2 NCR selected because it could
not be determined if VSR is possible (container.2.cdsf)

Virtual Synthetic Replication (VSR)

Virtual Synthetic replication is a single stream replication process that leverages the previously replicated backup so that only the changes are required to be scanned and sent to the target Data Domain. Once those changes have been sent, the target Data Domain is able to virtually synthesize a full backup, using the backup from the previous replication as a base file, stitching it together with the newly replicated changes. This is in contrast to the NCR/AMS process which requires the full backup be scanned. VSR is supported in Avamar 7.1/DDOS 5.5 and up.

Key Features of VSR:

  • Single Data Domain stream consumed per concurrent Avamar process.
  • Introduced in Avamar 7.1/DDOS 5.5
  • By default Avamar will attempt to user VSR whenever possible.
  • Replicates only the changed data.
  • Highly network efficient.
  • Ideal for low change rate clients.
  • Only compatible with NDMP, VMware Image, and File system backups.

Requirements for VSR:

  • All of these conditions must be met, otherwise replication will default back to NCR:
    • Both the replication source and the replication destination Avamar servers must be at AV7.1 or
    • Both the replication source and the replication destination Data Domain systems must be at DD OS
      5.5 or later.
    • Backup dataset must be compatible with VSR; i.e., backup dataset must be VMware image, file
      system, or NDMP backups to AV/DD.
    • “Base” backup, which is the immediate previous backup of that dataset, must exist on replication
    • Replication method must be set to default or to force VSR.
    • If the backup leverages the snapview process then all of the partials must also be replicated in the
      correct order for VSR to work correctly. Please see the additional notes on VSR section for further details.
      When attempting to replicate a backup, the initial replication will leverage NCR to seed the base file on to the replication target. Once the base files are located on the target Data Domain, VSR will be used assuming all VSR requirements are met. As long as VSR continues to work correctly the base file is then updated every time the replication completes.

Example of Avamar choosing VSR replication:

2016-03-15 07:37:17 avtar Info <6654>: Replicating backup 471, 100.0 GB
2016-03-15 07:37:17 avtar Info <40198>: id:19 VSR selected because file
size is too small for NCR (16912D3D130B98D763593BD4DCD0BC586766FE46)
2016-03-15 07:37:17 avtar Info <40198>: id:20 VSR selected because file
size is too small for NCR (04C560CC90AC7A626F4E7866573CF93D7D32EFE6)
2016-03-15 07:37:17 avtar Info <40198>: id:21 VSR selected because file
size is too small for NCR (9D78E4B1CD2DF728D7A45B2E374D6D28A853ED2C)
2016-03-15 07:37:17 avtar Info <40198>: id:22 VSR selected because file
size is too small for NCR (41A1BDB73F3BB615E03A95FC3758CDE1A2725E6F)
2016-03-15 07:37:17 avtar Info <40198>: id:23 VSR selected because file
size is too small for NCR (1D6784C9E27240132EB7FBFEE3C0D4136D81AA8D)
2016-03-15 07:37:17 avtar Info <40198>: id:24 VSR selected because file
size is too small for NCR (FC295BC80B73D67BFE437291253D46FAD42650C2)
2016-03-15 07:37:17 avtar Info <40198>: id:25 VSR selected because file
size is too small for NCR (93B06F7711E1F64111BCAD4FBE966E051655EB94)
2016-03-15 07:37:17 avtar Info <40198>: id:26 VSR selected because file
size is too small for NCR (55A58F9CDCB82A6756853E3D84A3D2A6993F2C3D)
2016-03-15 07:37:17 avtar Info <40200>: id:27 VSR selected because all
base files are available (6122D9EF93CB95529D5E59327A3FF4789B0D0911)

Additional Notes on VSR
In some cases where Avamar leverages partial backups VSR will not always be a viable solution. By design, Avamar expires partial backups every 7 days. If a backup is not replicated with all of its partials within that 7 day time limit Avamar removes them. This typically affects large backup clients like NDMP.

For additional information on this issue please see one of the following KBs:

  • For Avamar 7.1.X KB 463951
  • For Avamar 7.2.X KB 463040

So now for the instructions of how to force the replication type to AMS/NCR:

  1. Edit the replication job that you want to modify
  2. Go to the Overview screen
  3. Click More Options
  4. Click More at the bottom right hand side of the window
  5. Input the following:
    1. [avtar]ddr-repl-method-control=N
      Where N is the desired value.
      Setting this value to 0 forces AMS/NCR
      Setting this value to 10 forces VSR
      By default this value is set to 5
    2. .Capture2
  6. Make sure to click the + sign to add the command

Avamar 7.3 with Data Domain

I upgraded the Avamar in my lab yesterday to the GA release of Avamar 7.3.  After the upgrade finished I had several errors pertaining to the integration with my Data Domain.  Errors on the Avamar were related to the certificate exchange between Avamar and Data Domain and also that a password was not specified.  In Data Domain System Manager there is an option under Administration->Access->Administrator Access that allows you to set a passphrase.  I set this passphrase on the Data Domain and went back to Avamar and generated a new SSH-Key and everything started reporting correctly again.  The specific error in Avamar was ‘System does not have any passphrase.  System passphrase needs to be set to proceed.’

DD Passphrase

Steps for editing Avamar are below (perform these steps after you set the passphrase on the Data Domain):

  1. Once logged into Avamar Administrator go to Server
  2. Go to Server Management tab
  3. Highlight the Data Domain and click the icon with the magnifying glass at the top to edit
  4. Click Ok to re-establish connection with Data Domain now that the passphrase has been set
  5. If you have an SSH key error after this you can regenerate the SSH key from the same screen by editing the Data Domain again

Screenshots of errors below:

Avamar Error 1

Avamar Error 2



Cannot browse CIFS share presented on Data Domain

I have had this issue a couple times recently and want to put the resolution out in case someone encounters this same issue.  SMB signing is disabled by default on Data Domain so if you if your organization requires SMB signing this will need to be configured on the Data Domain.

  1. Login to the DD as sysadmin
  2. Run the following command
    • cifs option set “server signing” auto

A couple of things to note from the DD documentation with SMB signing enabled:

  • SMB signing is supported from DDOS 5.2.4.
  • SMB signing is disabled by default for performance reasons. When enabled, this feature can cause
    throughput performance drops of from 29 percent (reads) to 50 percent (writes). Individual system
    performance will vary.

Hope this helps!

DDBoost for Microsoft Applications (SQL)

I got a chance to play around in the lab today using the new DDBoost for Microsoft Applications. This is a feature that gives the SQL DBAs the ability to use SQL Server Management Studio to perform their backups while still being able to use DDBoost. Prior to this being an option, if the company was using Data Domain to store their SQL backups using SQL jobs they had to use a CIFS share on the Data Domain. Using CIFS from a Data Domain is a perfect way to accomplish backups if the DBs want to control them; however, there is no dedup at the client side so the whole DB backup has to go over the network to the Data Domain before the Data Domain can dedup and compress. With DDBoost that dedup is offloaded to the server and only sends changed blocks across the network drastically decreasing network congestion. I have screenshots below of the setup procedure.

1. Download the Data Domain Boost for Microsoft Applications (either 32-bit or 64-bit) from support.emc.com

2. A password is required for extracting the zipped folder after download. This password would have been received with the software license purchased.

3. Once extracted, install the EMCDDBMA64.msi or EMCDDBMA.msi

4. After the install completes and you open SQL Server Management Studio you will click on the EMC DataDomain Boost for Microsoft Applications:

5. Once opened you will see where you can choose the server instance, database, recovery model, and backup type. Select the DB you want to backup and give it a name and description as well as an expire after date.

6. You will now setup the Data Domain with a Storage Unit and a user for that Storage Unit.

7. Now, back in SQL Server Management Studio, you will click the add button under Destination and fill in the appropriate information.

8. That’s it, you can now run a manual backup by selecting the Data Domain system you just added and clicking run.

9. You will get a completed job message once the job finishes.

10. Click on the restore tab and click the browse button to choose the Data Domain Server, then SQL Server Instance, the the Database to make sure you see the completed backup. This is also the same place you would come to do a restore of the Database.

11. If you want to schedule a job in SQL Server Management Studio to run automatically you would navigate to SQL Server Agent-Jobs, right click on Jobs and select new job. You would then create the job with a step to backup the Database. I have shown through screenshot and pasted the command needed to make this work, replace the appropriate sections with your SQL DB and Data Domain information.

Make sure the below command is all on one line in the job:
“c:\Program Files\EMC DD Boost Modules\DDBMA\bin\ddbmsqlsv.exe” -c SQLAO1.labclt.local -l full -a “NSR_DFA_SI=TRUE” -a “NSR_DFA_SI_USE_DD=TRUE” -a “NSR_DFA_SI_DD_HOST=” -a “NSR_DFA_SI_DD_USER=sql” -a “NSR_DFA_SI_DEVICE_PATH=/sqlao1” “MSSQL:master”

Once this job runs you will be able to see the backup by referring to step 10 above.

Change Sender Address Data Domain

The default sender address for Data Domain systems is HOSTNAME@domain.com. If you need to change this for any reason the steps are below:

via SSH logged in as sysadmin:
config set admin-email user@domain.com

via Data Domain System Manager:
1. Go to System Settings->General Configuration->System Properties
2. At the top right of that page click More Tasks->Set System Properties
3. Fill out the Admin Email section and click Ok
4. If you send a test email now you should see the sender address come from the new address you entered in the prior steps



Upgrading Data Domain systems integrated with Avamar systems


I received an email from EMC this morning regarding DDOS upgrades that are integrated with Avamar systems and thought I would pass this along.  If you have Avamar integrated with Data Domain please contact EMC support before you try and upgrade Data Domain DDOS or it could result in data loss/data unavailability.

Audience: Level 30 = Customers Article Type: ETA
Last Published: Wed Jul 23 17:12:17 GMT 2014 Validation Status: Final Approved


Article Content

Impact SEVERITY RATING: Critical: Data Loss, Data Unavailability
Issue Upgrading Data Domain systems integrated with Avamar systems to DD OS 5.4 or DD OS 5.5 may result in unavailability or loss of backup data on the Avamar systems if the OS release is not verified as compatible with the applicable integrated Avamar system and the published upgrade path for DD OS and Avamar integrated systems is not followed.
Environment EMC Hardware: Data Domain systems integrated with Avamar systems
EMC Hardware: Data Domain Deduplication Storage Systems
EMC Hardware: Avamar Server 6.0
EMC Hardware: Avamar Server 6.1
EMC Hardware: Avamar Server 7.0
EMC Hardware: Avamar Server 7.1
EMC Software: DD OS 5.2
EMC Software: DD OS 5.4
EMC Software: DD OS 5.5
Cause Different DD OS versions provide support for functions used by Avamar systems that use different interfaces.

When DD OS is upgraded to a version incompatible with the integrated Avamar versions, Avamar cannot access the data stored on the Data Domain system in the required fashion.

Resolution When considering DD OS upgrades, verify that the upgraded DD OS version is compatible with both the current Avamar Server version and the target Avamar Server version.
When considering OS upgrades for Avamar and Data Domain systems that are integrated, follow these steps to enhance compatibility:

  • Determine current and target Avamar Server and DD OS versions and use the diagram below to determine the proper upgrade steps.
  • DO NOT skip any intermediate steps or an incompatibility issue may result that could adversely disrupt the Avamar Server’s operation. Upgrading Avamar clients may also be required at each Avamar Server upgrade.

The correct upgrade path for DD OS and Avamar integrated systems is as follows:

User-added image

Upgrading DD OS or Avamar outside this sequence may result in backup data unavailability.

Refer to the Avamar / Data Domain Integration and Upgrade Manual for more information.

Contact Avamar Support for assistance upgrading the Avamar / Data Domain Integrated Systems.

Notes Severity Disclaimer:
For an explanation of Severity Ratings, refer to EMC Knowledgebase article 185920 EMC Technical Advisory (ETA) Severity Rating.

EMC Technical Advisory Information:
For an explanation of EMC Technical Advisories, refer to the EMC Knowledgebase article 16189 What is the EMC Technical Advisory (ETA) process?

Notes (Employees and Partners) If a customer experiences this issue because they upgraded their Data Domain system to a version incompatible with Avamar, the customer’s District Service Manager (DSM) and Account Manager must be contacted immediately, and a Code Red must be opened before any additional action is taken. Jeff Cameron, John Burton, and Baba Krishnankutty must also be informed.
Article Metadata

Product Data Domain, Data Domain Deduplication Storage Systems, Data Domain Software, DD OS 5.4, DD OS 5.5, DD OS,Avamar Server6,Avamar Server7
Shared Yes
Cloned From Article Number 000185791
Domain Articles Resolution Path
Domain Install/Upgrade/Config
Life Cycle GA
Legal Information