Signiant Support

Exclude Directories in a Synchronizing Transfer Print


Summary

When using a synchronizing transfer, there is a field named "Directories to Sync" that can be used to indicate which directories should be synchronized. Unfortunately, the value for this field must be hard coded as part of the package. That is, it is impossible to programatically specify the value for this field.

As such, creating a package that will synchronize a directory tree except for specific sub directories is a bit more difficult than a standard package. In order to accomplish this, you can use the Filename Mapping Command. The Filename Mapping Command accepts a stream of file names to its standard input. Each line in this stream represents a file to send to the target hosts. This command is typically used to change the file name or location on the target host(s) by sending a new file name to standard output. When the Filename Mapping Command generates the output "*** skip ***" in place of a file name, the agent is then instructed not to send this file. This behaviour can be exploited to skip directories during a synchronizing transfer. An example for UNIX systems is shown below.

Discussion

There are some limitations you should be aware of before using this approach:
  • The source agent must still generate a list of files in the directories the transfer is excluding. When the directories to exclude contain a large number of items, a considerable amount of time could be spent scanning these directories and evaluating each item. While processing these directories, no data will be sent to the target host(s).
  • Excluded directories, and files in these directories, will not be deleted from the target host(s) if they exist on the source. This is because both the source and target hosts compare their generated file lists to determine which files to delete. As the file is generated on both sides of the transfer, it will not be deleted on the target host(s). If you wish these directories and files to be deleted, you must do so in another manner (such as manually, using a Remote Command package, or using another process).

Additional Information

The following is a sample script that can be used as the Source Filename Mapping Command to exclude specific directories from a synchronizing transfer. This is intended for UNIX and Linux hosts, and uses simple Bourne shell commands. The source host must have a copy of 'cut' and 'grep' in its path, and this version of 'grep' must be able to handle regular expressions.

This script makes use of a data transfer variable %DirsToExclude%. This can either be set aspart of the package, or prompted at schedule time.

For this particular example, %DirsToExclude% should contain a space-separated list of directories to exclude. This example does not handle a case where a directory name contains a space, but can be enhanced to accomodate that situation if desired.

#!/bin/sh
EXCLUDES='%DirsToExclude%'
# Input lines to the target mapping command consist of four tokens:
# Filename, enclosed in quotes ("),
# Name of the source node,
# File size, in bytes
# String (file or directory) indicating the file type.
while read line
do
    # Get the filename
    FILENAME=`echo $line|cut -f 2 -d '"'`
    # Reset skip flag
    SKIP=0
    # Check the file for each directory named in EXCLUDES
    if [ '$EXCLUDES' ]; then
        for DIR in $EXCLUDES; do
            RC=`echo $FILENAME | grep ^${DIR}`
            if [ $RC ]; then
                # This file matches an excluded directory
                # set skip flag
                SKIP=1
             fi
        done
    fi
    if [ $SKIP -eq 1 ]; then
        # tell the agent to skip this file
        echo "*** skip ***";
     else
        # tell the agent to send the file
        echo $FILENAME
    fi
done