Signiant Support

13.3 Media Mover User's Guide Print


Media Mover Introduction

 

Media Mover provides accelerated file movement based on some standard automated templates that allow Signiant users to automate the transfer of files between agents.

media_mover.jpg

Media Mover is comprised of the following applications:

  • MediaDropBox
  • MediaAggregator
  • MediaDistributor
  • MediaReplicator

 

MediaDropBox

MediaDropBox transfers data between a single source location and one or more destination locations. It polls the specified location at a user-determined interval and sends any changes detected in the source location to the specified destination locations.

By default, the file readiness checks ensure that any files which are in use or busy (changed) during the transfer are skipped. Optionally, signed hashes are created by both the source and target agents. These signed hashes are placed (with a copy of the source and target agent's certificates) in a certified delivery log. This log is stored on the manager's filesystem and is viewable and accessible from the Manager UI. This option is valuable when you need to 'prove' that a file was transferred and normal logging operations are insufficient.

MediaAggregator

MediaAggregator retrieves files from multiple agents to a single target agent, and stores the files on its local disk or locally attached storage. The files transferred from the source agents may be stored on a per agent basis, or in a single flat folder on the target agent storage (either local disk or NAS). This is referred to as a pull file transfer template.

MediaDistributor

MediaDistributor allows you to schedule a simple push distribution in which files are transferred from one source machine to one or more target machines.

MediaReplicator

MediaReplicator is a bulk file transfer solution that allows you to schedule a distribution from a source agent to one or more target agents (with an option to synchronize the target with the source). As it is designed for dealing with large numbers of files (i.e., thousands or millions) it does not have the file readiness checks that are present with MediaDropbox and MediaDistributor. Thus, it should be used when files are not growing in size in the source directory.

Media Mover Prompts

To create a job using one of the Media Mover solutions, do the following:

 

  1. From the Manager, expand the Media Mover menu and select one of the following:
    • MediaDropBox - recommended for transferring changing files from a single source directory on one agent to one or more target agents.
    • MediaAggregator - recommended for retrieving changing files from one or more source agents to a single target agent.
    • MediaDistributor - recommended for transferring multiple directories from a single source agent to one or more target agents.
    • MediaReplicator - recommended for transferring large numbers of static files from a single source agent to one or more target agents.
  2. Click Add and enter the required information into the prompts.

 

The following table describes the job prompts associated with the Media Mover solutions. Not all prompts are valid for all solutions. Mandatory prompts are indicated on the job form by an asterisk.

Prompt Description
Job Name

A unique name for this job. This must start with a letter and can only contain underscores to separate words. This name should help operators to recognize the functions of this job (e.g., Send_Media_to_Site2).

Drop Box / Source Specification
Source Agent/Drop Box Agent

The agent(s) from which you are transferring the data.

Source Data or Source Directory or Drop Box Directory

The source folder for files to transfer. Type the folder in the prompt, or click the computer icon to browse and select an agent folder. The source folder specified may be in the following format:

  • Windows root drives, e.g., C:\ApplicationData
  • UNIX root drives, e.g., /home
  • Windows UNC path names, e.g., \\machine1\applicationdata

Leaving the Source Data prompt empty will use the agent's default folder.

Include File Patterns

Allows users to specify which files are transferred by filtering on the names to include. Use a comma to separate multiple filters. By default, an asterisk appears in the prompt, to include all file names/types. For example, if you type *.doc, *.ppt, in the prompt, the transfer wills include files with these extensions.

Exclude File Patterns

Allows users to specify which files are excluded from transfer. Use a comma to separate multiple filters. If the source directory for this jobs is also being used as the target directory for another job, you should add #work_file#* and #chkpt_file#*.

Exclude Subdirectories

To exclude sub-folders, choose from the following options:

  • None (all sub-folders will be transferred - none will be excluded)
  • All (no sub-folders will be transferred - all of them will be excluded)
  • Specified (a list of sub-folders to exclude - type sub-folders in the Subdirectories to Exclude prompt, described below)
Subdirectories to Exclude

Sub-folders may be excluded by specifying them in this prompt. Multiple entries must be separated with a comma. When the job runs, all folders that match those specified in the Subdirectories to Exclude prompt is excluded. Note that normal behavior is to exclude sub-folders that match regardless of where they appear in the folder path. Using the anchoring expression (@) changes this behavior to anchor the exclude folder path.

For example, if you specify a source folder of C:\data\docs, and an exclude folder of temp, any sub-folders called temp will be excluded, even those nested within another sub-folder. (For example the sub-folder C:\data\docs\publish\release\temp would be excluded as well.)

If you who want to exclude a folder only at a certain level can use the @ symbol to anchor the exclude folder path at the starting source folder level. For example, specifying @temp in the above example would mean that the C:\data\docs\temp folder is excluded, but the C:\data\docs\publish\release\temp folder is included.

If you instead wanted to make sure just the C:\data\docs\publish\release\temp folder is excluded, but wanted to have a source folder path of c:\data\docs,you need to type C:\data\docs in the source folder prompt and @publish\release\temp in the exclude prompt. Special characters allow you to make use of pattern matching on the folder path. You must escape special characters with a backslash to be matched literally. Characters include the following:

 

  • * (matches zero or more characters)
  • ? (matches any single character)
  • [...] (matches any one of the enclosed characters - for example, [ch] would match the characters "c" or "h")
  • A pair of digit, lowercase or uppercase characters separated by a hyphen '-' denotes a range set and matches any character sorted in the range. If the first character following the '[' is '^' or '!', then any character not enclosed is matched.
  • Use commas to specify multiple distinct patterns.

 

File Readiness Check Type

Specify the kind of check type an agent uses to determine if a file is ready to be transferred (i.e., the file is not in use). For some configurations, you may or may not wish to select the file readiness check option:

  1. For Windows agents and local Windows file systems, there is no need to use the file readiness check, as the core file in use· behavior is sufficient.
  2. For Windows agent and NAS Windows file system, as long as the NAS file system properly supports file locking for in-use files, the readiness check is unnecessary. However, it is still available in case the NAS file system does not properly support file locking.
  3. For non-Windows agents of any kind (Linux, UNIX, MacOS). If the local or network file system supports file locking, the readiness check is not required. Determine this on a case-by-case basis.

Choose one of the following:

  • None - Files that are in use are skipped by default.
  • Stream Growing Files - This transfers in-progress files before the file is complete. The file is constantly monitored and any data is transferred when it is discovered. This is useful when you need to support the delivery of large files. The Readiness Check Delay value is used to specify the amount of time in seconds that must pass without any file changes for the file to be declared complete.
  • Consecutive Seek (recommended) – Reads the last few bytes at the end of the file, waits for delay period number of seconds and checks again. If the last few bytes match in both cases the file is sent. Very useful on large files where the MD5 option would simply take too long. 
  • MD5 Hash - Use an MD5 hash to compute a hash on all files under the drop-box folder. After the wait time specified in the Readiness Check Delay prompt, the hashes are recomputed. Only files with identical hashes are transferred. Note that job initialization may take a great deal of time if large files are present.
  • Date/Time & Size - Use a difference in date/time or size on the file to compare its readiness. Only files that have identical date/time & size are transferred.
Readiness Check Delay

The amount of time (in seconds) to pause between the two phases of the specified file readiness check type ("MD5 Hash", "Consecutive Seek", or "Date/Time & Size") for an open file. The file is transferred only if the results of the two checks are identical. The default value is 10 seconds. If a large number of files are being checked, a value of 0 seconds may be sufficient, since there will likely be sufficient time between the first and second checks of a given file to determine if its content has changed without inserting an additional delay.

When File Readiness Check Type is set to Stream Growing Files, this is the time in seconds that must pass without any file changes for the file to be declared complete.

Health Check Delay

This applies only when File Readiness Check Delay is set to Stream Growing Files. The time in seconds that the transfer waits after the end of the file is detected before transferring the file based on the size specified in Header Rewrite Size. The default value is 10 seconds.

Header Rewrite Size

This applies only when File Readiness Check Delay is set to Stream Growing Files. Specifies the number of bytes to transfer in the header file. The default value is 1 million bytes.

Source Data Action After Successful Transfer

Specifies removal of files or folders from the dropbox folder after a successful transfer: Choose from one of the following:

  • None (do not delete any source files or empty folders)
  • Move data to another location (source files are moved to the location specified in Source Data Location After Successful Transfer. Note: if a location is not specified, the files are not moved.)
  • Delete Transferred Files (delete the files)
  • Delete Empty Directories (delete the directories)
  • Delete Transferred Files and Empty Directories (delete both transferred files and empty directories)
  • Delete Transferred Files and Directories (delete both transferred files and all directories immediately after transfer)

There are certain requirements with the empty folders options: the folders must contain no files (they may contain other empty folders that will be removed as well). The folder cannot have been modified within the previous 5 minutes; if it has, it will not be removed.

Source Directories to Exclude from Deletion Scan

Allows you to specify a comma-separated list of directories that you do not want to delete.

Source Data Location After Successful Transfer

Allows you to specify a location for the files to be moved to after a successful transfer. This must be specified when Source Data Action After Successful Transfer is set to Move data to another location. Note: if this prompt is not configured, then the files are not moved to another location. This prompt is only applicable when Source Data Action After Successful Transfer is set to Move data to another location.

Skip Source File Not Found On Send

Allows you to skip files that are not found during a transfer. The default value is Yes.

Skip Files With 0 Byte Size

Allows you to skip files that have 0 byte size and are not ready for transfer. Such files can be created during the copying of data. When set to Yes, these files are skipped. The default value is No.

Source File Data Allows you to filter files by Date or Time, based on when the file was last modified, accessed, or when the file was created.s
Destination Specification
Target Agent(s)

The agent(s) to which you are transferring the files.

Target Directory

Specifies the folder to store the data from the source agents. This prompt will default to the 'default folder' as configured on the agent. If the folder does not exist it will be created. For example:

  • Windows root drives, e.g., C:\ApplicationData
  • UNIX root drives, e.g., /home
  • Windows UNC path names, e.g., \\machine1\applicationdata
Directory Mapping Options

Which, if any, items are added to the target directory path during the transfer (for example, Source Host or Job Date/Time, and so on).

If neither item is checked, all files will be put into a single parent folder. Files can be separated in directories named by source agent or date/time or both by selecting the appropriate control.

Flatten Directory Structure

Flattening your directory structure will take all files from a transfer and write them to the destination directory without using the folder structure from the origin.

Maximum Job Date-Time Directories

If the Directory Mapping Option "Job Date/Time" has been selected, the Maximum Job Date-Time Directories setting will determine the maximum number of job date/time folders to be maintained on the target system. For example, with the default value of 10, once more than 10 job date/time folders are created in the target directory, the oldest job date/time directory will be deleted so that a maximum of 10 will exist at any given time. Selectable values range from 1 through 1000 plus a value of "Unlimited", which instructs the job to never prune the job date/time directories.

Remove Expired Files on Target

Specifies whether files on the destination are deleted when they reach the end of a user-specified expiration period (see Expiration Period (Days), below). Once the files are transferred to the target location, the directory is examined for files that have a modified time greater than the expiry time specified. Files that match the criteria are removed. The expiry time is specified in days. The removal of the expired files is done at Target Initialization.

Remove Expired Directories on Target

Specifies whether directories on the destination are deleted when they reach the end of the user-specified expiration period (see Expiration Period below). Once the files are transferred to the target location, the directory is examined for a modified time greater than the expiry time specified. Directories that match the criteria are removed. The expiry time is specified in days. The removal of the expired folders is done at Target Initialization.

Expiration Period

Specifies the number of days files can exist on the destination before they are removed, if Remove Expired Files on Destination (described above) is set to Yes.

Transport Options
Simultaneous Transfer Streams

Set the number of consecutive transfer streams to use for file transfers.

Use UDP Control Channel

Allows you to specify that job control information will be communicated using UDP instead of TCP. This includes both the communication between the Manager and the controlling agent, and the communication between the controlling agent and the slave agent(s). By default "Use UDP Channel" is set to No.

This setting applies to all Manager/agent job control communication across the entire job. UDP versus TCP communication for agent administration via the GUI is controlled separately as part of agent configuration.

When "Use UDP Control Channel" is set to "Yes", all file transfers are also forced to utilize UDP WAN acceleration. That is, the "Use WAN Accelerator" setting is overridden. Thus it is not possible to have job control using TCP and data transfer using UDP.

This functionality is intended to be used for firewall traversal when the TCP port cannot be opened. If this is not a requirement, then it is recommended that "Use UDP Control Channel" be set to "No".

Use WAN Accelerator

Selecting Yes means that when the job runs, it will use UDP as the underlying transport, and will attempt to use all available bandwidth up to the user-specified bandwidth maximum. This configuration can result in a faster transfer rate than if it is not selected. The Use WAN Accelerator option should be used on high latency high bandwidth networks where throughput is a top priority.

When running a job with "Use WAN Accelerator" set to "Yes", with a low bandwidth ceiling, or flow, or low throughput, the transfer will be switched to TCP. (The cutoff value is 57200 bits/second.)

Note that bandwidth throttles may also be employed by other network devices and policies (e.g., QoS), therefore, a bandwidth throttle (or target maximum) defined here may not be achievable. If you are having difficulty achieving a particular bandwidth target ensure that other policies are not impacting your ability to reach the desired throughput.

WAN Accelerator Aggressiveness

Indicates how sensitive the job will be to other network traffic when it is running. Choose one of the following:

  • High: The agent will always attempt to send data at the Bandwidth Ceiling rate and not share with other network traffic.

  • Medium: The agent will always attempt to send data at the Bandwidth Ceiling rate, however, if other traffic is detected, the agent will share the network resources but never fall below the bandwidth floor (see below).

  • Low: The agent will always attempt to send data at the Bandwidth Ceiling rate, however, if other traffic is detected, the agent will drop its transfer rate to the bandwidth floor (see below) right away, and slowly attempt to creep back up to the target throughput rate.

Bandwidth Ceiling

The maximum rate at which the source will send the data on the network. Typically this is the maximum speed of the network (i.e., 100Mbps, 10Mbps, etc.). "0" specifies auto-detection for the best bandwidth rate. Testing has shown that specifying a ceiling is faster though - as long as it is set correctly. This is because if the ceiling is set, the transfer immediately starts at that rate, rather than ramping up.

Bandwidth Floor

The minimum rate at which the data should be sent. If WAN Accelerator Aggressiveness is set to Medium or Low, the Bandwidth Floor value specified here will be used. "0" specifies auto-detection for the best bandwidth rate.

Bandwidth Throttle by Time of Day

Allows you to further specify the amount of bandwidth the job uses based on the time of day, with a start and end time, days of the week and the limit (specified in bytes per second to a maximum of whatever the CPU can handle, or a percentage of a selected network connection). Values are set using GMT time.

For example, users could specify that the job uses 50% of a 56 Kbps link between 9:00 and 5:00 Monday through Friday, 75% between 5:00 pm and 6:00 am, and 100% on the weekend. Click the plus icon ("+") to add more bandwidth throttles, or the "x" to delete a throttle.

Note that once a job has started, all bandwidth throttles are applied at the times based on the Daylight Savings Time (DST) in effect when the job started. If DST changes while the job is running, bandwidth time of day changes may be off by the time change value (plus or minus an hour) after the time change.

Note that bandwidth throttles may also be employed by other network devices and policies (e.g., QoS), therefore, a bandwidth throttle (or target maximum) defined here may not be achievable. If you are having difficulty achieving a particular bandwidth target ensure that other policies are not impacting your ability to reach the desired throughput.

Advanced Options
Encryption Level

The encryption level selected will dictate how securely the information is transmitted over the network (note: the file/package is always stored in the original format).  Encryption is isolated to the 'on the wire' portion of the transfer only and does not affect what is stored on the file system whatsoever.

Allows specification of the encryption level from the following values:

  • No Encryption - signed
  • No Encryption – unsigned
  • Encryption On

 

The default value is No Encryption – unsigned . Note that mutual authentication is always used, regardless of the encryption level specified. Encryption is done "in-stream" and not on disk prior to sending.

The No encryption – signed option transfers unencrypted (plain text) data, but includes the SSL protocol's message digest calculation and signing to ensure data stream integrity.

The No encryption – unsigned option allows the transfer of data, after the initial SSL authentication of the endpoints, to proceed with no encryption, no message digest computation and no signing of the message digests (normally included with the SSL protocol). This mode of operation is only for raw performance, since it makes no guarantee of the integrity of the data stream other than the default one has with a normal network channel. The underlying TCP protocol can guarantee the integrity of messages across each single network hop, but has no facilities for detecting a man-in-the-middle attack.

The Encryption On option uses the following maximum or high encryption level: Cipher=AES256-SHA, Keysize=256/256.

Delivery Mode

Specifies the mode to transfer files:

  • Fast - Do not create temporary work files. This mode is optimized for speed.
  • Normal - Use temporary work files during transfer. Temporary files allow an interrupted transfer to resume from where the interruption occurred. The creation of temporary files adds time to the transfer, but provides resiliency.
  • Log File Name - Log file names transferred. Logs are stored on the Manager.
  • Certify File Delivery - Create agent certified delivery log of files transferred.
Compress Files

If set to Yes, the source agent compresses each file before sending it. The files will be uncompressed automatically on the target agent(s). The degree of compression depends on the type of data you are transferring. The following are typical rates of compression for different types of data:

  • Plain text: 70-95%
  • TIFF images: 20-40%
  • Binary files: 0-5%

The default value is No. Choose this option if you are running over a low speed WAN link (e.g., less than 3 Mbps). This option is NOT recommended for LAN or high speed links.

Follow Symbolic Links

Specifies whether the end target of a symbolic link or the symbolic link itself is transferred to the target. If a symbolic link references a directory and Follow Symbolic Links is enabled, a directory is transferred; otherwise, if Follow Symbolic Links is disabled, the symbolic link itself is transferred. Select Yes to transfer the files or directory contents to which the symbolic links point. Select No to transfer only the symbolic links themselves. This field has a default value of No.

Windows versions prior to Windows 2008 do not support symbolic links, and these agent treat the arrival of a symbolic link at a Windows target as an error, causing the transfer to fail, unless you set the "follow symbolic links" option to Yes. When set to Yes, the file to which the symbolic link points is transferred under the name of the symbolic link.

Incremental Transfer

If set to Yes, only changed bytes of files will be transferred, not the entire file. Typically used in low bandwidth situations. Choose this if you are running over a low speed WAN link.

Synchronize Target with Source

The Synchronize option enables the agent “synchronizing transfer” mode which involves a deletion phase on the target agent. If enabled, all content in the target folder matching the source file/folder inclusion criteria, but that did not exist in the source folder, is deleted. For example, if source inclusions is simply “*” (or blank, which equates to “*”) then all content in the target folder that did not exist in the source will be deleted following the transfer phase. However, if source inclusions is “*.txt” then only the .txt files in the target folder that did not exist in the source folder will be deleted following the transfer.

Be very careful when enabling the synchronize option due to the fact that it deletes content at the target end. This deleted data is not recoverable via Signiant.

Enabling this option is incompatible with the Follow Symbolic Links option.

Verify for sufficient disk space before Job transfers start

Select Yes to check the remote disk space on the target agent before the Job transfer starts. This ensures that the transfer is not aborted due to insufficient disk space.

Verify if target directory exists

Select Yes to verify that the target directory exists on the target agent before the Job transfer starts. This ensures that the transfer is not aborted due to a nonexistent directory.

Notification and Logging
Email Condition

The circumstances in which an e-mail is sent. Choose from the following:

  • Always – An e-mail is always sent when a job runs
  • Always Exclude Empty Transfers - An e-mail is sent only for complete transfers.
  • On Transfer – An e-mail is sent only when the transfer occurs
  • Never – No e-mail is sent.
Email To

The email address of the person to receive the logging report. Separate multiple addresses with a comma.

Email CC

Specifies the email address(es) to receive a carbon copy email notification message when the job runs. Separate multiple addresses with a comma.

Email Bcc

Specifies the email address(es) to receive a blind carbon copy email notification message when the job runs. Separate multiple addresses with a comma.

Email Job Failure Report To

The email address of the person to receive the failure report. Separate multiple addresses with a comma.

Number of consecutive failures

Set the number of consecutive transfer failures required before a warning is sent to the notification recipients.

Log Detail Level

The type of logging information for this job. Choose from Error, Warn (warning), Info (Information) or Debug. Debug provides the greatest level of detail while Error provides the least.

SNMP
SNMP Trap Receivers

The list of hostnames and/or IP addresses of systems which are listening for SNMP traps (normally central network management consoles).

SNMP Trap Community String

The SNMP community string (i.e. password) to be sent in the trap.

SNMP Trap Types

The job completion statuses on which to sent SNMP traps. Choices are:

  • Job Success
  • Job Failure
Schedule
Frequency

Specifies how often the job runs. Users can choose from the following options:

  • None - Use for jobs that run only at irregular, user-defined times.
  • Hourly, Daily, Weekly, Monthly, Yearly - Run the job once every selected interval.
  • Monthend - Run the job once every last day of the month.
  • Once - Run the job only once, when first created.

Note that editing the frequency of a job that is currently in a queued state within a queued job group will have no effect on the job until the next time the job is run.

Start Date/Time

The date and time at which you want the job to run.

Note that editing the start date/time of a job that is currently in a queued state within a queued job group will have no effect on the job until the next time the job is run.

Time Zone

Specifies the time zone in which the displayed times are set. For example, if an Eastern Time zone specifies a start time of 9:00 am and a time zone of PST, the job runs at 9:00 am Pacific Standard Time or noon Eastern Standard Time.

Priority

Specifies the priority of the job.

When used in conjunction with a Resource Control, the priority determines the placement of the job within the resource queue. A job with a higher priority is placed before a job with a lower priority.

The User can choose from the following priorities:

  • Low
  • Medium
  • High
  • Urgent
  • Immediate

The default value is Medium.

Finish Before

The date and time at which you want the job to be completed.

When used in conjunction with a Resource Control, this attribute determines the placement of the job within the resource queue. A job with an earlier Finish Before time is placed before a job with a later Finish Before time.

Interrupt on Failure

When a component in the workflow fails, the workflow will halt allowing for operator intervention. If the workflow is interrupted and an alarm condition of any kind is defined that applies to that particular job, an alarm is displayed on the Alarms page indicating that the workflow is in an Interrupted state.