Signiant Support

13.3 Object Mover User's Guide Print


Object Mover Introduction

Object Mover allows you to easily manage transfers into and out of your on-premise object storage. Object Mover works seamlessly with your on-premise object storage and gives you flexibility in how you store, manage, and transfer content.

Object Mover is comprised of the following applications:

  • ObjectDropBox
  • ObjectUploader
  • ObjectDownloader
  • ObjectReplicator

 

ObjectDropBox

ObjectDropBox transfers data between a single source location and one or more on-premise object storage locations. It polls the specified location at a user-determined interval and sends any changes detected in the source location to the specified destination locations.

By default, the file readiness checks ensure that any files which are in use or busy (changed) during the transfer are skipped. Optionally, signed hashes are created by both the source and target agents. These signed hashes are placed (with a copy of the source and target agent's certificates) in a certified delivery log. This log is stored on the Signiant Manager filesystem and is viewable and accessible from the Signiant Manager UI. This option is valuable when you need to verify that a file was transferred and normal logging operations are insufficient.

ObjectUploader

ObjectUploader retrieves files from multiple agents to a single target agent, and stores the files on the designated on-premise object storage server. The files transferred from the source agents may be stored on a per agent basis, or in a single flat folder on the target storage server. This is referred to as a pull file transfer template.

ObjectDownloader

ObjectDownloader retrieves files from your on-premise object storage servers and transfers these files to your local or network attached storage.These transfers can be scheduled, encrypted and managed from within the Signiant Manager.

ObjectReplicator

ObjectReplicator is a bulk file transfer solution that allows you to schedule a distribution from a source on-premise object storage server to one or more on-premise object storage servers (with an option to synchronize the target with the source). As it is designed for dealing with large numbers of files (i.e., thousands or millions) it does not have the file readiness checks that are present with ObjectDropbox and ObjectDownloader. Thus, it should be used when files are not growing in size in the source directory.

Configuring and Managing Object Mover

Platform Support

Object Mover is supported on the following platforms:

  • Mac OSX 10.10 and higher
  • RedHat Linux 7, CentOS 7 and higher

 

When using Object Mover on RedHat 7/CentOS 7 agents, the agent must use binaries built for RedHat 7. See the Agent Installation User's Guide, Chapter 3: Signiant Agent Upgrade on RedHat 7 and CentOS 7.

Configure Object Mover Jobs

To create a job using one of the Object Mover solutions, do the following:

 

  1. From the Manager, expand the Object Mover menu and select one of the following:
    • ObjectDropBox - transfers data between a single source location and one or more on-premise object storage locations.
    • ObjectUploader - retrieves files from multiple agents to a single target agent, and stores the files on the designated on-premise object storage server.
    • ObjectDownloader - retrieves files from your on-premise object storage servers and transfers these files to your local or network attached storage.
    • ObjectReplicator -a bulk file transfer solution that allows you to schedule a distribution from a source on-premise object storage server to one or more target agents
  2. Click Add and enter the required information.

 

The following table describes the job prompts associated with the Object Mover solutions. Not all prompts are valid for all solutions. Mandatory prompts are indicated on the job form with an asterisk.

Prompt Description

Job Name

A unique name for this job. This must start with a letter and can only contain underscores to separate words. This name should help operators to recognize the functions of this job (e.g., UploadTransfers).

Drop Box Source / Source Specification

Drop Box / Source Agent

The agent(s) from which you are transferring the data.

Drop Box Directory / Source Data

The source folder for files to transfer. Type the folder in the prompt, or click the computer icon to browse and select an agent folder. The source folder specified may be in the following format:

  • UNIX root drives, e.g., /home

Leaving the prompt empty will use the agent's default folder.

Source Object Storage

To configure your Source Object Storage, configure the following fields:  Storage Server, Bucket, SubFolder, Bucket Access Style, Access Key, and Secret Key. See the prompt descriptions in Destination Specification below.

Include File Patterns

Allows users to specify which files are transferred by filtering on the names to include. Use a comma to separate multiple filters. By default, an asterisk appears in the prompt, to include all file names/types. For example, if you type *.doc, *.ppt, in the prompt, the transfer wills include files with these extensions.

Exclude File Patterns

Allows users to specify which files are excluded from transfer. Use a comma to separate multiple filters. If the source directory for this jobs is also being used as the target directory for another job, you should add #work_file#* and #chkpt_file#*.

Exclude Subdirectories

To exclude sub-folders, choose from the following options:

  • None (all sub-folders will be transferred - none will be excluded)
  • All (no sub-folders will be transferred - all of them will be excluded)
  • Specified (a list of sub-folders to exclude - type sub-folders in the Subdirectories to Exclude prompt, described below)

Subdirectories to Exclude

Sub-folders may be excluded by specifying them in this prompt. Multiple entries must be separated with a comma. When the job runs, all folders that match those specified in the Subdirectories to exclude prompt will be excluded. Note that normal behavior is to exclude sub-folders that match regardless of where they appear in the folder path. Using the anchoring expression (@) changes this behavior to anchor the exclude folder path.

For example, if you specify a source folder of C:\data\docs, and an exclude folder of temp, any sub-folders called temp will be excluded, even those nested within another sub-folder. (For example the sub-folder C:\data\docs\publish\release\temp would be excluded as well.)

If you who want to exclude a folder only at a certain level can use the @ symbol to anchor the exclude folder path at the starting source folder level. For example, specifying @temp in the above example would mean that the C:\data\docs\temp folder would be excluded, but the C:\data\docs\publish\release\temp folder would be included.

If you instead wanted to make sure just the C:\data\docs\publish\release\temp folder was excluded, but wanted to have a source folder path of c:\data\docs, the user would need to type C:\data\docs in the source folder prompt and @publish\release\temp in the exclude prompt. Special characters allow you to make use of pattern matching on the folder path. You must escape special characters with a backslash to be matched literally. Characters include the following:

 

  • * (matches zero or more characters)
  • ? (matches any single character)
  • [...] (matches any one of the enclosed characters - for example, [ch] would match the characters "c" or "h")
  • A pair of digit, lowercase or uppercase characters separated by a hyphen '-' denotes a range set and matches any character sorted in the range. If the first character following the '[' is '^' or '!', then any character not enclosed is matched.
  • Use commas to specify multiple distinct patterns.

 

File Readiness Check Type

Specify the kind of check type an agent uses to determine if a file is ready to be transferred (i.e., the file is not in use).

For non-Windows agents of any kind (Linux, UNIX, MacOS). If the local or network file system supports file locking, the readiness check is not required. Determine this on a case-by-case basis.

Choose one of the following:

  • None - Files that are in use are skipped by default.
  • Stream Growing Files - This transfers in-progress files before the file is complete. The file is constantly monitored and any data is transferred when it is discovered. This is useful when you need to support the delivery of large files. The Readiness Check Delay value is used to specify the amount of time in seconds that must pass without any file changes for the file to be declared complete.
  • Consecutive Seek (recommended) – Reads the last few bytes at the end of the file, waits for delay period number of seconds and checks again. If the last few bytes match in both cases the file is sent. Very useful on large files where the MD5 option would simply take too long. 
  • MD5 Hash - Use an MD5 hash to compute a hash on all files under the drop-box folder. After the wait time specified in the Readiness Check Delay prompt, the hashes are recomputed. Only files with identical hashes are transferred. Note that job initialization may take a great deal of time if large files are present.
  • Date/Time & Size - Use a difference in date/time or size on the file to compare its readiness. Only files that have identical date/time & size are transferred.

Readiness Check Delay

The amount of time (in seconds) to pause between the two phases of the specified file readiness check type ("MD5 Hash", "Consecutive Seek", or "Date/Time & Size") for an open file. The file is transferred only if the results of the two checks are identical. The default value is 10 seconds. If a large number of files are being checked, a value of 0 seconds may be sufficient, since there will likely be sufficient time between the first and second checks of a given file to determine if its content has changed without inserting an additional delay.

When File Readiness Check Type is set to Stream Growing Files, this is the time in seconds that must pass without any file changes for the file to be declared complete.

Source Deletions After Successful Transfer

Specifies removal of files or folders from the dropbox folder after a successful transfer: Choose from one of the following:

  • None (do not delete any source files or empty folders)
  • Transferred Files (remove the files)
  • Empty folders (remove folders that are left over after the source files have been removed)
  • Transferred Files and Empty Directories (remove both transferred files and empty folders)
  • Transferred Files and Directories ( remove both transferred files and all folders immediately after transfer)

There are certain requirements with the empty folders options: the folders must contain no files (they may contain other empty folders that will be removed as well). The folder cannot have been modified within the previous 5 minutes; if it has, it will not be removed.

Source Directories to Exclude from Deletion Scan

Allows you to specify a comma-separated list of directories that you do not want to delete.

Skip Source File Not Found On Send

Allows you to skip files that are not found during a transfer. The default value is Yes.

Destination Specification

Target Agent(s)

The agent(s) to which you are transferring the files.

Target Object Storage

Select a Storage Profile and subfolder to use for this Object Mover job.

You can configure Storage Profiles on the Administration>Storage Profiles menu.

Transport Options
Simultaneous Transfer Streams Use the drop-down list to specify the number of active simultaneous transfer streams. The default value is 10.

Use UDP Control Channel

Allows you to specify that job control information will be communicated using UDP instead of TCP. This includes both the communication between the Manager and the controlling agent, and the communication between the controlling agent and the slave agent(s). By default Use UDP Channel is set to No.

This setting applies to all Manager/agent job control communication across the entire job. UDP versus TCP communication for agent administration via the GUI is controlled separately as part of agent configuration.

When Use UDP Control Channel is set to Yes, all file transfers are also forced to utilize UDP WAN acceleration, meaning that the Use WAN Accelerator setting is overridden and it is not possible to have job control using TCP and data transfer using UDP.

This functionality is intended to be used for firewall traversal when the TCP port cannot be opened. If this is not a requirement, then it is recommended that Use UDP Control Channel be set to No.

Use WAN Accelerator

Selecting Yes means that when the job runs, it will use UDP as the underlying transport, and will attempt to use all available bandwidth up to the user-specified bandwidth maximum. This configuration can result in a faster transfer rate than if it is not selected. The Use WAN Accelerator option should be used on high latency high bandwidth networks where throughput is a top priority.

When running a job with Use WAN Accelerator set to Yes, with a low bandwidth ceiling, or flow, or low throughput, the transfer will be switched to TCP. (The cutoff value is 57200 bits/second.)

Note that bandwidth throttles may also be employed by other network devices and policies (e.g., QoS), therefore, a bandwidth throttle (or target maximum) defined here may not be achievable. If you are having difficulty achieving a particular bandwidth target ensure that other policies are not impacting your ability to reach the desired throughput.

WAN Accelerator Aggressiveness

Indicates how sensitive the job will be to other network traffic when it is running. Choose one of the following:

  • High: The agent will always attempt to send data at the Bandwidth Ceiling rate and not share with other network traffic.

  • Medium: The agent will always attempt to send data at the Bandwidth Ceiling rate, however, if other traffic is detected, the agent will share the network resources but never fall below the bandwidth floor (see below).

  • Low: The agent will always attempt to send data at the Bandwidth Ceiling rate, however, if other traffic is detected, the agent will drop its transfer rate to the bandwidth floor (see below) right away, and slowly attempt to creep back up to the target throughput rate.

Bandwidth Ceiling

The maximum rate at which the source will send the data on the network. Typically this is the maximum speed of the network (i.e., 100Mbps, 10Mbps, etc.). 0 specifies auto-detection for the best bandwidth rate. Testing has shown that specifying a ceiling is faster though - as long as it is set correctly. This is because if the ceiling is set, the transfer immediately starts at that rate, rather than ramping up.

Bandwidth Floor

The minimum rate at which the data should be sent. If WAN Accelerator Aggressiveness is set to Medium or Low, the Bandwidth Floor value specified here will be used. 0 specifies auto-detection for the best bandwidth rate.

Bandwidth Throttle by Time of Day

Allows you to further specify the amount of bandwidth the job uses based on the time of day, with a start and end time, days of the week and the limit (specified in bytes per second to a maximum of whatever the CPU can handle, or a percentage of a selected network connection). Values are set using GMT time.

For example, users could specify that the job uses 50% of a 56 Kbps link between 9:00 and 5:00 Monday through Friday, 75% between 5:00 pm and 6:00 am, and 100% on the weekend. Click the plus icon ("+") to add more bandwidth throttles, or the "x" to delete a throttle.

Note that once a job has started, all bandwidth throttles are applied at the times based on the Daylight Savings Time (DST) in effect when the job started. If DST changes while the job is running, bandwidth time of day changes may be off by the time change value (plus or minus an hour) after the time change.

Note that bandwidth throttles may also be employed by other network devices and policies (e.g., QoS), therefore, a bandwidth throttle (or target maximum) defined here may not be achievable. If you are having difficulty achieving a particular bandwidth target ensure that other policies are not impacting your ability to reach the desired throughput.

Advanced Options

Encryption Level

The encryption level selected will dictate how securely the information is transmitted over the network (note: the file/package is always stored in the original format).  Encryption is isolated to the 'on the wire' portion of the transfer only and does not affect what is stored on the file system whatsoever.

Allows specification of the encryption level from the following values:

  • No Encryption - signed
  • No Encryption – unsigned
  • Encryption On

 

The default value is No Encryption – unsigned . Note that mutual authentication is always used, regardless of the encryption level specified. Encryption is done n-stream and not on disk prior to sending.

The No encryption – signed option transfers unencrypted (plain text) data, but includes the SSL protocol's message digest calculation and signing to ensure data stream integrity.

The No encryption – unsigned option allows the transfer of data, after the initial SSL authentication of the endpoints, to proceed with no encryption, no message digest computation and no signing of the message digests (normally included with the SSL protocol). This mode of operation is only for raw performance, since it makes no guarantee of the integrity of the data stream other than the default one has with a normal network channel. The underlying TCP protocol can guarantee the integrity of messages across each single network hop, but has no facilities for detecting a man-in-the-middle attack.

The Encryption On option uses the following maximum or high encryption level: Cipher=AES256-SHA, Keysize=256/256.

Delivery Mode

Specifies the mode to transfer files:

  • Fast - Do not create temporary work files. This mode is optimized for speed.
  • Normal - Use temporary work files during transfer. Temporary files allow an interrupted transfer to resume from where the interruption occurred. The creation of temporary files adds time to the transfer, but provides resiliency.
  • Log File Name - Log file names transferred. Logs are stored on the Manager.
  • Certify File Delivery - Create agent certified delivery log of files transferred.
Notification and Logging

Email Condition

The circumstances in which an e-mail is sent. Choose from the following:

  • Always – An e-mail is always sent when a job runs
  • Always Exclude Empty Transfers - An e-mail is sent only for complete transfers.
  • On Transfer – An e-mail is sent only when the transfer occurs
  • Never – No e-mail is sent.

Email To

The email address of the person to receive the logging report. Separate multiple addresses with a comma.

Email CC

Specifies the email address(es) to receive a carbon copy email notification message when the job runs. Separate multiple addresses with a comma.

Email Bcc

Specifies the email address(es) to receive a blind carbon copy email notification message when the job runs. Separate multiple addresses with a comma.

Email Job Failure Report To

The email address of the person to receive the failure report. Separate multiple addresses with a comma.

Log Detail Level

The type of logging information for this job. Choose from Error, Warn (warning), Info (Information) or Debug. Debug provides the greatest level of detail while Error provides the least.

Schedule

Frequency

Specifies how often the job runs. Users can choose from the following options:

  • None - Use for jobs that run only at irregular, user-defined times.
  • Hourly, Daily, Weekly, Monthly, Yearly - Run the job once every selected interval.
  • Monthend - Run the job once every last day of the month.
  • Once - Run the job only once, when first created.

Note that editing the frequency of a job that is currently in a queued state within a queued job group will have no effect on the job until the next time the job is run.

Start Date/Time

The date and time at which you want the job to run.

Note that editing the start date/time of a job that is currently in a queued state within a queued job group will have no effect on the job until the next time the job is run.

Time Zone

Specifies the time zone in which the displayed times are set. For example, if an Eastern Time zone specifies a start time of 9:00 am and a time zone of PST, the job runs at 9:00 am Pacific Standard Time or noon Eastern Standard Time.

Priority

Specifies the priority of the job.

When used in conjunction with a Resource Control, the priority determines the placement of the job within the resource queue. A job with a higher priority is placed before a job with a lower priority.

The User can choose from the following priorities:

  • Low
  • Medium
  • High
  • Urgent
  • Immediate

The default value is Medium.

Finish Before

The date and time at which you want the job to be completed.

When used in conjunction with a Resource Control, this attribute determines the placement of the job within the resource queue. A job with an earlier Finish Before time is placed before a job with a later Finish Before time.

Interrupt on Failure

When a component in the workflow fails, the workflow will halt allowing for operator intervention. If the workflow is interrupted and an alarm condition of any kind is defined that applies to that particular job, an alarm is displayed on the Alarms page indicating that the workflow is in an Interrupted state.

Configure Agent Groups

To allow Object Storage uploads and downloads, you must edit the appropriate Agent group(s) and turn on this permission.

To enable Object Storage uploads and downloads, do the following:

  1. In the Manager, select Agents > Groups and select the appropriate Agent.
  2. Click Edit.
  3. On the Object Storage tab, select Object Storage Uploads/Downloads Enabled.
  4. Click OK.

Managing Object Mover

To view the number of object storage-enabled agents you have in your system, run Health Check and open the Agents report and/or the Main report. In the Main report, on-premise object store-enabled agent information is in the Manager Ecosystem table.