Signiant Support

13.1 Flight Workflow User's Guide Print



Flight Introduction

Signiant Flight is a file transfer service that accelerates the movement of large files into and out of cloud storage. This document describes the implementation of Signiant Flight within Signiant Managers and Agents. The Signiant Flight implementation for Managers and Agents uses the Flight_Reference_Workflows job template. This template provides two workflows: Flight_Upload and Flight_Download.

Flight Upload

The Flight_Upload workflow is used to create jobs which transfer files and folders from one or more Signiant agents to either Microsoft Azure or Amazon S3. Based on configuration properties, this job uploads the files and folders that you designated for transfer. This transfer takes advantage of Signiant's WAN acceleration technology, maximizing the transfer throughput to your cloud storage solution. In the event that the transfer is interrupted, the transfer resumes automatically at the point of content interruption. For example, if a 10 GB file transfer is interrupted at 8 GB, the transfer resumes at the 8 GB point of interruption.

Flight Download

The Flight_Download workflow is used to create jobs which transfer files and folders from either Amazon S3 or Microsoft Azure to one or more Signiant agents. Based on configuration properties, this job downloads the list of files and folders from your cloud storage solution to the designated Signiant agent(s). This transfer takes advantage of Signiant's WAN acceleration technology, maximizing the transfer throughput to your cloud storage solution. In the event that the transfer is interrupted, the transfer resumes automatically at the point of content interruption. For example, if a 10 GB file transfer is interrupted at 8 GB, the transfer resumes at the 8 GB point of interruption.

Cloud Storage Configuration

Signiant Flight works with Amazon S3 and Microsoft Azure. During the configuration process, you are prompted to enter your cloud storage server credentials and details to ensure secure and efficient transfers.


Using Flight

To use Flight you must do the following:

  1. Download the latest Signiant Flight CLI from the Signiant website and place it on the Signiant agent in the Signiant bin folder. To learn about the Signiant Flight CLI utility, read the Signiant Flight CLI Guide.
  2. Using the Flight workflows, create jobs to upload and/or download content. See Creating Flight Transfer Jobs.

Creating Flight Transfer Jobs

To create Flight transfer jobs using the reference workflows, do the following:

  1. In the Signiant Manager, select Flight >Flight Uploads and click Add.
  2. In Job Name, type a name for the job.
  3. Configure the Flight Uploads job, refer to Flight Uploads Prompts for the prompt details.
  4. Select Flight Downloads and click Add.
  5. In Job Name, type a name for the job.
  6. Configure the Flight Downloads job, refer to Flight Downloads Prompts for the prompt details.

Flight Uploads Prompts

The following table describes the prompts for the Flight Uploads job.

Prompt Description
Source Content

Source Agent

The agent where the files are uploaded from. This can be a single agent or a load-balanced agent group. This is a mandatory prompt.

Source Data

List of files/folders to be transferred. This is a list of absolute paths (or paths relative to the Base Folder) and may be in SigListXML format or comma-separated. If this list contains a path to a single file, and that path ends with ._manifest_, then it is considered a manifest file and is read by the job to determine the actual list of files/folders to be transferred. UNC paths are supported. If the Base Folder is specified, then these paths are considered relative to the Base Folder. A combination of absolute and relative paths is not supported. UNC paths are supported on Windows.

This is a mandatory prompt.

Delete Source Files After Upload

Select Yes or No from the drop-down list. When set to Yes, files or folders are removed from the source after a successful transfer.

Delete Empty Source Folders After Upload

Select Yes or No from the drop-down list. When set to Yes, the empty source folder is deleted after the files and folders are uploaded.

Source Content Filters

Include File Patterns

Allows users to specify which files are transferred by filtering on the names to include. Use a comma to separate multiple filters. By default, an asterisk appears in the prompt, to include all file names and types. For example, if you type *.doc, *.ppt, in the prompt, the transfer includes files with these extensions.

Exclude File Patterns

Allows users to specify which files are excluded from the transfer. Use a comma to separate multiple filters. If the source directory for this job is also being used as the target directory for another job, you should add #work_file#* and #chkpt_file#*.

Exclude Subdirectories

Allows you to control how subfolders are excluded.

None: all subfolders will be transferred

All: no subfolders will be transferred

Specified (a list of subfolders to exclude - type the subfolder names in the Subdirectories to Exclude prompt)

Subdirectories to Exclude

Subfolders can be excluded by specifying them in this prompt. Multiple entries must be separated with a comma. When the job runs, all folders that match those specified in the Subdirectories to Exclude prompt are excluded. Typical behavior is to exclude subfolders that match this prompt regardless of where they appear in the folder path. When you use the anchoring expression (@) the behavior is now to anchor the exclude folder path.

For example, if you specify a source folder of C:\data\docs, and an exclude folder of temp, any subfolders called temp are excluded, including those nested within another subfolder. (For example the subfolder C:\data\docs\publish\release\temp is excluded as well.)

If you who want to exclude a folder only at a certain level, use the @ symbol to anchor the exclude folder path at the starting source folder level. For example, specifying @temp in the above example means that the C:\data\docs\temp folder is excluded, but the C:\data\docs\publish\release\temp folder is included.

To exclude C:\data\docs\publish\release\ but have a source folder path of C:\data\docs, the user needs to type C:\data\docs in the Base Folder prompt and @publish\release\temp in the Subdirectories to Exclude prompt. Special characters allow you to make use of pattern matching on the folder path. To match special characters literally, you must escape special characters. Characters include the following:

* (matches zero or more characters)

? (matches any single character)

[...] (matches any one of the enclosed characters - for example, [ch] would match the characters "c" or "h")

A pair of digit, lowercase or uppercase characters separated by a hyphen '-' denotes a range set and matches any character sorted in the range. If the first character following the '[' is '^' or '!', then any character not enclosed is matched. Use commas to specify multiple distinct patterns.

File Date Filter

Used to filter files based on date or unit of time. To configure this option select:

  • Filter on: select None, When File Was Last Modified, When File Was Last Accessed, or When File Was Created.
  • By Date: enable this option and specify your date range.
  • By Unit of Time: enable this option and specify the time range.

File Readiness Check Type

Specify the check type used by an agent to determine if a file is ready to be transferred. Depending on your configuration, you may or may not want to enable the file readiness check option.

For Windows agents and local Windows file systems, there is no need to use the file readiness check, because the core file in use is sufficient.

For Windows agent and NAS Windows file system, as long as the NAS file system properly supports file locking for in-use files, the readiness check is unnecessary. However, it is still available in the event the NAS file system does not properly support file locking.

For non-Windows agents of any kind (Linux, UNIX, MacOS), if the local or network file system supports file locking, the readiness check is not required. Determine this on a case-by-case basis.

Readiness Check Delay

The amount of time (in seconds) to pause between the two phases of the specified file readiness check type (MD5 Hash, Consecutive Seek, or Date/Time & Size) for an open file. The file is transferred only if the results of the two checks are identical. If a large number of files are being checked, a value of 0 seconds may be sufficient, since there will likely be sufficient time between the first and second checks of a given file to determine if its content has changed without inserting an additional delay. The default value is 10 seconds.

Target Storage

Cloud Vendor

Select Amazon S3 or Microsoft Azure to define the cloud storage you're using.

This is a mandatory prompt.

Storage Specification Mechanism

Select Flight Storage Config ID or Cloud Vendor Credentials. When Flight Storage Config ID is selected, the object storage and access credentials are managed using https://manage.signiant.com. When Cloud Vendor Credentials is selected, the credentials are specified in the job.

If you select Flight Storage Config ID, you must enter this value below in Flight Storage Config ID. If you select Cloud Vendor Credentials, you must specify either the Amazon S3 or Microsoft Azure properties.

Flight Storage Config ID

Enter the Flight Storage Config ID. You must enter this value if you selected Flight Storage Config ID in the Storage Specification Mechanism list.

Folder Structure Preservation

Specify how to manage folder structure during the upload of files and folders. From the drop-down list select: Preserve Top-Level Folders and Subfolders, Preserve Subfolders Only, or Flatten Folder Structure. The workflow transfer can: preserve the structure of all uploaded folders including the base folder (Preserve Top-Level Folders and Subfolders), preserve the structure of the subfolders only and not the base folder (Preserve Subfolders Only) or flatten all content so that no folder structure is transferred to the object container (Flatten Folder Structure). Note: if Source Data is a manifest file, Folder Structure Preservation must be set to Flatten Folder Structure.

Optional Target Subfolder

By default all uploaded content is placed relative to the root of the object storage. When an Optional Target Subfolder is specified, all content is placed relative to this subfolder rather than the root. This subfolder is created if it does not exist.

Flight Service

Flight Service Primary Entry Point

Select your primary direct end point server. This specifies the specific locations or regions that the Flight CLI connects to when performing the upload. It is generally recommended that the Flight Service Primary Entry Point be at the location that is geographically closest to where the object storage container is located, not necessarily the point that is closest to the source Signiant Agent.

Flight Service Backup Entry Point

Select the backup entry point server. This list includes the regional server aliases. This attribute can be specified as a geographic region spanning multiple Signiant Flight clusters, providing further high-availability and fault-tolerance. The Flight Service Backup Entry Point is provided for high-availability, in the unlikely event that the primary endpoint is not available, the backup is used.

Flight API Key

Enter your Flight API Key. You must create the Flight API Key using the Signiant Console. Use https://manage.signiant.com to access the Signiant Console.

This is a mandatory prompt.

Amazon S3 Credentials

AWS Access Key

Enter your AWS Access Key.

AWS Secret Key

Enter your AWS Secret Key.

S3 Bucket

Enter your S3 Bucket name. This is the folder into which the files are placed. This location must exist; it is not created automatically.

Microsoft Azure Credentials

Azure Storage Account Name

Enter your Azure Storage Account Name.

Azure Access Key

Enter your Azure Access Key.

Azure Container

Enter your Azure Container name. This is the folder into which the files are placed. This location must exist, it is not created automatically.

Encryption Options

Encrypt Uploaded Files

Select Yes or No from the drop-down list. All Flight transfers are encrypted in transit and when Encrypt Uploaded Files is set to Yes, all content uploaded to the cloud is encrypted. This allows you to choose an extra measure of security for sensitive material. The Encryption Key is required to decrypt this content.

Encryption Key

Enter your Encryption Key. This is used when encrypting the content. The Encryption Key must be a 64-character hex string (0....9,A-F). For encryption to work correctly, you must also configure the Initialization Vector.

Initialization Vector

Enter your Initialization Vector. This is used when uploading content. The Initialization Vector must be a 32-character hex string (0...9, A-F). When Encryption Key is configured, you must also configure Initialization Vector.

Transport Options

Protocol

Specifies either a specific protocol UDP, TCP, HTTP for the file transfer, or All. All means to try UDP first, then TCP, and finally HTTP. The default value is All.

Bandwidth Throttle

Allows you to specify the amount of bandwidth the job uses. Note that bandwidth throttles may also be employed by other network devices and policies (e.g., QoS), therefore, a bandwidth throttle (or target maximum) defined here may not be achievable. If you are having difficulty achieving a particular bandwidth target ensure that other policies are not impacting your ability to reach the desired throughput.

Use the drop-down list to select the rate and the bandwidth slider icon to calculate the amount of bandwidth as a percentage of the standard bandwidth maximums.

Transport Restart Attempts

The number of restarts on a transfer failure before the job exits in error. If zero, then no restarts are attempted. The workflow does not retry on invalid storage credentials or other conditions under which a retry has no chance of succeeding. The default value is 3.

Transport Port Usage

Set to Single or Range. By default the transfer uses a single port (49221) for firewall efficiency. For enhanced throughput, set this to Range. If outbound firewall rules are set, then a port range of (49221-49420) must be added to the firewall. The default value is Single.

Relay Agents

If the network connection to the Flight Server must be routed via a Signiant relay agent (typically for DMZ navigation), then enter one or more Relay Agent addresses (IP address or hostname) through which the network connection is to be routed. If multiple relay agents are selected, all routes to the Flight Server are tried, and the first to respond is used.

HTTP Proxy

If you have an outbound HTTP proxy, specify the HTTP Proxy server URL and port. This is required for the job key check. This proxy is not used for HTTP fallback transfers. The specified value should be in the format: http://<host>:<port>

Notification and Logging

Success Email Condition

The circumstances in which an email is sent. Choose from the following:

  • Never: no email is sent
  • Always: an email is always sent when a job runs
  • On Transfer: an email is sent only when the transfer occurs

Email To

The email address(es) who receive the logging report. Separate multiple email addresses with a comma.

Email Cc

Specifies the email addresses(es) who receive a carbon copy email notification message when the job runs. Separate multiple email addresses with a comma.

Email Bcc

Specifies the email addresses(es) who receive a blind carbon copy email notification message when the job runs. Separate multiple email addresses with a comma.

Email Failure Report To

Type the email address of the person who will receive failure reports. Separate multiple addresses with a comma.

Log Detail Level

From the drop-down list, select the amount of information based on severity, that is logged during the transfer. The options are: Error, Warn, Info, or Debug. The default is Info. Debug provides the greatest level of detail while Error provides the least.

Create File Delivery Log

Controls the creation of the delivery log. Select Yes or No from the drop-down list.

Job Schedule

Frequency

Specifies how often the job runs. Note that editing the frequency of a job that is currently in a queued state within a queued job group has no effect on the job until the next time the job is run.

The options are:

None: use for jobs that run only at irregular, user defined times.

Hourly, Daily, Weekly, Monthly, Yearly: run the job once every selected interval.

Monthend: run the job once every last day of the month.

Once: run the job only once, when first created.

Start Date/Time

The date and time at which you want the job to run. Note that editing the start date/time of a job that is currently in a queued state within a queued job group has no effect on the job until the next time the job is run.

Time Zone

Specifies the time zone in which the displayed times are set. For example, if an Eastern Time Zone specifies a 9:00 a.m start time and a time zone of Pacific Standard Time, the job runs at 9:00 a.m Pacific Standard Time.

This is a mandatory prompt.

Interrupt on Failure

When a job in the workflow fails, the workflow stops and waits for administrator intervention. If the workflow is interrupted and an alarm condition of any kind is defined that applies to that particular job, an alarm is displayed on the Alarms page indicating that the workflow is in an interrupted state.

Priority

Specifies the priority of the job. When used in conjunction with a Resource Control, the priority determines the placement of the job within the resource queue. A job with a higher priority is placed before a job with a lower priority. The options are: 1 (low), 2 (medium), 3 (high), 4 (urgent), or 5 (immediate). The default value is 2. (For information about resource controls, see Understanding Resource Controls in the Manager User's Guide.)

This is a mandatory prompt.

Finish Before

The date and time at which you want the job to be completed. When used in conjunction with a Resource Control, this attribute determines the placement of the job within the resource queue. A job with an earlier Finish Before time is placed before a job with a later Finish Before time. (For information about resource controls, see Understanding Resource Controls in the Manager User's Guide.)

Flight Downloads Prompts

The following table describes the prompts for the Flight Downloads job.

Prompt Description
Source Content

Cloud Vendor

Select Amazon S3 or Microsoft Azure to define the cloud storage vendor.

This is a mandatory prompt.

Storage Specification Mechanism

Select Flight Storage Config ID or Cloud Vendor Credentials. When Flight Storage Config ID is selected, the object storage and access credentials are managed using https://manage.signiant.com. When Cloud Vendor Credentials is selected, the credentials are specified in the job.

If you select Flight Storage Config ID, you must enter this value below in Flight Storage Config ID. If you select Cloud Vendor Credentials, you must specify either the Amazon S3 or Microsoft Azure properties.

Flight Storage Config ID

Enter the Flight Storage Config ID. You must enter this value if you selected Flight Storage Config ID in the Storage Specification Mechanism list.

Source Data

List of files and/or folders to be transferred. This is a list of absolute paths (or paths relative to the Target Folder) and may be in SigListXML format or comma-separated. A combination of absolute and relative paths is not supported.

If this list contains a path to a single file, and that path ends with ._manifest_, then it is considered to be a manifest file and is read by the job to determine the actual list of files/folders to be transferred. A UNC path to the manifest file is supported on Windows. The manifest path must be an absolute path.

This is a mandatory prompt.

Target Storage

Target Agent

Select an Agent from the drop-down list from which the files/folders are downloaded.

This is a mandatory prompt.

Target Folder

Destination folder for the downloaded files/folders.

This is a mandatory prompt.

Flight Service

Flight Service Primary Entry Point

Select your primary direct end point server. This specifies the specific locations or regions that the Flight CLI connects to when performing the upload. It is generally recommended that the Flight Service Primary Entry Point be at the location that is geographically closest to where the object storage container is located, not necessarily the point that is closest to the source Signiant Agent.

Flight Service Backup Entry Point

Select the backup entry point server. This list includes the regional server aliases. This attribute can be specified as a geographic region spanning multiple Signiant Flight clusters, providing further high-availability and fault-tolerance. The Flight Service Backup Entry Point is provided for high-availability, in the unlikely event that the primary endpoint is not available, the backup is used.

Flight API Key

Enter your Flight API Key. You must create the Flight API Key using the Signiant Console. Use https://manage.signiant.com to access the Signiant Console.

This is a mandatory prompt.

Amazon S3 Credentials

AWS Access Key

Enter your AWS Access Key.

AWS Secret Key

Enter your AWS Secret Key.

S3 Bucket

Enter your S3 Bucket name. This is the folder into which the files are placed. This location must exist; it is not created automatically.

Microsoft Azure Credentials

Azure Storage Account Name

Enter your Azure Storage Account Name.

Azure Access Key

Enter your Azure Access Key.

Azure Container

Enter your Azure Container name. This is the folder into which the files are placed. This location must exist, it is not created automatically.

Decryption Options

Decrypt Download Files

Select Yes or No to control file decryption. When set to Yes, content that has been encrypted by the FlightUpload component is decrypted.

Decryption Key

Enter the Decryption Key to use when decrypting the downloaded content. This must be a 64-character hex string (0...9, A-F). The Initialization Vector must also be configured.

Initialization Vector

Enter the Initialization Vector to use when decrypting the downloaded content. This must be a 32-character hex string (0...9, A-F). This must be provided in order to decrypt files.

Transport Options

Protocol

Specifies either a specific protocol UDP, TCP, HTTP for the file transfer, or All. All means to try UDP first, then TCP, and finally HTTP. The default value is All.

Bandwidth Throttle

Allows you to specify the amount of bandwidth the job uses. Note that bandwidth throttles may also be employed by other network devices and policies (e.g., QoS), therefore, a bandwidth throttle (or target maximum) defined here may not be achievable. If you are having difficulty achieving a particular bandwidth target ensure that other policies are not impacting your ability to reach the desired throughput.

Use the drop-down list to select the rate and the bandwidth slider icon to calculate the amount of bandwidth as a percentage of the standard bandwidth maximums.

Transport Restart Attempts

The number of restarts on a transfer failure before the job exits in error. If zero, then no restarts are attempted. The workflow does not retry on invalid storage credentials or other conditions under which a retry has no chance of succeeding. The default value is 3.

Transport Port Usage

Set to Single or Range. By default the transfer uses a single port (49221) for firewall efficiency. For enhanced throughput, set this to Range. If outbound firewall rules are set, then a port range of (49221-49420) must be added to the firewall. The default value is Single.

Relay Agents

If the network connection to the Flight Server must be routed via a Signiant relay agent (typically for DMZ navigation), then enter one or more Relay Agent addresses (IP address or hostname) through which the network connection is to be routed. If multiple relay agents are selected, all routes to the Flight Server are tried, and the first to respond is used.

HTTP Proxy

If you have an outbound HTTP proxy, specify the HTTP Proxy server URL and port. This is required for the job key check. This proxy is not used for HTTP fallback transfers. The specified value should be in the format: http://<host>:<port>

Notification and Logging

Success Email Condition

The circumstances in which an email is sent. Choose from the following:

  • Never: no email is sent
  • Always: an email is always sent when a job runs
  • On Transfer: an email is sent only when the transfer occurs

Email To

The email address(es) who receive the logging report. Separate multiple email addresses with a comma.

Email Cc

Specifies the email addresses(es) who receive a carbon copy email notification message when the job runs. Separate multiple email addresses with a comma.

Email Bcc

Specifies the email addresses(es) who receive a blind carbon copy email notification message when the job runs. Separate multiple email addresses with a comma.

Email Failure Report To

Type the email address of the person who will receive failure reports. Separate multiple addresses with a comma.

Log Detail Level

From the drop-down list, select the amount of information based on severity, that is logged during the transfer. The options are: Error, Warn, Info, or Debug. The default is Info. Debug provides the greatest level of detail while Error provides the least.

Create File Delivery Log

Controls the creation of the delivery log. Select Yes or No from the drop-down list.

Job Schedule

Frequency

Specifies how often the job runs. Note that editing the frequency of a job that is currently in a queued state within a queued job group has no effect on the job until the next time the job is run.

The options are:

None: use for jobs that run only at irregular, user defined times.

Hourly, Daily, Weekly, Monthly, Yearly: run the job once every selected interval.

Monthend: run the job once every last day of the month.

Once: run the job only once, when first created.

Start Date/Time

The date and time at which you want the job to run. Note that editing the start date/time of a job that is currently in a queued state within a queued job group has no effect on the job until the next time the job is run.

Time Zone

Specifies the time zone in which the displayed times are set. For example, if an Eastern Time Zone specifies a 9:00 a.m start time and a time zone of Pacific Standard Time, the job runs at 9:00 a.m Pacific Standard Time.

This is a mandatory prompt.

Interrupt on Failure

When a job in the workflow fails, the workflow stops and waits for administrator intervention. If the workflow is interrupted and an alarm condition of any kind is defined that applies to that particular job, an alarm is displayed on the Alarms page indicating that the workflow is in an interrupted state.

Priority

Specifies the priority of the job. When used in conjunction with a Resource Control, the priority determines the placement of the job within the resource queue. A job with a higher priority is placed before a job with a lower priority. The options are: 1 (low), 2 (medium), 3 (high), 4 (urgent), or 5 (immediate). The default value is 2. (For information about resource controls, see Understanding Resource Controls in the Manager User's Guide.)

This is a mandatory prompt.

Finish Before

The date and time at which you want the job to be completed. When used in conjunction with a Resource Control, this attribute determines the placement of the job within the resource queue. A job with an earlier Finish Before time is placed before a job with a later Finish Before time. (For information about resource controls, see Understanding Resource Controls in the Manager User's Guide.)


Limitations and Error Handling

Restrictions and Limitations

The following restrictions and limitations apply to the Flight Uploads workflow:

  • When a file specified to be transferred does not exist, the job logs the missing file as an error and continues to transfer remaining files. The job exits with a success.

The following restrictions and limitations apply to the Flight Downloads workflow:

  • When a file specified to be transferred does not exist, the job logs the missing file as an error and continues to transfer the remaining files. The job exits with a success.
  • Metadata files associated with a specific file may not get transferred unless specified explicitly or contained within a folder that is transferred.

Error Handling

When an error is encountered during a file transfer that prevents the transfer or if there is a configuration error, Flight exits in error. In all other error conditions, Flight retries all transfers until the retry limit is met.

All errors occurring during a file transfer are written to the job log.

When the job log level is set to debug, the manifest file and Flight transport log are not deleted.