Signiant Support

13.1 FTP Integration User's Guide Print



FTP Integration and Configuration

The FTP sample workflows allow users to employ Signiant media transfer technology when a Signiant agent does not reside at the destination (or source). Agents can invoke the open protocol FTP to send or retrieve files.

In addition to the correct hardware and software platforms required for a Signiant Manager and agents, the Signiant FTP media integrator solution requires a configured FTP server, that you can connect to using the Signiant FTP solution. Signiant is compatible with most of the common FTP servers.


FTP Components

The DropBoxToFTP and FTPRetriever components specifies the parameters relating to scheduling the workflow, such as when and how the job should run. The FileList and FTPPut components are grouped together as part of the execution sequence that transfers files via FTP to the FTP edge server and to deliver them to specified recipients on the FTP network. The FTPGet component retrieves files from an FTP server on a regular basis. The JobReport and JobReportException components specify information relating to a job report about the FTP job, including recipients, email subject line, number of report versions to maintain, and so on. An exception hierarchy allows you to specify conditions under which the report is sent (for example, if one of the component fails, the report may go to different recipients than if the entire job succeeds).

Best Practices

In cases where you want to transfer a large number of files (several hundred or more), you may want to first combine the files into a single compressed (zipped) file to improve transfer performance. After creating the compressed file using methods appropriate to your operating system, send the compressed file as you would any other file. You may want to notify the recipients that they will need to uncompress the file after they receive it.

Signiant FTPRetriever provides a solution whereby files can be retrieved from an FTP server on a regular basis. As the FTP server has limited functionality compared to a Signiant agent, far fewer options are available to ensure that files are not in use on the FTP server when they are being retrieved. Users of the FTPRetriever solution should take this into consideration. There are a few limitations with the FTPRetriever solution:

  • No "Skip on Failure" functionality for transferring multiple files. If the first one or two files fail, the entire FTP transfer fails.
  • No symbolic link support. A symbolic link in the source directory causes the FTP transfer to quit. Use the actual path to the file.
  • No file statistics appear on the status page for an FTP transfer. File Volume, Data Volume and Performance display no information.
Note: In cases where you want to retrieve a large number of files (several hundred or more), you may want to ask the provider of the media to bundle the files into a single "zip" file.

FTP Prompts

The FTP sample workflows included with a Signiant FTP license are FTPDropBox and FTPRetriever. FTPDropBox supports job scheduling and a watch folder. Every complete file within the watch folder directory will be transferred to an FTP server, based on a user-defined frequency. By default, any files which are in use or busy (being changed) during the transfer will be skipped.

In order to run an FTP job, you need to first select a job group within which to place the job . You will require access to at least one job group. To run either of the FTP sample workflows, follow these steps:

  1. From the Manager, select Jobs>Groups.
  2. Select one of the job groups in the list by double-clicking on it. A list of jobs already contained in that job group appears.
  3. From the job menu, click "Add". The "Add Job" dialog appears.
  4. From the "Job Template Library" drop-down list, select "FTP_Workflows".
  5. From the "Job Template" dropdown, select "DropBoxToFTP" to send files to an FTP server (or "FTPRetriever" to retrieve files from an FTP Server).
  6. Fill in prompts as described in the table below.

The following table describes the prompts associated with FTPGet and FTPPut. Mandatory prompts are indicated with an asterisk.

FTPGet FTPPut Prompt Description
x x Job Name

A unique name for this job. This must start with a letter and can only contain underscores to separate words. This name should help operators to recognize the functions of this job (e.g., Send_Media_to_Site2).

Drop Box Specification
  x *Drop Box Agent

The agent(s) from which you are transferring the data.

  x *Drop Box Folder

The source folder for files to transfer. Type the folder in the prompt, or click the computer icon to browse and select an agent folder. The source folder specified may be in the following format:

  • Windows root drives, e.g., C:\ApplicationData
  • UNIX root drives, e.g., /home
  • Windows UNC path names, e.g., \\machine1\applicationdata

Leaving the Source Data prompt empty will use the agent's default folder.

  x Include File Names/Types

Allows users to specify which files are transferred by filtering on the names to include. Use a comma to separate multiple filters. By default, an asterisk appears in the prompt, to include all file names/types. For example, if you type *.doc, *.ppt, in the prompt, the transfer wills include files with these extensions.

  x Exclude File Names/Types

Allows users to specify which files are excluded from transfer. Use a comma to separate multiple filters. If the source directory for this jobs is also being used as the target directory for another job, you should add #work_file#* and #chkpt_file#*.

  x Exclude Subfolders

To exclude sub-folders, choose from the following options:

  • None (all sub-folders will be transferred - none will be excluded)
  • All (no sub-folders will be transferred - all of them will be excluded)
  • Specified (a list of sub-folders to exclude - type sub-folders in the Subdirectories to Exclude prompt, described below)
  x Subfolders to Exclude

Sub-folders may be excluded by specifying them in this prompt. Multiple entries must be separated with a comma. When the job runs, all folders that match those specified in the Subdirectories to exclude prompt will be excluded. Note that normal behavior is to exclude sub-folders that match regardless of where they appear in the folder path. Using the anchoring expression (@) changes this behavior to anchor the exclude folder path.

For example, if you specify a source folder of C:\data\docs, and an exclude folder of temp, any sub-folders called temp will be excluded, even those nested within another sub-folder. (For example the sub-folder C:\data\docs\publish\release\temp would be excluded as well.)

If you who want to exclude a folder only at a certain level can use the @ symbol to anchor the exclude folder path at the starting source folder level. For example, specifying @temp in the above example would mean that the C:\data\docs\temp folder would be excluded, but the C:\data\docs\publish\release\temp folder would be included.

If you instead wanted to make sure just the C:\data\docs\publish\release\temp folder was excluded, but wanted to have a source folder path of c:\data\docs, the user would need to type C:\data\docs in the source folder prompt and @publish\release\temp in the exclude prompt. Special characters allow you to make use of pattern matching on the folder path. You must escape special characters with a backslash to be matched literally. Characters include the following:

 

  • * (matches zero or more characters)
  • ? (matches any single character)
  • [...] (matches any one of the enclosed characters - for example, [ch] would match the characters "c" or "h")
  • A pair of digit, lowercase or uppercase characters separated by a hyphen '-' denotes a range set and matches any character sorted in the range. If the first character following the '[' is '^' or '!', then any character not enclosed is matched.
  • Use commas to specify multiple distinct patterns.

 

  x File Readiness Check Type

Specify the kind of check type an agent uses to determine if a file is ready to be transferred (i.e., the file is not in use). For some configurations, you may or may not wish to select the file readiness check option:

  1. For Windows agents and local Windows file systems, there is no need to use the file readiness check, as the core file in use· behavior is sufficient.
  2. For Windows agent and NAS Windows file system, as long as the NAS file system properly supports file locking for in-use files, the readiness check is unnecessary. However, it is still available in case the NAS file system does not properly support file locking.
  3. For non-Windows agents of any kind (Linux, UNIX, MacOS). If the local or network file system supports file locking, the readiness check is not required. Determine this on a case-by-case basis.

Choose one of the following:

  • None - Files that are in use are skipped by default.
  • Consecutive Seek (recommended) - Reads the last few bytes at the end of the file, waits for delay period number of seconds and checks again. If the last few bytes match in both cases the file is sent. Very useful on large files where the MD5 option would simply take too long. 
  • MD5 Hash - Use an MD5 hash to compute a hash on all files under the drop-box folder. After the wait time specified in the Readiness Check Delay prompt, the hashes are recomputed. Only files with identical hashes are transferred. Note that job initialization may take a great deal of time if large files are present.
  • Date/Time & Size - Use a difference in date/time or size on the file to compare its readiness. Only files that have identical date/time & size are transferred.
  x Readiness Check Delay

The amount of time (in seconds) to pause between the two phases of the specified file readiness check type ("MD5 Hash", "Consecutive Seek", or "Date/Time & Size") for an open file. The file is transferred only if the results of the two checks are identical. The default value is 10 seconds. If a large number of files are being checked, a value of 0 seconds may be sufficient, since there will likely be sufficient time between the first and second checks of a given file to determine if its content has changed without inserting an additional delay.

  x Source Deletions After Successful Transfer

Specifies removal of files or folders from the dropbox folder after a successful transfer: Choose from one of the following:

  • None (do not delete any source files or empty folders)
  • Transferred Files (remove the files)
  • Empty folders (remove folders that are left over after the source files have been removed)
  • Transferred Files and Empty Directories (remove both transferred files and empty folders)
  • Transferred Files and Directories ( remove both transferred files and all folders immediately after transfer)

There are certain requirements with the empty folders options: the folders must contain no files (they may contain other empty folders that will be removed as well). The folder cannot have been modified within the previous 5 minutes; if it has, it will not be removed.

  x Source Directories to Exclude from Deletion Scan

Allows you to specify a comma-separated list of directories that you do not want to delete.

FTP Specification
x x *FTP Server Hostname

The name of the FTP server. The name must be resolvable by DNS. You can enter the name of the IP address of the server.

x x FTP Port

The port number the FTP server uses. Typically, this is port 21.

x   FTP Base Folder

The directory where files should be copied.

  x FTP Target Folder

The directory where files should be copied.

x x *FTP Username

The login user name for the FTP server.

x x FTP Password

The password for the user defined above.

x   Source Data

The source folder for files to transfer. Type the folder in the prompt, or click the computer icon to browse and select an agent folder. The source folder specified may be in the following format:

  • Windows root drives, e.g., C:\ApplicationData
  • UNIX root drives, e.g., /home
  • Windows UNC path names, e.g., \\machine1\applicationdata

Leaving the Source Data prompt empty will use the agent's default folder.

x   Include Files

Allows users to specify which files are transferred by filtering on the names to include. Use a comma to separate multiple filters. By default, an asterisk appears in the prompt, to include all file names/types. For example, if you type *.doc, *.ppt, in the prompt, the transfer wills include files with these extensions.

x   Include Folders

Allows users to specify which folders are retrieved (searched for retrieval). If the FTP server does not support recursion for the 'username' provided this option will not work.

x   Exclude Files

Allows users to specify which files are excluded from transfer. Use a comma to separate multiple filters. If the source directory for this jobs is also being used as the target directory for another job, you should add #work_file#* and #chkpt_file#*.

x   Exclude Folders

Allows users to specify which folders are not searched for retrieval.

x   Recurse Subfolders

Allow user to specify if the solution should descend into folders below the folder associated with the 'username'. This is a privilege not normally provided to FTP user accounts; therefore, it is recommended that you verify this capability with the FTP server administrator before enabling it. No errors will be provided if the workflow is unable to descend into folders.

x   Case Sensitive Match

Specify whether your search matches files exactly.

  x Recreate Folder Structure

Should the solution attempt to recreate the directory structure from the source agent onto the target FTP server.

  x Folder Mapping Options

Specifies whether the solution creates directories on the target that identify the FTP Server hostname (or IP address) and job date/time. This is a useful option when the same-named files are being transferred repeatedly. Choose from the following:

  • Source Host - creates a subdirectory with the name of the FTP Server from which the files were transferred.
  • Job Date/Time - creates a subdirectory corresponding to the current time in 24 hour GMT. The format used is YYYY-MM-DD_HHMM-SS.
x x Fail on FTP File Errors

Specifies conditions under which the component will exit in error. Choose from the following:

  • On Component Completion - if file transfer errors occur, the component exits in error, but only after all files have been transferred (or attempted to transfer).
  • On First Error - the component exits in an error state on the first file transfer error.
  • Never - the component exits with a success even if file transfer errors have occurred.
x x FTP Timeout

Allows you to specify how long the workflow will wait without activity before failing in error.

x x FTP Block Size

Specifies the block size used for FTP transfers.

x x File Buffer Size

Specifies the buffer size that the component will use when processing the file during a transfer.

Notification and Logging
x x Email Job report To

The email address of the person who you want to receive the logging report. Separate multiple addresses with a comma.

x x Email Subject

Text that appears in the subject prompt of the report email message.

x x Email Condition

Specifies when a job report is emailed to users specified in the Email Job Report To prompt. Choose from the following:

  • Always - Users receive a job report after every job run.
  • On Error - Users receive a job report after a job run that reports an error exit code.
  • Never - Users never receive job reports.
x x Log Detail Level

The type of logging information for this job. Choose from Error, Warn (warning), Info (Information) or Debug. Debug provides the greatest level of detail while Error provides the least.

Schedule
x x Frequency

Specifies how often the job runs. Users can choose from the following options:

  • None - Use for jobs that run only at irregular, user-defined times.
  • Hourly, Daily, Weekly, Monthly, Yearly - Run the job once every selected interval.
  • Monthend - Run the job once every last day of the month.
  • Once - Run the job only once, when first created.

Note that editing the frequency of a job that is currently in a queued state within a queued job group will have no effect on the job until the next time the job is run.

x x Start Date/Time

The date and time at which you want the job to run.

Note that editing the start date/time of a job that is currently in a queued state within a queued job group will have no effect on the job until the next time the job is run.

x x Time Zone

Specifies the time zone in which the displayed times are set. For example, if an Eastern Time zone specifies a start time of 9:00 am and a time zone of PST, the job runs at 9:00 am Pacific Standard Time or noon Eastern Standard Time.