Signiant Support

13.4 Workflow Gateway User's Guide Print


Workflow Gateway

The Signiant Workflow Gateway is designed as a two-stage dropbox application, where any workflow can be launched against content that is placed in the dropbox. The first stage collects the content placed in the dropbox and prepares a secondary workflow to run against that content. The content may be specified as single files, grouped together in folders or defined explicitly through XML files placed in the root folder of the dropbox.

The advantages over a single job monitoring a dropbox folder (e.g. MediaDropBox) include:

  • Specific file sets can be specified for each launched workflow job, rather than simply processing all files in the dropbox folder on each scheduled run.
  • File readiness checks can be run against each defined file set and the associated workflows are only launched when all referenced content is available. Therefore, for workflows with content that is ready, these are launched and those with incomplete content are delayed.
  • Multiple workflow jobs are launched in parallel, so processing of content is not held up due to a single long-running job. NOTE: In order to maximize performance of the Signiant Manager, the use of queued job groups is recommended when using Workflow Gateway.
  • Jobs are retried until successful (up to a maximum of five attempts) with a configurable retry time between failed runs.
  • Sub-folders within the dropbox folder are utilized which indicate the state of the launched jobs, without the need to log in to the Signiant Manager. Folders exist for active jobs, jobs which have completed successfully, jobs which have been scheduled for retry and jobs which have ended in error after all retries.

The advantages over initiating workflows directly via SOAP against specific content include:

  • No need for knowledge of Signiant job control web services. Just place the content into the monitored folder and it automatically gets turned into a Signiant job running the specified workflow against the specified content.
  • No need to embed plain-text passwords in a script or configuration file. The default user and password used to launch the workflow jobs is configured as part of the Workflow Gateway Monitor job.

The Workflow Gateway DropBox Monitor workflow is contained in the Workflow_Gateway_DropBox_Monitor Job Template.

The workflow consists of three components:

  • WorkflowGatewayDropBoxMonitor start component: Provides the prompt interface for defining the dropbox monitoring jobs.
  • WorkflowGatewayMonitorAndLaunch command component: Does all of the processing of the dropbox folder content.
  • MonitorReport job report component: Generates and sends a job report if the WorkflowGatewayMonitorAndLaunch component has reported an error. This generally means that the component failed to run, or that jobs could not be created or started. For jobs that are successfully created and started, job-specific notification is the responsibility of the workflow run in the given job.

Workflow Gateway Dropbox Monitors

Once files have been placed in the dropbox, they are examined on a periodic basis as specified by the DropBox Monitor job. The files are evaluated against the package composition criteria. Once a set of files match the criteria the files and folders are moved into a subfolder of the dropbox folder named _Active.

A separate job is then started for the package content. This job will handle creation of the actual package, copying of the files in the package to the Media Exchange repository and the subsequent delivery of the package to its destination recipient(s) and/or channels. The job name will include an alphanumeric portion of the file and/or folder in the package to aid in identification through the Manager GUI.

The DropBox Monitor job will create as many package processing jobs on each run as required to satisfy the files found in the dropbox. When the files in the package have been successfully copied to the Media Exchange package repository, distributed, and delivered, they are deleted from the _Active folder.

When multiple files are being copied into a folder for placement in a package, it would be possible for the dropbox monitor to run and package the files before they have all arrived. To prevent this, a folder named _Stage is created in the dropbox folder. File and folders, placed in this directory, are ignored by the dropbox monitoring job. Once the user has copied the appropriate content, then it will be moved to the top level dropbox folder. Since file moving is an instantaneous operation, there is no chance of any files being missed.

The package name is based on the package composition:

  • Individual files: The package name is based on the file name in the package.
  • Folder: The package name is based on the folder name.
  • XML defined: The package name is as specified in the name of the XML file.

Workflow Gateway Dropbox Monitor Jobs

To configure the Workflow Gateway Dropbox Monitor jobs do the following:

  1. In the Manager, select Workflow Gateway > Dropbox Monitors.
  2. Click Add and the Add Job screen is displayed.

The following section describes the prompts associated with creating a Workflow Gateway Dropbox Monitor job.

  • Job Name: Type the name for the new job. This is a mandatory prompt.
  • DropBox Settings
    • DropBox Agent: Select from the drop-down list, the hostname of the agent on which the dropbox is located. This agent does not need to be Media Exchange enabled. This is a mandatory prompt.
    • DropBox Folder: Select the monitored folder into which files and/or folders will be placed. The default user for the DropBox Agent must have access to this folder. Specifically, if a Windows UNC path is used, then a real user, not NT authority\system, must be used. This is a mandatory prompt.
  • Workflow Launch Mode
    • Launch Workflow For Each: Defines how the file sets for the Secondary workflow are defined:
      • Individual File and/or Subfolder: The secondary workflow is to be run against each file in the root of the dropbox as well as each subfolder in the root of the dropbox.
      • Individual File: The secondary workflow is to be run against each file in the root of the dropbox.
      • Subfolder: The secondary workflow is to be run against each folder in the root of the dropbox.
      • Job Definition File: The file set is defined within an XML file that can be found in the root of the dropbox.
  • File/Subfolder Options
    • File/Subfolder Options: Defines how files and subfolders are managed:
      • Preserve Subfolders for Individual Files (Job Definition File Mode): If this option is selected, any files specified explicitly using the XML manifest will have their relative subfolder structure maintained when passed to the secondary workflow. Note that any files specified with absolute paths (outside of the DropBox folder) will not have their subfolder structure preserved when passed to the secondary workflow. NOTE: Applies to XML Launch Type ONLY.
      • Use File Include/Exclude Patterns (Individual File Mode): If this option is selected and the launch type is by individual file, inclusion and exclusion lists can be specified for determining the files to process.
    • File Include Patterns: Used when Use File Include/Exclude File Patterns is selected to specify the patterns to include. Blank or “*” indicates to include everything. Partial wildcards can be used (e.g. *.mov). Separate multiple patterns with commas.
    • File Exclude Patterns: Used when Use File Include/Exclude File Patterns is selected to specify the patterns to exclude. Partial wildcards can be used (e.g. *.tmp). Separate multiple patterns with commas.
    • File Readiness Check Delay: The amount of time, in seconds, to pause between the two phases of the file readiness checks.
  • Job Creation Options
    • Create Jobs as User: The default userid to use for creating the workflow jobs. This can be overridden on a per-job basis. This is a mandatory prompt.
    • Password: The default password to use for creating the workflow jobs. This can be overridden on a per-job basis. This is a mandatory prompt.
    • Default Job Group: Select from the drop-down list, the default job group in which to create the workflow jobs. This can be overridden on a per-job basis.
    • Job Name Prefix: A prefix to use for all created jobs. This can be left blank to indicate no prefix.
    • Job Name Suffix: Select Auto-Generated Numeric to generate a unique numeric suffix for the job name or select None to not create a job name suffix for the job.
    • Use UDP Control Channel: By default this is set to No, ensuring job control information is communicated using TCP instead of UDP. This prompt determines whether the file transfer from the dropbox to the Media Exchange repository uses a TCP or UDP control channel for communication between the Manager and the dropbox agent, as well as between the dropbox and Media Exchange repository agents. In either case, the actual file transfer will utilize WAN acceleration (UDP) if licensed.
    • Encryption Level: The encryption level selected dictates how securely information is transmitted over the network (Note: the file/package is always stored in the original format). Encryption is restricted to the file transfer portion of the job and does not affect how the files are stored on the filesystem. Select one of the following levels from the drop-down list:
      • High
      • Medium
      • Low
      • No Encryption - signed: transfers unencrypted (plain text) data, but includes the SSL protocol's message digest calculation and signing to ensure data stream integrity.
      • No Encryption - unsigned: this default value, allows the transfer of data, after the initial SSL authentication of the endpoints, to proceed with no encryption, no message digest computation and no signing of the message digests (normally included with the SSL protocol). This mode of operation is only for raw performance, since it makes no guarantee of the integrity of the data stream other than the defaults used on a normal network channel. The underlying TCP protocol can guarantee the integrity of messages across each single network hop, but has no facilities for detecting a man-in-the-middle attack.
  • Job Retry Options
    • Maximum Number of Retries: The number of times to retry each failed workflow job. Select a value between 0 and 5. The default is 5.
    • Retry 1 Interval: The amount of time allowed after the first job failure before trying again. The default is Immediate.
    • Retry 2 Interval: The amount of time allowed after the second job failure before trying again. The default is 5 minutes.
    • Retry 3 Interval: The amount of time allowed after the third job failure before trying again. The default is 1 hour.
    • Retry 4 Interval: The amount of time allowed after the fourth job failure before trying again. The default is 2 hours.
    • Retry 4 Interval: The amount of time allowed after the fifth job failure before trying again. The default is 1 day.
    • Launched Job Failure Action: This setting controls whether the Monitor job ends in error if any of the launched jobs have ended in error (after all retries) or whether all error handling is to be handled by the launched jobs. The options are Fail Monitor Job After Launched Job Fails All Retries or None - Launched Jobs Will Handle Failure Conditions.
  • Job Notification and Logging Options
    • Email Job Launch Failures To: A list of email addresses to send failure messages to if the monitor job ends in error, or if errors occur when trying to create or run the workflow jobs.
    • Email Long Running Job Notifications To: A list of email addresses to send notifications to if the monitor job detects launched jobs that have been running for longer than a specified amount of time.
    • Long Running Job Threshold: The amount of time for evaluating whether a running job qualifies as a long running job. The default is None.
    • Monitor Job Log Detail Level: From the drop-down list, select the amount of information, based on severity, that is logged during the job. Choose Error, Warn (Warning), Info (Information) or Debug. Debug provides the greatest level of detail while Error provides the least.
  • Job Folder Maintenance Options
    • Delete Successful Older Than: The frequency at which the job-specific subfolders under the Success folder are deleted. The default is 1 Day.
    • Delete Errored Older Than: The frequency at which the job-specific subfolders under the Errored folder are deleted. The default is 1 Month.
    • Delete Stale Older Than: The frequency at which the job-specific subfolders under the Stale folder are deleted. The default is 1 Month.
    • Stale Period: The amount of time for evaluating whether an unsatisfied job definition XML file is considered stale. The default is 1 Day.
  • Job Maintenance Options
    • Delete Jobs Older Than: Specifies when jobs should be deleted. The default is 1 Month.
  • Dropbox Monitor Frequency
    • Frequency: Use these options to specify how often the job runs. Select from the following options:
      • None: use for jobs that run only at irregular, user-defined times.
      • Hourly, Daily, Weekly, Monthly, Yearly: run the job once every selected interval.
      • Monthend: run the job once every last day of the month.
      • Once: run the job only once, when first created.

      Note that editing the frequency of a job that is currently in a queued state within a queued job group has no effect on the job until the next time the job is run.

    • Start Date/Time: Use the calendar to specify the date and time at which you want the job to run. Note that editing the start date/time of a job that is currently in a queued state within a queued job group will have no effect on the job until the next time the job is run.
    • Time Zone: From the drop-down list, select the time zone in which the displayed times are set. For example, if an Eastern Time zone specifies a start time of 9:00 am and a time zone of PST, the job runs at 9:00 am Pacific Standard Time or noon Eastern Standard Time.

Workflow Gateway Processing

Monitoring Package Processing

To simplify the monitoring of the jobs created for package processing, three job views are accessible via the menu. Under the Workflow Gateway menu, a sub menu exists named Delivery Jobs. This sub menu contains menu items corresponding to the following three job views:

  1. Running: View all jobs created for package processing by the Workflow Gateway Dropbox Monitor that are currently running.
  2. Errored: View all jobs created for package processing by the Workflow Gateway Dropbox Monitor that have failed.
  3. All: View all jobs created for package processing by the Workflow Gateway Dropbox Monitor.

Package Processing Job Group

The jobs that process each package are placed in the job group WorkflowGatewayMonitorAndLaunch. All packages submitted at the same time will be processed in parallel. If the Manager is licensed for Resource Management, then the queuing feature can be enabled on the job group. In this scenario, the parallel processing of packages is based on the queuing configuration. See Resource Controls in the Manager User's Guide for further information on this feature.

Note: The automatic processing of packages can generate substantial workload on the agents.  If you expect more than 10 packages will be created during any polling cycle of the package processor, it is recommended that you purchase the Resource Management feature.  Otherwise, you may need to adjust the processes that submit files into the Media Gateway application's dropboxes.


Workflow Gateway Dropbox Subfolders

The  dropbox is composed of the following processing subfolders:

  • _Active: This folder contains one subfolder for each launched job that has yet to be run or has only been run once and has not completed processing. If the content was defined in a job definition file, that file will have been moved from the dropbox root folder to the jobname subfolder also. The jobname subfolder name will match the job name exactly.
  • _Retry: This folder contains one subfolder for each launched job that failed its initial run but has not exceeded the maximum number of retries. Job subfolders and their contents get moved from _Active to _Retry after the first failed job run. The jobname subfolder name will match the job name exactly.
  • _Errored: This folder contains one subfolder for each launched job that failed after the maximum number of retries. Job subfolders and their contents get moved from _Retry to _Errored after the final failed job run. The jobname subfolder name will match the job name exactly.
  • _Success: This folder contains one subfolder for each launched job that has completed successfully. Job subfolders and their contents get moved from either _Active or _Retry to _Success after a successful job run. The jobname subfolder name will match the job name exactly.
  • _Stage: This folder is intended as a staging area where files can be placed in preparation for processing. All content in this folder is ignored by the Monitor Job. The idea is that if a large set of files is going to take significant time to write to the dropbox folder, then it may be better to write them all to the _Stage folder and then move them all at once (with the job definition file if applicable) to the dropbox root folder.
  • _System: This folder contains control files used by the monitor to track the state of launched jobs. It may also contain the optional job defaults file defaults.siggateway.xml.
  • _Stale: This folder contains the XML job definition files that have not had their source content satisfied (all files/folders present and ready) within a configurable period.

Workflow Gateway XML Package Definition

Job Definition File

This section outlines the job definition file used when the monitor workflow has been configured to use Job Definition File mode as well as the global job defaults XML file, which is applicable to the per-file and per-folder workflow modes.

In Job Definition File mode, Workflow Gateway is driven with XML files using the naming convention <job_name>.siggateway.xml. These files contain the information required to define a Signiant job, although some of the optional fields can be specified as prompts to the Workflow Gateway Monitor job (e.g. User, Password, Job Group).

The following XML fragment outlines the structure of the job definition files. Items in bold are required, items in italics are optional. Each element is defined in more detail in the table below.

<SigniantWorkflowGatewayXML version="1.0">
     <Job>
         <JobName></JobName>
         <JobTemplateLibrary></JobTemplateLibrary>
         <JobTemplate></JobTemplate
         <JobUser></JobUser>
         <JobPassword></JobPassword>
         <JobGroup></JobGroup>
              <Inputs>
                  <Input name=''></Input>
                 -
                 <Input name=''></Input>
             </Inputs>
             <SourceContent1>
                   <Path> </Path>
                   -
                   <Path> </Path>
            </SourceContent1>
            -
            <SourceContent10>
                  <Path> </Path>
                 -
                 <Path> </Path>
           </SourceContent10>
   </Job>
</SigniantWorkflowGatewayXML>

Job Definition Elements
Element Description

JobName

Defines the base for the job name. If no JobName element is specified, the job name is either derived from the <job_name>.siggateway.xml file (in job definition file mode) or from the single file or subfolder name (in per-file/per-folder modes). A unique numbered suffix is added, as is the job name prefix defined in the Gateway Monitor (which can be blank). This is an optional element.

JobTemplateLibrary

The job template library containing the workflow to launch. This is required element.

JobTemplate

The name of the start component of the workflow to launch. This is a required element.

JobUser

The userid used during job creation. If JobUser is not specified, the value set in the Gateway Monitor will be used. This is an optional element.

JobPassword

The password used to create the job. If JobPassword is not specified, the value set in the Gateway Monitor will be used. This is an optional element.

JobGroup

The job group in which to create the job in. If JobGroup is not specified, the value set in the Gateway Monitor will be used. This is an optional element.

Input

Each input/prompt to the workflow for which a value is to be set requires an Input element, where the "name" attribute defines the name as it appears with the Show Web Services Info screen on the Workflow Canvas. The value to set is the content of the element. Note that certain variables are supported as content values. This is a required element.

SourceContent1...10

Files and/or folders on the source system which must be checked for existence and file readiness should be defined using the SourceContentN "bundles". Up to 10 such bundles are supported, numbered SourceContent1 through SourceContent10. Each of these bundles can have one or multiple Path elements. The given job is only scheduled for launch when all content identified in these bundles has been verified to exist and have passed file readiness checks. Each of these bundles can be substituted into the contents of an Input element using variable substitution values. This is an optional element.

Supported Job Definition File Variables

The Inputs section of the job definition file supports the use of certain variables as content values. These variables are evaluated by the Workflow Gateway Monitor and substituted with the appropriate values prior to job creation.

The following table describes the support job definition variables:

Variable Name Description

%SourceAgent%

Evaluates to the hostname of the source agent (where the Workflow Gateway Monitor is running).

%SourceFolder%

Evaluates to the job-specific folder (not the DropBox folder) where the source content referenced in the job definition file has been moved. This is effectively the working folder for each launched job.

%SourceContent1%

Evaluates to a SigListXML format file list corresponding to the SourceContent1 bundle.

%SourceContent2%

Evaluates to a SigListXML format file list corresponding to the SourceContent2 bundle.

%SourceContent3%

Evaluates to a SigListXML format file list corresponding to the SourceContent3 bundle.

%SourceContent4%

Evaluates to a SigListXML format file list corresponding to the SourceContent4 bundle.

%SourceContent5%

Evaluates to a SigListXML format file list corresponding to the SourceContent5 bundle.

%SourceContent6%

Evaluates to a SigListXML format file list corresponding to the SourceContent6 bundle.

%SourceContent7%

Evaluates to a SigListXML format file list corresponding to the SourceContent7 bundle.

%SourceContent8%

Evaluates to a SigListXML format file list corresponding to the SourceContent8 bundle.

%SourceContent9%

Evaluates to a SigListXML format file list corresponding to the SourceContent9 bundle.

%SourceContent10%

Evaluates to a SigListXML format file list corresponding to the SourceContent10 bundle.

Variable names are not case-sensitive: %sourceagent%, %SOURCEAGENT% and %SourceAgent% expand as expected.

Relative vs Absolute Paths

The <Path> elements within each SourceContent1..10 bundle can be specified as either relative paths (no leading drive letter or slashes) or as absolute paths (leading slash, drive letter qualified, UNC paths). Note: relative and absolute paths are handled differently:

Relative paths represent files and folders which have been placed directly in the Workflow Gateway Monitor root folder (the DropBox folder). Prior to the creation of the job, such relative content is moved to the job-specific subfolder (under _Active or _Retry). This is done so that the launched job is able to find the files since they are in the job’s working folder. If the option Preserve Subfolders for Individual Files is selected, any relative files specified explicitly in the XML manifest have their relative subfolder structure maintained when passed to the secondary workflow.

Absolute paths represent files and folders which are outside of the Workflow Gateway Monitor root folder (the DropBox folder). Unlike relative files, these files are not copied to the job-specific subfolder. Because they are referenced using an absolute path, the launched job has the correct information for locating them.

Sample Job Definition File

The following is an example of a job definition file which makes use of source content bundles and variables:

<SigniantWorkflowGatewayXML version="1.0"> <Job> <JobTemplateLibrary>Media_Mover_Workflows</JobTemplateLibrary> <JobTemplate>MediaTrigger</JobTemplate> <Inputs> <Input name='MediaTrigger.Source.SourceAgent'>%SourceAgent%</Input> <Input name='MediaTrigger.Target.SourceDirectory'>%SourceFolder%</Input> <Input name='MediaTrigger.Source.SourceData'>%SourceContent1%</Input> <Input name='MediaTrigger.Target.TargetAgents'>imac1-win.ott.signiant.com</Input> <Input name='MediaTrigger.Target.TargetDirectory'>C:\incoming</Input> <Input name='MediaTrigger.Schedule.timezone'>GMT</Input> <Input name='MediaTrigger.NotificationAndLogging._sp_log_severity'>3</Input> </Inputs> <SourceContent1> <Path>/Folder1</Path> <Path>C:\temp\test1.mov</Path> <Path>\\imac1-win\temp\test2.mov</Path> <Path>test3.mov</Path> </SourceContent1> </Job> </SigniantWorkflowGatewayXML>

Per-File and Per-Folder Modes

In the per-file and per-subfolder modes of Workflow Gateway, there is no unique job definition file associated with each file or folder as there is with the job definition file mode. However, the per-file and per-folder modes do require the existence of the global job defaults file. See Global Job Defaults File for details on the global job defaults file.

Global Job Defaults File

In job definition file mode, the global job defaults file can be used to minimize the amount of information required to be specified in the job definition files. The global job defaults file has exactly the same format as the job definition files and may contain any or all of the same information. Prior to job creation, the content of the global job defaults file is merged with each job definition file to create the job definitions.

Note: in the event that the same attribute appears in both the global job defaults file and a job definition file, the value in the job definition file has precedence.

The global job defaults file is optional when using job definition file mode, but is mandatory when using the per-file and per-folder modes. The file is required with per-file and per-folder modes because the global job defaults file is the only source for job definition parameters.

The global job defaults file is placed in the _System subfolder of the DropBox folder and must be named defaults.siggateway.xml.

Sample Global Job Defaults File

The following example shows the use of a global job defaults file where nearly all of the job information has been specified. The only information not specified are the actual SourceContent elements.

<SigniantWorkflowGatewayXML version="1.0"> <Job> <JobTemplateLibrary>Media_Mover_Workflows</JobTemplateLibrary> <JobTemplate>MediaTrigger</JobTemplate> <Inputs> <Input name='MediaTrigger.Source.SourceAgent'>%SourceAgent%</Input> <Input name='MediaTrigger.Target.SourceDirectory'>%SourceFolder%</Input> <Input name='MediaTrigger.Source.SourceData'>%SourceContent1%</Input> <Input name='MediaTrigger.Target.TargetAgents'>imac1-win.ott.signiant.com</Input> <Input name='MediaTrigger.Target.TargetDirectory'>C:\incoming</Input> <Input name='MediaTrigger.Schedule.timezone'>GMT</Input> <Input name='MediaTrigger.NotificationAndLogging._sp_log_severity'>3</Input> </Inputs> </Job> </SigniantWorkflowGatewayXML>

In per-file and per-subfolder modes, the variable %SourceContent1% is used to represent the single file or single folder being processed for the given job. The above global job defaults file represents a complete configuration for the launch of a MediaTrigger job on a per-file or per-folder basis.

Sample Minimal Job Definition File

Using the above global job defaults file, the job-specific definition file only needs to define the SourceContent elements, such as:

<SigniantWorkflowGatewayXML version="1.0"> <Job> <SourceContent1> <Path>/Folder1</Path> <Path>C:\temp\test1.mov</Path> <Path>\\imac1-win\temp\test2.mov</Path> <Path>test3.mov</Path> </SourceContent1> </Job> </SigniantWorkflowGatewayXML>

Sample Effective Job Definition XML

In job definition file mode, prior to job creation the contents of the global job defaults file and job definition file are merged, resulting in the following effective XML:

<SigniantWorkflowGatewayXML version="1.0"> <Job> <JobName>This is my job name</JobName> <JobTemplateLibrary>Media_Mover_Workflows</JobTemplateLibrary> <JobTemplate>MediaTrigger</JobTemplate> <Inputs> <Input name='MediaTrigger.Source.SourceAgent'>%SourceAgent%</Input> <Input name='MediaTrigger.Target.SourceDirectory'>%SourceFolder%</Input> <Input name='MediaTrigger.Source.SourceData'>%SourceContent1%</Input> <Input name='MediaTrigger.Target.TargetAgents'>imac1-win.ott.signiant.com</Input> <Input name='MediaTrigger.Target.TargetDirectory'>C:\incoming</Input> <Input name='MediaTrigger.Schedule.timezone'>GMT</Input> <Input name='MediaTrigger.NotificationAndLogging._sp_log_severity'>3</Input> </Inputs> <SourceContent1> <Path>/Folder1</Path> <Path>C:\temp\test1.mov</Path> <Path>\\imac1-win\temp\test2.mov</Path> <Path>test3.mov</Path> </SourceContent1> </Job> </SigniantWorkflowGatewayXML>