Fan Out File Transfer

Copy a file from a single source to multiple targets.

Pattern Definition

Problem How do I transfer a data file from a single source to multiple targets where source and target may be Cloud based or on premises?
Context This pattern is applicable when there is a desire to transfer a file from one source to multiple targets. The source and targets may be Cloud-based or on premises.
Forces
  • A single file needs to be transferred to multiple targets.
  • Processing of the source file may create different files for the different targets.
  • Data Integration Platform may provide access to sources or targets that would not otherwise be accessible.
Solution The pattern employs one Data Integration Platform regardless of the location of the source and targets.
ConsequencesPros:
  • Allows multiple target files to be created with only one read of the source file.
  • Reduces bandwidth required since the source file is read only once.
  • Reduces processing by sharing the source to target file processing.
Cons:
  • Makes failure analysis more complex since one read causes multiple writes.

Interaction View

Cloud Platform with Cloud / On-Premise Source

Transfer data (via file) from a source that is Cloud-based or on-premises to multiple targets.

Interactions

  1. The Data Integration Platform reads the source file. The source file may be on Cloud-based storage on on-premises storage.
  2. The Data Integration Platform processes the file and writes the file to Cloud-based storage.
  3. The Data Integration Platform processes the file (if necessary) and writes the file to another Cloud-based storage.
  4. The Data Integration Platform processes the file (if necessary) and writes the file to on-premises storage.
  5. The Data Integration Platform processes the file (if necessary) and writes the file to another on-premises storage.

Notes

  1. The on-premises file system must be accessible from the Public Cloud.
  2. This example shows four target files, but there could be any number of Cloud-based and on premises target files.

On-Premise Platform with Cloud / On-Premise Source

Transfer data (via file) from a source that is Cloud-based or on premises to multiple targets.

Interactions

  1. The on-premises Data Integration Platform reads the source file. The source file may be on Cloud-based storage on on-premises storage.
  2. The on-premises Data Integration Platform processes the file and writes the file to on-premises storage.
  3. The on-premises Data Integration Platform processes the file (if necessary) and writes the file to another on-premises storage.
  4. The on-premises Data Integration Platform processes the file (if necessary) and writes the file to Cloud-based storage.
  5. The on-premises Data Integration Platform processes the file (if necessary) and writes the file to another Cloud-based storage.

Notes

  1. his example shows four target files, but there could be any number of Cloud-based and on premises target files.

Cloud Source with Agent

Transfer data (via file) from a source that is Cloud-based to multiple targets.

Interactions

  1. The Data Integration Platform reads the Cloud-based source file.
  2. The Data Integration Platform processes the file and writes the file to Cloud-based storage.
  3. The Data Integration Platform processes the file (if necessary) and writes the file to another Cloud-based storage.
  4. The Data Integration Platform processes the file (if necessary) and sends the file to the on-premises Integration Agent.
  5. The Integration Agent writes the file to on-premises storage.
  6. The Data Integration Platform processes the file (if necessary) and sends the file to the on-premises Integration Agent.
  7. The Integration Agent writes the file to another on-premises storage

Notes

  1. The Integration Agent provides secure communication between on premises and Public Cloud. The Integration Agent establishes an encrypted connection to the Data Integration Platform by calling into the Data Integration Platform. There is no need to “punch a hole” in the on-premises firewall.
  2. This example shows four target files, but there could be any number of Cloud-based and on premises target files.

On-Premise Source with Agent

Transfer data (via file) from a source that is on-premises to multiple targets.

Interactions

  1. The Integration Agent reads the on-premises file.
  2. The Integration Agent sends the file to the Data Integration Platform.
  3. The Data Integration Platform processes the file and writes the file to Cloud-based storage.
  4. The Data Integration Platform processes the file (if necessary) and writes the file to another Cloud-based storage.
  5. The Data Integration Platform processes the file (if necessary) and sends the file to the on-premises Integration Agent.
  6. The Integration Agent writes the file to on-premises storage.
  7. The Data Integration Platform processes the file (if necessary) and sends the file to the on-premises Integration Agent.
  8. The Integration Agent writes the file to another on-premises storage.

Notes

  1. The Integration Agent provides secure communication between on premises and Public Cloud. The Integration Agent establishes an encrypted connection to the Data Integration Platform by calling into the Data Integration Platform. There is no need to “punch a hole” in the on-premises firewall.
  2. This example shows four target files, but there could be any number of Cloud-based and on premises target files.