Consolidate File Transfer

Consolidate files from multiple sources into a single target

Pattern Definition

Problem How do I consolidate data from multiple source files to a single target file where source and target may be Cloud based or on premises?
Context This pattern is applicable when there is a desire to consolidate data from multiple files into a single target file. The source and targets may be Cloud-based or on premises.
Forces
  • Data from multiple files needs to be consolidated into a single target file.
  • Target file may contain only a subset of the data in the source files.
  • Data Integration Platform may provide access to sources or targets that would not otherwise be accessible.
Solution The pattern employs one Data Integration Platform regardless of the location of the source and targets.
ConsequencesPros:
  • Allows multiple source files to be merged into one target file.
Cons:
  • Makes failure analysis more complex since multiple reads result in only one file being written.

Interaction Views

  • Cloud Platform with Cloud / On-Premise Target
  • On-Premise Platform with Cloud / On-Premise Target
  • Cloud Target with Agent
  • On-Premise target with Agent

Cloud Platform with Cloud / On-Premise Target

Consolidate data from multiple source files into a single target file. Source and target may be Cloud-based or on-premises.

Interactions

  1. The Data Integration Platform reads and processes a Cloud-based source file.
  2. The Data Integration Platform reads and processes another Cloud-based source file.
  3. The Data Integration Platform reads and processes an on-premises source file.
  4. The Data Integration Platform reads and processes another on-premises source file.
  5. The Data Integration Platform writes the output file to the target storage. The target file may be on Cloud-based storage on on-premises storage

Notes

  1. The on-premises file system must be accessible from the Public Cloud.
  2. This example shows four source files, but there could be any number of Cloud-based and on premises files read and processed in any order.

On-Premise Platform with Cloud / On-Premise Target

Consolidate data from multiple source files into a single target file. Source and target may be Cloud-based or on-premises.

Interactions

  1. The on-premises Data Integration Platform reads and processes a Cloud-based source file.
  2. The on-premises Data Integration Platform reads and processes another Cloud-based source file.
  3. The on-premises Data Integration Platform reads and processes an on-premises source file.
  4. The on-premises Data Integration Platform reads and processes another on-premises source file.
  5. The on-premises Data Integration Platform writes the target file to storage. The target file may be on Cloud-based storage on on-premises storage.

Notes

  1. This example shows four source files, but there could be any number of Cloud-based and on premises files read and processed in any order.

Cloud Target with Agent

Consolidate data from multiple source files into a single Cloud-based target file. Source files may be Cloud-based or on-premises.

Interactions

  1. The Data Integration Platform reads and processes a Cloud-based source file.
  2. The Data Integration Platform reads and processes another Cloud-based source file.
  3. The Integration Agent reads an on-premises source file.
  4. The Integration Agent sends the file to the Data Integration Platform which processes the file.
  5. The Integration Agent reads another on-premises source file.
  6. The Integration Agent sends the file to the Data Integration Platform which processes the file.
  7. The Data Integration Platform writes the file to the Cloud-based target file.

Notes

  1. The Integration Agent provides secure communication between on premises and Public Cloud. The Integration Agent establishes an encrypted connection to the Data Integration Platform by calling into the Data Integration Platform. There is no need to “punch a hole” in the on-premises firewall.
  2. This example shows four source files, but there could be any number of Cloud-based and on premises files read and processed in any order.

On-Premise target with Agent

Consolidate data from multiple source files into a single on-premises target file. Source files may be Cloud-based or on-premises.

Interactions

  1. The Data Integration Platform reads and processes a Cloud-based source file.
  2. The Data Integration Platform reads and processes another Cloud-based source file.
  3. The Integration Agent reads an on-premises source file.
  4. The Integration Agent sends the file to the Data Integration Platform which processes the file.
  5. The Integration Agent reads another on-premises source file.
  6. The Integration Agent sends the file to the Data Integration Platform which processes the file.
  7. The Data Integration Platform send the target file to the Integration Agent.
  8. The Integration Agent writes the target file to on-premises storage.

Notes

  1. The Integration Agent provides secure communication between on premises and Public Cloud. The Integration Agent establishes an encrypted connection to the Data Integration Platform by calling into the Data Integration Platform. There is no need to “punch a hole” in the on-premises firewall.
  2. This example shows four source files, but there could be any number of Cloud-based and on premises files read and processed in any order.