Cloud File Transfer

Copy a file from a source to a target (where both the source and target can be Cloud-based or on-premises) using a Cloud-based Data Integration Platform.

Pattern Definition

Problem How do I transfer a data file from source to a target where source and target could be Cloud-based or on premises with minimal on premises impact?
Context This pattern is applicable when there is a desire to transfer files from Cloud to Cloud, Cloud to on premises, on premises to Cloud, or on premises to on premises. The Cloud-based source or target may be SaaS, PaaS, or IaaS.
Forces
  • Sources and targets are Cloud-based or on premises.
  • Cloud-based sources and targets may be SaaS, applications hosted on PaaS or IaaS, PaaS, or IaaS (e.g. storage).
  • Nexus of file transfers is intra-Cloud but file transfers between Clouds and on premises are also required.
  • Data Integration Platform can access sources and targets that are on premises or in another Cloud.
  • Limited need for file transfers between on-premises sources and targets.
  • There is a desire to leverage Cloud for file transfer capabilities.
Solution This pattern employs a Data Integration Platform only in the Cloud. This provides the quickest way to deploy a Data Integration Platform providing file transfer capability between the Cloud and on premises, Cloud to Cloud, and on premises to on premises.
ConsequencesPros:
  • Provides the quickest file transfer approach.
  • Only one Data Integration Platform to manage.
  • Good solution when the majority of sources and/or targets are hosted in the same Cloud as the Data Integration Platform.
  • Only one Data Integration Platform to manage.
Cons:
  • Could cause heavy network traffic between Clouds if sources and targets are in Clouds other that the Cloud hosting the Data Integration Platform.
  • Limits the options for transferring files to targets in another Cloud.
  • Not a good solution if on premises to on premises file transfer is common.
  • Limits the options for connecting to the on-premises sources and targets.

Interaction Views

Upload On-Premises to Cloud

Transfer data (via file) from an on-premises application to SaaS or Cloud-based application.

Interactions

  1. The Data Integration Platform notifies the on-premises application to export a file. This notification could be time based or business event driven. The notification could also be replaced by a timed export from the on-premises application.
  2. The on-premises application exports data to a file.
  3. The Data Integration Platform retrieves the file.
  4. The Data Integration Platform processes the file and writes a file to Cloud-based storage.
  5. The Data Integration Platform notifies the Cloud-based application that the file is ready for processing. The Cloud-based application could be SaaS, or an application running on PaaS or IaaS. The notification could also be replaced by a timed import or polling from the Cloud-based application.
  6. The Cloud-based application reads the file and processes the contents.

Notes

  1. The on-premises file system must be accessible from the Public Cloud.
  2. For notification to work, the on-premises application must be accessible from the Public Cloud.
  3. The notifications to the applications are shown as a single call, but there could be multiple API calls to export or import the files.

Upload On Premises to Cloud with Agent

Transfer data (via file) from an on-premises application to SaaS or Cloud-based application.

Interactions

  1. The Data Integration Platform communicates with the on-premises Integration Agent to initiate the transfer. This notification could be time based or business event driven.
  2. The Integration Agent notifies the on-premises application to export a file. The notification could also be replaced by a timed export from the on-premises application.
  3. The on-premises application exports data to a file.
  4. The Integration Agent retrieves the file.
  5. The Integration Agent sends the file to the Data Integration Platform
  6. Then the Data Integration Platform processes the file and writes a file to Cloud-based storage.
  7. The Data Integration Platform notifies the Cloud-based application that the file is ready for processing. The Cloud-based application could be SaaS, or an application running on PaaS or IaaS. The notification could also be replaced by a timed import or polling from the Cloud-based application.
  8. The Cloud-based application reads the file and processes the contents.

Notes

  1. The Integration Agent provides secure communication between on premises and Public Cloud. The Integration Agent establishes an encrypted connection to the Data Integration Platform by calling into the Data Integration Platform. There is no need to “punch a hole” in the on-premises firewall.
  2. The notifications to the applications are shown as a single call, but there could be multiple API calls to export or import the files

Download Cloud to On Premises

Transfer data (via file) from a SaaS or Cloud-based application to an on-premises application.

Interactions

  1. The Data Integration Platform notifies the Cloud-based application to export a file. The Cloud-based application could be SaaS, or an application running on PaaS or IaaS. This notification could be time based or business event driven. The notification could also be replaced by a timed export from the Cloud-based application.
  2. The Cloud-based application exports data to a file.
  3. The Data Integration Platform retrieves the file.
  4. The Data Integration Platform processes the file and writes a file to on-premises storage.
  5. The Data Integration Platform notifies the on-premises application that the file is ready for processing. The notification could also be replaced by a timed import or polling from the on-premises application.
  6. The on-premises application reads the file and processes the contents.

Notes

  1. The on-premises file system must be accessible from the Public Cloud.
  2. For notification to work, the on-premises application must be accessible from the Public Cloud.
  3. The notifications to the applications are shown as a single call, but there could be multiple API calls to export or import the files.

Download Cloud to On Premises with Agent

Transfer data (via file) from a SaaS or Cloud-based application to an on-premises application.

Interactions

  1. The Data Integration Platform notifies the Cloud-based application to export a file. The Cloud-based application could be SaaS, or an application running on PaaS or IaaS. This notification could be time based or business event driven. The notification could also be replaced by a timed export from the Cloud-based application
  2. The Cloud-based application exports data to a file.
  3. The Data Integration Platform retrieves the file.
  4. The Data Integration Platform processes the file and sends the file to the on-premises Integration Agent.
  5. The Integration Agent writes the file to on-premises storage.
  6. The Data Integration Platform initiates a notification by calling the on-premises Integration Agent.
  7. The Integration Agent notifies the on-premises application that the file is ready for processing. The notification could also be replaced by a timed import or polling from the Cloud-based application.
  8. The on-premises application reads the file and processes the contents.

Notes

  1. The Integration Agent provides secure communication between on premises and Public Cloud. The Integration Agent establishes an encrypted connection to the Data Integration Platform by calling into the Data Integration Platform. There is no need to “punch a hole” in the on-premises firewall.
  2. The notifications to the applications are shown as a single call, but there could be multiple API calls to export or import the files.

Upload Cloud to Cloud

Transfer data (via file) from a SaaS or Cloud-based application to another SaaS or Cloud-based application.

Interactions

  1. The Data Integration Platform notifies the Cloud-based application to export a file. The Cloud-based application could be SaaS, or an application running on PaaS or IaaS. This notification could be time based or business event driven. The notification could also be replaced by a timed export from the Cloud-based application.
  2. The Cloud-based application exports data to a file.
  3. The Data Integration Platform retrieves the file.
  4. The Data Integration Platform processes the file and writes a file to a Cloud-based storage.
  5. The Data Integration Platform notifies the Cloud-based application that the file is ready for processing. The Cloud-based application could be SaaS, or an application running on PaaS or IaaS. The notification could also be replaced by a timed import or polling from the Cloud-based application.
  6. The Cloud-based application reads the file and processes the contents.

Notes

  1. The files must be accessible from the Public Cloud.
  2. The notifications to the applications are shown as a single call, but there could be multiple API calls to export or import the files.

Download Cloud to Cloud

Transfer data (via file) from a SaaS or Cloud-based application to another SaaS or Cloud-based application.

Interactions

  1. The Data Integration Platform notifies the Cloud-based application to export a file. The Cloud-based application could be SaaS, or an application running on PaaS or IaaS. This notification could be time based or business event driven. The notification could also be replaced by a timed export from the Cloud-based application.
  2. The Cloud-based application exports data to a file.
  3. The Data Integration Platform retrieves the file.
  4. The Data Integration Platform processes the file and writes a file to a Cloud-based storage.
  5. The Data Integration Platform notifies the Cloud-based application that the file is ready for processing. The Cloud-based application could be SaaS, or an application running on PaaS or IaaS. The notification could also be replaced by a timed import or polling from the Cloud-based application.
  6. The Cloud-based application reads the file and processes the contents.

Notes

  1. The files must be accessible from the Public Cloud.
  2. The notifications to the applications are shown as a single call, but there could be multiple API calls to export or import the files.