Federated File Transfer

Copy a file from a Cloud-based source to a Cloud-based target using Data Integration Platforms in both Clouds.

Pattern Definition

Problem How do I transfer a data file from a cloud-based source to a target in another Cloud where both Clouds use a Data Integration Platform?
Context This pattern is applicable when there is a desire to transfer files from a Cloud-based source to a Cloud-based target. The source and target may be SaaS, PaaS, or IaaS.
Forces
  • All file transfers are between Cloud-based sources and targets.
  • No need for file transfers to or from on-premises.s.
  • Applications may be SaaS or can be applications hosted on PaaS or IaaS.
  • Nexus of file transfers is intra-Cloud but file transfers between Clouds are also required.
  • Data Integration Platform may provide access to sources or targets that would not otherwise be accessible.
Solution The pattern employs one Data Integration Platform in each Cloud that has sources or targets.
ConsequencesPros:
  • Provides the greatest flexibility
  • Encapsulates intra-Cloud file transfers reducing Cloud-to-Cloud network traffic.
Cons:
  • Increased complexity compared to having only one Data Integration Platform
  • Multiple Data Integration Platforms to manage.
  • Different skill sets for different Data Integration Platforms are required.
  • Potential performance impact due to file transfers going through two Data Integration Platforms.

Interaction View

Upload Cloud to Cloud

Transfer data (via file) from a SaaS or Cloud-based application to SaaS or Cloud-based application in another cloud

Interactions

  1. The Data Integration Platform notifies the Cloud-based application to export a file. The Cloud-based application could be SaaS or an application running on PaaS or IaaS. This notification could be time based or business event driven. The notification could also be replaced by a timed export from the Cloud-based application.
  2. The Cloud-based application exports data to a file.
  3. The Data Integration Platform retrieves the file.
  4. The Data Integration Platform processes the file and sends the file to the other Data Integration Platform.
  5. The Data Integration Platform processes the file and writes a file to a Cloud-based storage.
  6. The Data Integration Platform notifies the Cloud-based application that the file is ready for processing. The Cloud-based application could be SaaS, or an application running on PaaS or IaaS. The notification could also be replaced by a timed import or polling from the Cloud-based application.
  7. The Cloud-based application reads the file and processes the contents.

Notes

  1. The notifications to the applications are shown as a single call, but there could be multiple API calls to export or import the files.
  2. The Data Integration Platforms tend to support:
    • REST with HTTP Basic or OAuth security, or
    • SOAP with WS Security using username and timestamp.

Upload By Reference Cloud to Cloud

Transfer data (via file) from a SaaS or Cloud-based application to SaaS or Cloud-based application in another cloud by passing the file by reference. 

Interactions

  1. The Data Integration Platform notifies the Cloud-based application to export a file. The Cloud-based application could be SaaS or an application running on PaaS or IaaS. This notification could be time based or business event driven. The notification could also be replaced by a timed export from the Cloud-based application.
  2. The Cloud-based application exports data to a file.
  3. The Data Integration Platform creates a handle for the file.
  4. The Data Integration Platform passes the file handle to the Data Integration Platform.
  5. The Data Integration Platform reads the file from Cloud-based storage.
  6. The Data Integration Platform processes the file and writes a file to a Cloud-based storage.
  7. The Data Integration Platform notifies the Cloud-based application that the file is ready for processing. The Cloud-based application could be SaaS, or an application running on PaaS or IaaS. The notification could also be replaced by a timed import or polling from the Cloud-based application.
  8. The Cloud-based application reads the file and processes the contents.

Notes

  1. The Data Integration Platform must support pass by reference file transfers.
  2. The notifications to the applications are shown as a single call, but there could be multiple API calls to export or import the files.
  3. The Data Integration Platform tend to support:
    • REST with HTTP Basic or OAuth security, or
    • SOAP with WS Security using username and timestamp.

Download Cloud to Cloud

Transfer data (via file) from SaaS or Cloud-based application to a SaaS or Cloud-based application in another cloud.

Interactions

  1. The  Data Integration Platform notifies the Cloud-based application to export a file. The Cloud-based application could be SaaS, or an application running on PaaS or IaaS. This notification could be time based or business event driven. The notification could also be replaced by a timed export from the Cloud-based application.
  2. The Cloud-based application exports data to a file.
  3. The Data Integration Platform retrieves the file.
  4. The Data Integration Platform processes the file and sends the file to Data Integration Platform in another Cloud.
  5. The Data Integration Platform processes the file and writes a file to a Cloud-based storage.
  6. The Data Integration Platform notifies the Cloud-based application that the file is ready for processing. The Cloud-based application could be SaaS or an application running on PaaS or IaaS. The notification could also be replaced by a timed import or polling from the Cloud-based application.
  7. The Cloud-based application reads the file and processes the contents.

Notes

  1. The notifications to the applications are shown as a single call, but there could be multiple API calls to export or import the files.
  2. The Data Integration Platforms tend to support:
    • REST with HTTP Basic or OAuth security, or
    • SOAP with WS Security using username and timestamp.

Download By Reference Cloud to Cloud

Transfer data (via file) from an SaaS or Cloud-based application to a SaaS or Cloud-based application in another cloud passing the file by reference. 

Interactions

  1. The Data Integration Platform notifies the Cloud-based application to export a file. The Cloud-based application could be SaaS, or an application running on PaaS or IaaS. This notification could be time based or business event driven. The notification could also be replaced by a timed export from the Cloud-based application.
  2. The Cloud-based application exports data to a file.
  3. The Data Integration Platform creates a handle for the file.
  4. The Data Integration Platform passes the file handle to the Data Integration Platform.
  5. The Data Integration Platform reads the file from Cloud-based storage.
  6. The Data Integration Platform processes the file and writes a file to a Cloud-based storage.
  7. The Data Integration Platform notifies the Cloud-based application that the file is ready for processing. The Cloud-based application could be SaaS, or an application running on PaaS or IaaS. The notification could also be replaced by a timed import or polling from the Cloud-based application.
  8. The Cloud-based application reads the file and processes the contents.

Notes

  1. The Data Integration Platform must support pass by reference file transfers.
  2. The notifications to the applications are shown as a single call, but there could be multiple API calls to export or import the files.
  3. The Data Integration Platforms tend to support:
    • REST with HTTP Basic or OAuth security, or
    • SOAP with WS Security using username and timestamp.