LEGO® for the Cloud: Architecting Complex Solutions with Simple AWS Patterns (Part-5 covering Patterns for Uploading Data)

Patterns for Uploading Data

Write Proxy Pattern

High speed uploading to internet storage

Problem : Performance problem when writing large amounts of data to Internet Storage as it supports only HTTP.

Solution : Implement upload proxy pattern supporting UDP and Compression / de-compression logic

Note: Refer S3 Transfer acceleration for better option.

Legacy; better options available now

Storage Index Pattern

Increasing the efficiency of Internet Storage

Problem : High-speed search functions are not provided, requiring you to take some action on the application side when retrieving a data table for a specific user or when retrieving data for a specific range of dates.

Solution : Create and store meta-data at the same time into a KVS that has excellent searching performance, and then that data is used as an index.

NOTE: AWS Athena (serverless) for analytics

Direct object Upload Pattern

Simplifying the upload procedure

Problem : In some cases, the upload process involves a high server-side load (particularly in terms of the network load), requiring a virtual server that is dedicated to uploads

Solution : Leave the upload process to the Internet storage.

NOTE:

S3 is strongly consistent, now