Introduction to S3Proxy and SwiftProxy Andrew Gaul Bounce Storage 20 May 2015 Problem ● Applications written against single cloud provider, e.g., Amazon S3 ○ Either using REST API or vendor SDK ● Application → S3 protocol → Amazon ● What happens when you want to move to OpenStack Swift? ○ Rewrite application using another REST API or vendor SDK? Partial solution ● Portable, multi-cloud toolkits ○ fog, jclouds, libcloud, and pkgcloud ● Allow source code portability between providers ○ Amazon S3, EMC Atmos, Google Cloud Storage, Microsoft Azure, and OpenStack Swift ● Application → jclouds → any provider ● Requires modifying application S3Proxy and SwiftProxy ● Allow access to other object stores without rewriting your application ○ Instead you change the endpoint ● Provide S3 and Swift REST frontends ○ Separate server processes ● Apache jclouds provides many backends ● Application → S3Proxy → any provider Features ● ● ● ● Create, delete, and list buckets Put, get, remove, and list objects Multi-part upload Store and retrieve content and user metadata ● Server-side copy ● Multi-object delete How S3Proxy and SwiftProxy work ● S3 put object PUT /kitten.jpg HTTP/1.1 Host: photos.s3.amazonaws.com Authorization: AWS {account}:{signature} ● Apache jclouds put object blobStore.putBlob("photos", blob); ● Swift put object PUT /v1/{account}/photos/kitten.jpg HTTP/1.1 X-Auth-Token: {token} Custom middlewares ● Consider using S3Proxy and SwiftProxy without protocol translation ○ S3 application → S3Proxy → Amazon ● Having a proxy allows implementing custom middlewares ○ Authentication, compression, logging, QoS, quotas, tiering, etc. ● No application or provider support required Other uses ● Local testing with filesystem backend ○ Replace Amazon S3 ○ Replace Swift All In One ● Use S3-based FUSE file systems ○ s3fs, s3ql, s3backer, etc. ● Substitute for swift3 middleware on public providers, e.g., Rackspace CloudFiles ● Yours! Looking forward ● ● ● ● ● Object versioning Server side encryption Expand ACL support Partial Amazon Glacier support Improve test coverage ○ Ceph s3-tests ○ Swift integration tests Summary ● S3Proxy and SwiftProxy offer protocol translation and allow custom middlewares ● Apache-licensed, Java implementations ● https://github.com/bouncestorage/s3proxy ● https://github.com/bouncestorage/swiftproxy