A Study on using Google Cloud Storage with the S3 Compatibility API

Sections

1. Where things break ( TL;DR )
2. Server Side Config
3. Client Side Config
4. Code Samples

Where things break

ACLs have 2 Properties
1. Grantees ( Who gets access )
2. Scope ( How much access do they get )
ACLs are of 2 types
1. Canned ACLs ( Predefined Scopes & Grantees )
2. Custom ACLs ( Custom Scope & Grantees )
ACLs can be Applied at 2 Levels
1. Bucket
2. Object
| AWS Canned ACL            | GCP Canned ACL            | Applies |
|---------------------------|---------------------------|---------|
| private | private | Both |
| public-read | public-read | Both |
| public-read-write | public-read-write | Both |
| aws-exec-read | - | Both |
| authenticated-read | authenticated-read | Both |
| bucket-owner-read | bucket-owner-read | Object |
| bucket-owner-full-control | bucket-owner-full-control | Object |
| log-delivery-write | - | Bucket |
| - | project-private | Both |
2 Types of CORS 
1. Simple
2. Preflighted
| AWS CORS       | GCP CORS        |
|----------------|-----------------|
| AllowedHeaders | ResponseHeaders |
| AllowedMethods | Methods |
| AllowedOrigins | Origins |
| MaxAgeSeconds | MaxAgeSec |
ClientError: An error occurred (MalformedLifecycleConfiguration) when calling the PutBucketLifecycleConfiguration operation: The XML you provided was not well-formed or did not validate against our published schema.
ClientError: An error occurred (MalformedLifecycleConfiguration) when calling the PutBucketLifecycleConfiguration operation: The XML you provided was not well-formed or did not validate against our published schema.
2 Scenarios
1. Single Part Upload
2. Multi-Part Upload
Types of File Integrity Checks
1. CRC32C
2. MD5
3. ETags
|     | Single Part | Multi-Part                            |
|-----|-------------|---------------------------------------|
| GCP | MD5, CRC32C | b64 encoded CRC32C (CRC32Cs of Parts) |
| AWS | MD5 | Hex encoded MD5 ( MD5 of Parts ) |

Server Side Configuration

Client Side Configuration

s3_resource = boto3.resource(service_name='s3', endpoint_url=GCP_URL, aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY, region_name=GCP_REGION_NAME)
session = Session()
s3_session = session.resource(service_name='s3', endpoint_url=GCP_URL, aws_access_key_id=ACCESS_KEY, aws_secret_access_key=ACCESS_KEY, region_name=GCP_REGION_NAME)
s3_client = boto3.client('s3', endpoint_url= GCP_URL, aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY, region_name=GCP_REGION_NAME)
cat /etc/boto.cfg
[Credentials]
aws_access_key_id = ACCESS_KEY
aws_secret_access_key = SECRET_KEY
s3_host = storage.googleapis.com

Code Samples

1. Create Bucket
2. Multipart Upload
3. Signed URLs
4. Object Versioning Enable
import boto3s3 = boto3.resource(service_name='s3', endpoint_url=GCP_URL, aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY, region_name=GCP_REGION_NAME)
s3.create_bucket(Bucket = BUCKET_NAME, CreateBucketConfiguration= {'LocationConstraint': GCP_REGION_NAME })
import boto3
from boto3.s3.transfer import TransferConfig
s3 = boto3.resource(service_name='s3', endpoint_url=GCP_URL, aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY, region_name=GCP_REGION_NAME)
config = TransferConfig(multipart_threshold=1024 * 25,
max_concurrency=10,
multipart_chunksize=1024 * 25,
use_threads=True)
file_path = os.path.dirname(__file__) + FILE_NAMEs3_resource.Object(BUCKET_NAME, OBJECT_NAME).upload_file(file_path,
ExtraArgs={'ContentType': 'xxx/yyy'},
Config=config)
import boto3s3 = boto3.resource(service_name='s3', endpoint_url=GCP_URL, aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY, region_name=GCP_REGION_NAME)response = s3.meta.client.generate_presigned_url('get_object', Params={'Bucket': BUCKET_NAME, 'Key': OBJECT_NAME}, ExpiresIn=EXPIRATION)print(response.data)
import boto3s3 =boto3.resource(service_name='s3', endpoint_url=GCP_URL, aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY, region_name=GCP_REGION_NAME)versioning = s3.BucketVersioning(BUCKET_NAME)
versioning.enable()
print(versioning.status())

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store