Amazon S3 分布式存储的 python 接口实现

Amazon s3 是一种分布式的对象存储。用键值对的方式，来存储数据。其中，存入的所有数据都是一个对象（object），每一个对象都有一个键（key）存在。

具有非常方便的 web 访问接口，以及权限控制。

下面是具体读、写、判断三个接口的具体实现

1. 创建一个 bucket

class ImageFeatIO:
    __read_singleton = None
    __write_singleton = None
    __read_count = 0
    __write_count = 0

    def __init__(self):
        pass

    @staticmethod
    def get_write_instance():  #创建一个 bucket 的单例
        if ImageFeatIO.__write_singleton is None:
            paras = get_config('config')
            access_key = paras['access_key']
            secret_key = paras['secret_key']
            write_host = paras['file_write_host']
            conn = boto.connect_s3(
                aws_access_key_id=access_key,
                aws_secret_access_key=secret_key,
                host=write_host, is_secure=False,
                calling_format=boto.s3.connection.OrdinaryCallingFormat()
            )
            bucket_name = paras['bucket_name']
            bucket = conn.get_bucket(bucket_name)
            ImageFeatIO.__write_singleton = bucket
        return ImageFeatIO.__write_singleton

2. 写入文件

写入文件的时候，可以进行权限控制

官方说明如下：

1. Create a custom ACL that grants specific rights to specific users. At the moment, the users that are specified within grants have to be registered users of Amazon Web Services so this isn’t as useful or as general as it could be.
2. Use a “canned” access control policy. There are four canned policies defined:
1. private: Owner gets FULL_CONTROL. No one else has any access rights.
2. public-read: Owners gets FULL_CONTROL and the anonymous principal is granted READ access.
3. public-read-write: Owner gets FULL_CONTROL and the anonymous principal is granted READ and WRITE access.
4. authenticated-read: Owner gets FULL_CONTROL and any principal authenticated as a registered Amazon S3 user is granted READ access

def write_image_feature_to_file(id, imageFeaturestring):
    bucket = ImageFeatIO.get_write_instance()
    k = Key(bucket)
    k.key = id
    res = k.set_contents_from_string(imageFeaturestring)
    k.set_acl('authenticated-read')
    return res

3.读取（下载）文件

s3 提供了一种非常方便的 web 服务接口，可以从任何地方以 http 协议获取数据

def read_image_feature_from_file(id):
    url_head = ImageFeatIO.get_read_instance()
    url = url_head + '/' + id  #url= http://img.3weijia.com/bucket_name/key
    request = urllib2.Request(url=url)
    response = urllib2.urlopen(request, timeout=20)
    res_unicode = unicode(response.read())
    res = res_unicode.encode('unicode-escape').decode('string_escape')
    return res

4. 判断键值是否存在

def if_image_feature_exist(id):
    bucket = ImageFeatIO.get_write_instance()
    key = bucket.get_key(id)

    return key is not None

参考：

http://boto.readthedocs.io/en/latest/s3_tut.html

https://www.cnblogs.com/web424/p/6840207.html