Skip to main content

2021-18 Weekly Report

Hello everyone, here is the weekly report for AOS, range from 2021-05-06 to 2021-05-08.

Due to the International Workers' Day, we only have three workdays this week. However, we still made lots of effort and managed to get some results.

Library#

Appender Support#

Appender is used to append content to the tail of file.

This week, @JinnyYi finished implement appender support in Implement Appender Support. Great work!

Idempotent Storager Delete Operation#

We use Delete to handle all object delete operations, but their behavior is not unified and well-defined.

So @Xuanwo made a proposal: AOS-46: Idempotent Storager Delete Operation for this, and @JinnyYi is now leading the implementation.

For more details, please refer to Implement AOS-46: Idempotent Storager Delete Operation.

Additional Error Specification#

In this week, @xxchan has finished his first proposal: AOS-47: Additional Error Specification in our team! Congratulations!

After a study of our existing implementations in different services, @xxchan did a great job for this proposal, which is designed to offer users unified experience when handle errors with go-storage.

For more details, please refer to Implement AOS-47: Additional Error Specification.

Service Registry Support#

For now, every service implement the function like:

func New(pairs ...typ.Pair) (typ.Servicer, typ.Storager, error) {}
func NewServicer(ps ...types.Pair) (types.Servicer, error) {}
func NewStorager(ps ...types.Pair) (types.Storager, error) {}

Users need to handle types by themselves.

So @Xuanwo made a proposal: AOS-48: Service Registry to refactor the functions which make it easier to init a Storager.

The proposal is still a draft and not implemented yet.

Add CreateDir Operation#

Applications need the ability to create a directory, and for now we did not offer a unified way to handle this.

So @Xuanwo made a proposal: AOS-49: Add CreateDir Operation to allow user create a directory in the same way.

The Direr interface has been added in go-storage, and more implementations in different services is now help wanted. You are welcome to make your own PR for this.

For more details, please refer to Implement AOS-49 Add CreateDir Operation.

Community#

Summer 2021 of Open Source Promotion Plan#

So far, over 30 students have shown their interest in our projects. You are welcome to keep an eye on our forum: https://forum.aos.dev/, where all event-related announcements will be posted.

For more details, please refer to https://aos.dev/community/events/ospp-summer-2021.


2021-17 Weekly Report

Hello everyone, here is the weekly report for AOS, range from 2021-04-25 to 2021-04-30.

Library#

Appender Support#

Appender is used to append content to the tail of file.

This week, we implement Appender for the services that support this feature. CommitAppend is introduced to finish an append procedure as service like dropbox needs a close to mark this appends process has been finished.

Server Side Encryption Support#

Server Side Encryption, a.k.a., SSE is a feature to encrypt data on server side, which is used by various applications.

@xxchan was responsible for completing the SSE support for the services this week, including Implement SSE support, SEE-S3 sample code and related documents. SSE blog shows what is SSE and how to use SSE in go-storage.

Error Handling#

Expected errors are errors that implementer expected. Those errors should be defined with enough comments and any changes to them should be documented.

AOS-11 has specified error handling, but our service is not strict with its requirements when implemented. To make error handling more user friendly, @xxchan is leading the support in Network/IO Error should be expected so that we can retry them.

Make Delete idempotent#

The current delete behavior should be different for fs and object storage services.

For dropbox alike storage services, delete a file that is uploading, such as Appender procedure unfinished, will returns an error, which is different with file system and object storage. So we need to unify the behavior.

Please keep an eye on our track issue: Make Delete idempotent.

Application#

dm#

dm is a data migration service and a long time effort.

To ensure that task recovery carried out according to the plan, even if it is interrupted for many times, Prnyself finished Proposal: Add support for stop and resume task this week and will start the demo.

Hi-Fi of dm is under designing. The Hi-Fi prototype interaction is on trial and welcome feedback.

Community#

Summer 2021 of Open Source Promotion Plan#

Open Source Promotion Plan, a.k.a., OSPP, encourage everyone to pay close attention to open source software and open source community, aiming to cultivate and explore more outstanding developers.

This week we applied to participate in OSPP - Summer 2021 and posed the subject . @Xuanwo, @Prnyself and @xxchan will take part in the subject, and we will recruit contributors from student groups.

For details, please refer to https://aos.dev/community/events/ospp-summer-2021.


What's SSE and how to use it

Data protection includes protecting data while in transit (as it travels to and from the server) and at rest (while it is stored on disk). You can do the following to protect your data when using storage services:

  • Secure Socket Layer/Transport Layer Security (SSL/TLS): Protect data in transit.
  • Client-Side Encryption: You manage the encryption keys, encryption process, and related tools all yourself. This can protect data both in transit and at rest, and this is transparent to the server.
  • Server-Side Encryption: Request the server to encrypt your data before it is persisted to disk.

Recently, we implemented SSE support in go-storage, and let's talk about Server-Side Encryption (shorten to SSE) here.

What is SSE#

Generally, the server will use AES-256, a symmetric-key algorithm, to encrypt your data. (Exceptionally, Aliyun Object Storage supports SM-4.) AES-256 uses a 32-byte binary key, and the same key is used in encryption and decryption. You have three mutually exclusive options, depending on how you choose to manage the encryption keys.

  • SSE with Keys Managed by Cloud Service Provider
  • SSE with Customer Master Keys (CMKs) Stored in Key Management Service, often called SSE-KMS
  • SSE with Customer-Provided Keys, often called SSE-C

Note: not all service providers provide all of the above.

SSE with Keys Managed by Cloud Service Provider#

In this way, the server will manage the keys, and the encryption is transparent to you. Some service providers (e.g., Google Cloud Storage and Azure) set this as default behavior and require no setup or configuration (and no extra charge), so your data is protected silently at rest by SSE! The others require you to request SSE explicitly at the time of object creation by adding a header, or configure a bucket-level policy. Generally, reading an encrypted object requires no special configuration.

SSE with Customer Master Keys (CMKs) Stored in Key Management Service (SSE-KMS)#

In this way, you ask the service provider's Key Management Service to generate and store CMKs which can be managed by you (e.g., establishing and maintaining their key policies, enabling and disabling them). Then you can request SSE with the CMK (by specifying its key-id in the header) at the time of object creation, or configure a bucket-level policy. When reading an encrypted object, the client doesn't need to give the key-id, but must have access to the CMK.

SSE with Customer-Provided Keys (SSE-C)#

In this way, you create and store encryption keys yourself. Different from Client-Side Encryption, in SSE-C you send unencrypted data along with the encryption key to the server, and then the server uses the key to encrypt your data, but doesn't store the key. When reading an encrypted object, you will have to carry the encryption key again to let the server decrypt your data with the key. In other words, you manage the encryption keys, and the server manages the encryption process and related tools.

How to Use SSE in go-storage#

Since different services support different SSE options and have different behaviors, SSE-related pairs are considered service pairs. So you will have to check which options are provided for your specific service first. We listed supported options and related pairs in our service docs. You'd better first choose the SSE option you want to use, and then ignore other pairs in case of being confused with the usage of the pairs.

We recommend using SSE with DefaultPairs, where you only provide SSE-related options during NewStorager and can write the same logic for different services. Take s3 for example. The code snippets come from go-storage-example

func NewS3SseS3() (types.Storager, error) {
return s3.NewStorager(
append(S3Pairs(),
s3.WithDefaultStoragePairs(s3.DefaultStoragePairs{
Write: []types.Pair{
// Required, must be AES256
s3.WithServerSideEncryption(s3.ServerSideEncryptionAes256),
},
}))...,
)
}
func NewS3SseKms(keyId string, context map[string]string, bucketKeyEnabled bool) (types.Storager, error) {
ctx, _ := json.Marshal(context)
return s3.NewStorager(
append(S3Pairs(),
s3.WithDefaultStoragePairs(s3.DefaultStoragePairs{
Write: []types.Pair{
// Required, must be aws:kms
s3.WithServerSideEncryption(s3.ServerSideEncryptionAwsKms),
// Required
//
// Example value: 1234abcd-12ab-34cd-56ef-1234567890ab
s3.WithServerSideEncryptionAwsKmsKeyID(keyId),
// Optional
//
// An encryption context is an optional set of key-value pairs that can contain additional contextual information about the data. https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html#encryption-context
s3.WithServerSideEncryptionContext(base64.StdEncoding.EncodeToString(ctx)),
// Optional, S3 Bucket Key settings will be used if this is not specified.
//
// S3 Bucket Keys can reduce your AWS KMS request costs by decreasing the request traffic from Amazon S3 to AWS KMS. https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html#sse-kms-bucket-keys
s3.WithServerSideEncryptionBucketKeyEnabled(bucketKeyEnabled),
},
}))...,
)
}
func NewS3SseC(key []byte) (types.Storager, error) {
return s3.NewStorager(
append(S3Pairs(),
s3.WithDefaultStoragePairs(s3.DefaultStoragePairs{
Write: []types.Pair{
// Required, must be AES256
s3.WithServerSideEncryptionCustomerAlgorithm(s3.ServerSideEncryptionAes256),
// Required, your AES-256 key, a 32-byte binary value
s3.WithServerSideEncryptionCustomerKey(key),
},
// Now you have to provide customer key to read encrypted data
Read: []types.Pair{
// Required, must be AES256
s3.WithServerSideEncryptionCustomerAlgorithm(s3.ServerSideEncryptionAes256),
// Required, your AES-256 key, a 32-byte binary value
s3.WithServerSideEncryptionCustomerKey(key),
},
}))...,
)
}

2021-16 Weekly Report

Hello everyone, here is the weekly report for AOS, range from 2021-04-19 to 2021-04-23.

Firstly, let's welcome the new hands: xxchan. He joined in our team since 2021-04-19 and have made many improvements of our core library go-storage.

Library#

fs.FS and http.FileSystem Support#

fs.FS is the new file system abstraction that introduced in go 1.16. And http.FileSystem is the file system abstraction used in net/http.

In pull request pkg: Implement wrapper for fs.FS and http.FileSystem, we introduce a fswrap package to convert our Storager to a fs.FS or http.FileSystem.

func HttpFs(s types.Storager) http.FileSystem {}
func Fs(s types.Storager) fs.FS {}

We can use our Storager in stdlib directly for now!

Multiparter Support#

Multipart is widely used in object storage services to upload huge files (up to 50TB). In this week, we added integration tests for Multiparter in go-integration-test. And we will implement multipart for all services that support this feature.

Keep an eye on our track issue: Implement Multipart Support.

Appender Support#

Appender is used to append content to the tail of file.

This week, we also implement the integration tests for Appender in go-integration-test. And @JinnyYi will implement appender support in Implement Appender Support.

Server Side Encryption Support#

Server Side Encryption, a.k.a., SSE is a feature to encrypt data on server side. In order to meet pingcap/dumpling needs, we spent a lot of time to make sure they implemented correctly.

Our new crew member @xxchan is leading the support in Implement SSE support.

Community#

This week go-storage's star is 106, and we got our first user's question about document.

So we start a new project called go-storage-example, we will add more examples in this repo, please let us know what you need in discussions!


2021-15 Weekly Report

This is the weekly report for AOS, range from 2021-04-12 to 2021-04-16.

Application#

dm#

This week we started design review. https://www.figma.com/file/tZBW1fMDLlcdFpaHJYih9B.

And we pre-released: v0.2.0.

In this pre-release we:

  • add create task dialog in ui #40
  • refactor into grpc services #51

Library#

go-storage#

This week we fixed code generation by Increase stablility of generated code (#526)

And add UnimplementedStub to have forward compatible implementations #524, in which we introduced a breaking change by UnimplementedStub.

So we will make sure all service implement it after we realease go-storage v3.5.0 in next weeks. Roadmap

Create API Support#

We implement Create API for all services:

New release go-service-s3#

This week we released: v1.1.0

In this release we:

Community#

Working with pingcap/dumpling#

pingcap/dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).

dumpling's community has approved the proposal: Use aos-dev/go-storage to replace storage.ExternalStorage at 04/15's meeting

To meet the needs of dumpling's community, we will be focusing on the following:

Working with QingStor/qsftpd#

QingStor/qsftpd is a FTP server that persists all data to QingStor Object Storage.

It will adopt go-storage and transfer to AOS org.

New hands#

Welcome new hands on board!

2021-11 Weekly Report

This is the weekly report for AOS, range from 2021-03-15 to 2021-03-21.

Application#

dm#

This week we implemented the data flow related logic and next week we plan to implement the upper level business logic so that users can create and run a migration task.

Library#

go-storage#

This week we released: v3.4.2

In this release we implement multipart related fields like:

  • MultipartNumberMaximum
  • MultipartSizeMaximum
  • MultipartSizeMinimum

So we can get service restrictions on multipart.

Community#

We set up following sites:

We had a meeting of roadmap of 2021 Q2, slides: https://docs.google.com/presentation/d/1jC30T4S-eFrtIUoj9QStv8UDXq65UJaJmP0R2y-HqXo/