Skip to main content

One post tagged with "updates"

View All Tags

2021-18 Weekly Report

Hello everyone, here is the weekly report for AOS, range from 2021-05-06 to 2021-05-08.

Due to the International Workers' Day, we only have three workdays this week. However, we still made lots of effort and managed to get some results.

#

Appender 支持#

Appender 被用于将内容附加到文件尾部。

This week, @JinnyYi finished implement appender support in Implement Appender Support. Great work!

Idempotent Storager Delete Operation#

We use Delete to handle all object delete operations, but their behavior is not unified and well-defined.

So @Xuanwo made a proposal: AOS-46: Idempotent Storager Delete Operation for this, and @JinnyYi is now leading the implementation.

For more details, please refer to Implement AOS-46: Idempotent Storager Delete Operation.

Additional Error Specification#

In this week, @xxchan has finished his first proposal: AOS-47: Additional Error Specification in our team! Congratulations!

After a study of our existing implementations in different services, @xxchan did a great job for this proposal, which is designed to offer users unified experience when handle errors with go-storage.

For more details, please refer to Implement AOS-47: Additional Error Specification.

Service Registry Support#

For now, every service implement the function like:

func New(pairs ...typ.Pair) (typ.Servicer, typ.Storager, error) {}
func NewServicer(ps ...types.Pair) (types.Servicer, error) {}
func NewStorager(ps ...types.Pair) (types.Storager, error) {}

Users need to handle types by themselves.

So @Xuanwo made a proposal: AOS-48: Service Registry to refactor the functions which make it easier to init a Storager.

The proposal is still a draft and not implemented yet.

Add CreateDir Operation#

Applications need the ability to create a directory, and for now we did not offer a unified way to handle this.

So @Xuanwo made a proposal: AOS-49: Add CreateDir Operation to allow user create a directory in the same way.

The Direr interface has been added in go-storage, and more implementations in different services is now help wanted. You are welcome to make your own PR for this.

For more details, please refer to Implement AOS-49 Add CreateDir Operation.

社区#

开源软件供应链点亮计划 - 暑期2021#

So far, over 30 students have shown their interest in our projects. You are welcome to keep an eye on our forum: https://forum.aos.dev/, where all event-related announcements will be posted.

For more details, please refer to https://aos.dev/community/events/ospp-summer-2021.


2021-17 Weekly Report

Hello everyone, here is the weekly report for AOS, range from 2021-04-25 to 2021-04-30.

#

Appender 支持#

Appender 被用于将内容附加到文件尾部。

This week, we implement Appender for the services that support this feature. CommitAppend is introduced to finish an append procedure as service like dropbox needs a close to mark this appends process has been finished.

服务器端加密支持#

Server Side Encryption, a.k.a., SSE is a feature to encrypt data on server side, which is used by various applications.

@xxchan was responsible for completing the SSE support for the services this week, including Implement SSE support, SEE-S3 sample code and related documents. SSE blog shows what is SSE and how to use SSE in go-storage.

Error Handling#

Expected errors are errors that implementer expected. Those errors should be defined with enough comments and any changes to them should be documented.

AOS-11 has specified error handling, but our service is not strict with its requirements when implemented. To make error handling more user friendly, @xxchan is leading the support in Network/IO Error should be expected so that we can retry them.

Make Delete idempotent#

The current delete behavior should be different for fs and object storage services.

For dropbox alike storage services, delete a file that is uploading, such as Appender procedure unfinished, will returns an error, which is different with file system and object storage. So we need to unify the behavior.

Please keep an eye on our track issue: Make Delete idempotent.

应用#

dm#

dm is a data migration service and a long time effort.

To ensure that task recovery carried out according to the plan, even if it is interrupted for many times, Prnyself finished Proposal: Add support for stop and resume task this week and will start the demo.

Hi-Fi of dm is under designing. The Hi-Fi prototype interaction is on trial and welcome feedback.

社区#

开源软件供应链点亮计划 - 暑期2021#

Open Source Promotion Plan, a.k.a., OSPP, encourage everyone to pay close attention to open source software and open source community, aiming to cultivate and explore more outstanding developers.

This week we applied to participate in OSPP - Summer 2021 and posed the subject . @Xuanwo, @Prnyself and @xxchan will take part in the subject, and we will recruit contributors from student groups.

For details, please refer to https://aos.dev/community/events/ospp-summer-2021.


What's SSE and how to use it

Data protection includes protecting data while in transit (as it travels to and from the server) and at rest (while it is stored on disk). You can do the following to protect your data when using storage services:

  • Secure Socket Layer/Transport Layer Security (SSL/TLS): Protect data in transit.
  • Client-Side Encryption: You manage the encryption keys, encryption process, and related tools all yourself. This can protect data both in transit and at rest, and this is transparent to the server.
  • Server-Side Encryption: Request the server to encrypt your data before it is persisted to disk.

Recently, we implemented SSE support in go-storage, and let's talk about Server-Side Encryption (shorten to SSE) here.

What is SSE#

Generally, the server will use AES-256, a symmetric-key algorithm, to encrypt your data. (Exceptionally, Aliyun Object Storage supports SM-4.) AES-256 uses a 32-byte binary key, and the same key is used in encryption and decryption. You have three mutually exclusive options, depending on how you choose to manage the encryption keys.

  • SSE with Keys Managed by Cloud Service Provider
  • SSE with Customer Master Keys (CMKs) Stored in Key Management Service, often called SSE-KMS
  • SSE with Customer-Provided Keys, often called SSE-C

Note: not all service providers provide all of the above.

SSE with Keys Managed by Cloud Service Provider#

In this way, the server will manage the keys, and the encryption is transparent to you. Some service providers (e.g., Google Cloud Storage and Azure) set this as default behavior and require no setup or configuration (and no extra charge), so your data is protected silently at rest by SSE! The others require you to request SSE explicitly at the time of object creation by adding a header, or configure a bucket-level policy. Generally, reading an encrypted object requires no special configuration.

SSE with Customer Master Keys (CMKs) Stored in Key Management Service (SSE-KMS)#

In this way, you ask the service provider's Key Management Service to generate and store CMKs which can be managed by you (e.g., establishing and maintaining their key policies, enabling and disabling them). Then you can request SSE with the CMK (by specifying its key-id in the header) at the time of object creation, or configure a bucket-level policy. When reading an encrypted object, the client doesn't need to give the key-id, but must have access to the CMK.

SSE with Customer-Provided Keys (SSE-C)#

In this way, you create and store encryption keys yourself. Different from Client-Side Encryption, in SSE-C you send unencrypted data along with the encryption key to the server, and then the server uses the key to encrypt your data, but doesn't store the key. When reading an encrypted object, you will have to carry the encryption key again to let the server decrypt your data with the key. In other words, you manage the encryption keys, and the server manages the encryption process and related tools.

How to Use SSE in go-storage#

Since different services support different SSE options and have different behaviors, SSE-related pairs are considered service pairs. So you will have to check which options are provided for your specific service first. We listed supported options and related pairs in our service docs. You'd better first choose the SSE option you want to use, and then ignore other pairs in case of being confused with the usage of the pairs.

We recommend using SSE with DefaultPairs, where you only provide SSE-related options during NewStorager and can write the same logic for different services. Take s3 for example. The code snippets come from go-storage-example

func NewS3SseS3() (types.Storager, error) {
return s3.NewStorager(
append(S3Pairs(),
s3.WithDefaultStoragePairs(s3.DefaultStoragePairs{
Write: []types.Pair{
// Required, must be AES256
s3.WithServerSideEncryption(s3.ServerSideEncryptionAes256),
},
}))...,
)
}
func NewS3SseKms(keyId string, context map[string]string, bucketKeyEnabled bool) (types.Storager, error) {
ctx, _ := json.Marshal(context)
return s3.NewStorager(
append(S3Pairs(),
s3.WithDefaultStoragePairs(s3.DefaultStoragePairs{
Write: []types.Pair{
// Required, must be aws:kms
s3.WithServerSideEncryption(s3.ServerSideEncryptionAwsKms),
// Required
//
// Example value: 1234abcd-12ab-34cd-56ef-1234567890ab
s3.WithServerSideEncryptionAwsKmsKeyID(keyId),
// Optional
//
// An encryption context is an optional set of key-value pairs that can contain additional contextual information about the data. https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html#encryption-context
s3.WithServerSideEncryptionContext(base64.StdEncoding.EncodeToString(ctx)),
// Optional, S3 Bucket Key settings will be used if this is not specified.
//
// S3 Bucket Keys can reduce your AWS KMS request costs by decreasing the request traffic from Amazon S3 to AWS KMS. https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html#sse-kms-bucket-keys
s3.WithServerSideEncryptionBucketKeyEnabled(bucketKeyEnabled),
},
}))...,
)
}
func NewS3SseC(key []byte) (types.Storager, error) {
return s3.NewStorager(
append(S3Pairs(),
s3.WithDefaultStoragePairs(s3.DefaultStoragePairs{
Write: []types.Pair{
// Required, must be AES256
s3.WithServerSideEncryptionCustomerAlgorithm(s3.ServerSideEncryptionAes256),
// Required, your AES-256 key, a 32-byte binary value
s3.WithServerSideEncryptionCustomerKey(key),
},
// Now you have to provide customer key to read encrypted data
Read: []types.Pair{
// Required, must be AES256
s3.WithServerSideEncryptionCustomerAlgorithm(s3.ServerSideEncryptionAes256),
// Required, your AES-256 key, a 32-byte binary value
s3.WithServerSideEncryptionCustomerKey(key),
},
}))...,
)
}

2021-16 周报

大家好, 这里是 AOS的每周报告, 范围从 2021-04-19 至 2021-04-23。

首先,让我们欢迎新人加入: xxchan. 自从加入我们的团队以来,对我们的核心库 go-storage 做了许多改进。

#

fs.FS 和 http.FileSystem 支持#

fs.FS 是在 go 1.16 中引入的新的文件系统抽象。 http.FileSystemnet/http 中使用的文件系统抽象。

在 PR pkg: Implement wrapper for fs.FS and http.FileSystem 中, 我们引入了一个 fswrap 软件包来将我们的 Storager 转换为 fs.FShttp.FileSystem

func HttpFs(s types.Storager) http.FileSystem {}
func Fs(s types.Storager) fs.FS {}

我们现在可以直接在标准库中使用我们的 Storager

分段上传支持#

Multipart 被广泛用于对象存储服务中用来上传大文件(最大可达 50 TB)。 这个星期,我们添加了 Multiparter 的集成测试 go-integration-test。 我们将为支持这一功能的所有服务实现 Multiparter。

请关注我们的跟踪 Issue: Implement Multipart Support

Appender 支持#

Appender 被用于将内容附加到文件尾部。

本周,我们还在 go-integration-test 中为 Appender 增加了集成测试。 @JinnyYi 将在 PR 中 Implement Appender Support 实现Appender 支持。

服务器端加密支持#

服务器端加密,a.k.a..,SSE是在服务器端加密数据的一个功能。 为了满足 pingcapt/dumpling 的需要,我们花费了很多时间来确保正确地实现它们。

我们的新组员 @xxchan 正在领导该事项 Implement SSE support

社区#

本周 go-storage的 Star 是106,我们得到了我们的第一个 用户关于文档 的问题。

所以我们开始一个新的项目,名为 go-storage-example。我们将在这个仓库中添加更多的示例, 请在 讨论 中告诉我们你需要什么!


2021-15 Weekly Report

This is the weekly report for AOS, range from 2021-04-12 to 2021-04-16.

Application#

dm#

This week we started design review. https://www.figma.com/file/tZBW1fMDLlcdFpaHJYih9B.

And we pre-released: v0.2.0.

In this pre-release we:

  • add create task dialog in ui #40
  • refactor into grpc services #51

Library#

go-storage#

This week we fixed code generation by Increase stablility of generated code (#526)

And add UnimplementedStub to have forward compatible implementations #524, in which we introduced a breaking change by UnimplementedStub.

So we will make sure all service implement it after we realease go-storage v3.5.0 in next weeks. Roadmap

Create API Support#

We implement Create API for all services:

New release go-service-s3#

This week we released: v1.1.0

In this release we:

Community#

Working with pingcap/dumpling#

pingcap/dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).

dumpling's community has approved the proposal: Use aos-dev/go-storage to replace storage.ExternalStorage at 04/15's meeting

To meet the needs of dumpling's community, we will be focusing on the following:

Working with QingStor/qsftpd#

QingStor/qsftpd is a FTP server that persists all data to QingStor Object Storage.

It will adopt go-storage and transfer to AOS org.

New hands#

Welcome new hands on board!

2021-11 Weekly Report

This is the weekly report for AOS, range from 2021-03-15 to 2021-03-21.

Application#

dm#

This week we implemented the data flow related logic and next week we plan to implement the upper level business logic so that users can create and run a migration task.

Library#

go-storage#

This week we released: v3.4.2

In this release we implement multipart related fields like:

  • MultipartNumberMaximum
  • MultipartSizeMaximum
  • MultipartSizeMinimum

So we can get service restrictions on multipart.

Community#

We set up following sites:

We had a meeting of roadmap of 2021 Q2, slides: https://docs.google.com/presentation/d/1jC30T4S-eFrtIUoj9QStv8UDXq65UJaJmP0R2y-HqXo/