Feature request: add support for bucket related operations

[dentiny]

Hi team, I'm wondering if we could add bucket related operations? For example, creation, deletion, list, etc.
I search in the github issues / PRs / keywords, for example `git grep bucket | grep create", and cannot find any clue.

Context:

• I'm using iceberg-rust to work on my project, within which there's FileIO to manage file-oriented IO operations
• But it doesn't support directory-related operations, and at the moment there's no plan and motivation to enhance, see discussion thread here: Ideas: add directory support for `FileIO` iceberg-rust#1246
• To test iceberg related functionalities, I need to (1) use opendal for necessary directory-related operations; and (2) s3 client for s3 / minio related bucket related operations => 3 IO related libraries => 3 different set of APIs and configurations

In my opinion, it would be nice if opendal could provide a unified interface for all IO operations;
as of now, I think the library abstraction layer is file, directory and object; would appreciate if we could add bucket into consideration also.

Let me know if I mis-understand, thank you for the help and happy to discuss more about it!

[Xuanwo]

Hi team, I'm wondering if we could add bucket related operations? For example, creation, deletion, list, etc.

Personally, I don't think this falls within the scope of opendal, as opendal mainly focuses on data plane operations rather than management plane operations.

To me, it's impossible to create a portable API for management across different storage services. Data plane operations, such as read, write, and list, are generally similar. But management functions vary greatly, including policies, ACLs, lifecycle management, website hosting, and more.

To test iceberg related functionalities, I need to (1) use opendal for necessary directory-related operations; and (2) s3 client for s3 / minio related bucket related operations => 3 IO related libraries => 3 different set of APIs and configurations

Most of the time, we treat S3 buckets as relatively static resources. Therefore, we can create a bucket ahead of time and use different prefixes as needed. I’m happy to help implement this change on the Iceberg side so we can consistently use the same bucket for testing, rather than attempting to create one programmatically.

[dentiny]

Thanks for the answer! Kinda expected it :(