S3 next level: offload, backup plus also data-sharing, querying?

[hpvd]

There are several discussions/issues about utilizing S3 for nats/jetstream

• Support S3 API for JetStream #5486
• S3 Compatibility for NATS Object Store #4871

=> What do you think of taking S3 -support directly to the next level,
by making early design choices to utilize it for
offload and backup
but also for data-sharing and querying?

Data-sharing and quering (3rd party usage) could be made possible by

1. storing messages in S3 directly in the standard open tablet format
   (-> parquet, see https://duckdb.org/2025/01/22/parquet-encodings.html)

2. one could read these parquet files from multiple S3 buckets e.g. via duckdb
   https://duckdb.org/docs/guides/network_cloud_storage/s3_import.html

3. easily combine them e.g. using DuckDB
   https://duckdb.org/docs/data/multiple_files/overview.html#parquet

4. combining different schemas by position or name if needed
   https://duckdb.org/docs/data/multiple_files/combining_schemas

5. and security could be addressed when using standard parquet encryption (Parquet Modular Encryption) specification
   https://duckdb.org/docs/data/parquet/encryption.html

raw idea inspired by
https://streamnative.io/blog/stream-table-duality-and-the-vision-of-enabling-data-sharing

[ABFocke]

I really like the idea.
would be something similar to warpstream.

[hpvd]

if you like it, plz up-vote this topic :-)

[hpvd]

from #4871 (comment)

Most general approach (basic)

config:
for every subject, no matter of its use case (standard, system, KV, Object…)

connected external storage

• S3
• ...

S3-storage

• bucket name
• region
• ....

mode

• duplicate
• offload

offload threshold

• message age > [0..] seconds
• message quantity > [0..]
• subject size > [0.. ] byte

allowing to start with "0" may enable use-cases purely relying on S3 (for this subject)

[hpvd]

allowing to start with "0" may enable use-cases purely relying on S3 (for this subject)

-> for all use-cases where performance is not important but cost have to be as low as possible...

[ohanf]

allowing to start with "0" may enable use-cases purely relying on S3 (for this subject)

-> for all use-cases where performance is not important but cost have to be as low as possible...

I would not start with 0, especially if you are assuming it will be cheaper. Object storage (ie S3) is not a file system, notably you can not "append" to files once they have been uploaded. Providers also charge per operation, listing in particular can become quite expensive if you are not careful.

The overhead of writing a new file for every message would be non-viable: tons of tiny files, amplification of API operations (direct impact to cost) and writing parquet headers. Some amount of buffering would need to be done before writing, and you may want to consider "compaction" jobs even after files have been uploaded to further increase density and thus storage+cost efficiency. Using S3 for storage is becoming increasingly popular for live systems, especially in the obserability and ETL spaces, and they have all run into these problems and have similar solutions.

Look into technologies like Grafana's mimir and loki which are good Go based examples of storing live data in object storage. There is also the fledgling InfluxDB 3.0 (written in Rust) which has a similar approach. I believe there is even literature around how some Kafka connectors handle collecting messages into parquet and writing to S3 (although I don't have the link handy). I don't know how Kafka (and maybe RedPanda) does their own offloading, but that could be something to reference as well as I believe it has first class support.

[hpvd]

@ohanf many thanks for sharing this valuable knowledge incl examples and hints where to look into!
So far I have only experience using S3 for backups and file exchange...