Iceberg Ruby

Apache Iceberg for Ruby

🦆 Also check out SeaDuck

Installation

Add this line to your application’s Gemfile:

gem "iceberg"

Getting Started

Create a client for an Iceberg catalog

catalog = Iceberg::RestCatalog.new(uri: "http://localhost:8181", default_namespace: "main")

Create a namespace if needed

catalog.create_namespace("main")

Create a table

table =
  catalog.create_table("events") do |t|
    t.bigint "id"
    t.float "value"
  end

Append data

table.append([{"id" => 1, "value" => 3.0}, {"id" => 2, "value" => 4.0}])

Or with Polars

df = Polars::DataFrame.new({"id" => [1, 2], "value" => [3.0, 4.0]})
table = catalog.create_table("events", schema: df.schema)
table.append(df)

Or with Nanoarrow

arr =
  Nanoarrow::Array.new(
    [{"id" => 1, "value" => 3.0}, {"id" => 2, "value" => 4.0}],
    Nanoarrow.struct({"a" => Nanoarrow.int64, "b" => Nanoarrow.string})
  )
table = catalog.create_table("events", schema: arr.schema)
table.append(arr)

Load a table

table = catalog.load_table("events")

Query a table

table.to_a
# or
table.to_polars.collect
# or
table.scan.to_arrow

Catalog Types

REST

Iceberg::RestCatalog.new(
  uri: "http://localhost:8181"
)

S3 Tables

Iceberg::S3TablesCatalog.new(
  arn: "arn:aws:s3tables:..."
)

Glue

Iceberg::GlueCatalog.new(
  warehouse: "s3://my-bucket"
)

SQL

Iceberg::SqlCatalog.new(
  uri: "postgres://localhost:5432/iceberg",
  warehouse: "s3://my-bucket"
)

Memory

Iceberg::MemoryCatalog.new(
  warehouse: "/tmp/warehouse"
)

Namespaces

List namespaces

catalog.list_namespaces

Create a namespace

catalog.create_namespace("main")

Check if a namespace exists

catalog.namespace_exists?("main")

Get the properties of a namespace

catalog.namespace_properties("main")

Update a namespace

catalog.update_namespace("main", properties: {})

Drop a namespace

catalog.drop_namespace("main")

Tables

List tables

catalog.list_tables("main")

Create a table

catalog.create_table("events") do |t|
  t.integer "id"
  t.float "value"
end

Load a table

catalog.load_table("events")

Check if a table exists

catalog.table_exists?("events")

Rename a table

catalog.rename_table("events", "events2")

Register a table

catalog.register_table("events", "metadata.json")

Drop a table

catalog.drop_table("events")

Static Tables

Load a static table

Iceberg::StaticTable.new("metadata.json")

History

View the changelog

Contributing

Everyone is encouraged to help improve this project. Here are a few ways you can help:

• Report bugs
• Fix bugs and submit pull requests
• Write, clarify, or fix documentation
• Suggest or add new features

To get started with development:

git clone https://github.com/ankane/iceberg-ruby.git
cd iceberg-ruby
bundle install
bundle exec rake compile:dev

# memory catalog
bundle exec rake test:memory

# REST catalog
docker run -p 8181:8181 apache/iceberg-rest-fixture
bundle exec rake test:rest

# SQL catalog
createdb iceberg_ruby_test
bundle exec rake test:sql

# S3 Tables catalog
bundle exec rake test:s3tables

# Glue catalog
bundle exec rake test:glue