Overhaul API (#5)

* The public constructor for the `KWayMerger` is now the new `kway_merge` function.
  `KWayMerger` is public, but unexported.
* Instead of the `F` parameter (and argument to its constructor), `kway_merge`
  uses the same ordering API as Base's sorting functions.
* `KWayMerger{T}` now iterates `@NamedTuple{from_iter::Int, value::T}`, to reduce
  the risk of users conflating the two elements of the tuple.

Author: jakobnissen

CHANGELOG.md

@@ -7,4 +7,15 @@ This project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html
 ## UNRELEASED
 * Add content here that have been merged, but not made it to a release yet.
 
+## [0.2.0]
+### Breaking changes
+* The public constructor for the `KWayMerger` is now the new `kway_merge` function.
+  `KWayMerger` is public, but unexported.
+* Instead of the `F` parameter (and argument to its constructor), `kway_merge`
+  uses the same ordering API as Base's sorting functions.
+* `KWayMerger{T}` now iterates `@NamedTuple{from_iter::Int, value::T}`, to reduce
+  the risk of users conflating the two elements of the tuple.
+
+
 ## [0.1.0]
+* Initial release

Project.toml

@@ -1,6 +1,6 @@
 name = "KWayMerges"
 uuid = "f29e91c7-719d-4dbc-8870-0ce36bf055b7"
-version = "0.1.0"
+version = "0.2.0"
 authors = ["Jakob Nybo Nissen <[email protected]>"]
 
 [compat]

README.md

@@ -6,56 +6,62 @@
 
 Implementation of k-way merge.
 
-This package implements the `KWayMerger` type.
-It is a stateful, lazy iterator of the elements in an iterator of iterators.
-The elements of the inner iterators will be yielded in an order given by a predicate optionally passed to `KWayMerger` (default: `isless`).
-Therefore, if the inner iterators are sorted by the predicate, the output of the `KWayMerger` is also guaranteed to be sorted.
+This package exports the function `kway_merge`.
+It constructs a `KWayMerger` - a stateful, lazy iterator of the elements in an iterator of iterators.
+The elements of the inner iterators will be yielded in order, as specified by the optional ordering (default: `Forward`).
+Therefore, if the inner iterators are sorted by the order, the yielded elements of the `KWayMerger` is also guaranteed to be sorted.
 
-The primary purpose of `KWayMerger` is to efficiently merge N sorted iterables into one sorted stream.
+The primary purpose of `kway_merge` is to efficiently merge N sorted iterables into one sorted stream.
 
-The iterator yields `(i::Int, x)` tuples, where `x` is the next element of one of the iterators, and `i` is the 1-based index of the iterator that yielded `x`:
+The iterator yields `@NamedTuple{from_iter::Int, value::T}`, where the value field has the next element of one of the iterators, and the from_iter field contains the 1-based index of the iterator that yielded the value:
 
 ```julia
-julia> it = KWayMerger([[2, 3], [1, 4]]);
+julia> it = kway_merge([[2, 3], [1, 4]]);
 
 julia> first(it)
-(2, 1)
+(from_iter = 2, value = 1)
 
-julia> println(collect(it))
+julia> println(map(Tuple, it))
 [(1, 2), (1, 3), (2, 4)]
 ```
 
 The function `peek` can be used to check the next element without advancing the iterator:
 
 ```julia
-julia> it = KWayMerger([1]);
+julia> it = kway_merge([1]);
 
 julia> peek(it)
-(1, 1)
+(from_iter = 1, value = 1)
 
 julia> first(it)
-(1, 1)
+(from_iter = 1, value = 1)
 
 julia> peek(it) === nothing
 true
 ```
 
 ## Documentation
-This package's public functionality are the `KWayMerger` type, and its `Base.peek` method.
+This package's public functionality are the `kway_merge` function, the (unexported) `KWayMerger` type, and its `Base.peek` method.
 See their docstrings for more details.
 
 ## Performance
-When merging I iterables with a total length of N:
+When merging I iterables:
 * A `KWayMerger` allocates O(I) space upon construction 
 * Producing each element takes O(log(I)) time
 
-Therefore, merging I sorted iterables with N total elements using a KWayMerger therefore takes O(N * log(I)) time.
-It is generally faster than flattening the iterators and sorting, when I << N.
+Therefore, merging I sorted iterables with N total elements using `kway_merge` takes O(N * log(I)) time.
+This is similar to the O(N * log(N)) time taken for comparison-based sorts.
+That's no co-incidence: One can take a list with N elements, separate it into N 1-element lists, then merge them with a kway-merge. That is a variant of merge sort.
+
+However, compared to a comparison-based sort like quicksort, using a kway merge has the following differences:
+* Usually, we have I << N, and therefore, kway merge is usually faster.
+* For large I, quicksort is faster in practice because its overhead per element is smaller.
+
 Note that Julia uses radix sort for integers, which sorts in O(N), and therefore usually beats a k-way merge.
 
 ## Contributing
 We appreciate contributions from users including reporting bugs, fixing
-issues, improving performance and adding new features.
+issues, improving performance and adding new fea oftentures.
 
 Take a look at the [contributing files](https://github.com/BioJulia/Contributing)
 detailed contributor and maintainer guidelines, and code of conduct.

checklist.md

@@ -1,135 +1,170 @@
 module KWayMerges
 
-export KWayMerger
+using Base.Order: Ordering, Forward, ord, lt
+
+export kway_merge
+
+@static if VERSION >= v"1.11.0"
+    eval(Meta.parse("public KWayMerger"))
+end
 
 include("heap.jl")
 
 """
-    KWayMerger{T, I, F}(f::F, iterators)
-    KWayMerger{T, I}(iterators)
-    KWayMerger(f, iterators)
-    KWayMerger(iterators)
+    KWayMerger{T, I, O, S}
+
+Stateful iterator of a k-way merge of multiple iterators of the same type.
+Constructed using [`kway_merge`](@ref).
+
+The type parameters are:
+* `T`: Element type of iterators
+* `I`: Iterator type
+* `O`: Ordering, subtype of `Base.Ordering`
+* `S`: Type of state of iterators
+"""
+struct KWayMerger{T, I, O <: Base.Ordering, S}
+    ordering::O
+    iterators::Vector{I}
+    states::Vector{S}
+    heap::Vector{@NamedTuple{from_iter::Int, value::T}}
+end
+
+"""
+    kway_merge( 
+        iterators;
+        lt=isless,
+        by=identity,
+        rev::Bool=false,
+        order::Base.Order.Ordering=Base.Order.Forward
+    )
+    kway_merge(::Type{T}, ::Type{T}, iterators; kwargs...)
+    kway_merge(::Type{T}, ::Type{T}, ordering::Ordering, iterators)
 
 Create a stateful iterator which does a k-way merge between multiple
 iterators of the same type.
 
-This iterator yields `(index::Int, x::T)` elements, where `x` is the next element from
-one of the iterators, and `index` is the 1-based index of the iterator that yielded `x`.
-The elements `x` are chosen from among the iterators such that, among all elements which
-are the next element of the iterators, the element is chosen which is the smallest
-according to the predicate `f::F`, which defaults to `isless`.
-
-This implies that if all iterators are sorted by `f`, the yielded will be in sorted
-order.
+This iterator yields `@NamedTuple{from_iter::Int, value::T}` elements, where `value` is the
+next element from one of the iterators, and `from_iter` is the 1-based index of the iterator
+that yielded `value`.
+The element `value` is chosen among the iterators such that, among all elements which
+are the next element of the iterators, the element is chosen which is the first
+according to the ordering.
+This implies that if all iterators are sorted by `f`, the yielded will be in sorted order.
 Hence, a `KWayMerger` is typically used to combined multiple sorted arrays
 into one sorted array.
 
+The ordering is given by the keywords `by`, `lt`, `rev` and `order` - these are the
+same as for `Base.sort!`.
+
+
 # Examples
 ```jldoctest
 julia> arrs = [[1,6], [2], [5,7], [3,4,8]];
 
-julia> it = KWayMerger(arrs);
+julia> it = kway_merge(arrs);
+
+julia> first(it, 2)
+2-element Vector{@NamedTuple{from_iter::Int64, value::Int64}}:
+ (from_iter = 1, value = 1)
+ (from_iter = 2, value = 2)
 
-julia> print(collect(it))
-[(1, 1), (2, 2), (4, 3), (4, 4), (3, 5), (1, 6), (3, 7), (4, 8)]
+julia> print(map(Tuple, it))
+[(4, 3), (4, 4), (3, 5), (1, 6), (3, 7), (4, 8)]
 ```
 
 # Extended help
-The type parameters are:
-* `F`: Type of function used to compare the elements. It defaults
-  to `typeof(Base.isless)`
-* `T`: Element type of iterators
-* `I`: Iterator type
-* `S`: Type of state of iterators
-
 All iterators must be of the same type. For the constructors which don't pass
 in `T` and `I` explicitly, `Base.eltype` is used
 to determine them; since its default implementation
 returns `Any`, explicitly passing them may be needed for good performance for some
 iterators.
 
 `S` is derived automatically, but this must be a fixed type;
-iterators that use states of multiple different types may
-not be supported by `KWayMerger`.
+iterators that use states of multiple different types during iteration may
+not be supported.
 """
-struct KWayMerger{T, I, F, S}
-    f::F
-    iterators::Vector{I}
-    states::Vector{S}
-    heap::Vector{Tuple{Int, T}}
-end
-
-function KWayMerger{T, I, F}(f::F, iterators) where {T, I, F}
+function kway_merge(::Type{T}, ::Type{I}, ordering::O, iterators) where {T, I, O}
     iters = vec(collect(iterators))
     states = nothing
-    things = Tuple{Int, T}[]
+    things = @NamedTuple{from_iter::Int, value::T}[]
     for i in eachindex(iters)
         it = iterate(iters[i])
         isnothing(it) && continue
-        (thing::T, state) = it
+        (value::T, state) = it
         if isnothing(states)
             states = Vector{typeof(state)}(undef, length(iters))
         end
-        push!(things, (i, thing))
+        push!(things, (; from_iter = i, value))
         states[i] = state
     end
-    heapify!(f, things)
+    heapify!(ordering, things)
     states = if isnothing(states)
         Vector{Union{}}(undef, length(iters))
     else
         states
     end
-    return KWayMerger{T, I, F, eltype(states)}(f, iters, states, things)
+    return KWayMerger{T, I, O, eltype(states)}(ordering, iters, states, things)
 end
 
-function KWayMerger{T, I}(iterators) where {T, I}
-    return KWayMerger{T, I, typeof(isless)}(isless, iterators)
+function kway_merge(
+        ::Type{T},
+        ::Type{I},
+        iterators;
+        lt = isless,
+        by = identity,
+        rev::Bool = false,
+        order::Base.Ordering = Forward,
+    ) where {T, I}
+    ordering = ord(lt, by, rev, order)
+    return kway_merge(T, I, ordering, iterators)
 end
 
-KWayMerger(iterators) = KWayMerger(isless, iterators)
-
-function KWayMerger(f::F, iterators) where {F}
+function kway_merge(iterators; kwargs...)
     I = eltype(typeof(iterators))
     T = eltype(I)
-    return KWayMerger{T, I, F}(f, iterators)
+    return kway_merge(T, I, iterators; kwargs...)
 end
 
 # We could technically know this, but KWayMerger is stateful, and
 # Julia's iterator length works badly with stateful iterators.
 Base.IteratorSize(::Type{<:KWayMerger}) = Base.SizeUnknown()
-Base.eltype(::Type{<:KWayMerger{T}}) where {T} = Tuple{Int, T}
+Base.eltype(::Type{<:KWayMerger{T}}) where {T} = @NamedTuple{from_iter::Int, value::T}
 
 function Base.iterate(x::KWayMerger, ::Nothing = nothing)
     isempty(x.heap) && return nothing
-    (i, item) = @inbounds x.heap[1]
-    iterator = @inbounds x.iterators[i]
-    state = @inbounds x.states[i]
+    top = @inbounds x.heap[1]
+    iterator = @inbounds x.iterators[top.from_iter]
+    state = @inbounds x.states[top.from_iter]
     it = iterate(iterator, state)
     if it === nothing
-        @inbounds heappop!(x.f, x.heap)
+        @inbounds heappop!(x.ordering, x.heap)
     else
         (new_item, new_state) = it
-        @inbounds x.states[i] = new_state
-        @inbounds heapreplace!(x.f, x.heap, (i, new_item))
+        @inbounds x.states[top.from_iter] = new_state
+        @inbounds heapreplace!(
+            x.ordering,
+            x.heap,
+            (; from_iter = top.from_iter, value = new_item)
+        )
     end
-    return ((i, item), nothing)
+    return (top, nothing)
 end
 
 Base.isempty(x::KWayMerger) = isempty(x.heap)
 Base.isdone(x::KWayMerger) = isempty(x.heap)
 
 """
-    peek(x::KWayMerger{T})::Union{Tuple{Int, T}, Nothing}
+    peek(x::KWayMerger{T})::Union{@NamedTuple{from_iter::Int, value::T}, Nothing}
 
 Get the first element of `x` without advancing the iterator, or `nothing` if the
 iterator is empty.
 
 # Examples
 ```jldoctest
-julia> it = KWayMerger([[3, 4], [2, 7]]);
+julia> it = kway_merge([[3, 4], [2, 7]]);
 
 julia> peek(it)
-(2, 2)
+(from_iter = 2, value = 2)
 
 julia> collect(it); # exhaust stateful iterator