TypedTables.Table as a sink

[non-Jedi]

I was playing around with using Transducers with TypedTables.jl. It works relatively well out of the box:

julia> t = Table(a=rand(1:100, 10), b=rand(10))
Table with 2 columns and 10 rows:
      a   b
    ┌──────────────
 1  │ 68  0.0588238
 2  │ 84  0.316647
 3  │ 35  0.619855
 4  │ 12  0.583985
 5  │ 23  0.155771
 6  │ 92  0.44839
 7  │ 13  0.547894
 8  │ 57  0.283057
 9  │ 81  0.361922
 10 │ 56  0.368723

julia> f = Filter(x -> iseven(x.a)) |> Map(x -> (a=x.a, b=2x.b))
Filter(Main.λ❓) |>
    Map(Main.λ❓)

julia> t2 = Table(collect(f, t))
Table with 2 columns and 5 rows:
     a   b
   ┌─────────────
 1 │ 68  0.117648
 2 │ 84  0.633293
 3 │ 12  1.16797
 4 │ 92  0.896779
 5 │ 56  0.737446

The one problem I see here is calling collect before the Table constructor causes unnecessary copying (creates a Vector{NamedTuple{(:a, :b),Tuple{Int64,Float64}}} which then has to be converted to a NamedTuple{(:a, :b),Tuple{Vector{Int64},Vector{Float64}}}). Using collect only works because Tables.jl defines a Vector{T} where {T<:NamedTuple} as a table.

Any ideas for making Transducers more general in this way to avoid unnecessary copying?

[non-Jedi]

I'm a dummy. copy! can be used directly for this. I thought Tables were meant to be wholly immutable, but they define both push! and empty!.

julia> t2 = copy!(f, similar(t, 0), t)
Table with 2 columns and 4 rows:
     a   b
   ┌─────────────
 1 │ 12  0.899638
 2 │ 96  1.48997
 3 │ 84  1.96983
 4 │ 82  0.385463

[non-Jedi]

Would it be worthwhile to define Base.copy(f::Transducer, x) = Base.copy!(f, similar(x, 0), x) to make this kind of thing more ergonomic?

[tkf]

copy(f::Transducer, x) sounds like a good idea. Another possible name is map. But whether or not it is the right thing to depend on what Base decides to do with map(f, dict): JuliaLang/julia#5794

BTW, I think copy!(f, similar(t, 0), t) would fail if you change row type in f (e.g., add a column). Ideally, something like

foldl(push!!, xf, input_table; init=Table())

should work. But Table() throws and push!! is not defined for Table.