Cannot deserialize nested schema after adding optional field

[ericphanson]

This is very similar to #94 but less scary because it throws an error (and this is actually how I realized that bug existed).

using Legolas
using Legolas: @schema, @version

@schema "test.issue" Issue

@version IssueV1 begin
    a::Int
end


@schema "test.parent" Parent
@version ParentV1 begin
    x::IssueV1
end


table = [ParentV1(; x=IssueV1(; a=1))]
Legolas.write("issue.arrow",table,  ParentV1SchemaVersion())

# New Julia session

using Legolas
using Legolas: @schema, @version

@schema "test.issue" Issue

@version IssueV1 begin
    a::Int
    b::Union{Missing, Int}
end


@schema "test.parent" Parent
@version ParentV1 begin
    x::IssueV1
end

Legolas.read("issue.arrow").x[1]

throws

ERROR: MethodError: no method matching get(::Int64, ::Symbol, ::Missing)

Closest candidates are:
  get(::Union{Tables.AbstractColumns, Tables.AbstractRow}, ::Union{Integer, Symbol}, ::Any)
   @ Tables ~/.julia/packages/Tables/AcRIE/src/Tables.jl:193
  get(::IdDict{K, V}, ::Any, ::Any) where {K, V}
   @ Base iddict.jl:101
  get(::Base.ImmutableDict, ::Any, ::Any)
   @ Base dict.jl:808
  ...

Stacktrace:
 [1] IssueV1(row::Int64)
   @ Main ~/Legolas.jl/src/schemas.jl:601
 [2] fromarrow(#unused#::Type{IssueV1}, x::Int64)
   @ ArrowTypes ~/.julia/packages/ArrowTypes/aTFES/src/ArrowTypes.jl:168
 [3] getindex(s::Arrow.Struct{IssueV1, Tuple{Arrow.Primitive{Int64, Vector{Int64}}}}, i::Int64)
   @ Arrow ~/.julia/packages/Arrow/ZPy8W/src/arraytypes/struct.jl:49
 [4] top-level scope
   @ REPL[9]:1
 [5] top-level scope
   @ ~/.julia/juliaup/julia-1.9.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.9/REPL/src/REPL.jl:1416

Again the issue is the fromarrow definition in

Legolas.jl/src/schemas.jl

Line 649 in b86444c

$Arrow.ArrowTypes.fromarrow(::Type{<:$R}, $(keys(record_fields)...)) = $R(; $(keys(record_fields)...))

which expects exactly the current fields to be present to be deserialized (even optional ones).

We can change that line to

        $Arrow.ArrowTypes.fromarrow(::Type{<:$R}, $(field_kwargs...)) = $R(; $(keys(record_fields)...))

which will work as long as new optional schemas are always added to the end of the list of columns (and the order doesn't change). This works by changing the signature to make those positional arguments optional, of the form a=missing, b=missing, etc. I'm not sure if that is acceptable or not, because it relies on the ordering. Xref #94

[ericphanson]

One workaround here is to add new optional fields by declaring a whole new version of the nested schema, i.e.

@version IssueV2 begin
    a::Int
    b::Union{Missing, Int}
end

and add

Legolas.accepted_field_type(::ParentV1SchemaVersion, ::Type{IssueV1}) = Union{<:IssueV1, <:IssueV2}

to allow old parent tables to continue to validate, and update the field in the parent constructor:

@version ParentV1 begin
    x::IssueV2 = IssueV2(x)
end

[jrevels]

just calling out the example in the OP is "not supported" since adding the b::Union{Missing,Int} field would necessitate a version bump to Issue, and then you enter the regime described in your second comment where things work fine.

Adding b::Any without bumping the Issue version should indeed work, though. But probably a low priority bug in practice given that I doubt this use case is super common