Replace all full(X) calls with convert(Array, X) and migrate tests

[Sacha0]

This PR reconstitutes and rebases #17079. #17079 was reverted in #17564 due to JuliaStats/DataArrays.jl#208; the series of PRs associated with #17660 should prevent this PR from causing JuliaStats/DataArrays.jl#208 to reoccur (see also #12153 (comment)). Best!

[nalimilan]

Any reason not to use the compact Array(A) form instead?

[Sacha0]

Any reason not to use the compact Array(A) form instead?

Good catch! No reason. I wrote convert(Array, by force of habit. Array( seems much better. Will fix. Thanks!

[Sacha0]

Replaced convert(Array, with Array( everywhere relevant. Also fixed a rebase mistake, so this PR now passes CI. (The lone AV x86_64 failure was a git-related error in the pkg tests.) Best!

[tkelman]

which should these be linking to?

[Sacha0]

convert? Would ...may be converted into a regular matrix with [convert(Array, )](:func:convert) (or Array() for short)... be better? Thoughts? Thanks!

[tkelman]

wouldn't zeros(size(A)) be equivalent and allocate less?

[Sacha0]

Good call. Fixed on push. Thanks!

[tkelman]

excess space in Array( A)

[Sacha0]

Good catch. Fixed on push. Thanks!

[tkelman]

what's going on here?

[kshyatt]

Are you asking why we have two full calls? It's because the first full goes LDLt -> Tridiagonal, and the second goes Tridiagonal -> Matrix

[tkelman]

ah right my "use a different name for..." issue. what type does Array(::LDLt) return?

[Sacha0]

Derp is what's going on here. Fixed on push. (Array(::LDLt) returns an Array.) Thanks!

[kshyatt]

If it's consistent with the old API it should be one of Tridiagonal or
SymTridiagonal

On Wed, Oct 12, 2016, 11:41 AM Tony Kelman notifications@github.com wrote:

@tkelman commented on this pull request.

In test/linalg/tridiag.jl #18850:

@@ -119,7 +119,7 @@ for elty in (Float32, Float64, Complex64, Complex128, Int)
invFsv = Fs\vv
x = Ts\vv
@test x ≈ invFsv

•        @test full(full(Tldlt)) ≈ Fs
  

•        @test Array(Array(Tldlt)) ≈ Fs
  

ah right my "use a different name for..." issue. what type does
Array(::LDLt) return?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#18850, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAyk43tKILsHBdhRWBxSP5EPGdJuabNuks5qzSnegaJpZM4KR2SL
.

[tkelman]

the return type of full(::LDLt{eltype, arraytype}) on 0.4 was the arraytype. Need to check on 0.5. Converting to Array should always return an Array (and full on 0.6-dev as of a few weeks ago does this), so this operation that returns the arraytype from a factorization doesn't have its own name right now.

[kshyatt]

@tkelman probably better to open a new issue/find that old one for this, but... unfactorize?

[andreasnoack]

...or full 😄 . In one of the issues, it was concluded that using convert(Type{AbstractMatrix}, x::Factorization) would work for factorizations.

[tkelman]

Are the factorizations themselves AbstractArrays?

There's a bug here, the PR that reimplemented full in terms of convert did not do so correctly for the factorization types, and now this test change is not testing the original intent either.

[Sacha0]

In #12153 consensus was to replace full(factorization)'s behavior with convert(AbstractMatrix/AbstractArray, factorization). #17066 realized that change,

julia> A = SymTridiagonal(2*ones(4), ones(3));

julia> FA = factorize(A)
Base.LinAlg.LDLt{Float64,SymTridiagonal{Float64}}([2.0 0.5 0.0 0.0; 0.5 1.5 0.666667 0.0; 0.0 0.666667 1.33333 0.75; 0.0 0.0 0.75 1.25])

julia> convert(AbstractMatrix, FA)
4×4 SymTridiagonal{Float64}:
 2.0  1.0   ⋅    ⋅
 1.0  2.0  1.0   ⋅
  ⋅   1.0  2.0  1.0
  ⋅    ⋅   1.0  2.0

julia> convert(Array, FA)
4×4 Array{Float64,2}:
 2.0  1.0  0.0  0.0
 1.0  2.0  1.0  0.0
 0.0  1.0  2.0  1.0
 0.0  0.0  1.0  2.0

but @tkelman is correct that something is amiss with full(factorization)'s reimplementation as a child of convert(AbstractMatrix, factorization):

julia> full(FA)
4×4 Array{Float64,2}:
 2.0  1.0  0.0  0.0
 1.0  2.0  1.0  0.0
 0.0  1.0  2.0  1.0
 0.0  0.0  1.0  2.0

Having a look at that now (edit: found bug, PR forthcoming), but should be orthogonal to this PR. Thanks and best!

[Sacha0]

#18938 indicates that some of the tests touched in this PR should be more strict. Similarly, some of the full(X) calls presently replaced by Array(X) might rather mean AbstractArray(X) where X is a factorization.

Additionally, we never resolved the full(Triangular/Symmetric/Hermitian) question discussed in #17066, so some full(X) calls where X is one of those wrapper types might not mean Array(X).

Perhaps breaking this PR up into a series of smaller PRs would make this transition safer / smoother? (To decouple the simple, safe cases from the tricky ones?) Thoughts? Thanks and best!

[andreasnoack]

Additionally, we never resolved the full(Triangular/Symmetric/Hermitian) question discussed in #17066, so some full(X) calls where X is one of those wrapper types might not mean Array(X).

We are looking for a generic function that returns the wrapped array type with elements equal to the wrapper. That function used to be called full so the conclusion to use explicit type conversion instead of a generic function might have been too hasty. The name full might have been wrong but the functionality seems right.

An alternative solution would be to have a mattype function similar to the eltype. Then we could convert with mattype(A)(A) which should work for Symmetric etc. as well as Factorizations.

[Sacha0]

That function used to be called full so the conclusion to use explicit type conversion instead of a generic function might have been too hasty. The name full might have been wrong but the functionality seems right.

So far we've disambiguated two of full's meanings, convert(Array, X) and convert(AbstractArray, X). That we have not yet disambiguated full's third meaning does not imply that disambiguating the two preceding meanings was unwise. To the contrary, only through this disambiguation process have we plumbed the depths of full's depravity :). And the number and variety of ambiguities / challenges we've encountered only emphasizes full's shortcomings / the value of displacing it with better-defined entities.

An alternative solution would be to have a mattype function similar to the eltype. Then we could convert with mattype(A)(A) which should work for Symmetric etc. as well as Factorizations.

Yes, in #17066 we agreed that this approach to handling full(Triangular/Symmetric/...) might work well (but discussion faltered). Annotated matrices and factorizations are fundamentally different objects though, and full's semantics for factorizations (almost) uniformly conform to convert(AbstractArray, X) or convert(Array, X).

Seems we have consensus that something like convert(mattype(A), A) for full(Triangular/Symmetric/...) is the path forward (independent of the name mattype)?

Best!

[andreasnoack]

I don't question that a renaming was a good idea but I think there is a common pattern among the annotation types and the Factorizations that full was able to capture with some success. That I decided to use the same name as we're already using for densifying sparse matrices was probably not a good idea but I think dedicated generic function instead of convert was useful.

Thanks for reminding of #17066 (comment) in which I conclude that parenttype/mattype is problematic. A solution could be to define the mattype function recursively such that mattype(A::Symmetric)=mattype(A.data) and mattype{T}(::Matrix{T})=Matrix{T}, mattype(A:SubArray)=mattype(parent(A)). But again, this would lead to the behavior that full used to have for the annotation types and Factorizations so maybe renaming full for those types is the simpler solution.

[Sacha0]

The series of pull requests listed just above should cover all files in which all instances of full translate cleanly to Array/AbstractArray. In the remaining files, replacing some instances of full requires consensus on the point discussed above (what full for Symmetric/Hermitian/AbstractTriangular should translate to). Thoughts on how to resolve that issue so that we can move this effort forward? Thanks!

[andreasnoack]

Could a way forward be to try to define a storagetype/mattype/parenttype function for the annotation types? Then we can punt a bit on what this function should do for Factorizations. I think this xtype function should call itself for arrays with an underlying array (XTriangular, Symmetric, and SubArray) and return the array type for e.g. Array, SparseMatrixCSC, DistributedArray, and SArray. Would you be able to try out such a PR?

[Sacha0]

Sounds good, and shall do! Thanks!

[Sacha0]

With #12153 on the 1.0 milestone, perhaps this PR should receive a 1.0 tag as well?

[JeffBezanson]

In some cases would it make sense to convert to DenseArray? E.g. if you had a distributed sparse array and wanted to get a distributed dense array.

[Sacha0]

In some cases would it make sense to convert to DenseArray? E.g. if you had a distributed sparse array and wanted to get a distributed dense array.

Absolutely! :) Replacing full uniformly with Array as in this (outdated) pull request is incorrect. full should instead transform to Array, AbstractArray, DenseArray, or one of a number of other possibilities depending on context. (The later, smaller pull requests that reference this pull request (see above) transform full more appropriately / generally, though I imagine some of the transformations to Array in those pull requests could instead have been DenseArray or some other broader type.) Best!

[Sacha0]

Closed by #24250 and the series of preceding pull requests. Best!