Skip to content
This repository has been archived by the owner on Mar 12, 2021. It is now read-only.

BoundsError when initializing CuArrays #412

Closed
samo-lin opened this issue Sep 10, 2019 · 9 comments
Closed

BoundsError when initializing CuArrays #412

samo-lin opened this issue Sep 10, 2019 · 9 comments
Labels
bug

Comments

@samo-lin
Copy link

Describe the bug
I obtain a BoundsError when initializing CuArrays.

To Reproduce
The Minimal Working Example (MWE) for this bug:

> julia
julia> using CUDAdrv

julia> using CUDAnative

julia> using CuArrays
Internal error: encountered unexpected error in runtime:
BoundsError(a=Array{Core.Compiler.BasicBlock, (32,)}[
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=1, last=7), preds=Array{Int64, (1,)}[32], succs=Array{Int64, (1,)}[2]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=8, last=14), preds=Array{Int64, (1,)}[1], succs=Array{Int64, (2,)}[5, 3]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=15, last=20), preds=Array{Int64, (1,)}[2], succs=Array{Int64, (1,)}[4]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=21, last=21), preds=Array{Int64, (1,)}[3], succs=Array{Int64, (1,)}[7]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=22, last=28), preds=Array{Int64, (1,)}[2], succs=Array{Int64, (1,)}[6]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=29, last=29), preds=Array{Int64, (1,)}[5], succs=Array{Int64, (1,)}[7]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=30, last=47), preds=Array{Int64, (2,)}[4, 6], succs=Array{Int64, (2,)}[9, 8]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=48, last=49), preds=Array{Int64, (1,)}[7], succs=Array{Int64, (0,)}[]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=50, last=50), preds=Array{Int64, (1,)}[7], succs=Array{Int64, (1,)}[10]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=51, last=52), preds=Array{Int64, (1,)}[9], succs=Array{Int64, (1,)}[11]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=53, last=53), preds=Array{Int64, (1,)}[10], succs=Array{Int64, (1,)}[12]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=54, last=54), preds=Array{Int64, (1,)}[11], succs=Array{Int64, (1,)}[13]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=55, last=56), preds=Array{Int64, (1,)}[12], succs=Array{Int64, (1,)}[14]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=57, last=65), preds=Array{Int64, (1,)}[13], succs=Array{Int64, (1,)}[15]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=66, last=71), preds=Array{Int64, (1,)}[14], succs=Array{Int64, (2,)}[17, 16]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=72, last=74), preds=Array{Int64, (1,)}[15], succs=Array{Int64, (0,)}[]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=75, last=82), preds=Array{Int64, (1,)}[15], succs=Array{Int64, (2,)}[19, 18]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=83, last=84), preds=Array{Int64, (1,)}[17], succs=Array{Int64, (0,)}[]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=85, last=85), preds=Array{Int64, (1,)}[17], succs=Array{Int64, (1,)}[20]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=86, last=87), preds=Array{Int64, (1,)}[19], succs=Array{Int64, (1,)}[21]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=88, last=88), preds=Array{Int64, (1,)}[20], succs=Array{Int64, (1,)}[22]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=89, last=89), preds=Array{Int64, (1,)}[21], succs=Array{Int64, (1,)}[23]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=90, last=90), preds=Array{Int64, (1,)}[22], succs=Array{Int64, (1,)}[24]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=91, last=92), preds=Array{Int64, (1,)}[23], succs=Array{Int64, (1,)}[25]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=93, last=93), preds=Array{Int64, (1,)}[24], succs=Array{Int64, (1,)}[26]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=94, last=94), preds=Array{Int64, (1,)}[25], succs=Array{Int64, (1,)}[27]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=95, last=95), preds=Array{Int64, (1,)}[26], succs=Array{Int64, (2,)}[29, 28]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=96, last=98), preds=Array{Int64, (1,)}[27], succs=Array{Int64, (0,)}[]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=99, last=102), preds=Array{Int64, (1,)}[27], succs=Array{Int64, (2,)}[31, 30]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=103, last=103), preds=Array{Int64, (1,)}[29], succs=Array{Int64, (1,)}[32]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=104, last=104), preds=Array{Int64, (1,)}[29], succs=Array{Int64, (1,)}[32]),
  Core.Compiler.BasicBlock(stmts=Core.Compiler.StmtRange(first=105, last=107), preds=Array{Int64, (2,)}[30, 31], succs=Array{Int64, (1,)}[1])], i=(0,))
rec_backtrace at /buildworker/worker/package_linux64/build/src/stackwalk.c:94
record_backtrace at /buildworker/worker/package_linux64/build/src/task.c:246
jl_throw at /buildworker/worker/package_linux64/build/src/task.c:577
jl_bounds_error_ints at /buildworker/worker/package_linux64/build/src/rtutils.c:187
getindex at ./array.jl:731
jfptr_getindex_1594.clone_1 at /apps/daint/UES/jenkins/6.0.UP07/gpu/easybuild/software/Julia/1.0.4-CrayGNU-18.08-cuda-9.1/lib/julia/sys.so (unknown line)
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2162
replace_code_newstyle! at ./compiler/ssair/legacy.jl:80
optimize at ./compiler/optimize.jl:212
typeinf at ./compiler/typeinfer.jl:35
typeinf_ext at ./compiler/typeinfer.jl:574
typeinf_ext at ./compiler/typeinfer.jl:611
jfptr_typeinf_ext_1.clone_1 at /apps/daint/UES/jenkins/6.0.UP07/gpu/easybuild/software/Julia/1.0.4-CrayGNU-18.08-cuda-9.1/lib/julia/sys.so (unknown line)
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2162
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1537 [inlined]
jl_apply_with_saved_exception_state at /buildworker/worker/package_linux64/build/src/rtutils.c:257
jl_type_infer at /buildworker/worker/package_linux64/build/src/gf.c:253
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:1764 [inlined]
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1808
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2162
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1537 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:268
unknown function (ip: 0xffffffffffffffff)

julia>

Expected behavior
Initialization without error.

Build log

julia> import Pkg; Pkg.build()
 Resolving package versions...
   Cloning default registries into /users/omlins/.julia/1.0.4/daint-gpu/registries
   Cloning registry General from "https://github.com/JuliaRegistries/General.git"
[ Info: no packages to build

Environment details (please complete this section)
Details on Julia:

julia> versioninfo()
Julia Version 1.0.4
Commit 38e9fb7f80 (2019-05-16 03:38 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, haswell)
Environment:
  EBVERSIONJULIA = 1.0.4
  JULIA_DEPOT_PATH = /users/omlins/.julia/1.0.4/daint-gpu:/apps/daint/UES/jenkins/6.0.UP07/gpu/easybuild/software/Julia/1.0.4-CrayGNU-18.08-cuda-9.1/extensions:/apps/daint/UES/jenkins/6.0.UP07/gpu/easybuild/software/Julia/1.0.4-CrayGNU-18.08-cuda-9.1/local/share/julia:/apps/daint/UES/jenkins/6.0.UP07/gpu/easybuild/software/Julia/1.0.4-CrayGNU-18.08-cuda-9.1/share/julia
  EBJULIA_ENV_NAME = 1.0.4-daint-gpu
  JULIA_PROJECT = /users/omlins/.julia/1.0.4/daint-gpu/environments/1.0.4-daint-gpu
  EBROOTJULIA = /apps/daint/UES/jenkins/6.0.UP07/gpu/easybuild/software/Julia/1.0.4-CrayGNU-18.08-cuda-9.1
  EBDEVELJULIA = /apps/daint/UES/jenkins/6.0.UP07/gpu/easybuild/software/Julia/1.0.4-CrayGNU-18.08-cuda-9.1/easybuild/Julia-1.0.4-CrayGNU-18.08-cuda-9.1-easybuild-devel
  JULIA_LOAD_PATH = @:@#.#.#-daint-gpu:@stdlib
  EBEXTSLISTJULIA = MPI.jl-0.9.0,CUDAdrv.jl-3.1.0,CUDAnative.jl-2.3.0,CuArrays.jl-1.2.1

Julia packages:

  • CUDAdrv.jl-3.1.0
  • CUDAnative.jl-2.3.0
  • CuArrays.jl-1.2.1

CUDA: toolkit and driver version
26) cudatoolkit/9.1.85_3.18-6.0.7.0_5.1__g2eb7c52

omlins@nid04276:~> nvidia-smi
Tue Sep 10 12:55:25 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.44 Driver Version: 396.44 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla P100-PCIE... On | 00000000:02:00.0 Off | 0 |
| N/A 26C P0 29W / 250W | 0MiB / 16280MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

Additional context
@samo-lin samo-lin added the bug label Sep 10, 2019
@maleadt
Copy link
Member

maleadt commented Sep 10, 2019

Known issue in Julia 1.0, but otherwise harmless. Not present in Julia 1.1 or higher.

@maleadt maleadt closed this as completed Sep 10, 2019
@samo-lin
Copy link
Author

samo-lin commented Sep 10, 2019 •
edited
Loading

That is good news that it is not present in Julia 1.1 or higher. However the version 1.0.4 is the current long-term support (LTS) release; thus, I believe it would be important to fix this error. We are starting to support Julia on our supercomputer with a site-wide installation and most users will probably use the long-term support release...

@maleadt
Copy link
Member

maleadt commented Sep 10, 2019

Understandably; you'd need to file an upstream issue about that. Probably fairly easy to figure out by bisecting and grepping on the error message.

@samo-lin
Copy link
Author

Is this a bug in Julia rather than in CuArrays? (BTW, I tried with Julia 1.0.5; the bug is still there)

@maleadt
Copy link
Member

maleadt commented Sep 17, 2019

Yes, this is a bug in Julia, and hasn't been fixed in the 1.0 series. If you try 1.1 or 1.2, the issue will be gone. But again, if you ignore the visual noise CuArrays should work just fine on 1.0 too :-)

@samo-lin
Copy link
Author

OK, thanks. As this error message is harmless as you say, is there maybe an easy way to suppress the error message until this issue is fixed in Julia 1.0? A potential new Julia user might get an unjustified bad impression of your package when using CuArrays with Julia version 1.0. I know about the quality of the packages you are developing, but new users don't... We have now installed Julia and your packages site-wide and the first thing a new user sees when trying it out is this error message...

@maleadt
Copy link
Member

maleadt commented Sep 17, 2019

I'll have a go at bisecting the fix, maybe we can put it in the next 1.0.

@maleadt
Copy link
Member

maleadt commented Sep 18, 2019

Already reported at JuliaLang/julia#30093.
I bisected the fix to JuliaLang/julia#28878, which is fairly useless.

@samo-lin
Copy link
Author

Perfect! Thanks!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@maleadt @samo-lin