Usage of parallel threads not stable - ghdl (gcc backend)

[webermann]

Some tests failed if I use the -p option. Without this option all tests are OK. I tried some variants for NUM_THREADS and there where no working option.

Here is my console output for -p 4

python run.py -p 4
Starting test.tb_test_lib.test_function_log2
Starting test.tb_test_lib.test_function_log2ceil
Starting test.tb_test_lib.test_function_ibo
Starting test.tb_test_lib.test_function_byteswap
Assembler messages:
Error: can't open e~tb_test_lib-tb.s for reading: No such file or directory
ghdl: compilation error
fail (P=0 S=0 F=1 T=9) test.tb_test_lib.test_function_log2 (0.2 seconds)

Starting test.tb_test_lib.test_function_equal
gcc: error: e~tb_test_lib-tb.o: No such file or directory
ghdl: compilation error
fail (P=0 S=0 F=2 T=9) test.tb_test_lib.test_function_byteswap (0.2 seconds)

gcc: error: e~tb_test_lib-tb.o: No such file or directory
ghdl: compilation error
fail (P=0 S=0 F=3 T=9) test.tb_test_lib.test_function_log2ceil (0.2 seconds)

Starting test.tb_test_lib.test_function_min_max
Starting test.tb_test_lib.test_function_getBitOfArray
gcc: error: e~tb_test_lib-tb.o: No such file or directory
ghdl: compilation error
fail (P=0 S=0 F=4 T=9) test.tb_test_lib.test_function_ibo (0.2 seconds)

Starting test.tb_test_lib.test_function_boolean
Assembler messages:
Error: can't open e~tb_test_lib-tb.s for reading: No such file or directory
ghdl: compilation error
fail (P=0 S=0 F=5 T=9) test.tb_test_lib.test_function_boolean (0.2 seconds)

Starting test.tb_test_lib.test_function_ones_zeros
gcc: error: e~tb_test_lib-tb.o: No such file or directory
ghdl: compilation error
fail (P=0 S=0 F=6 T=9) test.tb_test_lib.test_function_min_max (0.2 seconds)

gcc: error: e~tb_test_lib-tb.o: No such file or directory
ghdl: compilation error
fail (P=0 S=0 F=7 T=9) test.tb_test_lib.test_function_equal (0.2 seconds)

gcc: error: e~tb_test_lib-tb.o: No such file or directory
ghdl: compilation error
fail (P=0 S=0 F=8 T=9) test.tb_test_lib.test_function_getBitOfArray (0.2 seconds)

pass (P=1 S=0 F=8 T=9) test.tb_test_lib.test_function_ones_zeros (0.4 seconds)

==== Summary ======================================================
pass test.tb_test_lib.test_function_ones_zeros (0.4 seconds)
fail test.tb_test_lib.test_function_log2 (0.2 seconds)
fail test.tb_test_lib.test_function_byteswap (0.2 seconds)
fail test.tb_test_lib.test_function_log2ceil (0.2 seconds)
fail test.tb_test_lib.test_function_ibo (0.2 seconds)
fail test.tb_test_lib.test_function_boolean (0.2 seconds)
fail test.tb_test_lib.test_function_min_max (0.2 seconds)
fail test.tb_test_lib.test_function_equal (0.2 seconds)

fail test.tb_test_lib.test_function_getBitOfArray (0.2 seconds)

pass 1 of 9

fail 8 of 9

Total time was 2.3 seconds

Elapsed time was 0.9 seconds

Some failed!

Without the -p option:
python run.py
Starting test.tb_test_lib.test_function_log2
pass (P=1 S=0 F=0 T=9) test.tb_test_lib.test_function_log2 (0.4 seconds)

Starting test.tb_test_lib.test_function_log2ceil
pass (P=2 S=0 F=0 T=9) test.tb_test_lib.test_function_log2ceil (0.4 seconds)

Starting test.tb_test_lib.test_function_ibo
pass (P=3 S=0 F=0 T=9) test.tb_test_lib.test_function_ibo (0.4 seconds)

Starting test.tb_test_lib.test_function_byteswap
pass (P=4 S=0 F=0 T=9) test.tb_test_lib.test_function_byteswap (0.4 seconds)

Starting test.tb_test_lib.test_function_equal
pass (P=5 S=0 F=0 T=9) test.tb_test_lib.test_function_equal (0.4 seconds)

Starting test.tb_test_lib.test_function_min_max
pass (P=6 S=0 F=0 T=9) test.tb_test_lib.test_function_min_max (0.4 seconds)

Starting test.tb_test_lib.test_function_getBitOfArray
pass (P=7 S=0 F=0 T=9) test.tb_test_lib.test_function_getBitOfArray (0.4 seconds)

Starting test.tb_test_lib.test_function_boolean
pass (P=8 S=0 F=0 T=9) test.tb_test_lib.test_function_boolean (0.4 seconds)

Starting test.tb_test_lib.test_function_ones_zeros
pass (P=9 S=0 F=0 T=9) test.tb_test_lib.test_function_ones_zeros (0.4 seconds)

==== Summary ======================================================
pass test.tb_test_lib.test_function_log2 (0.4 seconds)
pass test.tb_test_lib.test_function_log2ceil (0.4 seconds)
pass test.tb_test_lib.test_function_ibo (0.4 seconds)
pass test.tb_test_lib.test_function_byteswap (0.4 seconds)
pass test.tb_test_lib.test_function_equal (0.4 seconds)
pass test.tb_test_lib.test_function_min_max (0.4 seconds)
pass test.tb_test_lib.test_function_getBitOfArray (0.4 seconds)
pass test.tb_test_lib.test_function_boolean (0.4 seconds)

pass test.tb_test_lib.test_function_ones_zeros (0.4 seconds)

pass 9 of 9

Total time was 3.8 seconds

Elapsed time was 3.8 seconds

All passed!

[kraigher]

What version of GHDL are you using? When adding support for GHDL to VUnit we discovered this problem with parallel test execution using the same top level. This problem was reported to the GHDL team and they fixed before we added official support of GHDL to VUnit.

[kraigher]

Also what GHDL backend (mcode/llvm/gcc) are you using? It could be that this problem exists for GCC backend which we (the VUnit developers) have not tested that much with. The LLVM backend is the one I mainly test with.

[webermann]

Here is the ghdl output:

ghdl --version
GHDL 0.33 (20150921) [Dunoon edition]
Compiled with GNAT Version: GPL 2014 (20140331)
GCC back-end code generator
Written by Tristan Gingold.

Copyright (C) 2003 - 2015 Tristan Gingold.
GHDL is free software, covered by the GNU General Public License. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[kraigher]

I will see if I can try the GCC backend to reproduce this. Do you have the opportunity to try the LLVM version?

[webermann]

Not at the moment... may I can try to install it on monday together with our IT.

[kraigher]

I have reported the problem to the GHDL maintainer: ghdl/ghdl#8

[kraigher]

@webermann Also a pro-tip for VUnit:
When testing stuff that does not have much state so that you do not get much convenience from starting a new simulation for each test in a test bench you can tell VUnit to run all tests in a test bench in the same simulation which will go often go faster. That is what we often use for testing function/procedures/packages rather than entities. Just add the following comment pragma to the test bench:

-- vunit_pragma run_all_in_same_sim

Remember to use the correct test pattern with a while-loop for it to work though:

while test_suite loop
  reset_stuff;
  if run("test1") then
  -- ...  
  elsif run("test2") then
  -- ...  
  end if;
end loop;

Also remember that you are yourself responsible for ensuring test-case independence since they are not run in independent simulations.

[webermann]

Uuhh, thanks for the tip, thats much faster!

[kraigher]

@webermann I found the problem in VUnit. We did not provide the -o flag to GHDL with the GCC backend only with LLVM backend. This is fixed in v0.48.2.

[webermann]

Thanks a lot!!