Framework: Switch CUDA AT2 build to be non-UVM and enable tests

[sebrowne]

@trilinos/framework

Motivation

Want to align the CUDA AT2 build with the old AutoTester one.

Related Issues

https://sems-atlassian-son.sandia.gov/jira/browse/TRILFRAME-673

[sebrowne]

The CUDA tests look good, with four exceptions, detailed here: https://sems-cdash-son.sandia.gov/cdash/viewTest.php?onlyfailed&buildid=211376

@trilinos/intrepid2 I show that failing test was set to RUN SERIAL for CUDA builds, I can do that here as well if that's still what we want to do.
@trilinos/panzer that test is on for all other configs, no obvious framework-side issues to me.
@trilinos/rol that test is on for all other configs, no obvious framework-side issues to me.
@trilinos/stratimikos we had that test disabled for our non-CUDA container as well, but again, nothing really obvious from our side.

If any developers from the tagged teams can provide any insight for the four failing tests (and they do fail reliably), it would be much appreciated! I can turn them off, but I wanted to at least do SOME due diligence and see what the community thinks.

[CamelliaDPG]

@trilinos/intrepid2 I show that failing test was set to RUN SERIAL for CUDA builds, I can do that here as well if that's still what we want to do.

Yes, please. The MonolithicExecutable test is one that has a lot of test cases, and some of them are intensive, so that sharing compute resources with other tests can lead to timeouts. We use RUN SERIAL to mitigate.

[rppawlo]

@cgcgcg - would you mind taking a look at the panzer/mini-em failure here? Looks to be a linear solver issue similar to what you have fixed in the past.

[cgcgcg]

I see this message in the output of the failing Stratimikos and Panzer tests:

--------------------------------------------------------------------------
The call to cuIpcGetMemHandle failed. This means the GPU RDMA protocol
cannot be used.
  cuIpcGetMemHandle return value:   1
  address: 0x42b363c80
Check the cuda.h file for what the return value means. Perhaps a reboot
of the node will clear the problem.
--------------------------------------------------------------------------