Update Alpaka to version 1.1.0 #8957

fwyzard · 2024-01-23T10:10:01Z

Major changes include:

introduce SYCL/oneAPI support for CPUs and Intel GPUs
make alpaka platforms full objects, and rename from Pltf to Platform
make the vendor random number generators support optional
refactor Vec type, and add .x(), .y(), .z() accessors
enable support for asynchronous memory operations in ROCm 5.3 and later
change all CUDA warp operations to synchronise all threads
remove ALPAKA_ASSERT_OFFLOAD, introduce ALPAKA_ASSERT_ACC
implement copysign(), fma(), log2(), log10() math functions
simplify offset and pitch APIs
add support for CUDA 12.2, 12.3 and ROCm 5.6, 5.7, 6.0
improve error messages related to kernel launches
rework thread pool and callback threads
fix alpaka::wait(device, event) function for CUDA/HIP GPUs
implement alpaka-ls

Disable the vendor specific random number generators in Alpaka (NVIDIA cuRAND, AMD rocRAND, Intel DPL).

cmsbuild · 2024-01-23T10:10:29Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for branch IB/CMSSW_14_0_X/master.

@cmsbuild, @aandvalenzuela, @smuzaffar, @iarspider can you please review it and eventually sign? Thanks.
@antoniovilela, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2024-01-23T10:10:29Z

cms-bot internal usage

fwyzard · 2024-01-23T10:10:42Z

This will require a corresponding update in CMSSW.

fwyzard · 2024-01-23T13:29:00Z

please test with cms-sw/cmssw#43772

cmsbuild · 2024-01-23T13:52:00Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e16e15/36983/summary.html
COMMIT: bf32ea7
CMSSW: CMSSW_14_0_X_2024-01-22-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8957/36983/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

Requested to quit.
Requested to quit.
Requested to quit.
* The action "build-external+alpaka+1.1.0-2f48beae08b1557551668b2b5e1c5bd3" was not completed successfully because Failed to build alpaka. Log file in /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/alpaka/1.1.0-2f48beae08b1557551668b2b5e1c5bd3/log. Final lines of the log file:
warning: line 36: It's not recommended to have unversioned Obsoletes: Obsoletes: external+alpaka+1.1.0-2f48beae08b1557551668b2b5e1c5bd3
error: Bad source: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/SOURCES/external/alpaka/1.1.0-2f48beae08b1557551668b2b5e1c5bd3/1.1.0-2f48beae08b1557551668b2b5e1c5bd3.tar.gz: No such file or directory

* The action "install-external+alpaka+1.1.0-2f48beae08b1557551668b2b5e1c5bd3" was not completed successfully because The following dependencies could not complete:
build-external+alpaka+1.1.0-2f48beae08b1557551668b2b5e1c5bd3
* The action "build-cms+cmssw-tool-conf+60.0-f22e1c72d73ca12b8bf57d758886aeb1" was not completed successfully because The following dependencies could not complete:
install-external+alpaka+1.1.0-2f48beae08b1557551668b2b5e1c5bd3

Major changes include: - introduce SYCL/oneAPI support for CPUs and Intel GPUs - make alpaka platforms full objects, and rename from Pltf to Platform - make the vendor random number generators support optional - refactor Vec type, and add .x(), .y(), .z() accessors - enable support for asynchronous memory operations in ROCm 5.3 and later - change all CUDA warp operations to synchronise all threads - remove ALPAKA_ASSERT_OFFLOAD, introduce ALPAKA_ASSERT_ACC - implement copysign(), fma(), log2(), log10() math functions - simplify offset and pitch APIs - add support for CUDA 12.2, 12.3 and ROCm 5.6, 5.7, 6.0 - improve error messages related to kernel launches - rework thread pool and callback threads - fix alpaka::wait(device, event) function for CUDA/HIP GPUs - implement alpaka-ls

…A cuRAND, AMD rocRAND, Intel DPL)

cmsbuild · 2024-01-23T14:06:01Z

Pull request #8957 was updated.

fwyzard · 2024-01-23T14:06:01Z

please test with cms-sw/cmssw#43772

fwyzard · 2024-01-23T14:39:24Z

please abort

fwyzard · 2024-01-23T14:39:37Z

please test with cms-sw/cmssw#43772

cmsbuild · 2024-01-23T22:13:31Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e16e15/36990/summary.html
COMMIT: 59d4d94
CMSSW: CMSSW_14_0_X_2024-01-23-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8957/36990/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@ericcano Ports prefixScan, OneToManyAssoc and HistoContainer from CUDAUtilities. cmssw#43064

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e16e15/36990/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e16e15/36990/git-merge-result

Comparison Summary

Summary:

You potentially removed 73 lines from the logs
Reco comparison results: 5 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3247526
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3247504
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 200 log files, 161 edm output root files, 48 DQM output files
TriggerResults: no differences found

fwyzard · 2024-02-11T22:54:15Z

please test with cms-sw/cmssw#43772 for el8_aarch64_gcc12

fwyzard · 2024-02-11T22:54:44Z

please test with cms-sw/cmssw#43772 for el8_ppc64le_gcc12

cmsbuild · 2024-02-12T06:15:45Z

-1

Failed Tests: HeaderConsistency UnitTests RelVals AddOn GpuUnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e16e15/37355/summary.html
COMMIT: 59d4d94
CMSSW: CMSSW_14_1_X_2024-02-09-2300/el8_ppc64le_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8957/37355/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e16e15/37355/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e16e15/37355/git-merge-result

Unit Tests

I found 1 errors in the following unit tests:

---> test testPhysicsToolsSelectorUtilsPythonTestsDriver had ERRORS

RelVals

11634.0A fatal system signal has occurred: abort signal
12434.0A fatal system signal has occurred: abort signal
12434.7A fatal system signal has occurred: abort signal

Expand to see more relval errors ...

AddOn Tests

hlt_mc_GRun

A fatal system signal has occurred: abort signal

hlt_mc_GRun

----- Begin Fatal Exception 12-Feb-2024 04:48:33 CET-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling File::sysopen()
Exception Message:
Failed to open the file 'RelVal_Raw_GRun_MC.root'
   Additional Info:
      [a] Input file file:RelVal_Raw_GRun_MC.root could not be opened.
      [b] open() failed with system error 'No such file or directory' (error code 2)
----- End Fatal Exception -------------------------------------------------

hlt_mc_GRun

----- Begin Fatal Exception 12-Feb-2024 04:53:33 CET-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling File::sysopen()
Exception Message:
Failed to open the file 'RelVal_Raw_GRun_MC.root'
   Additional Info:
      [a] Input file file:RelVal_Raw_GRun_MC.root could not be opened.
      [b] open() failed with system error 'No such file or directory' (error code 2)
----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

GPU Unit Tests

I found 14 errors in the following unit tests:

---> test alpakaTestAtomicPairCounterCudaAsync had ERRORS
---> test alpakaTestIndependentKernelCudaAsync had ERRORS
---> test alpakaTestKernelCudaAsync had ERRORS
and more ...

cmsbuild · 2024-02-12T08:29:49Z

-1

Failed Tests: HeaderConsistency UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e16e15/37356/summary.html
COMMIT: 59d4d94
CMSSW: CMSSW_14_1_X_2024-02-09-2300/el8_aarch64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8957/37356/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e16e15/37356/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e16e15/37356/git-merge-result

Unit Tests

I found 3 errors in the following unit tests:

---> test testSiStripDQM_OfflineTkMap had ERRORS
---> test GCPall had ERRORS
---> test testTrackAnalysis had ERRORS

fwyzard · 2024-02-12T10:44:01Z

please test with cms-sw/cmssw#43772

fwyzard · 2024-02-12T10:44:30Z

please test with cms-sw/cmssw#43772 for el8_ppc64le_gcc12

fwyzard · 2024-02-12T10:44:45Z

please test with cms-sw/cmssw#43772 for el8_aarch64_gcc12

cmsbuild · 2024-02-12T14:49:01Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e16e15/37358/summary.html
COMMIT: 59d4d94
CMSSW: CMSSW_14_1_X_2024-02-11-2300/el8_aarch64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8957/37358/install.sh to create a dev area with all the needed externals and cmssw changes.

cmsbuild · 2024-02-12T21:15:03Z

-1

Failed Tests: UnitTests GpuUnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e16e15/37359/summary.html
COMMIT: 59d4d94
CMSSW: CMSSW_14_1_X_2024-02-11-2300/el8_ppc64le_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8957/37359/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 1 errors in the following unit tests:

---> test testPhysicsToolsSelectorUtilsPythonTestsDriver had ERRORS

GPU Unit Tests

I found 14 errors in the following unit tests:

---> test Hits_testCudaAsync had ERRORS
---> test ZVertexSoA_testCudaAsync had ERRORS
---> test TrackSoAHeterogeneousAlpaka_testCudaAsync had ERRORS
and more ...

cmsbuild · 2024-02-12T21:45:50Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e16e15/37364/summary.html
COMMIT: 59d4d94
CMSSW: CMSSW_14_1_X_2024-02-12-1100/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8957/37364/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 2 lines to the logs
Reco comparison results: 45 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3248626
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3248604
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 200 log files, 161 edm output root files, 48 DQM output files
TriggerResults: no differences found

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 34 differences found in the comparisons
DQMHistoTests: Total files compared: 3
DQMHistoTests: Total histograms compared: 39740
DQMHistoTests: Total failures: 1698
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 38042
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 2 files compared)
Checked 8 log files, 10 edm output root files, 3 DQM output files
TriggerResults: no differences found

smuzaffar · 2024-02-12T21:56:34Z

+externals

looks good, x86_64, aarch64, ppc64le externals built fine.

cmsbuild · 2024-02-12T21:56:53Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_14_1_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @antoniovilela, @sextonkennedy, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)
Notice This PR was tested with additional Pull Request(s), please also merge them if necessary: cms-sw/cmssw#43772

cmsbuild · 2024-02-13T11:35:09Z

REMINDER @antoniovilela, @sextonkennedy, @rappoccio: This PR was tested with cms-sw/cmssw#43772, please check if they should be merged together

antoniovilela · 2024-02-13T13:06:13Z

+1

cmsbuild added tests-pending externals-pending pending-signatures orp-pending labels Jan 23, 2024

fwyzard mentioned this pull request Jan 23, 2024

Update Alpaka to version 1.1.0 cms-sw/cmssw#43772

Merged

cmsbuild added tests-started requires-external and removed tests-pending labels Jan 23, 2024

cmsbuild added tests-rejected and removed tests-started labels Jan 23, 2024

fwyzard added 2 commits January 23, 2024 15:05

Disable the vendor specific random number generators in Alpaka (NVIDI…

59d4d94

…A cuRAND, AMD rocRAND, Intel DPL)

fwyzard force-pushed the IB/CMSSW_14_0_X/master_alpaka_1.1.0 branch from bf32ea7 to 59d4d94 Compare January 23, 2024 14:05

cmsbuild added tests-pending and removed tests-rejected requires-external labels Jan 23, 2024

cmsbuild added tests-started requires-external and removed tests-pending labels Jan 23, 2024

cmsbuild removed the tests-started label Jan 23, 2024

This was referenced Feb 12, 2024

Update Alpaka to version 1.1.0 [14.0.x] cms-sw/cmssw#43943

Merged

Update Alpaka to version 1.1.0 [14.0.x] #9010

Merged

cmsbuild added tests-approved and removed tests-started labels Feb 12, 2024

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Feb 12, 2024

cmsbuild added orp-approved and removed orp-pending labels Feb 13, 2024

cmsbuild merged commit 58d64ce into cms-sw:IB/CMSSW_14_1_X/master Feb 13, 2024
26 of 29 checks passed

fwyzard deleted the IB/CMSSW_14_0_X/master_alpaka_1.1.0 branch March 6, 2024 11:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Alpaka to version 1.1.0 #8957

Update Alpaka to version 1.1.0 #8957

fwyzard commented Jan 23, 2024

cmsbuild commented Jan 23, 2024

cmsbuild commented Jan 23, 2024 •

edited

Loading

fwyzard commented Jan 23, 2024

fwyzard commented Jan 23, 2024

cmsbuild commented Jan 23, 2024

cmsbuild commented Jan 23, 2024

fwyzard commented Jan 23, 2024

fwyzard commented Jan 23, 2024

fwyzard commented Jan 23, 2024

cmsbuild commented Jan 23, 2024

fwyzard commented Feb 11, 2024

fwyzard commented Feb 11, 2024

cmsbuild commented Feb 12, 2024

cmsbuild commented Feb 12, 2024

fwyzard commented Feb 12, 2024

fwyzard commented Feb 12, 2024

fwyzard commented Feb 12, 2024

cmsbuild commented Feb 12, 2024

cmsbuild commented Feb 12, 2024

cmsbuild commented Feb 12, 2024

smuzaffar commented Feb 12, 2024

cmsbuild commented Feb 12, 2024

cmsbuild commented Feb 13, 2024

antoniovilela commented Feb 13, 2024

Update Alpaka to version 1.1.0 #8957

Update Alpaka to version 1.1.0 #8957

Conversation

fwyzard commented Jan 23, 2024

cmsbuild commented Jan 23, 2024

cmsbuild commented Jan 23, 2024 • edited Loading

fwyzard commented Jan 23, 2024

fwyzard commented Jan 23, 2024

cmsbuild commented Jan 23, 2024

External Build

cmsbuild commented Jan 23, 2024

fwyzard commented Jan 23, 2024

fwyzard commented Jan 23, 2024

fwyzard commented Jan 23, 2024

cmsbuild commented Jan 23, 2024

Comparison Summary

fwyzard commented Feb 11, 2024

fwyzard commented Feb 11, 2024

cmsbuild commented Feb 12, 2024

Unit Tests

RelVals

AddOn Tests

GPU Unit Tests

cmsbuild commented Feb 12, 2024

Unit Tests

fwyzard commented Feb 12, 2024

fwyzard commented Feb 12, 2024

fwyzard commented Feb 12, 2024

cmsbuild commented Feb 12, 2024

cmsbuild commented Feb 12, 2024

Unit Tests

GPU Unit Tests

cmsbuild commented Feb 12, 2024

Comparison Summary

GPU Comparison Summary

smuzaffar commented Feb 12, 2024

cmsbuild commented Feb 12, 2024

cmsbuild commented Feb 13, 2024

antoniovilela commented Feb 13, 2024

cmsbuild commented Jan 23, 2024 •

edited

Loading