Update CUDA to support Pascal, Volta, Turing, Ampere and Lovelace GPUs #8767

fwyzard · 2023-10-18T09:24:56Z

No description provided.

fwyzard · 2023-10-18T09:25:02Z

enable gpu

fwyzard · 2023-10-18T09:25:06Z

please test

cmsbuild · 2023-10-18T09:25:20Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for branch IB/CMSSW_13_3_X/master.

@iarspider, @smuzaffar, @aandvalenzuela can you please review it and eventually sign? Thanks.
@antoniovilela, @sextonkennedy, @rappoccio you are the release manager for this.
cms-bot commands are listed here

smuzaffar · 2023-10-18T21:15:59Z

@fwyzard , this did not help. GPU relval still failed with the same error as in IB

PR: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e1f586/35252/runTheMatrixGPU-results/12434.587_TTbar_14TeV+2023_Patatrack_AllTripletsGPU_Validation/step2_TTbar_14TeV+2023_Patatrack_AllTripletsGPU_Validation.log
IB: https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc12/CMSSW_13_3_GPU_X_2023-10-17-2300/pyRelValMatrixLogs/run/12434.587_TTbar_14TeV+2023_Patatrack_AllTripletsGPU_Validation/step2_TTbar_14TeV+2023_Patatrack_AllTripletsGPU_Validation.log#/

cmsbuild · 2023-10-19T00:23:58Z

-1

Failed Tests: RelVals-GPU GpuUnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e1f586/35252/summary.html
COMMIT: b93cfa0
CMSSW: CMSSW_13_3_X_2023-10-17-2300/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8767/35252/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e1f586/35252/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e1f586/35252/git-merge-result

RelVals-GPU

12434.58712434.587_TTbar_14TeV+2023_Patatrack_AllTripletsGPU_Validation/step2_TTbar_14TeV+2023_Patatrack_AllTripletsGPU_Validation.log

GPU Unit Tests

I found 2 errors in the following unit tests:

---> test testTFVisibleDevicesCUDA had ERRORS
---> test testEigenGPUNoFit_t had ERRORS

Comparison Summary

Summary:

You potentially added 9 lines to the logs
Reco comparison results: 62637 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3357400
DQMHistoTests: Total failures: 155151
DQMHistoTests: Total nulls: 269
DQMHistoTests: Total successes: 3201958
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.10599999999999998 KiB( 49 files compared)
DQMHistoSizes: changed ( 10224.0 ): -0.308 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 12634.0 ): 0.288 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 250202.181 ): -0.288 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): 0.978 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 7.3 ): -0.564 KiB SiStrip/MechanicalView
Checked 214 log files, 167 edm output root files, 50 DQM output files
TriggerResults: found differences in 18 / 48 workflows

smuzaffar · 2023-10-19T10:43:48Z

please test

smuzaffar · 2023-10-19T12:02:57Z

test parameters:

full_cmssw = true
workflows_gpu = 160.03502,12434.504,12434.503

smuzaffar · 2023-10-19T12:04:35Z

please test

cmsbuild · 2023-10-19T17:34:58Z

-1

Failed Tests: GpuUnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e1f586/35285/summary.html
COMMIT: b93cfa0
CMSSW: CMSSW_13_3_X_2023-10-19-1100/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8767/35285/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Unit Tests

I found 2 errors in the following unit tests:

---> test testTFVisibleDevicesCUDA had ERRORS
---> test testEigenGPUNoFit_t had ERRORS

Comparison Summary

Summary:

You potentially removed 137 lines from the logs
Reco comparison results: 11 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3357400
DQMHistoTests: Total failures: 12
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3357366
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 214 log files, 167 edm output root files, 50 DQM output files
TriggerResults: no differences found

GPU Comparison Summary

Summary:

You potentially added 160 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 52 differences found in the comparisons
DQMHistoTests: Total files compared: 2
DQMHistoTests: Total histograms compared: 19678
DQMHistoTests: Total failures: 2007
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 17671
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 1 files compared)
Checked 15 log files, 14 edm output root files, 2 DQM output files
TriggerResults: found differences in 1 / 1 workflows

smuzaffar · 2023-10-19T17:39:35Z

please test for el8_aarch64_gcc12

smuzaffar · 2023-10-19T17:42:54Z

please test for el8_ppc64le_gcc12

smuzaffar · 2023-10-19T17:45:19Z

@fwyzard , gpu tests for x86_64 worked this time. Unit tests failure are

testEigenGPUNoFit_t was already there due to eigen update.
testTFVisibleDevicesCUDA: Tensorflow /cuda part was disabled for gcc12 that is why this test is failing.

are you happy with current list of cuda archs? once aarch64/ppc64le tests are done then I can merge this

fwyzard · 2023-10-19T21:58:17Z

@smuzaffar

are you happy with current list of cuda archs? once aarch64/ppc64le tests are done then I can merge this

the current list should cover all GPUs we currently have access to.
To be more complete, we could add the architectures for the A40/A10 (sm_87) and H100 (sm_90) GPUs:

-%define cuda_arch 60 70 75 80 89
+%define cuda_arch 60 70 75 80 87 89 90

Do you have some idea of how much that would increase the build time and binary size?

cmsbuild · 2023-10-19T22:27:04Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e1f586/35296/summary.html
COMMIT: b93cfa0
CMSSW: CMSSW_13_3_X_2023-10-18-2300/el8_aarch64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8767/35296/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e1f586/35296/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e1f586/35296/git-merge-result

cmsbuild · 2023-10-20T00:13:23Z

-1

Failed Tests: UnitTests RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e1f586/35297/summary.html
COMMIT: b93cfa0
CMSSW: CMSSW_13_3_X_2023-10-18-2300/el8_ppc64le_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8767/35297/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e1f586/35297/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e1f586/35297/git-merge-result

Unit Tests

I found 1 errors in the following unit tests:

---> test testCSCTriggerMapping had ERRORS

RelVals

24900.024900.0_CloseByPGun_CE_H_Coarse_Scint+2026D98/step2_CloseByPGun_CE_H_Coarse_Scint+2026D98.log
24896.024896.0_CloseByPGun_CE_E_Front_120um+2026D98/step2_CloseByPGun_CE_E_Front_120um+2026D98.log
24834.024834.0_TTbar_14TeV+2026D98/step2_TTbar_14TeV+2026D98.log

Expand to see more relval errors ...

fwyzard · 2023-10-20T06:39:13Z

@smuzaffar let's merge this, we can add more architectures later.

smuzaffar · 2023-10-20T12:11:11Z

@smuzaffar let's merge this, we can add more architectures later.

sure, once 13.3.0.pre4 is uploaded then I will merge this.

Do you have some idea of how much that would increase the build time and binary size?

CMSSW/lib binary size increased by ~3% (39MB: from 1189MB to 1228MB). I do not have access to individual build time for each package but overall full cmssw build time is nearly same between 150-155mins

smuzaffar · 2023-10-20T12:14:50Z

+externals

cmsbuild · 2023-10-20T12:15:07Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_13_3_X/master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @antoniovilela, @rappoccio, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

Update CUDA to support Pascal, Volta, Turing, Ampere and Lovelace GPUs

b93cfa0

cmsbuild added externals-pending pending-signatures tests-started orp-pending labels Oct 18, 2023

fwyzard mentioned this pull request Oct 18, 2023

GPU tests are failing after moving to GCC12 cms-sw/cmssw#42997

Closed

cmsbuild added tests-rejected and removed tests-started labels Oct 19, 2023

cmsbuild added tests-started and removed tests-rejected labels Oct 19, 2023

cmsbuild added tests-rejected and removed tests-started labels Oct 19, 2023

This was referenced Oct 20, 2023

Skip EcalRawData collection in ECAL DQM in GPU WFs cms-sw/cmssw#43058

Merged

Add ECAL portable data formats, collections and conditions for alpaka cms-sw/cmssw#42930

Merged

smuzaffar merged commit fe6c909 into cms-sw:IB/CMSSW_13_3_X/master Oct 20, 2023
23 of 26 checks passed

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Oct 20, 2023

cmsbuild mentioned this pull request Oct 21, 2023

[do not merge] testing root errors #8776

Closed

fwyzard deleted the IB/CMSSW_13_3_X/master_update_cuda_arch branch December 20, 2023 10:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update CUDA to support Pascal, Volta, Turing, Ampere and Lovelace GPUs #8767

Update CUDA to support Pascal, Volta, Turing, Ampere and Lovelace GPUs #8767

fwyzard commented Oct 18, 2023

fwyzard commented Oct 18, 2023

fwyzard commented Oct 18, 2023

cmsbuild commented Oct 18, 2023

smuzaffar commented Oct 18, 2023

cmsbuild commented Oct 19, 2023

smuzaffar commented Oct 19, 2023

smuzaffar commented Oct 19, 2023 •

edited

Loading

smuzaffar commented Oct 19, 2023

cmsbuild commented Oct 19, 2023

smuzaffar commented Oct 19, 2023

smuzaffar commented Oct 19, 2023

smuzaffar commented Oct 19, 2023

fwyzard commented Oct 19, 2023

cmsbuild commented Oct 19, 2023

cmsbuild commented Oct 20, 2023

fwyzard commented Oct 20, 2023

smuzaffar commented Oct 20, 2023

smuzaffar commented Oct 20, 2023

cmsbuild commented Oct 20, 2023

Update CUDA to support Pascal, Volta, Turing, Ampere and Lovelace GPUs #8767

Update CUDA to support Pascal, Volta, Turing, Ampere and Lovelace GPUs #8767

Conversation

fwyzard commented Oct 18, 2023

fwyzard commented Oct 18, 2023

fwyzard commented Oct 18, 2023

cmsbuild commented Oct 18, 2023

smuzaffar commented Oct 18, 2023

cmsbuild commented Oct 19, 2023

RelVals-GPU

GPU Unit Tests

Comparison Summary

smuzaffar commented Oct 19, 2023

smuzaffar commented Oct 19, 2023 • edited Loading

smuzaffar commented Oct 19, 2023

cmsbuild commented Oct 19, 2023

GPU Unit Tests

Comparison Summary

GPU Comparison Summary

smuzaffar commented Oct 19, 2023

smuzaffar commented Oct 19, 2023

smuzaffar commented Oct 19, 2023

fwyzard commented Oct 19, 2023

cmsbuild commented Oct 19, 2023

cmsbuild commented Oct 20, 2023

Unit Tests

RelVals

fwyzard commented Oct 20, 2023

smuzaffar commented Oct 20, 2023

smuzaffar commented Oct 20, 2023

cmsbuild commented Oct 20, 2023

smuzaffar commented Oct 19, 2023 •

edited

Loading