No execuation time comparison available for PRs

[mandrenguyen]

Since a few months we are not able to see the CPU impact of a given pull request, which used to be possible with the enable profiling option in the Jenkins tests.
This is a bit problematic for integrating new features, as we won't easily be able to keep track of changes in performance until a pre-release is built.
The issue seems to come from igprof, which apparently can no longer really be supported.
One suggestion from @gartung is to try to move to VTune.

[mandrenguyen]

assign core, reconstruction

[cmsbuild]

New categories assigned: core,reconstruction

@Dr15Jones,@jfernan2,@makortel,@mandrenguyen,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

A new Issue was created by @mandrenguyen Matthew Nguyen.

@rappoccio, @antoniovilela, @sextonkennedy, @makortel, @smuzaffar, @Dr15Jones can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[gartung]

The problem is that the cmsRun process itself gets a segfault while being profiled by Igprof. The same segfault might happen when being profiled with Vtune.

[makortel]

In case IgProf+cmsRun combination crashes, is any information on the job timings saved that can be used for comparison?

[gartung]

Usually the FastTimerService job completes and the average per module is contained in the raw json file if the resources piechart is not readable.

[gartung]

The IgprofService dumps the profile after the first, middle and next to last event. The first one might not have enough data to be meaningful.

[gartung]

@mandrenguyen Can you point me to a PR so I can look at the logs.

[mandrenguyen]

@gartung The last one we tried with profiling enabled was:
#43107

[makortel]

The crashes under profilers are quite likely caused by the memory corruption inside Tensorflow (when ran through IgProf or VTune) that has been investigated in #42444.

[VinInn]

The FastTimer Service should suffice. Still It seems not active in RelVals

[mmusich]

The issue seems to come from igprof, which apparently can no longer really be supported.
[...] is to try to move to VTune.

for my education is this replacement documented somewhere ?
I still see igprof listed in the RecoIntegration CMS Twiki.

[jfernan2]

@mmusich it is expected that VTune gives the same problem as igprof, so the replacement has not been done.
Indeed this is a real showstopper for RECO developments since we cannot monitor the time profiling in PRs

[mmusich]

so the replacement has not been done.
Indeed this is a real showstopper for RECO developments since we cannot monitor the time profiling in PRs

I see, that's bad news. I gather the same holds true for user checks when developing (regardless of the time profiling in PRs)

[makortel]

Is the most burning problem that there is no timing information (entire job, per module) or that the real IgProf/VTune profile (with function-level information) is missing (because of crash)?

[mmusich]

Is the most burning problem that there is no timing information (entire job, per module) or that the real IgProf/VTune profile (with function-level information) is missing (because of crash)?

for me (personally) at least, having the function level information would be really helpful.

[jfernan2]

IMHO the crash of igprof/Vtune is a problem although there is timing info from FastTimer module, but the real issue is not having a comparison of baseline time performance vs baseline+PR, which force us to detect a posteriori total increases in the profiles when a prerelease is built, and then figure out which PR(s) were responsible....

Perhaps a comparison script based on FastTimer output could be useful even if not optimal, do you think this is possible @gartung ?
Thanks

[gartung]

Yes it would be possible. In fact there is a script already that merges two FastTimer output files
https://github.com/fwyzard/circles/blob/master/scripts/merge.py
Changing that to diff two files should be possible.

[gartung]

You can try this script

https://raw.githubusercontent.com/gartung/circles/master/scripts/diff.py

[gartung]

If you add enable profiling as a comment on the pull request the FastTimer service is run as part of the profiling.