Javadoc validation

How the javadoc validation test works

This is an attempt to document the process I'm not familiar with

please fix if anything here is wrong or incomplete!

Test artifacts

Javadoc errors are logged under "Javadoc Logs" page on each platform build, like in 4.27.

If the javadoc error files are not empty, org.eclipse.releng.tests.BuildTests.testJavadocLogs will report a fail.

The logs about javadoc execution itself are under "Release engineering build logs" and there under "mb220_buildSdkPatch.sh.log".

Touch_doc_bundles

This is a workaround for stopping comparator to throwaway newly built bundles.

Build Process

There are two items in the build process we need to be aware of

Calculation of qualifier

The qualifier is calculated as timestampof latest file change among the source files for that bundle. Basically the last commit timestamp for that bundle. Please note: The timestamp is in UTC for plugins and New York time for features

Workings of comparator

Towards the end of building a bundle comparator kicks in. here are the steps it follows

1. Identify if there are any bundles with the same version and qualifier combination in comparator baseline. Comparator baseline is defined by comparator.repo property in the parent pom. example <comparator.repo>https://download.eclipse.org/eclipse/updates/4.28-I-builds</comparator.repo>
2. If not found retain currently built bundle
3. If found
   1. Compare the contents in baseline bundle and current built bundle. Report any differences. These differences are reposted as comparator errors
   2. Replace currently built bundle with the one from comparator baseline. This happens irrespective of whether we have differences in old and newly built bundles

Building of doc bundles

In three doc bundles we include generated javadoc from the bundle sources. While calculating the qualifier we don't consider the java doc sources. This causes qualifier to remain same across multiple builds. Since the qualifier remains same at the end of the build the comparator replaces new javadoc with old javadoc. To avoid replacement by comparator we need to bump up version or qualifier.

The job https://ci.eclipse.org/releng/job/Touch_doc_bundles/ bumps up qualifier so the comparator will not find a baseline version. This we will retain newer bundle.

TODO

How the build is triggered and where to find tasks output which produce error files like jdt.doc.isv.javadoc.txt

Expected flow

Javadoc generation uses a independent javadoc executable and runs as a separate process. One need to add all the required dependencies in the classpath used by javadoc executable. This done in the Options.txt files inside documentation bundles projects. The projects are under https://github.com/eclipse-platform/eclipse.platform.common/tree/master/bundles

Example https://github.com/eclipse-platform/eclipse.platform.common/pull/19/files

See also https://wiki.eclipse.org/How_to_add_things_to_the_Eclipse_doc

Source code for javadoc bundles is located at https://github.com/eclipse-platform/eclipse.platform.common. javadoc is generated using ant script buildDoc.xml located in each of the java doc plugin source. we use eclipse antrunner to run this script. The specific location where we actually generate is https://github.com/eclipse-platform/eclipse.platform.common/blob/54455a117960237d69a9d840e61872641cafdaaa/bundles/org.eclipse.jdt.doc.isv/buildDoc.xml#L141. The javadoc executable will generate a doc.bin.log file with all the errors javadoc executable found. This gets renamed appropriately with this line https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/1bbda0a96dc04e93bdc1bb91813a5d98b5659653/cje-production/mbscripts/mb300_gatherEclipseParts.sh#L247

How to reproduce build locally

This is work in progress

1. Build SDK
2. After that, see mb220_buildSdkPatch.sh and JenkinsJobs/Builds/I_build.groovy
3. Run something like below (to be tested)

export MAVEN_OPTS="-Xmx6G"
export CJE_ROOT="/path/to/eclipse.platform.releng.aggregator/eclipse.platform.releng.aggregator/cje-production"
export logDir="$CJE_ROOT/buildlogs"
mkdir -p $logDir
cd /path/to/eclipse.platform.releng.aggregator/eclipse.platform.releng.aggregator/cje-production/mbscripts
./mb220_buildSdkPatch.sh $CJE_ROOT/buildproperties.shsource 2>&1 | tee $logDir/mb220_buildSdkPatch.sh.log