KnownIssues.md

Known Issues

Known Issues are GitHub issues that are created to report and track known build or test errors.

Build Analysis shows a list of Known Issues for any build or test error that matches an existing Known Issue. This helps the developers to identify when a build break is not specific to their build. This information is displayed at the top of Build Analysis.

All reported Known Issues can be found on Known Issues project.

When and how to report a Known Issue

A Known Issue should be reported when you find a build or test error that is not caused by your changes and that affects or could affect more builds.

There are two types of Known Issues:

• Infrastructure Known Issue: An infrastructure issue is an issue that is not specific to your repository and that requires investigation by the engineering services (@dotnet/dnceng)
• Repository Known Issue: A repository issue is an issue that occurs in a particular repository and that should be investigated by the repository owners.

There are two ways to report a Known Issue: one is via Build Analysis and the other one is manually.

Reporting Known Issues via Build Analysis

1. In Build Analysis, you will see links for the type of issue you want to open.
2. Click on the link of the type of issue that suits the situation better.
3. A template is going to appear for you and most of this information should be already prefilled.
4. Fill out the JSON blob accordingly

Reporting a Known Issue manually

1. Decide if you need to open a repository issue or infrastructure issue

2. Open a new issue, choosing the repository in which you are opening the issue based on following rule:

   • Infrastructure issue - dnceng
   • Repository issue - In the repo in which the issue is happening

3. Add the label Known Build Error. (If the label is not available on the repository, follow the instructions to get onboard)

4. Copy and paste the template

   ## Build Information
   Build: <!-- Add link to the build with the reported error. -->
   Leg Name: <!-- Add the name of the impacted leg. -->
   
   <!-- Error message template  -->
   ## Error Message
   <!-- Fill for repository issues. For infrastructure issues the engineering services (@dotnet/dnceng) is going to fill it. -->
   ```json 
   { 
       "ErrorMessage": "",
       "BuildRetry": false,
       "ErrorPattern": "",
       "ExcludeConsoleLog": false
   } 
   ```

5. Fill out the JSON blob accordingly

Filling out Known Issues JSON blob

1. Fill the "Error message / Error pattern" section.
2. If the issue reported can be solved by retrying the build you can consider setting the "Build Retry" configuration to true
3. You should set ExcludeConsoleLog to true if you want to exclude console logs from Known Issues matching
4. You are done but what happens after a Known Issue is created?

Note: You need to escape special characters to make your JSON valid, to do it you can use online tools like freeformatter.

For example, if your error message is text "using" quotes

You need to enter text \"using\" quotes as the ErrorMessage value

You can use the preview tab to validate your JSON blob, GitHub will highlight invalid characters.

Fill out Known Issues with a list of errors

This section describes how to specify multiple error messages / error patterns to search for in order, as an AND condition, and every message/pattern being exclusive to a single line.

• To use the feature, you need to pass a list of strings as part of the JSON.

  For example, an ErrorMessage will look like this:

  {
  "ErrorMessage": ["Assert.True() Failure", "Actual:   False"]
  }

  And an ErrorPattern will look like this:

  {
  "ErrorPattern": ["Assert.True() .+", "Actual:.*False"]
  }

• The list works as an AND and they must be in order. Every issue in the list will try to match a line.

• The first issue in the list will be searched first, and then if the first issue matches, the second issue will be searched in the subsequent lines.

• In order to count as a match, all issues in the list must match.

• This works for ErrorMessage and ErrorPattern, but they should not be mixed. If you use ErrorMessage, then all issues in the list must match the ErrorMessage, and the same for ErrorPattern.

Note that every pattern/message will be searched per line, NOT multiple lines.

Example

Consider the following error message:

Assert.True() Failure
Expected: True
Actual:   False

In order to match this error message, you can use the following JSON:

{
  "ErrorMessage": ["Assert.True() Failure", "Actual:   False"]
}

Even though we didn't put the Expected: True line, the other lines are in there and in the right order, so it will match.

However, if we use the following JSON:

{
  "ErrorMessage": ["Assert.True() Failure", "Actual: True"]
}

The last line doesn’t match, so it will not match the error message.

How the matching process works between a Known Issue and a build/test error

The Known Issues feature identifies build and test errors and matches them with open Known Issues. It uses String.Contains to compare the errors with the “ErrorMessage” property of Known Issues, or regex matching if an “ErrorPattern” property is provided.

The matching process and its limitations depend on the type of error:

• Build error: For build errors, the feature searches for matches in two sources:

  • Azure DevOps error messages
  • Build log of failed jobs

• Test errors: For test errors, the feature analyzes the errors only when the build has 1000 or fewer failing tests. It also limits the analysis to 100 Helix logs. This limitation is due to the high cost of analyzing tests and Helix logs. The feature uses the following information to find matches:

  • Error message
  • Stack trace
  • Helix log for Helix tests

How to fill out a Known Issue error section

For the error matching to work, you need to provide an error message or an error pattern.

{
    "ErrorMessage":"",
    "ErrorPattern": "",
}

String matching

The ErrorMessage value needs to be updated with an error message that meets the following requirements:

1. Match at least one of the error messages of the reported build. For more details see: How matching process works

2. It shouldn't include any unique identifier of the message e.g., machine, path, file name, etc.

   For example, for the following error:

   ##[error].dotnet/sdk/6.0.100-rc.1.21411.28/NuGet.RestoreEx.targets(19,5): error : (NETCORE_ENGINEERING_TELEMETRY=Restore) Failed to retrieve information about 'Microsoft.Extensions.Hosting.WindowsServices' from remote source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/c9f8ac11-6bd8-4926-8306-f075241547f7/nuget/v3/flat2/microsoft.extensions.hosting.windowsservices/index.json'.
   

   We could choose this part of the message: (NETCORE_ENGINEERING_TELEMETRY=Restore) Failed to retrieve information. This is a good option because it targets a particular problem but excludes details of the build, thus making it possible for this same error to match with other builds.

After selecting the message, fill ErrorMessage value:

{ 
    "ErrorMessage":"(NETCORE_ENGINEERING_TELEMETRY=Restore) Failed to retrieve information" 
}

Regex matching

If you need to match an error using a regex, you should use the ErrorPattern value.

In the following example, the regular expression The command .+ failed would match errors like The command 'open explorer' failed

{
    "ErrorPattern": "The command .+ failed",
}

We recommend you test your regular expression, to do it you can use regex101 tester (choose .NET (C#) flavor) with the following regex options:

• Single line
• Insentitive
• No backtracking

What happens after creating or updating a Known Issue

Known Issues scan all builds from the last 24 hours since the issue was opened or updated with the error message provided. It also scans all builds that fail after the issue is created.

Known issues analyzes both infrastructure issues (Known Issues in dotnet/dnceng) and repository issues (Known Issues in the pull request’s repository).

Additionally displays the Known Issues matches in the Build Analysis check. The example below shows that the feature found 4 matches for the issue “Tracking issue for CI build timeouts”.

Build retry functionality in Known Issues

Build Analysis can retry a build that fails with an error that matches a Known Issue. To enable this functionality, the Known Issue must have the ‘BuildRetry’ property set to true. This property indicates that the build failure may be solved by retrying the build.

The following is an example of a Known Issue with the ‘BuildRetry’ property:

{ 
    "ErrorMessage":"The agent did not connect within the alloted time",
    "BuildRetry": true
} 

In this example, if a build fails with the error “The agent did not connect within the allotted time” on its first attempt, the feature will retry the build.

However, this functionality has some limitations. It only retries the build if the failure occurs on the first attempt. If the failure happens on a later attempt, the build will not be retried.

This limitation is designed for two reasons:

1. If a build fails multiple times, the underlying cause may be something different.
2. Many builds are analyzed and retrying them repeatedly can be costly and problematic.

How to get onboard

1. This feature depends on the Build Analysis feature. Therefore, you need to have the Build Analysis GitHub application installed in the repo where you want to use Known Issues.
   To install the application, you can contact the .NET Core Engineering Services team
2. For infrastructure issues, there are no additional steps because they are part of dotnet/dnceng.
3. For repository issues, you need to Create a label on the repo where you want to open the Repository issue.
   The name of the label must be Known Build Error

Decide Infrastructure or Repository issue

• Infrastructure:
  • The issue is not exclusive to a repository
  • Needs to be investigated by the engineering services (@dotnet/dnceng)
  • It hasn't been reported on dnceng
• Repository:

  • The issue is happening in a particular repository

  • The error needs to be investigated by the repository owners.

  • The problem hasn't been reported in the REPOSITORY, use the following query to verify that:

    https://github.com/dotnet/<REPOSITORY>/issues?q=is%3Aopen+is%3Aissue+label%3A%22Known+Build+Error%22
    

    or see all repository issues at Repository issues board

Where to find the Known Issue on the Build Analysis

The Known Issues are listed at the top of the Build Analysis.

E.g.

_{Was this helpful?}

Telemetry

Build Analysis sends Known Issues telemetry to the engineeringdata Kusto DB. The telemetry includes data about all the matches between build breaks and Known Issues the automation was able to find.

Two tables are used to store this data, KnownIssues for build related matches and TestKnownIssues for test related matches. You can use the following columns, in both tables, to build queries:

• Build Id
• Build repository
• Known Issue Id
• Known Issue repository
• Pull request that triggered the build (if available)

The query below returns all builds affected by Known Issue 76454 in the runtime repo

KnownIssues
| where IssueId == 76454 and IssueRepository == "dotnet/runtime"

Known Issues board

People can look at active Known Issues using the .NET Core Engineering Services: Known Build Errors GitHub project. In the board, Known Issues are divided into tabs, one for infrastructure related issues, the second one for repo specific issues and the last one for all infrastructure issue that have been created (opened and closed).