Writes the test properties, CSSProjectStructure, CSSIteration, WorkItemIds and Description to the TRX log #1801

dotMorten · 2018-10-18T04:49:33Z

Description

Writes the description, CSS*, test properties and workitem ids to the TRX log in accordance with the vstst.xsd schema

Related issue

The missing traits were discussed in this issue: #98
This PR addressing the lack of CssProjectStructure, CssIteration, Description, Properties/traits and WorkItemIds. The generated TRX has been validated against the schema.

Here's an example of TRX output from a test with 4 properties, where 3 of them have special places to go in the TRX:

        [TestMethod]
        [Priority(4)]
        [TestProperty("Test prop", "Prop val")]
        [WorkItem(1234)]
        [Owner("Morten")]
        [Description("This test contains 4 properties and a priority")]
        [CssProjectStructure("cssprojectstruct_goes_here")]
        [CssIteration("iteration_goes_here")]
        public void TestWithProperties()
        {
        }

TRX:

<?xml version="1.0" encoding="utf-8"?>
<TestRun id="1dbc39c8-cd68-4684-aae1-3cf98dc97760" name="mort5161@MNIELSEN1 2018-10-17 22:48:14" runUser="AVWORLD\mort5161" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2018-10-17T22:48:14.4655780-07:00" queuing="2018-10-17T22:48:14.4655784-07:00" start="2018-10-17T22:48:11.1260893-07:00" finish="2018-10-17T22:48:14.6676358-07:00" />
  <TestSettings name="default" id="61058af8-5506-4b9d-b8f2-1d263543cf12">
    <Deployment runDeploymentRoot="mort5161_MNIELSEN1_2018-10-17_22_48_14" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="49d87ce0-a1b6-4497-a848-9df61cd0d438" testId="40b0cd91-e3c2-d2c7-c9de-8d3b3d7e3bd1" testName="TestWithProperties" computerName="MNIELSEN1" duration="00:00:00.0128826" startTime="2018-10-17T22:48:13.8234820-07:00" endTime="2018-10-17T22:48:13.9008950-07:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="49d87ce0-a1b6-4497-a848-9df61cd0d438" />
  </Results>
  <TestDefinitions>
    <UnitTest name="TestWithProperties" storage="unittests.dll" priority="4" id="40b0cd91-e3c2-d2c7-c9de-8d3b3d7e3bd1">
      <Css projectStructure="cssprojectstruct_goes_here" iteration="iteration_goes_here" />
      <Description>This test contains 4 properties and a priority</Description>
      <Owners>
        <Owner name="Morten" />
      </Owners>
      <Execution id="49d87ce0-a1b6-4497-a848-9df61cd0d438" />
      <Properties>
        <Property>
          <Key>Test prop</Key>
          <Value>Prop val</Value>
        </Property>
      </Properties>
      <WorkItemIDs>
        <WorkItem id="1234" />
      </WorkItemIDs>
      <TestMethod codeBase="UnitTests.dll" adapterTypeName="executor://mstestadapter/v2" className="UnitTests.Tests" name="TestWithProperties" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="40b0cd91-e3c2-d2c7-c9de-8d3b3d7e3bd1" executionId="49d87ce0-a1b6-4497-a848-9df61cd0d438" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Completed">
    <Counters total="1" executed="1" passed="1" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
  </ResultSummary>
</TestRun>

…nce with the vstst.xsd schema

…o written as properties

mayankbansal018 · 2018-10-19T01:58:00Z

@dotMorten , will this change existing trx schema? I'll take a deeper look once I'm back on Monday.

dotMorten · 2018-10-19T03:23:32Z

No this is just populating the properties already declared in the schema. I validated the output and it parses against the schema. There's nothing in there that isn't declared in the schema already

src/Microsoft.TestPlatform.Extensions.TrxLogger/ObjectModel/TestElement.cs

mayankbansal018

dotMorten · 2018-11-16T06:23:09Z

Updated the private field

singhsarab

I don't see any tests for these changes. Can we please add.

src/Microsoft.TestPlatform.Extensions.TrxLogger/ObjectModel/TestPropertyItems.cs

singhsarab · 2018-11-16T06:56:53Z

src/Microsoft.TestPlatform.Extensions.TrxLogger/Utility/Converter.cs

            testElement.ExecutionId = new TestExecId(executionId);
            testElement.ParentExecutionId = new TestExecId(parentExecutionId);
+            testElement.WorkItemIds = GetWorkItemIds(rockSteadyTestCase);
+
+            foreach (var trait in rockSteadyTestCase.Traits)


Description, WorkItem, CssIteration and CssProjectStructure, are they part of the Traits collection ?

They are all traits - however in the TRX report they have their own specific locations.

@dotMorten These are specific to MSTest framework, aren't they ? If yes, can we try avoid these hardcoded checks.
Also, these have not been part of the Traits on MSTestv1, but started showing up in MSTestv2 only and we have already fixed that issue in the adapter.

can we try avoid these hardcoded checks

These values were already hardcoded (apart from the new ones I added). Look for instance in the GetPriority(…) and GetOwner(…) methods.

these have not been part of the Traits on MSTestv1, but started showing up in MSTestv2 only and we have already fixed that issue in the adapter.

I'm not sure what you're getting at here or what you want me to do?

Principle we are trying to follow is TestPlatform should have minimal MSTest framework specific code.

These properties (Description, WorkItem, CssIteration and CssProjectStructure) should not have been part of the Traits collection, that was a bug in the adapter and has been fixed.

I really like the idea of Testproperty name-value available in the Trx. For these properties, I'd rather like them to be available as name value pairs like any other test property.

I agree we have Priority and Owner as first class nodes, but these are legacy and have been there in older framework (backward compatibility).

Considering Description for first class node still does make sense to me, as it can be extended to other testing frameworks.

I'm not clear on how these are MSTest specific in any way. The trait names are rather generic. However there's an existing xml schema that defines all these properties as specific non-trait fields. These aren't the only properties that are pulled out from the traits collection and saved to the TRX in their own fields. I'm following the TRX schema 1:1 here. If another test framework wants to use for instance "Description" all they have to do is report a trait as "Description".
Whether non-traits should have been stored as traits in the first place is a moot point - that ship has already sailed, and this PR merely builds on what is already there and couldn't really change without breaking anyone.

These properties (Description, WorkItem, CssIteration and CssProjectStructure) should not have been part of the Traits collection, that was a bug in the adapter and has been fixed

As I had pointed out in the PR related to that, that change was a significant breaking change that I'm assuming you'll be rolling back before the next release before you break anyone. This specific PR was done this way to avoid changing any existing behavior and follow the exact patterns that were already established, so I have a hard time understand what the problem here is.

[Updating the comment]

Thanks for the explanation.

My main concern was around use of Traits to get these properties.

And microsoft/testfx#482 This change is much needed as it broke the experience for multiple customers and these unwanted traits started showing up in the TestExplorer while grouping via Traits.

I know currently these properties are available only through traits and not as part of Store property (which is dictionary of testproperty and value) available in the TestCase object.
But, we can fix that in the adapter to start populating these properties.

Also, frameworks like Nunit today populate traits as TestProperty.
What I am proposing is we first fix the adapter to start populating these properties as part of the store, and not as Traits. Then consume these to populate the trx element

I have raised a PR for these test properties to be available in the TestCase object. Here is the PR: microsoft/testfx#538

dotMorten · 2018-11-16T07:47:13Z

I don't see any tests for these changes. Can we please add.

It's not clear to me how you test the output of a trx report. Please advise/help

singhsarab · 2018-11-20T13:27:42Z

Regarding the tests, We can add a few test cases in ConverterTests for verifying that we are able to convert to relevant properties in TestElement.

https://github.com/Microsoft/vstest/blob/master/test/Microsoft.TestPlatform.Extensions.TrxLogger.UnitTests/Utility/ConverterTests.cs

agehrke · 2019-03-25T08:07:27Z

I would really like to help to get output of traits implemented in trx. Can I create a new PR using @dotMorten's code that only outputs traits as key/value elements as he proposed, and thus skipping special handling of Description, WorkItem, CssIteration and CssProjectStructure? I will add some tests as mentioned in @singhsarab's last comment.

singhsarab · 2019-03-25T09:29:18Z

@agehrke I am sure @dotMorten would like that.
Please let me know if you need any help.

schibu007 · 2020-12-04T09:17:29Z

This part of the resulting XML seems to be wrong/other than former MSTest.exe created it. VS2019 is not able to show the Associated Work Items in its Test Result viewer.
Actual:

...
    <WorkItemIDs>
        <WorkItem id="1234" />
    </WorkItemIDs>
...

Expected:

...
   <Workitems>
       <Workitem>1234</Workitem>
   </Workitems>
...

schibu007 · 2020-12-04T09:21:12Z

Apart from my former comment, what's the status of this PR? We look forward to get support for the work items in trx file.

dotMorten added 5 commits October 17, 2018 21:24

Writes the test properties and workitem ids to the TRX log in accorda…

6f0a12f

…nce with the vstst.xsd schema

Added support for Descrition and ensure owner and priority aren't als…

ec5ddb4

…o written as properties

Ensure The two CSS traits doesn't get included in the test properties

ca7dfc7

Fix typo

a2296dc

Added support for CSS Project Structure and CSS Iteration properties

bc1b9a1

dotMorten changed the title ~~Writes the test properties, workitem ids and description to the TRX log~~ Writes the test properties, CSSProjectStructure, CSSIteration, WorkItemIds and Description to the TRX log Oct 18, 2018

singhsarab requested a review from mayankbansal018 October 23, 2018 12:32

mayankbansal018 added 4 commits October 24, 2018 20:18

Merge branch 'master' into TrxProperties

b2879a6

Merge branch 'master' into TrxProperties

f3b710c

Merge branch 'master' into TrxProperties

89d75e2

Merge branch 'master' into TrxProperties

3a833cf

mayankbansal018 requested a review from singhsarab November 16, 2018 05:55

mayankbansal018 reviewed Nov 16, 2018

View reviewed changes

src/Microsoft.TestPlatform.Extensions.TrxLogger/ObjectModel/TestElement.cs Outdated Show resolved Hide resolved

mayankbansal018 approved these changes Nov 16, 2018

View reviewed changes

Change private field to protected for consistency

5cc3494

singhsarab suggested changes Nov 16, 2018

View reviewed changes

Fixed a few nit issues

21858c4

Merge branch 'master' into TrxProperties

934ff29

agehrke mentioned this pull request Mar 25, 2019

Include traits from TestCase in TRX output #1968

Closed

Haplois mentioned this pull request Dec 5, 2020

Implement Workitem support in TRX logger #2666

Merged

Haplois closed this Dec 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Writes the test properties, CSSProjectStructure, CSSIteration, WorkItemIds and Description to the TRX log #1801

Writes the test properties, CSSProjectStructure, CSSIteration, WorkItemIds and Description to the TRX log #1801

dotMorten commented Oct 18, 2018 •

edited

Loading

mayankbansal018 commented Oct 19, 2018

dotMorten commented Oct 19, 2018

mayankbansal018 left a comment

dotMorten commented Nov 16, 2018

singhsarab left a comment

singhsarab Nov 16, 2018

dotMorten Nov 16, 2018

singhsarab Nov 16, 2018

dotMorten Nov 16, 2018 •

edited

Loading

singhsarab Nov 19, 2018

dotMorten Nov 20, 2018

singhsarab Nov 20, 2018 •

edited

Loading

singhsarab Dec 11, 2018

dotMorten commented Nov 16, 2018 •

edited

Loading

singhsarab commented Nov 20, 2018

agehrke commented Mar 25, 2019

singhsarab commented Mar 25, 2019

schibu007 commented Dec 4, 2020

schibu007 commented Dec 4, 2020

Writes the test properties, CSSProjectStructure, CSSIteration, WorkItemIds and Description to the TRX log #1801

Writes the test properties, CSSProjectStructure, CSSIteration, WorkItemIds and Description to the TRX log #1801

Conversation

dotMorten commented Oct 18, 2018 • edited Loading

Description

Related issue

mayankbansal018 commented Oct 19, 2018

dotMorten commented Oct 19, 2018

mayankbansal018 left a comment

Choose a reason for hiding this comment

dotMorten commented Nov 16, 2018

singhsarab left a comment

Choose a reason for hiding this comment

singhsarab Nov 16, 2018

Choose a reason for hiding this comment

dotMorten Nov 16, 2018

Choose a reason for hiding this comment

singhsarab Nov 16, 2018

Choose a reason for hiding this comment

dotMorten Nov 16, 2018 • edited Loading

Choose a reason for hiding this comment

singhsarab Nov 19, 2018

Choose a reason for hiding this comment

dotMorten Nov 20, 2018

Choose a reason for hiding this comment

singhsarab Nov 20, 2018 • edited Loading

Choose a reason for hiding this comment

singhsarab Dec 11, 2018

Choose a reason for hiding this comment

dotMorten commented Nov 16, 2018 • edited Loading

singhsarab commented Nov 20, 2018

agehrke commented Mar 25, 2019

singhsarab commented Mar 25, 2019

schibu007 commented Dec 4, 2020

schibu007 commented Dec 4, 2020

dotMorten commented Oct 18, 2018 •

edited

Loading

dotMorten Nov 16, 2018 •

edited

Loading

singhsarab Nov 20, 2018 •

edited

Loading

dotMorten commented Nov 16, 2018 •

edited

Loading