WIP: core(axe): new audits empty-table-header, frame-focusable-content #12536

connorjclark · 2021-05-22T00:59:46Z

Many bug fixes. A few new rules.

Need web.dev pages ... OR (hear me out...) we just link to axe docs...

connorjclark · 2021-05-22T01:00:49Z

lighthouse-core/test/results/artifacts/artifacts.json

@@ -2016,33 +2114,32 @@
    ],
    "incomplete": [
      {


dunno why color-contrast went bye-bye

dunno why color-contrast went bye-bye

hmm, didn't this change once before?

oh i guess it did and we decided to manually keep it around?

patrickhulce · 2021-05-22T20:55:37Z

lighthouse-core/audits/accessibility/aria-text.js

+  failureTitle: '`role=text` elements should not have focusable descendants',
+  // TODO: need web.dev article.
+  /** Description of a Lighthouse audit that tells the user *why* they should try to pass. This is displayed after a user expands the section to see more. No character length limits. 'Learn More' becomes link text to additional documentation. */
+  description: 'Browsers may ignore `role=text` if it contains focusable descendants.',


Suggested change

description: 'Browsers may ignore `role=text` if it contains focusable descendants.',

description: 'Browsers may ignore elements with `role=text` if they contain focusable descendants.',

hmm not sure on this one. browsers ignore the element _as far as role=text is concerned, which is why I put the focus on that. The element still exists, is visible, etc. so it isn't entirely ignored :)

lighthouse-core/audits/accessibility/aria-text.js

lighthouse-core/audits/accessibility/empty-table-header.js

lighthouse-core/audits/accessibility/aria-text.js

lighthouse-core/audits/accessibility/nested-interactive.js

lighthouse-core/config/default-config.js

patrickhulce · 2021-05-22T21:20:13Z

lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html

@@ -225,6 +225,21 @@ <h2>Do better web tester page</h2>
 <button class="small-button">Do something</button>
 <button class="small-button">Do something else</button>

+<!-- FAIL - frame-focusable-content -->
+<iframe tabindex="-1" src="focusable-iframe.html"></iframe>


use srcdoc instead to avoid the 1-line extra file? :)

That was my first try, didn't work! Not supported I suppose.

That was my first try, didn't work! Not supported I suppose.

interesting, are srcdoc iframes a unique origin and so not available in the parent via js (so axe wouldn't be able to see inside)?

brendankenny · 2021-05-24T17:17:56Z

lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html

+<!-- FAIL - aria-text -->
+<div role="text">
+  Still an <a href="#">interactive link</a> because of the author error.
+</div>


move to a11y_tester with the other accessibility smoke tests?

These are here to populate the sample LHR. Do we care about coverage there?

These are here to populate the sample LHR. Do we care about coverage there?

ah, got it. Ideally yes, so maybe just duplicate in there? It's very helpful to have yarn smoke a11y cover (almost?) everything in the accessibility category

lighthouse-core/gather/gatherers/accessibility.js

brendankenny · 2021-05-24T17:22:15Z

lighthouse-core/test/report/html/renderer/report-renderer-test.js

+          // TODO .lh-category-wrapper:nth-child(2) > .lh-category > .lh-clump--manual.lh-clump.lh-audit-group > summary
+          'nested-interactive': {enabled: false},


I guess this is the selector of the failure? This is going to be lost down here. How hard is the fix?

I don't know, it was Friday evening, this PR got big, and I just wanted to expedite feedback (since I knew there'd be quite a bit).

don't plan on pushing this PR w/o looking into this more. looks fun.

https://dequeuniversity.com/rules/axe/4.2/nested-interactive

so it seems that this rule fails b/c summary containing a link :/

I think this rule is not ready for LH–should we hide it, keep it to zero weight, or delete?

I think this rule is not ready for LH–should we hide it, keep it to zero weight, or delete?

I vote we delete it and aria-text but leave a TODO for nested-interactive so we update it next time if ready

lighthouse-core/test/report/html/renderer/report-renderer-test.js

Co-authored-by: Patrick Hulce <patrick.hulce@gmail.com>

brendankenny · 2021-05-25T20:11:49Z

lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html

+<!-- FAIL - aria-text -->
+<div role="text">
+  Still an <a href="#">interactive link</a> because of the author error.
+</div>


These are here to populate the sample LHR. Do we care about coverage there?

ah, got it. Ideally yes, so maybe just duplicate in there? It's very helpful to have yarn smoke a11y cover (almost?) everything in the accessibility category

brendankenny · 2021-05-25T20:13:19Z

lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html

@@ -225,6 +225,21 @@ <h2>Do better web tester page</h2>
 <button class="small-button">Do something</button>
 <button class="small-button">Do something else</button>

+<!-- FAIL - frame-focusable-content -->
+<iframe tabindex="-1" src="focusable-iframe.html"></iframe>


That was my first try, didn't work! Not supported I suppose.

interesting, are srcdoc iframes a unique origin and so not available in the parent via js (so axe wouldn't be able to see inside)?

brendankenny · 2021-05-25T20:16:19Z

lighthouse-core/audits/accessibility/aria-text.js

+'use strict';
+
+/**
+ * @fileoverview Ensures `role=text` is only used on elements with no focusable descendants.


Rob seemed like he was lukewarm about some of the possible aria-text gotchas and it's a best-practice, not wcag2a or wcag2aa. Should we drop for now?

rob said:

aria-text is kind of news to me and looks like it has a potential gotcha (like if you have a link inside of the heading) so...hm...idk

i said:

apparently aria-label is the role to use if element has focusable elements, not aria-text. altho TIL role=text isnt standard yet? https://incl.ca/roletext-presently-kinda-not-thing-sorta/#:~:text=here-,role%3D%22text%22%20isn,While,-that

or are you referring to something else?

role=text not being standard seems like a convincing reason to drop/hide/0-weight it.

yes, that's what I was trying to summarize without using many braincells, apparently :)

lighthouse-core/audits/accessibility/frame-focusable-content.js

brendankenny · 2021-05-25T20:25:16Z

lighthouse-core/audits/accessibility/nested-interactive.js

+
+const str_ = i18n.createMessageInstanceIdFn(__filename, UIStrings);
+
+class NestedInteractive extends AxeAudit {


if nested-interactive is problematic, I say we just put 'nested-interactive': {enabled: false}, in accessibility.js and put a TODO linked to your dequelabs/axe-core#2958. Looking at the history of nested-interactive, the discussion seems to be all around <button> and e.g. links inside elements that are role="checkbox", but then it became all presentational elements. We thought we were doing well in the report with our <details>, so I think in this case we at least need to figure out how we fix ourselves before we demand other people fix their own sites :)

lighthouse-core/audits/accessibility/empty-table-header.js

lighthouse-core/audits/accessibility/frame-focusable-content.js

brendankenny · 2021-05-25T20:45:06Z

lighthouse-core/test/results/artifacts/artifacts.json

@@ -2016,33 +2114,32 @@
    ],
    "incomplete": [
      {


dunno why color-contrast went bye-bye

hmm, didn't this change once before?

Co-authored-by: Brendan Kenny <bckenny@gmail.com>

brendankenny · 2021-05-27T15:44:37Z

@connorjclark @patrickhulce I think the main last thing here is deciding if we want to keep empty-table-header and frame-focusable-content in the 8.0 release. To summarize:

frame-focusable-content: from the axe test cases for it it does seem able to fail, but I can't get it to, even c/p the tests. Figure out how to make it fail? File a bug? Drop for now?
empty-table-header never fails, it only goes pass/incomplete/notApplicable. Since it's informative (based on what we did in core(audits): add handling of 'incomplete' results from axe-core #10072), it will show the error on incomplete, but the best the error message ever will be (with current axe) is Fix any of the following:\n Unable to determine if element has children.

edit: actually, we don't show that explanation (that's Better error messages in a11y test #12282), so the incomplete result isn't that bad. Weird that it's not scored for clear cases like the empty <th> though:

patrickhulce · 2021-05-27T16:00:37Z

frame-focusable-content: from the axe test cases for it it does seem able to fail, but I can't get it to, even c/p the tests. Figure out how to make it fail? File a bug? Drop for now?

Given time before 8.0, I'd say drop for now. We can always bring it back in a point release later.

empty-table-header never fails, it only goes pass/incomplete/notApplicable. Since it's informative (based on what we did in #10072), it will show the error on incomplete, but the best the error message ever will be (with current axe) is Fix any of the following:\n Unable to determine if element has children.

Does this mean even when the table header has text, it gets marked as incomplete? Or are you saying that in every case where it should be failing it shows up as incomplete instead? (Or somewhere in between) If it's just that the failing case is replaced with incomplete, I think that's fine. It's a bit weird to not be afailure but maybe it's not as big a deal 🤷 If it's any of the other two options. I'd also say drop.

brendankenny · 2021-05-27T16:15:19Z

Does this mean even when the table header has text, it gets marked as incomplete? Or are you saying that in every case where it should be failing it shows up as incomplete instead? (Or somewhere in between) If it's just that the failing case is replaced with incomplete, I think that's fine. It's a bit weird to not be afailure but maybe it's not as big a deal 🤷 If it's any of the other two options. I'd also say drop.

#10072 is a bit weird. The idea was that for axe rules that can't fail but get marked incomplete because they may be failing, we should make them scoreDisplayMode 'informative' so they'll still be shown on failure but not scored (since they might not be failing). That part is fine, but the passing case gets marked notApplicable, which is kind of weird because if you have a table with header text you think you should get the credit of passing 🤷

So in all it means

passing (but 0 weight) if you don't have any tables with headers (th or aria-based)
informative if you may have failed (with the result looking like the screenshot above)
notApplicable if you do have a table with a header with proper text

all that's fine, I guess? It's not part of wcag2a or wcag2aa which is why I'm more inclined to drop it, but I also understand the "that's fine" argument too :)

patrickhulce · 2021-05-27T16:17:14Z

passing (but 0 weight) if you don't have any tables with headers (th or aria-based)
informative if you may have failed (with the result looking like the screenshot above)
notApplicable if you do have a table with a header with proper text

lol so they flipped passing and notApplicable on this rule. I'd vote file a bug with axe and implement later if it's fixed.

patrickhulce · 2021-06-09T14:53:09Z

When we get back to this we should add an audit for select-name (fixes #12639). Seems like an oversight that this got missed. Maybe we should add an assertion that every axe rule that we're running has an audit?

jayaddison · 2021-06-19T19:43:16Z

So in all it means

* passing (but 0 weight) if you don't have any tables with headers (`th` or aria-based)

* informative if you may have failed (with the result looking like the screenshot above)

* notApplicable if you do have a table with a header with proper text

👋 hey @connorjclark @patrickhulce - I noticed this while looking back through some GitHub history.

After refreshing my memory and reading this thread, case three above (informative rule, passing elements on the page) does read as non-ideal. The relevant conditional is https://github.com/GoogleChrome/lighthouse/pull/10072/files#diff-7f8ed664776e43a5b44d4db584f9b7d12f8ce6a0588cd40ac3ed6338e6627dddR64.

I remember passes not being present in the Accessibility result structure, and at some point I was experimenting with adding that, before removing it again nearish merge time.

Footnotes:

I'm not sure I was (or am) best equipped to help out, but do feel an obligation to offer maintenance, so let me know if I can help (not required by any means, especially if this is further distraction).
core(audits): add handling of 'incomplete' results from axe-core #10072 was noisy, sorry about that
This comment is perhaps slightly defensive, but I do worry that I've potentially broken or downscored pages here, and that's ungreat

brendankenny · 2021-06-19T20:27:29Z

Really sorry if my comment came across as criticizing. The thing I was trying to say (but reading back, it's a bit too implicitly stated) is that the change was necessary to handle incomplete results, but that the notApplicable part left a subset of results from a single axe rule in a weird state because axe has one way of marking results as possible but not (currently) verifiable problems and lighthouse has its own way of doing the same, but they're not quite compatible. So sometimes you end up in this technically correct but hard to diagnose state for that one audit (two when we land empty-table-header).

I didn't raise any alarms specifically because this only affects form-field-multiple-labels and the audit is unscored no matter its result since it's scoreDisplayMode: 'informative', so we don't have to worry about any regression or anything.

Your thoughts on resolving the third state sound good to me, and maybe there's more we can do to help future updaters of axe in Lighthouse with knowing when to mark an audit informative, if only to shorten the process of trying to add every permutation of a test and having no clue as to why they cannot get that audit to fail :)

brendankenny · 2021-06-19T20:33:37Z

And there have been several people in the accessibility gatherer and axe-audit recently and have them fresh in our minds, so I don't think there's any problem with one of us taking the change (unless you want to) and hopefully we can just rope you in for consultation, if you have the time.

jayaddison · 2021-06-19T20:36:12Z

Thanks @brendankenny - nope, no problem in your messaging at all - I was worried by the idea of #10072 having had a wide negative impact, but am glad to hear and see that the number of informative rules is small. Phew :)

And yep, I recall that kind of venn-diagram-matching problem re: the slight differences between the way the two libraries model the world. A lot of my unease, I think, was that I'd lost whatever context I had since then, and that perhaps I hadn't understood the problem properly (still possible!).

I'll probably gladly opt-out of any further changes for now, a bit less time nowadays. Thank you though :)

connorjclark added 6 commits May 21, 2021 16:45

wip

01ca311

update

1273ca6

rm

c6c7175

more

32d4604

test

985f593

test

3158287

connorjclark requested a review from a team as a code owner May 22, 2021 00:59

connorjclark requested review from patrickhulce and removed request for a team May 22, 2021 00:59

google-cla bot added the cla: yes label May 22, 2021

connorjclark changed the title ~~WIP: deps(axe-core): upgrade to 4.2.1~~ deps(axe-core): upgrade to 4.2.1 May 22, 2021

devtools-bot assigned patrickhulce May 22, 2021

devtools-bot added the waiting4reviewer label May 22, 2021

connorjclark commented May 22, 2021

View reviewed changes

patrickhulce suggested changes May 22, 2021

View reviewed changes

patrickhulce added waiting4committer and removed waiting4reviewer labels May 22, 2021

brendankenny mentioned this pull request May 24, 2021

☂️ 👣 LH v8 burndown #11866

Closed

17 tasks

brendankenny reviewed May 24, 2021

View reviewed changes

Apply suggestions from code review

e3cf2fb

Co-authored-by: Patrick Hulce <patrick.hulce@gmail.com>

vercel bot deployed to Preview May 24, 2021 18:12 View deployment

strings

37e6bae

vercel bot deployed to Preview May 24, 2021 18:35 View deployment

brendankenny added the 8.0 label May 25, 2021

connorjclark added waiting4reviewer and removed waiting4committer labels May 25, 2021

brendankenny reviewed May 25, 2021

View reviewed changes

Apply suggestions from code review

c1268c9

Co-authored-by: Brendan Kenny <bckenny@gmail.com>

vercel bot deployed to Preview May 25, 2021 21:10 View deployment

patrickhulce removed the waiting4reviewer label May 25, 2021

dbw/devtools expectations

95012a2

vercel bot deployed to Preview May 27, 2021 15:37 View deployment

connorjclark mentioned this pull request May 27, 2021

deps(axe-core): upgrade to 4.2.1 #12575

Merged

connorjclark added chillin ❄ and removed 8.0 labels May 27, 2021

connorjclark changed the title ~~deps(axe-core): upgrade to 4.2.1~~ WIP: add new axe audits Jun 1, 2021

patrickhulce mentioned this pull request Jun 9, 2021

<select> elements with no label not picked up as an error #12639

Open

patrickhulce mentioned this pull request Jun 25, 2021

deps(axe-core): upgrade to 4.2.3 #12706

Merged

brendankenny mentioned this pull request Jun 25, 2021

tests: assert what axe checks matches our a11y audits #12699

Merged

paulirish mentioned this pull request Jun 29, 2021

new_audit(axe): input-button-name, role-img-alt, select-name #12700

Closed

connorjclark changed the title ~~WIP: add new axe audits~~ WIP: core(axe): new audits empty-table-header, frame-focusable-content Nov 1, 2021

patrickhulce removed their assignment Nov 15, 2021

connorjclark added 2 commits January 27, 2022 15:19

Merge remote-tracking branch 'origin/master' into axe-new

70d2999

update

86d7bbc

vercel bot deployed to Preview January 27, 2022 23:27 View deployment

update

2775ab0

vercel bot deployed to Preview January 27, 2022 23:34 View deployment

connorjclark closed this Apr 25, 2022

connorjclark mentioned this pull request Apr 25, 2022

audit the axe audits #13905

Closed

connorjclark mentioned this pull request Sep 6, 2022

Handle accessibility incomplete and violations simultaneously #14354

Open

1 task

connorjclark mentioned this pull request May 19, 2023

new_audit: table-fake-caption, html-xml-lang-mismatch, input-button-name #15098

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: core(axe): new audits empty-table-header, frame-focusable-content #12536

WIP: core(axe): new audits empty-table-header, frame-focusable-content #12536

connorjclark commented May 22, 2021 •

edited

Loading

connorjclark May 22, 2021

brendankenny May 25, 2021

connorjclark May 26, 2021

patrickhulce May 22, 2021

connorjclark May 24, 2021

patrickhulce May 22, 2021

connorjclark May 22, 2021

brendankenny May 25, 2021

brendankenny May 24, 2021

connorjclark May 24, 2021 •

edited

Loading

brendankenny May 25, 2021

brendankenny May 24, 2021

connorjclark May 24, 2021

connorjclark May 24, 2021 •

edited

Loading

connorjclark May 25, 2021

brendankenny May 25, 2021

brendankenny May 25, 2021

brendankenny May 25, 2021

brendankenny May 25, 2021

connorjclark May 25, 2021

brendankenny May 25, 2021

brendankenny May 25, 2021

brendankenny May 25, 2021

brendankenny commented May 27, 2021 •

edited

Loading

patrickhulce commented May 27, 2021

brendankenny commented May 27, 2021

patrickhulce commented May 27, 2021

patrickhulce commented Jun 9, 2021

jayaddison commented Jun 19, 2021

brendankenny commented Jun 19, 2021 •

edited

Loading

brendankenny commented Jun 19, 2021 •

edited

Loading

jayaddison commented Jun 19, 2021 •

edited by connorjclark

Loading

	description: 'Browsers may ignore `role=text` if it contains focusable descendants.',
	description: 'Browsers may ignore elements with `role=text` if they contain focusable descendants.',

		// TODO .lh-category-wrapper:nth-child(2) > .lh-category > .lh-clump--manual.lh-clump.lh-audit-group > summary
		'nested-interactive': {enabled: false},


		const str_ = i18n.createMessageInstanceIdFn(__filename, UIStrings);

		class NestedInteractive extends AxeAudit {

WIP: core(axe): new audits empty-table-header, frame-focusable-content #12536

WIP: core(axe): new audits empty-table-header, frame-focusable-content #12536

Conversation

connorjclark commented May 22, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

connorjclark May 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

connorjclark May 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brendankenny commented May 27, 2021 • edited Loading

patrickhulce commented May 27, 2021

brendankenny commented May 27, 2021

patrickhulce commented May 27, 2021

patrickhulce commented Jun 9, 2021

jayaddison commented Jun 19, 2021

brendankenny commented Jun 19, 2021 • edited Loading

brendankenny commented Jun 19, 2021 • edited Loading

jayaddison commented Jun 19, 2021 • edited by connorjclark Loading

connorjclark commented May 22, 2021 •

edited

Loading

connorjclark May 24, 2021 •

edited

Loading

connorjclark May 24, 2021 •

edited

Loading

brendankenny commented May 27, 2021 •

edited

Loading

brendankenny commented Jun 19, 2021 •

edited

Loading

brendankenny commented Jun 19, 2021 •

edited

Loading

jayaddison commented Jun 19, 2021 •

edited by connorjclark

Loading