hide internal types/traits from std docs via new #[doc(masked)] attribute #44026

QuietMisdreavus · 2017-08-22T01:43:16Z

Fixes #43701 (hopefully for good this time)

This PR introduces a new parameter to the #[doc] attribute that rustdoc looks for on extern crate statements. When it sees #[doc(masked)] on such a statement, it hides traits and types from that crate from appearing in either the "Trait Implementations" section of many type pages, or the "Implementors" section of trait pages. This is then applied to the libc/rand/compiler_builtins imports in libstd to prevent those crates from creating broken links in the std docs.

Like in #43348, this also introduces a feature gate, doc_masked, that controls the use of this parameter.

To view the std docs generated with this change, head to https://tonberry.quietmisdreavus.net/std-43701/std/index.html.

rust-highfive · 2017-08-22T01:43:20Z

r? @steveklabnik

(rust_highfive has picked a reviewer for you, use r? to override)

QuietMisdreavus · 2017-08-22T01:45:24Z

Tracking issue: #44027

kennytm · 2017-08-22T07:18:32Z

src/test/rustdoc/doc-masked.rs

+
+// @has doc_masked/struct.LocalStruct.html
+// @has - '//*[@class="impl"]//code' 'impl LocalTrait for LocalStruct'
+// @!has - '//*[@class="impl"]//code' 'impl Copy for LocalStruct'


Masking failed. I guess it's because Copy and usize are from libcore instead of libstd?

[00:50:50] ---- [rustdoc] rustdoc/doc-masked.rs stdout ---- [00:50:50] [00:50:50] error: htmldocck failed! [00:50:50] status: exit code: 1 [00:50:50] command: "/usr/bin/python2.7" "/checkout/src/etc/htmldocck.py" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc/doc-masked.stage2-x86_64-unknown-linux-gnu" "/checkout/src/test/rustdoc/doc-masked.rs" [00:50:50] stdout: [00:50:50] ------------------------------------------ [00:50:50] [00:50:50] ------------------------------------------ [00:50:50] stderr: [00:50:50] ------------------------------------------ [00:50:50] 18: @!has check failed [00:50:50] `XPATH PATTERN` did not match [00:50:50] // @!has - '//*[@class="impl"]//code' 'impl Copy for LocalStruct' [00:50:50] 24: @!has check failed [00:50:50] `XPATH PATTERN` did not match [00:50:50] // @!has - '//*[@id="implementors-list"]//code' 'impl LocalTrait for usize' [00:50:50] [00:50:50] Encountered 2 errors [00:50:50] [00:50:50] ------------------------------------------ [00:50:50] [00:50:50] thread '[rustdoc] rustdoc/doc-masked.rs' panicked at 'explicit panic', /checkout/src/tools/compiletest/src/runtest.rs:2511:8 [00:50:50] note: Run with `RUST_BACKTRACE=1` for a backtrace. [00:50:50] [00:50:50] [00:50:50] failures: [00:50:50] [rustdoc] rustdoc/doc-masked.rs [00:50:50] [00:50:50] test result: FAILED. 162 passed; 1 failed; 1 ignored; 0 measured; 0 filtered out

Oh, that makes more sense. I saw the travis failure last night, but chalked it up to getting the htmldocck rules wrong. I'll poke at this more and see what can be done about it.

ollie27 · 2017-08-22T21:33:47Z

src/libstd/lib.rs

 #[macro_use]
 #[macro_reexport(vec, format)]
 extern crate alloc;
 extern crate alloc_system;
 extern crate std_unicode;
+#[doc(masked)]
 extern crate libc;

 // We always need an unwinder currently for backtraces
 extern crate unwind;


unwind would also need to be doc(masked) to get rid of the impls for _Unwind_Reason_Code and _Unwind_Action on the Clone page for example.

kennytm · 2017-08-25T14:44:07Z

4 failures now 😒

[00:53:35] ---- [rustdoc] rustdoc/doc-masked.rs stdout ----
[00:53:35] 	
[00:53:35] error: htmldocck failed!
[00:53:35] status: exit code: 1
[00:53:35] command: "/usr/bin/python2.7" "/checkout/src/etc/htmldocck.py" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc/doc-masked.stage2-x86_64-unknown-linux-gnu" "/checkout/src/test/rustdoc/doc-masked.rs"
[00:53:35] stdout:
[00:53:35] ------------------------------------------
[00:53:35] 
[00:53:35] ------------------------------------------
[00:53:35] stderr:
[00:53:35] ------------------------------------------
[00:53:35] 18: @!has check failed
[00:53:35] 	`XPATH PATTERN` did not match
[00:53:35] 	// @!has - '//*[@class="impl"]//code' 'impl Copy for LocalStruct'
[00:53:35] 
[00:53:35] Encountered 1 errors
[00:53:35] 
[00:53:35] ------------------------------------------
[00:53:35] 
[00:53:35] thread '[rustdoc] rustdoc/doc-masked.rs' panicked at 'explicit panic', /checkout/src/tools/compiletest/src/runtest.rs:2435:8
[00:53:35] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[00:53:35] 
[00:53:35] ---- [rustdoc] rustdoc/extern-impl.rs stdout ----
[00:53:35] 	
[00:53:35] error: htmldocck failed!
[00:53:35] status: exit code: 1
[00:53:35] command: "/usr/bin/python2.7" "/checkout/src/etc/htmldocck.py" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc/extern-impl.stage2-x86_64-unknown-linux-gnu" "/checkout/src/test/rustdoc/extern-impl.rs"
[00:53:35] stdout:
[00:53:35] ------------------------------------------
[00:53:35] 
[00:53:35] ------------------------------------------
[00:53:35] stderr:
[00:53:35] ------------------------------------------
[00:53:35] 32: @has check failed
[00:53:35] 	`XPATH PATTERN` did not match
[00:53:35] 	// @has - '//code' 'impl Bar for fn()'
[00:53:35] 34: @has check failed
[00:53:35] 	`XPATH PATTERN` did not match
[00:53:35] 	// @has - '//code' 'impl Bar for extern "C" fn()'
[00:53:35] 36: @has check failed
[00:53:35] 	`XPATH PATTERN` did not match
[00:53:35] 	// @has - '//code' 'impl Bar for extern "system" fn()'
[00:53:35] 
[00:53:35] Encountered 3 errors
[00:53:35] 
[00:53:35] ------------------------------------------
[00:53:35] 
[00:53:35] thread '[rustdoc] rustdoc/extern-impl.rs' panicked at 'explicit panic', /checkout/src/tools/compiletest/src/runtest.rs:2435:8
[00:53:35] 
[00:53:35] ---- [rustdoc] rustdoc/inline_cross/issue-33113.rs stdout ----
[00:53:35] 	
[00:53:35] error: htmldocck failed!
[00:53:35] status: exit code: 1
[00:53:35] command: "/usr/bin/python2.7" "/checkout/src/etc/htmldocck.py" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc/inline_cross/issue-33113.stage2-x86_64-unknown-linux-gnu" "/checkout/src/test/rustdoc/inline_cross/issue-33113.rs"
[00:53:35] stdout:
[00:53:35] ------------------------------------------
[00:53:35] 
[00:53:35] ------------------------------------------
[00:53:35] stderr:
[00:53:35] ------------------------------------------
[00:53:35] 18: @has check failed
[00:53:35] 	`XPATH PATTERN` did not match
[00:53:35] 	// @has - '//code' "for &'a char"
[00:53:35] 
[00:53:35] Encountered 1 errors
[00:53:35] 
[00:53:35] ------------------------------------------
[00:53:35] 
[00:53:35] thread '[rustdoc] rustdoc/inline_cross/issue-33113.rs' panicked at 'explicit panic', /checkout/src/tools/compiletest/src/runtest.rs:2435:8
[00:53:35] 
[00:53:35] ---- [rustdoc] rustdoc/issue-29503.rs stdout ----
[00:53:35] 	
[00:53:35] error: htmldocck failed!
[00:53:35] status: exit code: 1
[00:53:35] command: "/usr/bin/python2.7" "/checkout/src/etc/htmldocck.py" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc/issue-29503.stage2-x86_64-unknown-linux-gnu" "/checkout/src/test/rustdoc/issue-29503.rs"
[00:53:35] stdout:
[00:53:35] ------------------------------------------
[00:53:35] 
[00:53:35] ------------------------------------------
[00:53:35] stderr:
[00:53:35] ------------------------------------------
[00:53:35] 18: @has check failed
[00:53:35] 	`XPATH PATTERN` did not match
[00:53:35] 	// @has - "//ul[@id='implementors-list']/li" "impl<T> MyTrait for T where T: Debug"
[00:53:35] 
[00:53:35] Encountered 1 errors
[00:53:35] 
[00:53:35] thread 'main' panicked at 'Some tests failed', /checkout/src/tools/compiletest/src/main.rs:322:21
[00:53:35] ------------------------------------------
[00:53:35] 
[00:53:35] thread '[rustdoc] rustdoc/issue-29503.rs' panicked at 'explicit panic', /checkout/src/tools/compiletest/src/runtest.rs:2435:8
[00:53:35] 
[00:53:35] 
[00:53:35] failures:
[00:53:35]     [rustdoc] rustdoc/doc-masked.rs
[00:53:35]     [rustdoc] rustdoc/extern-impl.rs
[00:53:35]     [rustdoc] rustdoc/inline_cross/issue-33113.rs
[00:53:35]     [rustdoc] rustdoc/issue-29503.rs
[00:53:35] 
[00:53:35] test result: FAILED. 159 passed; 4 failed; 1 ignored; 0 measured; 0 filtered out

QuietMisdreavus · 2017-08-25T14:52:05Z

@kennytm Yeah, I'm needing to push to be able to test, since htmldocck.py doesn't work on python 3. >_>

It seems I was overzealous in my masking. Regardless, even with that change, Clone is still left out because its DefId still points to libcore instead of libstd.

kennytm · 2017-08-25T14:59:30Z

@QuietMisdreavus Seems only a problem of import (I haven't actually tried to run it after patching). These diffs should make it work for Python 3:

diff --git a/src/etc/htmldocck.py b/src/etc/htmldocck.py
index a5449b748d..c8246bf727 100644
--- a/src/etc/htmldocck.py
+++ b/src/etc/htmldocck.py
@@ -110,11 +110,17 @@ import os.path
 import re
 import shlex
 from collections import namedtuple
-from HTMLParser import HTMLParser
+try:
+    from html.parser import HTMLParser
+except ImportError:
+    from HTMLParser import HTMLParser
 from xml.etree import cElementTree as ET
 
 # &larrb;/&rarrb; are not in HTML 4 but are in HTML 5
-from htmlentitydefs import entitydefs
+try:
+    from html.entities import entitydefs
+except ImportError:
+    from htmlentitydefs import entitydefs
 entitydefs['larrb'] = u'\u21e4'
 entitydefs['rarrb'] = u'\u21e5'
 entitydefs['nbsp'] = ' '

QuietMisdreavus · 2017-08-25T15:14:18Z

@kennytm Want to make a new PR for that? I'm sure many people would love you for it. :P

kennytm · 2017-08-25T15:27:01Z

@QuietMisdreavus Let me verify if it actually works on Python 3 first 😄

QuietMisdreavus · 2017-08-25T19:11:42Z

Thinking about it more, it seems that to make this work how the tests expect it to basically amounts to fixing #22083. Since the DefId for exported types points to the original crate, any reference to them later on, even if from the exported location, still points to that original crate. Add another thing to the pile of "reexport pains".

So... does that block this PR? It still works in the context of std, just not so well outside of it.

QuietMisdreavus · 2017-08-25T21:22:23Z

To help anyone who wants to see this effect when run on std and not just on my toy (broken) examples, i have a rendering of libstd docs here.

bors · 2017-08-26T17:54:02Z

☔ The latest upstream changes (presumably #44098) made this pull request unmergeable. Please resolve the merge conflicts.

QuietMisdreavus · 2017-08-30T14:37:19Z

cc @rust-lang/dev-tools because i just realized i never properly pinged the team.

This PR hides some internal types from appearing in the std docs by adding a new flag to the doc attribute, #[doc(masked)]. The problem is that the tests i wrote for this applied it to std itself, which doesn't completely work, since re-exported types aren't marked as coming from their exported crate.

The question i'm posing to the team (which may need to be relayed to other teams, come to think of it >_>) is whether we still want to land this, and how i should update the tests/docs if so. I think it's still useful in the context of the std docs, but until i/we can handle re-exports with this it's less useful outside of it.

(EDIT: also i fixed the merge conflict and squashed the commits some)

bors · 2017-09-03T00:56:29Z

☔ The latest upstream changes (presumably #44108) made this pull request unmergeable. Please resolve the merge conflicts.

nrc · 2017-09-05T05:08:49Z

We discussed this at today's dev-tools meetings. We should land this as is and iterate (since it is behind a feature flag). Better to fix the std links issue sooner and work towards something more generally useful.

rustdoc: add new "Implementations on Foreign Types" section to traits Demo screenshot: ![image](https://user-images.githubusercontent.com/5217170/29281219-c547f758-80e3-11e7-808f-49f592c65c5b.png) Full demo available at https://tonberry.quietmisdreavus.net/foreign-test/foreign_test/trait.CheckIt.html This PR splits the "Implementors" section on trait pages into two: First, for impls on types local to the crate, their impls are kept as-is, printing one line for the impl line, and any additional lines for associated types. However, for types external to the crate, they are now pulled up over the others and are printed (almost) like the summary impl on the type page itself. This gives any doc comments on these impls or methods to be exposed in the documentation. There's just one small problem, though: [libstd docs apparently surface impls for libc and rand, possibly among others](https://tonberry.quietmisdreavus.net/foreign-std/std/marker/trait.Copy.html#foreign-impls). This adds this section to pages in the std docs where we might not want them to show up in the first place. I think this is a bug distinct from this PR, but it does make it drastically apparent. ~~My question, then, is this: Do we want this here? Taking it out involves fixing which impls are visible to rustdoc, possibly specifically when rendering the std facade. I'm convinced this is fine to land as-is, since it adds a feature specifically for non-std crates (i'm thinking of things like `num` or related crates that implement things on primitives or std types as part of their functionality).~~ (EDIT: I have an open PR to fix this: #44026)

QuietMisdreavus · 2017-09-06T17:49:26Z

I've fixed the merge conflict. The travis failure is due to a stalled macOS build.

carols10cents · 2017-09-11T15:02:00Z

I restarted the one osx builder.

ping looks like this is ready for your review, @steveklabnik !

carols10cents · 2017-09-18T15:52:15Z

ping @steveklabnik @rust-lang/docs @rust-lang/dev-tools, this needs a review please!

steveklabnik · 2017-09-18T20:19:57Z

Looks good to me.

steveklabnik · 2017-09-18T20:20:10Z

@bors: r+

bors · 2017-09-18T20:20:12Z

📌 Commit 64f6111 has been approved by steveklabnik

bors · 2017-09-19T04:21:03Z

⌛ Testing commit 64f6111 with merge 9a00f3c...

hide internal types/traits from std docs via new #[doc(masked)] attribute Fixes #43701 (hopefully for good this time) This PR introduces a new parameter to the `#[doc]` attribute that rustdoc looks for on `extern crate` statements. When it sees `#[doc(masked)]` on such a statement, it hides traits and types from that crate from appearing in either the "Trait Implementations" section of many type pages, or the "Implementors" section of trait pages. This is then applied to the `libc`/`rand`/`compiler_builtins` imports in libstd to prevent those crates from creating broken links in the std docs. Like in #43348, this also introduces a feature gate, `doc_masked`, that controls the use of this parameter. To view the std docs generated with this change, head to https://tonberry.quietmisdreavus.net/std-43701/std/index.html.

bors · 2017-09-19T08:19:41Z

☀️ Test successful - status-appveyor, status-travis
Approved by: steveklabnik
Pushing 9a00f3c to master...

rust-highfive assigned steveklabnik Aug 22, 2017

QuietMisdreavus changed the title ~~[WIP] hide internal types/traits from std docs via new #{doc(masked)] attribute~~ hide internal types/traits from std docs via new #{doc(masked)] attribute Aug 22, 2017

steveklabnik changed the title ~~hide internal types/traits from std docs via new #{doc(masked)] attribute~~ hide internal types/traits from std docs via new #[doc(masked)] attribute Aug 22, 2017

kennytm reviewed Aug 22, 2017

View reviewed changes

ollie27 reviewed Aug 22, 2017

View reviewed changes

QuietMisdreavus mentioned this pull request Aug 23, 2017

rustdoc: add new "Implementations on Foreign Types" section to traits #43849

Merged

aidanhs added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Aug 23, 2017

kennytm mentioned this pull request Aug 25, 2017

Allow htmldocck.py to run using Python 3 #44086

Merged

QuietMisdreavus changed the title ~~hide internal types/traits from std docs via new #[doc(masked)] attribute~~ [WIP] hide internal types/traits from std docs via new #[doc(masked)] attribute Aug 25, 2017

QuietMisdreavus added the T-dev-tools Relevant to the dev-tools subteam, which will review and decide on the PR/issue. label Aug 25, 2017

QuietMisdreavus force-pushed the trimmed-std branch from dae2f61 to 9c31e80 Compare August 30, 2017 14:28

QuietMisdreavus mentioned this pull request Sep 2, 2017

Stable Rustdoc contains broken links (to compiler internal crates) #43701

Closed

QuietMisdreavus added 3 commits September 5, 2017 13:50

new attribute #[doc(masked)] to hide internal crates from std docs

c491e19

add feature gate doc_masked and tests

bb6de3c

update doc_masked docs and tracking issue

db09332

QuietMisdreavus force-pushed the trimmed-std branch from 9c31e80 to db09332 Compare September 5, 2017 18:51

remove tests for doc(masked)

64f6111

QuietMisdreavus changed the title ~~[WIP] hide internal types/traits from std docs via new #[doc(masked)] attribute~~ hide internal types/traits from std docs via new #[doc(masked)] attribute Sep 6, 2017

carols10cents added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 11, 2017

QuietMisdreavus removed the T-dev-tools Relevant to the dev-tools subteam, which will review and decide on the PR/issue. label Sep 18, 2017

bors merged commit 64f6111 into rust-lang:master Sep 19, 2017

Manishearth mentioned this pull request Sep 21, 2017

#[doc = $expr] in macro silently fails to generate docs #44730

Closed

kennytm mentioned this pull request Sep 21, 2017

rustdoc: Masked functions still appear in the search result. #44740

Closed

This was referenced Nov 29, 2017

Do not display hidden types, fixes issue 23912 #46359

Merged

rustdoc links to private structs, causing 404s #23912

Closed

QuietMisdreavus deleted the trimmed-std branch November 29, 2017 15:25

jyn514 mentioned this pull request Mar 10, 2023

Diagnostic message for std trait implementation includes unstable std type #108994

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

hide internal types/traits from std docs via new #[doc(masked)] attribute #44026

hide internal types/traits from std docs via new #[doc(masked)] attribute #44026

QuietMisdreavus commented Aug 22, 2017 •

edited

Loading

rust-highfive commented Aug 22, 2017

QuietMisdreavus commented Aug 22, 2017

kennytm Aug 22, 2017

QuietMisdreavus Aug 22, 2017

ollie27 Aug 22, 2017

kennytm commented Aug 25, 2017

QuietMisdreavus commented Aug 25, 2017

kennytm commented Aug 25, 2017

QuietMisdreavus commented Aug 25, 2017

kennytm commented Aug 25, 2017

QuietMisdreavus commented Aug 25, 2017

QuietMisdreavus commented Aug 25, 2017

bors commented Aug 26, 2017

QuietMisdreavus commented Aug 30, 2017 •

edited

Loading

bors commented Sep 3, 2017

nrc commented Sep 5, 2017

QuietMisdreavus commented Sep 6, 2017

carols10cents commented Sep 11, 2017

carols10cents commented Sep 18, 2017

steveklabnik commented Sep 18, 2017

steveklabnik commented Sep 18, 2017

bors commented Sep 18, 2017

bors commented Sep 19, 2017

bors commented Sep 19, 2017

hide internal types/traits from std docs via new #[doc(masked)] attribute #44026

hide internal types/traits from std docs via new #[doc(masked)] attribute #44026

Conversation

QuietMisdreavus commented Aug 22, 2017 • edited Loading

rust-highfive commented Aug 22, 2017

QuietMisdreavus commented Aug 22, 2017

kennytm Aug 22, 2017

Choose a reason for hiding this comment

QuietMisdreavus Aug 22, 2017

Choose a reason for hiding this comment

ollie27 Aug 22, 2017

Choose a reason for hiding this comment

kennytm commented Aug 25, 2017

QuietMisdreavus commented Aug 25, 2017

kennytm commented Aug 25, 2017

QuietMisdreavus commented Aug 25, 2017

kennytm commented Aug 25, 2017

QuietMisdreavus commented Aug 25, 2017

QuietMisdreavus commented Aug 25, 2017

bors commented Aug 26, 2017

QuietMisdreavus commented Aug 30, 2017 • edited Loading

bors commented Sep 3, 2017

nrc commented Sep 5, 2017

QuietMisdreavus commented Sep 6, 2017

carols10cents commented Sep 11, 2017

carols10cents commented Sep 18, 2017

steveklabnik commented Sep 18, 2017

steveklabnik commented Sep 18, 2017

bors commented Sep 18, 2017

bors commented Sep 19, 2017

bors commented Sep 19, 2017

QuietMisdreavus commented Aug 22, 2017 •

edited

Loading

QuietMisdreavus commented Aug 30, 2017 •

edited

Loading