Should work continue on current rustdoc?

[mgsloan]

Hi! Rather new to rust, so not sure how the community tends to approach these things. Overall there seems to be a focus on organized efforts. From https://internals.rust-lang.org/t/the-impl-period-newsletter-week-1/5971 , I noticed there's a page for improving rustdoc https://paper.dropbox.com/doc/Rustdoc-issue-roundup-WG-docs-rustdoc-ZSIIXNDGEPozTM9axn0BO , and clearly the author knows about this alternative rustdoc implementation

• Make Steve’s job harder by introducing more features that the new rustdoc has to implement to get to parity >_>

To me it certainly makes sense to spend some effort to fix ugly things that are easy. But it seems potentially wasteful to have a concerted impl effort on a tool that's likely to be deprecated. Is there community consensus for replacing rustdoc? How far away is it from reaching that point?

Hoping this is a reasonable place to discuss it. Feel free to close.

[mgattozzi]

Hey @mgsloan! First off welcome to the community! :D I'll leave a few thoughts here about why both are happening.

The original rustdoc has been around in some form or another since early on in the compiler, as such it's really old and shares a lot of compiler internals to do what it does. This makes it rather unweildy to work with as not everyone is willing or able to hack on compiler internals. Seeing as I've seen you around in the Haskell community I'm sure you know not everyone really wants to hack on GHC in their day to day. Tools like Haddock aren't intertwined with compiler internals and instead just parse source files for special Haddock comments to generate documentation.

That being said we too wish to move rustdoc away from this older implementation and it was decided that to do so in anyway would require a complete rewrite of the code. In this way we use rls-analysis to handle all the parsing of the source files and use it's output to generate documentation. The hope is once rls-analysis is stablized we can get this version of rustdoc on stable rust as well!

So why go through all of the work of updating the current one? Well the backlog of issues are still there, more so than any other issue category on the rustc repo. As such we'd still like to improve it and make the documentation great for production users! As it stands this tool is still a long way from being parity complete and until it is we'll need to continue work on the current rustdoc in the rustc repo while we bring this up to speed.

Unfortunate in some sense yes, but it's not all wasted effort. It's still improving the current users experience and I think that's a win regardless, even if it means we have to do a bit more work. I couldn't give you a how far away but we haven't even finished up all the backend parsing yet, so we're not close to displaying really much in your browser either.

Does this answer your questions? I'd be happy to answer them if I didn't at all.

[mgsloan]

Yep, that answers my question quite well, thanks @mgattozzi !

Tools like Haddock aren't intertwined with compiler internals and instead just parse source files for special Haddock comments to generate documentation.

I wish this was the case! Haddock is quite intwined with ghc, it basically pretends to be ghc, and uses ghc as a library to parse and type-check. So, to build and generate docs for your code, in a way you're compiling it twice. Here're the bits in haddock: https://github.com/haskell/haddock/blob/42e49d322d5ef3b6e13740642844b2ba57e96b95/haddock-api/src/Haddock/Interface.hs#L67

Definitely one of the reasons haddock doesn't see all that many contributions or changes. I really hope that this project can avoid a similar circumstance for rust!

Unfortunate in some sense yes, but it's not all wasted effort. It's still improving the current users experience and I think that's a win regardless, even if it means we have to do a bit more work. I couldn't give you a how far away but we haven't even finished up all the backend parsing yet, so we're not close to displaying really much in your browser either.

Gotcha! In that case, I can see that it makes good sense to work on both in parallel. If this project was closer to parity it'd make sense to try to direct most of the efforts toward it.

In this way we use rls-analysis to handle all the parsing of the source files and use it's output to generate documentation. The hope is once rls-analysis is stablized we can get this version of rustdoc on stable rust as well!

Is the goal to have it so that rust's rls format is relatively stable / versioned? This'd be quite nice, to allow the doc tool to work for many rustc versions. Seems like there needs to be some synchronization between the compiler implementation and rls-analysis's implementation. Is it worth considering having the compiler use the same datatypes for reading and writing these files, and providing something like rls-analysis as a library?

[mgattozzi]

I think @nrc would be the best person to ask your last question here since he's the lead for the rls. Rust guarantees backwards compatibility but not necessarily forward (we had a whole discussion about this recently) so as long as rls-analysis can handle the new stuff rustdoc should work for whatever version of Rust it's used with, or at least that's the idea.

[steveklabnik]

Thanks for a good discussion so far, and second what @mgattozzi said 😄

Basically, rls-analysis will end up shipping with the compiler, in my understanding, which will let it be used from stable Rust. Basically what you said:

the compiler use the same datatypes for reading and writing these files, and providing something like rls-analysis as a library?

However, the end goal is to get rid of the data files entirely, and move to the "compiler query interface". That's not something we have to worry about though, as basically, we only need to care about rls-analysis's interface. That stuff is its internal implementation details.

[nrc]

Is the goal to have it so that rust's rls format is relatively stable / versioned? This'd be quite nice, to allow the doc tool to work for many rustc versions. Seems like there needs to be some synchronization between the compiler implementation and rls-analysis's implementation. Is it worth considering having the compiler use the same datatypes for reading and writing these files, and providing something like rls-analysis as a library?

We basically already do all of this :-) The rls-data crate provides the 'schema' for the data and is published on crates.io (so is versioned). It is shared by rls-analysis and the compiler. rls-analysis is a library, and how the data gets from the compiler to rls-analysis should be of no concern to Rustdoc or the RLS (this is not 100% true, since the client must provide the data, but we're getting there).

[mgsloan]

Great! It sounds like things are on the right track here, very reasonable. Thanks for answering my questions and discussing.