The Final Pieces of the OCaml Documentation Puzzle

Jon Ludlam, University of Cambridge,
Gabriel Radanne, INRIA and
Leo White, Jane Street

Principles

Documentation should be...

Correct
Useful

Focus first on static html

from module signatures

This talk is about Odoc, a tool for generating documentation from OCaml source code, for which I'm the current overall maintainer. In particular I'd like to discuss some of the innovations we've been working on over the past year or so to make 'Odoc 2.0' that we believe will make this the best tool for documenting your OCaml projects.

Let's start by stating some principles of good documentation.

Documentation should be correct. I hope this is a pretty uncontroversial statement! It's worth saying though, as the currently released version of Odoc produces documentation that is, in a few cases, incorrect or suffers from broken links.

Secondly, it should be useful. By this I mean it's got to show you what you want or need to know in an obvious way.

I'd also like to state clearly how the generated documentation will be consumed. As with most other code-documentation tools, we will focus first on generating static html. As we'll see later, we can then adapt this for other output formats such as latex or man pages.

Finally, we'll be using Module signatures as the basis for generating our documentation.

Ocamldoc output

## Expansions

We need to produce **expansions** for:

- Includes
- Modules / module types, when they're not already signatures, e.g.
```ocaml
module M : MODTYPE with type t := u
module N : F(X).T
```
- Classes / Class types

Notes:
It's not just includes that we are going to need to expand. The signatures of modules and module types are often
described by referring to other module types; sometimes with additional type or module equalities or substitutions.
Sometimes they're not; they're just signatures, in which case we don't need to do any extra expanding,
We also need to do some similar expanding for classes and class types; though this is an area we are still
working on, so I'm not going into any detail on this today.

We've also got the option of what to do with the expansions. For includes, we put the expansions inline
in the page. For modules and module types, for HTML, we generally will create a new page with the contents. Different
output formats can decide how to deal with them in different ways too, which we'll come back to later.

We also need to consider how the expansions will affect the destinations of the various clickable
links. The process of deciding this is called 'resolution', and is done both for links found in code,
called 'paths' or 'fragments', and those found in the documentation comments, known as 'references'.

So how are we going to calculate these expansions?

## Use the compiler?

The compiler must have calculated the contents of
**Sexp\_with\_comparable**, can we just use that?

Yes, but it only produces a signature 'result'
- No comments!
- No includes, no other structure

Notes:
The first and most obvious thing to do is to see if we can just ask the compiler. Can we get away with just
getting the compiler to all the work and we then just present it? Well, yes and no. We do
indeed get the compiler to do a lot of work and Odoc operates using the typed trees as input -- that
is, rather than reading the mli files (or ml files if nobody has written an mli file), it will
read the cmti files (or cmt files). This has the extremely useful property of ensuring that
we're only dealing with 'correct' input.

However, the problem with these expansions is that what the compiler wants and what we need
are not entirely aligned. The compiler only really cares about the resulting signature, and not what
went into making the signature, whereas we really want to track the provenance of elements
in the signature so we can show where they came from and link back to their definitions.
What the compiler has calculated will have all the includes, the type and module
substitutions, the typeof expressions, simply replaced with their results.
Additionally all of the comments are gone, so the actual documentation is missing, which is definitely
a minus when it comes to producing the docs.

So we need to write our own code to do these expansions.

## Copy the compiler!

Odoc 1.5.1 had a more ad-hoc implementation, causing bugs and
making it hard to adopt new OCaml features

The master branch on github has a brand-new implementation based on
how the compiler does it.

Notes:
The currently released version of Odoc has very clever but quite bespoke
implementation of expansion and resolution. This was very good, but not
perfect, and also hard to understand. This made it tricky to keep up with the
compiler as it added new features and there were some long-standing issues
which were very hard to fix.

Over the past year or so we have implemented "the new model" to do the work
of expansion and resolution, which is based on how the compiler itself works,
whilst keeping better track of where signature elements come from.

One of the most important implications of this approach is that Odoc ends up inheriting
a two-pass approach, corresponding to 'compile' and 'link', which I will talk
a little bit more about in a second.

Let's look
at the output of Odoc 1.5.1 and the current master branch of Odoc working on
our previous example.

Odoc 1.5.1 output

I'm not using the real S-expression and Comparable modules here, just some stubs to show how it works. You can see here that the includes have been expanded to show their contents. However, you can also see a number of problems here. Firstly there's this odd Lib__Sexp.t - there are no source files containing double underscores! Secondly I did actually write a docstring for the function `of_string`, which has disappeared. Thirdly the destructive type substitution hasn't happened - the expansion of `Comparable.S` still has a `type t` in it, and the function `le` below contains links to that type t (if you click on them it takes you to the line above instead the top definition.

This fails on principle 1 - it's not correct! So let's now take a quick peek at what improvements we've made in master.

Odoc master output

Compile/Link

Compile : .cmti / .cmt / .cmi -> .odoc

Requires dependent odoc files (same as standard ocaml compile dependencies)

Link : .odoc -> .odocl

Requires odoc files for everything in the package and all dependent packages

Generate: .odocl -> .html / .tex / man

No other dependencies

In basing our implementation on how the compiler works we need to process our inputs in much the same way as the compiler does, and we've named them similarly. We start by 'compiling' the cmti files into odoc files. As for standard compilation, if your module depends upon any other, that module will have to have been compiled first to produce its odoc file.

Linking is then performed. When we link we resolve all of the references in doc comments, which could be to any module in this package or any dependencies. As such linking requires all of the odoc files for those to have been compiled.

Once the linking is finished we have self-contained odocl files. We're then in a position to use these to generate any of our output formats. A single odocl file will often result in multiple output files.

Challenges

Canonical/hidden modules
We don't expand module aliases...
...except when they're the canonical module
Intermediate expansions
Shadowed items
Abstract modules/module types
References are more expressive than Paths
Dependently typed modules

Canonical modules 1

x.mli

type t

y.mli

type t = X.t

lib.ml-gen

(** @canonical Lib.X *)
module X = Lib__X

(** @canonical Lib.Y *)
module Y = Lib__Y

The long-term goal in OCaml is to do namespacing using the module system. We'll pack all the modules to do with a particular library into one top-level module and not have to expose the individual modules that make it up. Once we've got first-class support for this in the language we won't have to worry about canonical modules, but for now Odoc has to help maintain the illusion that it's already here.

This is how it works today with dune.

We write our modules -- here, X and Y, as we'd like them to appear within the library namespace. We can refer between them as just X and Y as usual, as long as we don't cause dependency loops. When we come to compile them though, dune will tell the compiler to ignore their filenames and that their names are actually `Lib__X` and `Lib__Y`. Now, since this would break the inter-module paths like `X.t`, dune will actually first generate this new file `lib.ml-gen` that contains aliases from the short names 'X' and 'Y' to 'Lib__X' and 'Lib__Y'. Now, even though this file refers to lib__X and Y, it can be compiled before them with the help of the compiler flag 'no-alias-deps', because nothing in the file depends on the contents of those modules. The generated file makes a note that to link it will require lib__X and Y like normal, but obviously it can't record the hash that it would do normally. Then when we compile X and Y (pretending they are lib__X and lib__Y), we instruct the compiler to 'open' the module Lib, which brings the short names into scope, and allows the files to compile as if their names hadn't been mangled.

When we then want to use this library, we refer to modules `Lib.X` and `Lib.Y` and pretend that lib__X and lib__Y don't exist.

So, how does Odoc deal with this? Because we work the same way as the compiler, we deal with this in the same way that the compiler does! First, we 'compile' the generated lib module. We make note that the paths to 'lib__X' and 'lib__Y' are forward paths -- that is, we haven't yet compiled the odoc files corresponding to those modules -- and we write out `lib.odoc`. We then compile lib__X and lib__Y. When it looks at the X.t in module y, because it's operating on the typedtree (the cmti rather than mli), it knows that the 'X' is the 'X' from lib.ml. So it looks up module X there and notes that it's got a 'canonical' tag in front of it, and records this in the path - semantically this means that references to that 'X' should be referred to 'Lib.X', and therefore the reference to 'X.t' should go to 'Lib.X.t'

Now that odoc has compiled the files, we get to the link phase. Since we're trying to pretend that anything with a double underscore doesn't exist, we don't link those files, so the only file we actually link in this example is 'Lib'. We look at the module aliases here, and note that they are 'self-canonical' - that is, the canonical path to the module is in fact this module! We therefore expand the aliases, and splice in the signatures from lib__X and lib__Y. We then recursively link these expansions, and in this case we'll resolve the X.t path, looking up the canonical path so that the link will be to 'Lib.X.t'. We actually do check that the target exists to ensure we have no broken paths. For any links that can't be resolved, we'll report this and just output text rather than a link.

Canonical modules 2

x.mli

type t

y.mli

type t = X.t

lib__.ml-gen

(** @canonical Lib.X *)
module X = Lib__X

(** @canonical Lib.Y *)
module Y = Lib__Y

lib.mli

(* User-defined Lib module *)
module X = X
module Y = Y

This is a slightly more advanced namespacing example.

The difference here is that we've handwritten our 'lib.ml' file, and so the generated one is 'lib__.ml'. This is more expressive and allows for finer control over the top-level module, but now you have to be much more careful that the canonical links are correct. As you can see, the 'canonical' tags in the generated lib__ file are still pointing at 'Lib', so they're pointing into the hand-written file. If you've chosen not to expose a module, or you've changed its name, the canonical path won't resolve correctly and this can lead to unresolved paths. As I mentioned previously though, we do check everything and can report if this sort of thing is broken.

The document layer

Results from link phase is a typed syntax tree
Requires 'thinking in OCaml' to traverse

Solution: transform to generic document IR

Once we have finally done all the resolving and expanding, compiling and linking, the output from the model layer is effectively a syntax tree that fundamentally has the same shape as the typed tree from the original cmti file. You have to know quite precisely how OCaml works in order to traverse it effectively. To simplify backend logic and to reduce code duplication, we first transform this syntax tree into a document-oriented intermidate representation. This is designed such that backends can traverse it simply and concentrate on generating the appropriate output for the chosen backend type. Naturally different output formats have different capabilities; HTML can split things into small pages with links between them, whereas printed latex-based documents should not be treated in the same way. This intermediate format therefore has to be rich enough to support a variety of output formats, whilst being agnostic about the actual language being rendered. To add a new backend then is simply a question of providing a function to walk this tree and render the appropriate output given a provided formatter

Items and annotated code

Traditional agnostic document representations:
- Blocks (paragraphs, lists, tables, etc)
- Inline (styling, links, etc)
Code representation:
- Item (module, includes, values, methods, etc)
- Annotated code (code with extra metadata)
New backends simply traverse values of these types

Traditional document representations such as pandoc's internal data types usually have a split between block items, such as paragraphs, list, headings and so on, and inline elements representing styling like 'bold' or links or straightforward text.

Code is naturally more hierarchical than this so we have more of a tree structure, but broadly we have the same sort of split. We have 'Items', which are the equivalent to blocks and contain headings, documentation blocks, as well as declarations such as modules, type and values. We also represent 'includes' at this level to enable these the HTML renderer to interactively allow them to be opened or closed.

The declarations are then constructed from 'Annotated code', which represents code with additional metadata associated with it. This additional metadata may be a documentation block, an alternative representation - for example, modules might have a short and long expansion, the short being an abbreviated "sig ... end", or possibly a path to a module type, and the long being the full expansion including all of the declarations; a list of items again.

HTML output

Man page output

Future directions

OCaml manual, and deprecating ocamldoc
Search
docs.ocaml.org -- multiple versions
mdx

If you're interested, come and talk to us!

I'd like to talk a bit now about our future plans for odoc. It's an exciting time as with these recent improvements we're now in an excellent position to make some really interesting advances!

The first thing We've got planned is moving the rendering of standard library documentation for the OCaml manual to odoc and slowly deprecating ocamldoc. It's likely this will follow a similar path to camlp4 - moving it out of the main ocaml repository to be maintained separately. I'd be very interested to hear from anyone for whom this will present any challenges - we're eager to work with people to find solutions, so please get in touch!

We'd like to add search to odoc soon. This will likely depend upon handling implementation as well as interface files in order that we can rank the results according to amount of usage.

We'd like to revamp docs.ocaml.org as a central repository of docs for all versions of packages. This is an interesting challenge, not least because the contents of a package is dependent not just upon its own version, but quite possibly on the versions of the packages it depends upon.

How we will integrate with the rest of the OCaml documentation ecosystem, for example, mdx, is also something we'd like to address. For example, we're currently looking at mapping beetwen odoc syntax and mdx. Perhaps also even merlin for looking up docstrings?

Odoc 2.0 Key Contributors

Jon Ludlam, Leo White, Jules Aguillon - new model
Gabriel Radanne - Document IR, new HTML and man renderers
Florian Angeletti - Latex renderer

Odoc 1.x contributors

Anil Madhavapeddy, Bobby Priambodo, Craig Ferguson, Daniel Bünzli, David Sheets, Florian Angeletti, Geoff Reedy, Jack Feser, Jeremie Dimino, Kate, Kevin Ji, Leandro Ostera, Luke Czyszczonik, Mohamed Elsharnouby, Nik Graf, Patrick Stapfer, Ricky Vetter, Rizo Isrof, Rudi Grinberg, Thibault Suzanne, Thomas Refis, Ulrik Strid, Ulysse, Yotam Barnoy, dhcmrlchtdj, mikaello