diff --git a/docs/dev/documentation.md b/docs/dev/documentation.md new file mode 100644 index 00000000000..a1ef0a367e7 --- /dev/null +++ b/docs/dev/documentation.md @@ -0,0 +1,354 @@ +# DAOS Documentation + +Documentation on DAOS is available in the following places: + +* The [https://daos.io/](https://daos.io/) website contains general + information about the DAOS Foundation, a subproject of the + Linux Foundation. This website is maintained by the + DAOS Foundation and uses Wordpress for content management. + +* The DAOS Community Wiki [https://wiki.daos.io/](https://wiki.daos.io/) + is a redirection to + [https://daosio.atlassian.net/wiki/spaces/DC]( + https://daosio.atlassian.net/wiki/spaces/DC), + an Atlassian Wiki space. + +* Documentation for end users and administrators of DAOS is + available at [https://docs.daos.io/](https://docs.daos.io/). + This website provides versioned documentation for each DAOS release. + Its web pages are created from the contents in the + [daos-stack/daos/docs/](https://github.com/daos-stack/daos/tree/master/docs) + trees of the DAOS release branches in Github. + This website also cross-reference the "DAOS Internals" documentation + (in the "Developer Zone" section). + +* The `dmg` and `daos` commands have man-pages that are created at + DAOS build time through hidden `manpage` options of these commands. + The contents of these man-pages is embedded in the Go code for + the commands. The man-pages currently only output concatenated + lists of all subcommands and their options. + See + [src/control/cmd/dmg/README.md](https://github.com/daos-stack/daos/blob/master/src/control/cmd/dmg/README.md) + and + [src/control/cmd/daos/README.md](https://github.com/daos-stack/daos/blob/master/src/control/cmd/daos/README.md). + +* Most DAOS commands and subcommands have a `--help` option which + prints one-line descriptions of the (sub)command and its options. + The contents of these help texts is embedded in the Go code + (or C code) for the commands. + +* Developer-facing "DAOS Internals" documentation of the + DAOS software is maintained as Markdown files within the + [daos-stack/daos/src/](https://github.com/daos-stack/daos/tree/master/src) + source tree of the DAOS Github repository. + Make sure to access the release branch (`master`, `release/2.8`, etc.) + for the version of DAOS you are working with. + +* DAOS API documentation is generated from the DAOS source files using + `doxygen`. Running `doxygen` during the website deployment process for + [https://docs.daos.io/](https://docs.daos.io/) + creates HTML pages in `docs/doxygen/html/`, + which are then made available in the "Developer Zone" section of + [https://docs.daos.io/](https://docs.daos.io/). + +The following sections describe how the multi-versioned documentation +on [https://docs.daos.io/](https://docs.daos.io/) +is generated from the documentation Markdown files in the Github +`daos-stack/daos/doc/` tree (using the `mkdocs` and `mike` tools, +and the `doxygen` tool for the API documentation). + +## Configuration files in Github + +### Mkdocs configuration + +The [MkDocs](https://www.mkdocs.org/) tool is a static site generator +that creates web pages from Markdown files. +The `mkdocs` command is driven by a single configuration file, +[mkdocs.yml](https://github.com/daos-stack/daos/tree/master/mkdocs.yml) +in the toplevel directory of the DAOS project on github. +Among other project information, this file defines the +menu structure for the navigation bar of the website that is being created. +Each DAOS release branch uses its own version of `mkdocs.yml`, +as some information in that file is version specific or uses URLs +that are pointing to that specific DAOS version. + +!!! note +Absolute paths or hardcoded version numbers in the `mkdocs.yml` file +should **not** be changed to relative URLs/paths, +as that will break some of the processing by downstream tools. +These specifics need to be updated once when a new release branch is +created, and again when an existing release branch that was under +development reaches General Availability (GA). +Examples with detailed steps can be found below. + +The `mkdocs` tool creates a `sites` subdirectory in the toplevel +directory of the DAOS project on github, +in which the HTML documents for the static website are created. +To prevent this temporary subdirectory to be accidentally committed +into github, the +[.gitignore](https://github.com/daos-stack/daos/blob/master/.gitignore) +file contains a `site/` line. + +### Mike configuration + +The `mike` plugin is used with +[Materials for MkDocs](https://squidfunk.github.io/mkdocs-material/setup/setting-up-versioning/) +to generate versioned documentation for each of the DAOS release branches. +Its configuration information is stored in the `versions.json` file +that resides in a new `gh-pages` branch in a checked out version +of the DAOS repository. +Note that this file is **not** committed into the main DAOS-Stack +github repo. It only exists in the local `gh-pages` branch +on the system on which the webpages are generated. + +The `versions.json` file should **not** be edited directly. +The `mike` command should be used to define which versions of the +documentation are being created, which aliases should be set up, +and which version should be the default version to display. +See below for details. + +### Doxygen configuration + +The configuration file for the `doxygen` tool is +[Doxyfile](https://github.com/daos-stack/daos/blob/master/Doxyfile), +located in the toplevel directory of the DAOS project on github. + +## Installing the mkdocs Software + +On the machine where the DAOS documentation is to be built, +GIT needs to be set up so the `daos-stack/daos` project +can be checked out. +In addition, the MkDocs package and plugins need to be installed. +Running `pip install mike` should install the prerequisite packages. + +Depending on the operation system and Python environment, +prerquisite software and the mkdocs plugins may have to be explicitly +installed. For example, on a Windows laptop running Cygwin: + +``` +pip3 install alabaster sphinxcontrib-applehelp sphinxcontrib-devhelp sphinxcontrib-htmlhelp sphinxcontrib-jsmath sphinxcontrib-qthelp + +pip3 install --user mkdocs +pip3 install --user mkdocs-material +pip3 install --user mike +export PATH="$HOME/.local/bin:$PATH" +mike --version +``` + +## Checking out DAOS from github + +To create the DAOS documentation webpages, it is recommended to work +on a clone of the DAOS project on github that is **separate** from any +checked out version that may be used for code development work. + +``` +cd ~/dev +git clone git@github.com:daos-stack/daos.git daos-website +cd daos-website +git branch # should be on master branch +grep site .gitignore # should show "site/" +grep doxy .gitignore # should show "docs/doxygen/" +mike list # should show nothing on a new setup +``` + +## Creating the static website contents + +To create the static website contents, the following process is used. +For each DAOS release branch that should get included on the website: + +* the git release branch is checked out + +* running `doxygen` creates a temporary version of the doxygen documentation + for this release in the `docs/doxygen/html` subdirectory. + +!!! note +To prevent these files from being accidentally committed into github, +the [.gitignore](https://github.com/daos-stack/daos/blob/master/.gitignore) +file contains a `docs/doxygen/` line. + +* `mike deploy` is called with a title, a name for the release, + and optional alias name(s) for this release. + These will be used in the multi-versioned website's navigation system. + + - `mike` first uses `mkdocs` to create a temporary copy of the + static webpages for this release in the `site/` subdirectory. + + - `mike` will then copy that `site/` contents into the temporary `gh-pages` + branch, with a directory name that matches the release name + specified on the `mike deploy` invocation. + + - Any aliases for that release will be created as symlinks in the + `gh-pages` branch, pointing to the directory with the main release name. + +This process is repeated for each release. After all versions have been +processed this way, invoking `mike default` will set the version that is +displayed on the website by default. +That command creates a toplevel `index.html` in the `gh-pages` branch which +contains a redirect to the version that was selected as the default. + +A complete example for a website where DAOS 2.6 is the current "latest" +version, and the master branch is the "2.7" development branch for what +will eventually become DAOS 2.8: + +``` +mike delete --all + +git checkout master +rm -rf docs/doxygen/html 2>/dev/null +doxygen +mike deploy -t "v2.7 - master" master v2.7 v2.8 2>&1 | tee deploy-log.master.txt +rm -rf docs/doxygen/html + +git checkout release/2.6 +rm -rf docs/doxygen/html 2>/dev/null +doxygen +mike deploy -t "v2.6 - latest" v2.6 latest 2>&1 | tee deploy-log.v2.6.txt +rm -rf docs/doxygen/html + +git checkout release/2.4 +rm -rf docs/doxygen/html 2>/dev/null +doxygen +mike deploy -t "v2.4 - deprecated" v2.4 2>&1 | tee deploy-log.v2.4.txt +rm -rf docs/doxygen/html + +mike set-default latest +mike list +``` + +After `release/2.8` has been branched, but before it becomes generally +available (GA), the website version structure should be updated like this: + +``` +mike delete master # will also delete the v2.7 v2.8 aliases + +git checkout master +rm -rf docs/doxygen/html 2>/dev/null +doxygen +mike deploy -t "v2.9 - master" master v2.9 v3.0 2>&1 | tee deploy-log.master.txt +rm -rf docs/doxygen/html + +git checkout release/2.8 +rm -rf docs/doxygen/html 2>/dev/null +doxygen +mike deploy -t "v2.8 - rc" v2.8 rc 2>&1 | tee deploy-log.v2.8.txt +rm -rf docs/doxygen/html + +mike list +``` + +And finally, when DAOS version 2.8 is released this will be set +as the new `latest` release: + +``` +mike delete rc +mike delete latest + +git checkout release/2.8 +rm -rf docs/doxygen/html 2>/dev/null +doxygen +mike deploy -t "v2.8 - latest" v2.8 latest 2>&1 | tee deploy-log.v2.8.txt +rm -rf docs/doxygen/html +mike set-default latest + +mike list +``` + +It is useful to save the logs from the `mike deploy` invocation. +These logs will contain warnings about pages that exist but are not +linked anywhere in the navigation structure, and other issues. + +The new website can be tested locally by running `mike serve` on the machine +where the website is being created. This will start mike's built-in webserver. +The new contents can then be inspected by pointing a browser +(running on the same machine) to `http://localhost:8000/`. + +## Staging the new website contents to docs.daos.io + +Because the version aliases are symlinks, and because the website contents +consists of thousands of small files, copying it to the actual webserver +is best done by creating and transferring a tarfile, which can then be +un-tarred on the webserver and will keep the symlinks intact. + +As a best practice, a new website version is first deployed in a +staging area so it can be validated in the same webserver environment +in which the live website is running: + +``` +export WEBSITE="docs.daos.io" +export TARFILE="docs-daos-io.tgz" + +git checkout github-pages + +tar czvf $TAR versions.json index.html v?.? latest master + +ssh $WEBSERVER "rm -r docs.daos.io/staging && mkdir docs.daos.io/staging" +scp R$TARFILE $WEBSERVER:docs.daos.io/staging +ssh $WEBSERVER "cd docs.daos.io/staging && tar xz $TARFILE && rm $TARFILE" + +rm $TARFILE + +git checkout master +``` + +At this point, the new website can be inspected by using the +[https://docs.daos.io/staging/](https://docs.daos.io/staging/) URL. + +## Go-live of the new website contents on docs.daos.io + +To make the website in the staging area the active website, there's a small +`go-live.sh` script on the webserver that will archive the current live +version and replace it with the one in the staging area: + +``` +$ cat go-live.sh +D=`date "+%Y-%m-%d"` +mv docs.daos.io old-docs.daos.io.$D +mv old-docs.daos.io.$D/staging docs.daos.io +mkdir docs.daos.io/staging +``` + +## Miscellaneous tips and tricks + +### Beware of relative URLs in links + +The `mkdocs` tool creates a *directory* (with an `index.html` in it) +for each Markdown *file* that it processes. +So a documentation file like `docs/user/container.md` +in the github master branch will become this URL on the website: +[https://docs.daos.io/master/user/container/index.html](https://docs.daos.io/master/user/container/index.html). + +This additional directory level may break relative URLs in +hyperlinks in the Markdown files. On the other hand, adding that +additional directory level to those relative URLs will break the +links when looking at the (un-processed) Markdown files directly +in github. + +There is no good fix. In some places absolute paths to website URLs +are used to make sure those hyperlinks work in both the raw +Markdown and the rendered HTML pages on the website. The drawback +of this approach is that those URLs will contain hardcoded +release numbers, but those can be quickly updated by a +search and replace when a new release branch is created. + +### Resolving Spell Checker Issues + +The DAOS CI performs spell checking on PRs. +If the spell checker reports an error for a legitimate word +(or acronym), the error can be resolved by adding the term to the +wordlist that is maintained in the `utils/cq/words.dict` file. + +### Always call mike on a release branch, not gh-pages + +You must be in an actual DAOS release branch, +not in the `gh-pages` branch, to run mike commands. +Otherwise the tool will report errors like this: + +``` +$ git branch +* gh-pages + master + +$ mike list +error: [Errno 2] No such file or directory: 'mkdocs.yml'; pass --config-file or set --remote/--branch explicitly +``` + diff --git a/mkdocs.yml b/mkdocs.yml index 9ba01d1548d..95fe52b04e3 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -111,6 +111,7 @@ nav: - Developer Zone: - 'Dev Environment': 'dev/development.md' - 'Contributing': 'dev/contributing.md' + - 'Documentation': 'dev/documentation.md' - 'DAOS Internals': 'https://github.com/daos-stack/daos/blob/master/src/README.md' - 'DAOS API Documentation': 'https://docs.daos.io/v2.9/doxygen/html/index.html' # Attention: Don't change that doxygen path to a relative path, or mkdocs will stumble...