Lately I’ve been working on improving the build process for PuffinPlot. The goal is to have it fully buildable from scratch with a simple mvn package, with no dependencies other than those which can be fetched automatically by Maven. One of the obstacles to this was the manual, written in LaTeX, which means that anyone who wants to build the full package must manually install LaTeX and various related tools and extensions. There’s no automated way to produce decent HTML and PDF output from LaTeX source without stepping outside the JVM (things might have been different had NTS taken off). I therefore needed to convert the source to a form which could be processed by a Maven plugin. I decided on AsciiDoc, probably my favourite lightweight markup language. It’s well-defined, full-featured, and widely supported. The most popular AsciiDoc processor at present is Asciidoctor, written in Ruby. Conveniently for Java users, there is an official, jruby-based JVM port of Asciidoctor – AsciidoctorJ, complete with associated Maven plugins. (jruby turned out to be a useful addition to my Maven build environment in any case: I wrote a couple of jruby scripts which use jgit to process commit information during the build, for example in order to set the version number automatically in the ‘About’ dialog.)

The LaTeX source of the PuffinPlot manual is relatively clean, as LaTeX source goes – which is to say, there are only a few dozen lines of package includes, configuration, and macro definitions in the preamble. No automated process will produce a 100% satisfactory conversion to AsciiDoc, but a ~90% accurate conversion, followed by some manual clean-up, is feasible. I used the ever-capable pandoc for the initial conversion; as far as I could tell, it’s the only widely available tool that can convert directly from LaTeX to AsciiDoc. (Incidentally – and curiously, given its very wide file format support – pandoc doesn’t support AsciiDoc as an input format.) The only pandoc-incompatible command in the file was \input changes.tex, which I fixed by changing it to \input{changes.tex}. The raw pandoc output was more usable than I had expected: tabular environments, sectioning, and lists were all converted successfully. The most significant omissions were section cross-references and the bibliography.

Tidying up the automatically converted source consisted mainly of:

• Replacing LaTeX quotation marks ` and ' with '` and `' respectively. (AsciiDoc can of course handle the Unicode characters ‘ and ’ natively, but the equivalent ASCII digraphs seem to be preferred.)

• Correcting anchor and cross-reference syntax, which pandoc mangled a little in conversion. (Emacs macros were invaluable for this.)

• In several description lists which used custom macros in the \item[…​] parameter, the term disappeared in the AsciiDoc output. If I’d noticed this in time, I think it could have been corrected by modification of the LaTeX source. As it was, I only noticed once I had significantly modified the AsciiDoc, so I used Emacs macros to do some semi-automated transformation and copy-pasting of the missing content from the LaTeX file.

• LaTeX math: while AsciiDoc can handle LaTeX math, it wasn’t necessary in this case: the manual didn’t contain any formulae, and only used math mode for the occasional special character. I search-and-replaced these instances and substituted them with suitable UTF-8 characters.

• Tweaking table layouts.

• Citations and the bibliography.

• Various minor aesthetic reformatting of the source.

Converting the bibliography turned out to be a significant subproject, documented here. Asciidoctor has limited built-in bibliography support, but for proper handling of author-date citations and automated formatting of bibliography entries a plugin is required; I used asciidoctor-bibtex, which appears to be the most popular and up-to-date, but there are also asciidoctor-bibliography and asciidoc-bib.

There was one feature I had to forgo entirely in the transition: floating tables and figures. These are available as standard in LaTeX, but I could find no equivalent in Asciidoctor, either built in or with a plugin. (The search was complicated by the fact that AsciiDoc uses the term "float" to refer to the placement of images at the left or right edge of a text block, with the text flowing around them.) I don’t think that the PuffinPlot PDF manual suffered much from the removal of floats (and the HTML manual doesn’t suffer at all), but for many documents this limitation might be a deal-breaker when considering the source format.

The manual includes two SVG images, and here AsciiDoc allows for a cleaner build process than LaTeX. LaTeX can’t include SVG files directly, so the old build script called Inkscape to convert them to PDF (for PDF ouput) and PNG (for HTML output), introducing another heavyweight external dependency. Asciidoctor, on the other hand, can handle both output formats more elegantly: for HTML output, the SVG images are embedded directly; for PDF output the SVG can be rendered into PDF using prawn-svg. Handling font embedding for SVGs rendered to PDF is not entirely straightforward, but preferable to my LaTeX set-up (which required the font to be installed and available to Inkscape, thus adding yet another hard-to-automate dependency to the build). For now, the default fonts are good enough.

The PDF output quality is entirely adequate, but – unsurprisingly – can’t match the quality of LaTeX typography. The HTML output quality is definitely an improvement: TeX4ht, for all its ingenuity, still generates output with a distinctly 1990s feel to it. Asciidoctor, even without custom styling, produces something more sleek and modern; it also has the useful ability to inline images, resulting in a single self-contained HTML file.

The full configuration for the Asciidoctor plugin is (as ever with Maven XML) rather verbose, but not particularly complex:

<plugin>
  <groupId>org.asciidoctor</groupId>
  <artifactId>asciidoctor-maven-plugin</artifactId>
  <version>${asciidoctor.maven.plugin.version}</version>
  <dependencies>
    <dependency>
      <groupId>org.asciidoctor</groupId>
      <artifactId>asciidoctorj-pdf</artifactId>
      <version>${asciidoctorj.pdf.version}</version>
    </dependency>
    <!-- Comment this section to use the default jruby artifact provided by the plugin -->
    <dependency>
      <groupId>org.jruby</groupId>
      <artifactId>jruby-complete</artifactId>
      <version>${jruby.version}</version>
    </dependency>
    <!-- Comment this section to use the default AsciidoctorJ artifact provided by the plugin -->
    <dependency>
      <groupId>org.asciidoctor</groupId>
      <artifactId>asciidoctorj</artifactId>
      <version>${asciidoctorj.version}</version>
    </dependency>
  </dependencies>
  <configuration>
    <sourceDirectory>src/main/asciidoc</sourceDirectory>
    <!-- Attributes common to all output formats -->
    <attributes>
      <sourcedir>${project.build.sourceDirectory}</sourcedir>
    </attributes>
    <gemPath>${project.build.directory}/rubygems</gemPath>
    <requires>asciidoctor-bibtex</requires>
    <sourceDocumentName>manual.adoc</sourceDocumentName>
    <resources>
      <!-- We don't want to copy any resources. Omitting the resources
           section entirely copies all the resources in the asciidoc
           source directory. Including a resources section with an
           empty resources subsection omits the asciidoc resources but
           copies the contents of src/main/java (!). So we need this
           resources section which explicitly excludes src/main/java.
      -->
      <resource>
        <directory>${project.basedir}/src/main/java</directory>
        <excludes>
          <exclude>**/*</exclude>
        </excludes>
      </resource>
    </resources>
  </configuration>
  <executions>
    <execution>
      <id>generate-asciidoc-manual-pdf</id>
      <phase>prepare-package</phase>
      <goals>
        <goal>process-asciidoc</goal>
      </goals>
      <configuration>
        <backend>pdf</backend>
        <outputDirectory>${project.build.directory}/manual-pdf</outputDirectory>
        <sourceHighlighter>coderay</sourceHighlighter>
        <!-- Use `book` docType to enable title page generation -->
        <doctype>book</doctype>
        <attributes>
          <pdf-stylesdir>${project.basedir}/src/main/asciidoc/theme</pdf-stylesdir>
          <pdf-style>custom</pdf-style>
          <icons>font</icons>
          <pagenums/>
          <toc/>
          <idprefix/>
          <idseparator>-</idseparator>
        </attributes>
      </configuration>
    </execution>
    <execution>
      <id>generate-asciidoc-manual-html</id>
      <phase>prepare-package</phase>
      <goals>
        <goal>process-asciidoc</goal>
      </goals>
      <configuration>
        <backend>html5</backend>
        <outputDirectory>${project.build.directory}/manual-html</outputDirectory>
        <sourceHighlighter>coderay</sourceHighlighter>
        <embedAssets>true</embedAssets>
        <attributes>
          <imagesdir>images</imagesdir>
          <toc>left</toc>
          <icons>font</icons>
          <sectanchors>true</sectanchors>
          <!-- set the idprefix to blank -->
          <idprefix/>
          <idseparator>-</idseparator>
          <docinfo1>true</docinfo1>
        </attributes>
      </configuration>
    </execution>
  </executions>
</plugin>

The full POM is here, and the AsciiDoc manual source is here.

Overall, as with so many things, the conversion took far more time and effort than expected, but it was certainly a worthwhile investment: in the long run, the improvements in build automation and reproducibility will pay ample dividends.