Expert publishing blog opinions are solely those of the blogger and not necessarily endorsed by DBW.
Typography and beautiful books are dear to my heart, which is why I love to design books. When I receive a new manuscript, I like to spend my time working with its visual design rather than spending hours cleaning it up and structuring its content. I want to enjoy the typographical playground. But because many final manuscripts have been edited and re-edited many times, they often contain technical issues that cause design problems when the content is converted to an ebook.
As a software engineer, I knew there had to be a way to automate the process of preparing documents for publication. I had a vision of clicking a button and enjoying a cup of tea while I watch my computer do the work of converting an author’s manuscript into a file ready for an e-reader or printing press. With the idea of letting the a software program do the monotonous work, I set out to create a book conversion workflow. Building a comprehensive and intelligent workflow automation has been quite a journey. Here’s my story.
My First Ebooks
I created my first ebooks by hand several years ago. The first few manuscripts I designed came from people I know who wrote books and wanted to publish them in both e-book versions and print. The first book I created was Breath Of Love by Bhante Vimalaramsi. At first, I tried various web services and tools, but their results were not satisfying to me: the mess of the original manuscript was simply transferred into the ebook, there were rendering issues on different devices and apps. For example, Microsoft Word internally leaves quite a colorful mess of annotations in the text, including formatting spaces, empty formatting residue, funky inline images for list bullets, broken character encodings, and so forth. This messiness is more than just visual, it goes much deeper into the guts of the DOM (document object model) of a text editor’s internal document representation.
Tools seldom checked for issues in the text, and in the end the ebooks often wouldn’t validate against official standards. I tried the commonly available ebook conversion tools, they were much too tedious. I tried various websites. I tried Word’s “Save as EPUB.” In one ways the tools were all alike: when a mess goes in, a mess comes out. An EPUB file can be checked that it meets the IDPF standards (https://github.com/IDPF/epubcheck). Surprisingly, many generated EPUB files do not meet these standards, causing all sorts of render and usability issues. So I opened a text editor and went to work myself.
One by one, I arduously transcribed the paragraphs from the original Word manuscript into HTML, and with tedious handiwork I cleaned up their formatting issues and typos. Sometimes, the text styles had me guessing the book’s structure, but leafing back and forth through the manuscript helped me to understand what the author may have intended the book’s structure to be. From that manual transcription, it was then easy to build the EPUB scaffolding. There was my first ebook, which mostly took advantage of the default of e-readers styles. The ebook was clean and simple, it rendered well on whatever device I tried, and, most importantly, it validated perfectly.
In that same manner I worked through next second manuscripts, a few books by Katarina Cernozubov-Digman, and by Lesley Synge. Already I began to feel bored by the repetitiveness of the process. I felt like I was wasting my time doing work that my computer could do so much better, and I felt robbed of the joy of actually being creative and designing the book itself.
More Software Solutions
These are the moments when it’s tremendously handy to be an experienced software engineer. I set out to write myself a little program that would construct the EPUB scaffolding from the HTML file which I had produced so laboriously. Soon, I switched to using XML, the markup language on which HTML is based, because HTML wasn’t expressive enough for my needs. Industry-strength tools worked much better with XML.
I soon noticed that little mistakes snuck into my manual transcriptions. So I expanded my XML digestion with error checks to find spelling and punctuation problems, and to either fix them automatically or at least warn me about them.
Ebooks weren’t enough, though. I wanted to create PDF files that are ready to go to the printer and to produce files in other formats, for example ICML (InDesign) or JSON for web applications. Word and its clones just don’t produce well designed books, and I often looked at the original manuscripts wishing that they would look less ill-designed, less amateurish. I stumbled upon a tool that allows me to apply CSS styles to my XML files. CSS is de-facto standard for styling text. So I excitedly began to design and create print-ready PDF files that looked beautiful and professional.
Because both ebook and print book now came from the same file, I could make edits to a single source and then generate the final book in different formats automatically.
Separating Content and Structure
I realized the importance of separating a manuscript’s content and structure from its visual design, how content is really quite different and independent from its presentation. For example, when I view this document on a printed page, then I style that content for the printed page. But I also want to view that same content on a small screen (e.g. Kindle or iPhone). That means that I have to adjust how the same content is styled depending on the media. Looking at a book on a 6-inch black and white screen versus a large-format print coffee-table book is hugely different. That difference is purely presentational, not content. (See my blog “Of Carts and Horses”.)
However, with all the automatic creation of print and ebooks from XML, I still found myself spending long hours insipidly transcribing manuscripts into XML, stripping them of their visual mess, and cleaning them up manually.
Once again, I rolled up my software engineering sleeves and set out to make my life easier. This time around though, the task was not at all easy.
Think about it: an author writes and structures a book using chapters and sections, emphasizes text portions, references other material in the book or externally, elaborates on the text using footnotes and endnotes, and so forth. These different structure elements are styled visually to set them apart from one another, and to guide the reader through the book without becoming distracted.
Using AI to Structure Manuscripts
I wanted to automate the reversal of this creative design process: derive an intended hierarchical structure of the manuscript from its styled elements!
So, I have been busy for about a year now refining an intelligent structure and content extractor using machine learning approaches. As it turns out, this is not only an engineering challenge, but also an interesting academic research topic.
Today, my tools have become much better. But because no automatic classification is perfect, I still have to confirm or adjust the guesses of my software. But that’s only another mouse click, and I don’t have to put my cup down.
A Workflow For Tea and Creativity
My company, Bookalope, has turned into more than just my own ebook workflow. In addition to an interactive website I now beta-test integration into Adobe InDesign. I am getting where I wanted to be: I can click a button and sip my tea while I watch my laptop do hours of tedious manual work in mere minutes.
When I’m done with my tea, I can indulge in designing my book, secure in the knowledge that the rest—content extraction and production of the final ebook and print book—are all taken care of. And that’s what I really wanted all along…
I’ve started to make my tools available online. Automation is what Bookalope does best, so you too can pay more attention to what matters: enjoy your book design and your tea.
Jens Tröger is the author of the Bookalope tools and website (https://bookalope.net). By trade he is a software engineer and researcher with more than two decades of industry experience. Over the years he has worked for i.a. Microsoft and Intel, before he started out on his Bookalope adventure. A confessing librophile and typophile, he enjoys collecting and working with books in his spare time.