Converting Documents from PDF
to XML & MS Word

Avoiding The Pitfalls

If you're serious about converting from PDF, out-of-the-box solutions should stay-in-the-box, writes Mike Gross, Chief Technology Officer at Data Conversion Laboratory (DCL).

In today's electronic/Internet age, documents can be created using a vast array of text formatting, word processing, desktop publishing, and drawing tools. But Adobe's PDF format has rapidly become a standard way of distributing documents in electronic format. As a result, most documents that are published today exist as a PDF document at some step along the way, even if they are ultimately going to paper.

Although PDF represents an easy and convenient way to electronically represent the paper document, PDF is more correctly a page layout file format than it is a word processing or desktop publishing file format. Therefore, although some minor modifications to a document are possible via Adobe's Acrobat software, it is not intended as an actual source publishing tool (nor is it practical to use it as one).

There often arises a need to republish PDF documents, which requires converting from PDF into a format that is easier to work with. In this white paper, we will address some of the issues and problems you can expect to encounter when converting from PDF into a "source document" format.

Flavors of PDF

There are several different flavors of PDF. In this white paper we are looking at PDF Normal. You get PDF Normal when you produce a PDF document from a text publishing tool (via Acrobat Distiller or other PDF writers). Adobe Acrobat allows other flavors of PDF to contain raster images of each of the pages of the document (with or without some text in the background to allow text searching). These PDF documents are referred to as Image Only or as Image+Text. You get these when you scan paper documents (via Acrobat Exchange or some other method). They are typically used for delivering some form of PDF quickly. But Image Only and Image+Text PDF are not much different to converting from paper, in the sense that OCR issues arise, and so are beyond the scope of this white paper.

Fundamental Problems of Converting From PDF

Extracting Document Text

The good thing about starting with PDF as your source format is that the actual text of the document is stored in an easily accessible way - not just the character chosen, but the specific font (such as Times New Roman). The font weight and font size are also specified in the PDF, so that in most cases, the text that you extract from the PDF document will be completely accurate. But even in the case of document text, there are certain elements in a PDF document that can produce errors when put through the conversion process. These are:

Word Spaces — In most cases, the spaces between words are properly extracted from a PDF document. However, in some instances, the spacing between characters is such that a conversion program cannot completely know whether a space should be there. If the software guesses wrong, and inserts a space where there should not be one, you can end up with a word split in two when it was not meant to be. If, on the other hand, it does not insert a space where there was meant to be one, you can get two words incorrectly joined. Both of these are ugly errors in the converted document.

Hyphens — Since the end-of-line hyphens that appear in published documents are not distinguishable from other types of hyphens that appear within lines in the PDF, conversion software has a problem. Assuming that it has properly determined the end of a line, it knows that most end-of-line hyphens are probably meant to be soft hyphens (only there to make the page layout look nice), but might be hard-hyphens (that are always there in certain word pairs, such as 'life-cycle'). Most conversion tools use dictionary-based algorithms to try to decide whether to leave these hyphens or remove them. These algorithms work pretty well. But unfortunately, there is simply no way to get this right 100% of the time.

Emphasis, Super-, and Sub-scripting — Sometimes the way that the document is rendered in the PDF is done in a non-direct way, so that getting font emphasis (such as Bold and Underline) correct is not always a given. In addition, the way that vertical positioning can be done within a PDF is such that, like with word spacing, the extraction software has a tough time determining whether something is in fact super or subscripted, and it simply makes a guess.

Special Characters and Sub-fonting — When converting special characters (such as foreign symbols and mathematical symbols), the source document often makes use of unusual or proprietary fonts, and the special characters need to be converted to more standard representations (such as ISO character entities or Unicode character representations). Typically, conversion software builds character conversion tables, but it is simply not feasible or practical to have these built for every font that a conversion program may encounter, so some of these characters may convert improperly. A more difficult problem relates to PDF's ability to do font embedding. A user can ask that only the part of a font that is used in a document be stored in the PDF file. Sometimes, when this is done, the characters within the sub-font are referred to through an indirect table within the PDF document, thereby making conversion of these characters extremely difficult. Many conversion tools "choke" on these types of characters — often rendering gibberish.

The bottom line with text is that you can expect PDF conversion tools to get most of the text in your document correct. In some cases, it may get all of your text correct. But since the possibilities for error from all of the problem areas we mentioned above are real, it is a good idea to do some level of proofing on your documents to ensure that all of the text was extracted properly.

Extracting Document Structure

On the whole, extraction of text from a PDF document works pretty well. The big downside is the PDF document specifies text positions on paper, but not much else. In most cases there is no information about the structure of a document. To do a decent job, conversion software is forced to "reverse engineer" the structure. This involves educated guessing, which sometimes leads to mistakes.

Now let's discuss the structural elements that typically cause problems:

Multiple (Newspaper) Columns — Many PDF documents contain "newspaper" type columns of text. Unfortunately, there is usually no column boundary demarcation in the PDF. So software is forced to guess which text belongs in each column of the page and break it apart based on the page geometry. This task is often accomplished successfully by conversion tools. But some layouts give them problems, especially with short columns, where there is not much useful geometry information. Getting this wrong is particularly painful, because it will result in lines from separate paragraphs completely intermingled, resulting in a very ugly paragraph that is hard to comprehend, and is very difficult to clean up.

Text Flows — Some documents, such as magazine articles and textbooks, often have text boxes set off to the side, and commentary text that runs alongside paragraphs, in which case the text flow from paragraph to paragraph is not at all obvious. This is also a challenge for conversion software.

Paragraph Delineation — In most cases, there is nothing in the PDF document indicating where a paragraph ends (usually noted by inserting a hard return) and a new one begins, so this too is guessed at by software. Again, shorter paragraphs are harder to determine. You should not expect software to guess paragraph delineation correctly all the time. You may get two paragraphs running together as one, or one paragraph running as two. Paragraphs that span two pages are also difficult to deal with.

Page Header and Footers — Typical published pages contain headers and footers at the tops and bottoms of pages. This is usually information that, although it appears on every page, is not desirable for it to appear in the target documents where it appeared in the source. So conversion software needs to attempt to guess these elements.

Tables — One of the hardest document elements to deal with is tables and tabular material. Ideally, you want a conversion tool to faithfully represent the original table in the converted document. The reality is that there is so much to a table that getting it completely right is beyond what your expectations should be. Table attributes, such as delineating the columns and rows, header and body delineation, vertical and horizontal cell spanning, cell separators, and vertical and horizontal cell alignment, all require a certain amount of guessing. And given that even a human sometimes needs to read a table for a few minutes to infer the tables structure, it's just not reasonable to expect software to get this right. Even recognizing that something is a table versus some other element is not trivial.

Graphics — When there are graphics within a document, the conversion software will typically convert the graphic to some sort of raster image format. However, in some cases, guessing which parts of the page belong to the graphic, as well as determining what comprises the graphic caption, can be quite tricky.

Mathematical Equations — These are extremely complicated document elements, which are usually authored using sophisticated equation authoring tools. You should not expect these to be extracted from a source document. You will either need to leave them as images in the converted documents (which may be okay, depending on the situation); or you may have to have them rekeyed in your new source document, in whatever format you decide to use to represent mathematical equations.

Part Two: Issues related to specific target formats

So far, we discussed the various global issues to address when you convert from PDF. We didn't cover the specifics of any particular format since most common document formats need the same elements from the source document when you convert from PDF. For example, the logical elements in the source document - such as proper word spacing, de-hyphenation, paragraph borders, special characters, multiple columns, text flow, and table layout - all need to be identified.

In this section we will go further and look at several popular target formats and the issues specific to them.

MS Word

Most of the issues discussed in the first part of this white paper relate directly to MS Word as it is the most common program for authoring documents and is a natural target format for those PDF documents that will need to be maintained and modified. It's also the easiest program in which to make fixes to elements in the source document that weren't converted properly. The key item to be aware of is that while professionally authored Word documents would ideally use style sheets to maintain consistent looks within documents, the conversion programs normally do not apply styles. Therefore, besides the normal cleanup tasks, if you need the documents to conform to a specific style sheet, one of your cleanup tasks will be to go through and manually style the document paragraphs.

Besides the lack of styling, the other key issue is that it is often possible to virtually replicate the look of the original PDF, but in a form that's not necessarily maintainable in that way. This is especially true for tabular material, in which the look can be replicated with exact positioning of lines and other elements in a manner that will look correct, but which will be extremely difficult to edit and maintain.

RTF

If you convert to RTF, you can then import your converted documents to any authoring program that allows RTF import. What applies to MS Word holds true for RTF and most other desktop publishing and word processing programs.

XML & SGML

Both XML and SGML require similar tagging which needs to be done at two levels - tagging of the document structure, as well as tagging of content elements. While the structural issues discussed above apply directly, there is the additional issue of tagging to your specific DTD or Schema, which the generic conversion programs know nothing about. Furthermore, XML/SGML documents will also require tagging related to document content (such as section titles and cross referencing). This will need to be applied either manually or by software in a post process.

Some conversion programs produce an intermediate level "vanilla" XML, and you can then use an XSLT script to transform the intermediate document into final form. In all these transformations it's important to realize that you are constantly inferring information which doesn't appear explicitly in the document. This is an inherently difficult process. The conversion programs will typically represent XML and SGML tables using either CALS or HTML table formats. These do a good job of replicating the look and feel of the original PDF table, but you should expect to have to do some clean up on the marked up tables.

HTML

There aren't enough tags in generic HTML tagging to fully replicate the document structures you'd find in most moderately complex documents. As a result many conversions to HTML are approximations - at best. However, using Cascading Style Sheets (CSS) in the latest versions of HTML, you can produce quite sophisticated paragraph layouts. The use of CSS, however, is a double-edged sword. While some of the PDF conversion tools have a tendency to make use of these features to replicate the look of the original page, and accomplish that task admirably, they do so by using features that are very difficult to properly edit and maintain. For example, since the layout done for paper publishing is not necessarily the same layout that you'd use for a computer screen inside a web browser, you could have difficulties re-wrapping the text for the format (due to the way converted documents are coded). This is particularly true of tables in the original PDF document, which should be rendered using HTML table tagging, making them much more flexible and "re-wrapable" so that they can be more readily displayed on smaller display devices (such as a PDA).

Several approaches to PDF conversion

Because PDF was designed to be a print-layout format, and not intended to be editable, as discussed above, the PDF document conversion software has a lot to do. It is a field in which a considerable amount of work is being done, and it is getting better over time. But, the fact is, the job of PDF conversion is rarely done perfectly, and you should expect to have to do some cleanup.

There are currently several software tools that support conversion from PDF into specific target formats. Some of them are Acrobat plugins, others are standalone software programs, while others operate in a service bureau mode (you upload the files to them, and they send you back converted documents). There are even freeware programs available that will do some level of conversion on PDF documents.

The options and control available to the user also vary greatly. For many of the tools that support PDF conversion, the user simply asks the software to export the PDF document to a particular target format (this method offers some user controlled options, but generally there is very little user intervention). Other tools require the user to "zone" the document manually. In this case, the user is defining the page flow zones on the page, such as multiple columns and differentiating between graphics, tables, and text. There are also tools that do their own guessing of the zones on a page, and then allow the user to override the zones guessed at by the conversion software.

DCL's preference is this last way, where the software attempts to guess page zones and the user gets to override this. In general allowing user control of the document before it is completely decomposed is important. As already mentioned, these tools occasionally don't de-columnize a page or break apart a table properly. It is much harder to fix these elements after the output software has run.

It should be mentioned that one of the options available is the internal 'Save As' option available within Acrobat itself. Adobe Acrobat 6 has a whole array of formats you can save a document in. Plus it has a good degree of functionality. For example, it can find paragraphs, decolumnize text, and find and decompose tables. Depending on the complexity of your documents, this solution may suit your needs. Unfortunately, it doesn't allow user intervention after the page is "zoned" and before it is output. So you are forced to clean up those types of errors afterwards.

OCR software tools are another option. Some of the OCR tools that have been around for many years have recently added support for PDF Normal documents - and since they have fairly sophisticated page layout capabilities, they may produce decent results. You need to be aware, however, that they may simply be OCRing the page. In which case you may get text accuracy errors that normally would not be an issue with PDF Normal documents. Making use of the text layer in the PDF documents is a better choice then attempting to "recognize" the document text.

Conclusion

In summary, there are no magic bullets - the features you need supported depend greatly on the complexity of your source documents. With any of the available tools, you will need to test carefully to see how well they convert your particular materials. These tools have made great strides in recent years. But because you can do so much in a PDF document, the conversion process will become ever more complex. You may find that a particular tool does very well with most of your documents, but breaks down on others; so it may be only a partial solution.

The various tools may output to different target formats, but these are somewhat interchangeable (an HTML document, for instance, can be imported directly into MS Word). Therefore, the quality of how the program decomposes the various page elements is more important than the specific format that it saves its output in.

In recent years, Adobe has added the concept of re-flowable PDF constructs, so that publishing tools that produce PDF will be able to pass more information to the PDF documents. This is in Adobe's own best interests, as they would like to make their PDF documents more easily renderable on the ever more prevalent handheld devices.

Theoretically, this will aid PDF conversion software tools in doing their job. So PDF document conversion may get easier in the future.

Certain conversion tools make use of very specific features of the various output formats to re-duplicate the look of the original page. In most cases, you should avoid using these representations of the converted PDF documents, since they tend to sacrifice logical structure in favor of a specific appearance. This makes the converted documents harder to maintain and less "re-purposable."

Lastly, it should be reiterated that, except for simple documents, you should not expect to get perfect final output from these conversion tools. Some level of proofing, manual review, and cleanup will always be required - and you should plan for it.

Because there are no magic bullets, our approach at DCL is to constantly re-evaluate the various tools on the market so that we can incorporate the best of what's out there into what we do. But since for the foreseeable future tools are only part of the solution, and since what we do is deliverable ready-to-use-documents, we've built the review and cleanup processes as an integral part of our workflow. We've found that the key to providing quality content is to have the right level of intervention and quality control at the right moments.

Be first in line to read new white papers on PDF, XML, and data conversion. Subscribe to DCLnews, Data Conversion Laboratory's popular tech newsletter now!