Parsing Consensus

The idea for this experiment came from a few conversations at CyberCon26. I spoke with people who had asked LLMs questions about IAEA safety and nuclear security guidance and come away with an understanding that the publications did not support. The answers were not always obvious hallucinations. Some quoted genuine passages but lost the distinctions that tell an experienced reader what those passages mean and how much weight to give them.

That made me suspect the prompt was not the main problem. The model was being given the words without enough of the document around them, so I began looking for a better way to represent the source material. I had done something related during the Committee Draft stage of IEC 63096, when I built (poorly) a deterministic parser for SC45 WGA9 that generated Annex A from the structured content in the body of the draft. It seemed worth testing the same general approach here.

It is easy to upload a folder of PDFs. The difficult part is getting answers that remain useful and defensible once document structure begins to matter. A seemingly simple question can depend on much more than finding the right words. Is the passage a requirement, a recommendation, an example in an annex, or a qualification in a footnote? Does it come from the publication being discussed or from a contents page that repeats the same heading? Is it paragraph 4.32, a table row that happens to begin with 4.32, or a page number that has wandered into the extracted text?

These distinctions are routine for someone used to working with IAEA guidance. Once a PDF is flattened into a stream of text, many of them disappear.

So, I built the IAEA Guidance Parser to help preserve them. It turns local folders of IAEA publications into structured, searchable knowledge files for an LLM. I wanted each retrieved passage to arrive with the context a reader would normally have around it i.e. which publication it came from, where it appeared, and what status it carried.

The problem also mattered to me because I had previously worked as a scientific secretary for publications in the NSS. I saw how much work goes into choosing a term, placing a paragraph, and resolving the torrent of comments on any draft text. A paragraph is more than a convenient container for sentences. Its location, its relationship to other publications and its status within the document all affect how it should be read and applied. Those distinctions may appear almost invisible once the final PDF is published but they remain part of the substance created and upheld through the international consensus process.

Search was not the hard part

My first approach was the obvious one: upload either the extracted text or the raw PDFs to NotebookLM or a Custom GPT. That preserves the words but not necessarily the document. A chapter heading may be separated from the paragraphs it governs. A footnote explaining an exception may be retrieved without the sentence it qualifies. A contents entry may look indistinguishable from the real heading later in the publication.

PDFs make this particularly difficult. A PDF is often closer to a set of instructions for how a page should look than a structured representation of paragraphs, headings and footnotes. The page can appear perfectly ordered to a person while its text layer contains lines in an inconvenient or misleading reading order. Many of the semantic relationships are only expressed visually rather than encoded explicitly.

Without those visual signals, retrieval may find a relevant sentence and still support a misleading answer.

What the parser actually does

The parser itself is a deterministic Python program. It reads the PDF text layer, applies explicit recognition and repair rules, maintains state while moving through the publication and emits typed records. The LLM enters the process later, after those records have been exported and uploaded as knowledge.

That boundary is deliberate. I did not want a model to guess whether a sentence sounds like a requirement or whether a block looks like an annex. Those decisions can be made more consistently from the publication’s structure, so the parser, rather than the model, decides where paragraphs, annexes, tables, and requirements begin and end.

Ideally, this structure would be checked by a person or exported directly from the IAEA’s publishing system. As neither option was available—the IAEA’s system or my spare time—the parser reconstructs as much as it can from the published PDFs, largely through trial and error. When I found specific problems, I used an LLM (oh, the irony) to understand the PDF syntax, explore solutions and build regression tests so they would not reappear. This was far from ideal, and I eventually had to turn to my vibe coding LLM friend to restructure the repository before it became a single file containing an endless collection of regular expressions.

1. Extracting text from a format that was not designed for it

The parser opens each PDF and extracts its text page by page. For every page, it retains the physical PDF page number and, where it can identify one safely, the printed page number shown in the publication.

It then performs conservative clean-up. It normalises whitespace, removes recognised page-mastering artefacts, joins lines that appear to be artificial PDF wraps and repairs ordinary words split by end-of-line hyphenation. Dashes used in publication identifiers and annex paragraph numbers are preserved because changing them could alter the identifier.

The clean-up is deliberately a light touch. It reverses some of the artefacts introduced when a visually mastered page is converted back into a sequence of text lines, without changing the publication’s wording.

The current version reads the PDF’s existing text layer and does not perform OCR. If an older PDF contains damaged font mappings, or a scanned page has no usable text layer, the parser cannot reconstruct the intended characters. It can flag suspicious output, but the source still requires review or a separate OCR process.

2. Rebuilding structure with a small state machine

After extraction, the parser walks through the cleaned lines in order. It keeps track of its current position in the publication, including:

• the current region, such as front matter, body, appendix, annex or references;
• the current major heading and subheading;
• whether it is still inside the contents pages;
• the active paragraph or other prose block; and
• whether a table is open across one or more pages.

The order in which it tests each line matters. Tables are considered before ordinary paragraphs because a table cell can begin with something that looks exactly like a paragraph number. Region transitions are handled before prose so that an Annex heading changes the classification of everything that follows. Requirement headings are separated before a preceding paragraph can absorb them.

Footnotes are extracted without automatically ending the paragraph they interrupt. In many PDFs, the main sentence continues after the footnote, sometimes on the next page.

I had thought that with SPESS C guidance for drafters and a single template in use within the IAEA there would be a pretty consistent use of formatting across both series. However that’s not always the case. I’ve had to debug the parser until it was capable of recognising numbering patterns used across different generations of the series, including forms such as:

It also accepts several visually similar dash characters. The PDFs contain hyphens, non-breaking hyphens, figure dashes, en dashes and em dashes that look almost identical on the page but behave differently in text matching. This was already a publication problem long before anyone began blaming LLMs for their dashes.

Multi-line headings are joined before they update the section path. Contents entries are suppressed rather than mistaken for the beginning of the body. Running headers and page numbers are removed before paragraph reconstruction.

Some annexes contain curriculum outlines or reporting templates that are visually tables but have no conventional TABLE N. caption. For these, the parser can create a synthetic table record so that their outline numbers do not become dozens of false paragraphs.

When it reaches a structural boundary, the parser finalises the active element and outputs a record. Instead of dividing the text after a fixed number of characters or sentences, it follows the reference units used by the publications: one numbered paragraph, one requirement, one table, one footnote or one heading.

3. Inferring metadata conservatively

The parser also needs to identify the publication it is reading. It tries to infer the title, series number, document family, category, type, year and other identifiers from the first pages of the PDF. It avoids generic pages that list all categories in a series because those pages can otherwise be mistaken for the identity of the publication itself.

Where the PDF is ambiguous, the parser can fall back to the filename. A YAML configuration can override either source for documents that remain difficult to identify.

The resulting metadata records where each important value came from: configuration, PDF inference, filename or fallback. This makes it possible to distinguish a value read confidently from the publication from one supplied as a practical default.

Each source PDF is also hashed. The hash provides source traceability by identifying exactly which file produced each set of records. It makes no claim about the correctness of the publication itself.

4. Self-describing records

Each structural record contains:

• document identity and type;
• element type and number;
• source region;
• status and the reason for that status;
• section path;
• physical and printed page numbers;
• extracted text;
• parser confidence;
• diagnostic notes; and
• relationships between elements, such as a footnote and its paragraph.

The full machine-readable index is useful for auditing, testing and building other tools. For an LLM, the parser also produces a more compact Markdown export. A record looks roughly like this:

---
doc: GSR-PART-2
record: paragraph 4.32
status: Normative
region: Body
pdf: 27
section: 4. LEADERSHIP FOR SAFETY

...paragraph text...

The parser repeats the labels beside every record so that a passage retrieved on its own still identifies its source and status.

The compact Markdown does not contain every field from the structural record. Parser confidence, detailed status reasons and diagnostic notes remain available in the complete output and QA reports. The LLM-facing version keeps the information most useful during retrieval and citation.

Status is data, not decoration

The most important design decision was to preserve the status of the text. The Nuclear Security Series has a hierarchy of Fundamentals, Recommendations, Implementing Guides and Technical Guidance. The Safety Standards Series is organised into Safety Fundamentals, Safety Requirements and Safety Guides.

Different levels of these hierarchies use different forms of language. Requirements use “shall”, while guides generally use “should” when describing recommended measures or acceptable alternatives.

Status also varies within a publication. SPESS C explains that an appendix is integral to a publication and carries the same status as its body. Annexes and footnotes provide practical examples, additional information or explanation and are not integral parts of the main text. Section 1 establishes the background, objective, scope and structure rather than carrying the publication’s primary requirements, recommendations or guidance.

The parser represents these distinctions using three deliberately simple labels:

• Normative: substantive body content from Section 2 onwards, together with substantive appendix material;
• Informative: substantive annex material and footnotes; and
• Informational: front matter, Section 1 context, headings, references, glossary material, publication metadata and back matter.

The classification is derived from structure: where the element appears, what type of element it is and which section or region governs it. The wording of an individual sentence does not determine its status.

Here, “Normative” refers to the passage’s status within the IAEA publication as approved by the relevant committee. Whether it is legally binding is a separate question determined by the relevant national and international framework. LLMs are very good at combining related passages into fluent prose, and that fluency can hide the fact that the passages do not have equal status.

An annex example may be useful, but it should not quietly reappear in an answer as a requirement. A footnote may clarify a paragraph, but it should not become a new recommendation or “shall”. On the other side of that challenge, excluding all supporting material would discard valuable explanation.

My response to this? Including status on every record! It makes it harder for the model to treat an annex example as equivalent to a requirement.

What the LLM does with the records

Once uploaded, the files are indexed by the platform. The details vary, but the model generally receives only a selection of records for each question. That creates two failure modes: the right record may not be retrieved, or the model may misuse a record that was.

The parser cannot control either stage. It can only make records easier to find and harder to misread by keeping identifiers, section paths, status and page references beside the text. Exact identifiers and paragraph numbers also provide useful anchors when semantic retrieval alone is insufficient.

I therefore instruct the model to preserve distinctions between requirements, recommendations, guidance and examples, cite the publication and PDF page (so the PDF can be more easily referenced), and say when the uploaded material does not support an answer.

Working across the series

With the series represented as structured knowledge, one question can retrieve records from several publications. The model can follow a chain from principle to requirement or recommendation and then to detailed guidance while retaining the document, paragraph, section, page and status of each step.

This creates practical questions that ordinary document search handles poorly:

• What do the overarching requirements say about an issue, and how do the relevant guides elaborate on it?
• Which publications discuss the same responsibility from the perspectives of the State, regulator, and operator?
• Where does a recommendation originate, and which implementing or technical guidance develops it further?
• Do several publications use a term consistently?
• Which passages establish expectations, and which provide examples of implementation?

The questions I found most useful were those that crossed publication boundaries. As an example, I loaded the knowledge files into a Custom GPT and asked it to “Summarise where documents in the NSS are implictly describing a Sensitive Digital Asset (SDA) without explicitly using the term”. This is what it output:

Several IAEA Nuclear Security Series publications address the substance of sensitive digital assets without using that specific term. They discuss computer-based systems, sensitive information assets, digital instrumentation and control systems, networks, and other digital systems whose compromise could affect nuclear safety, nuclear security, nuclear material accounting and control, or emergency response. These references identify systems that may warrant SDA-like protection, but they do not automatically classify every system concerned as an SDA; that determination depends on the function supported, system dependencies, and the potential consequences of compromise.

• Protection of important computer-based systems: NSS-13 recommends protecting computer-based systems used for physical protection, nuclear safety, and nuclear material accountancy and control against compromise. This is the clearest precursor to the later SDA concept. See NSS-13, paras 4.10 and 5.19, PDF pp. 33 and 44.
• Sensitive information assets: NSS-20, NSS-23-G, NSS-29-G, and NSS-11-G (Rev. 1) address computer systems, networks, communications systems, digital media, and instrumentation and control systems that process, store, control, or transmit sensitive information. They emphasize protection of confidentiality, integrity, and availability, access control, and security throughout the information life cycle. This category is broader than SDAs because it can also include non-digital assets. See NSS-23-G, paras 2.9–2.12 and 6.17, PDF pp. 15–16 and 30; NSS-29-G, paras 4.47–4.52, section “Security of Information,” PDF pp. 36–37; and NSS-11-G (Rev. 1), paras 3.20–3.21, PDF pp. 24–25.
• Digital instrumentation and control systems: NSS-33-T provides computer-security guidance for instrumentation and control systems performing safety, security, or auxiliary functions. Such systems are a major potential SDA class because their compromise could prevent important functions from being performed. See NSS-33-T, paras 1.1–1.4, 1.10, and 1.12, sections “Background,” “Objective,” and “Scope,” PDF pp. 11–13.
• Cyber, blended, and insider attacks: NSS-10-G (Rev. 1) considers cyberattacks against safety, security, nuclear material accounting and control, and emergency-response systems, including supply-chain compromise and combined physical–cyber attacks. NSS-8-G (Rev. 1) similarly addresses insider compromise, remote connectivity, and the monitoring of sensitive computer assets. See NSS-10-G (Rev. 1), paras 2.19–2.20 and 7.12, sections “Potential Adversaries and Their Attributes and Characteristics” and “Developing Attack Scenarios,” PDF pp. 18 and 39; and NSS-8-G (Rev. 1), paras 3.9–3.10, 4.78, and 4.84, PDF pp. 19 and 35–37.
• Application in particular operational areas: Other publications apply related computer-security concepts to physical-protection computers and networks, computerized nuclear material accounting and control records, border-detection systems, transport tracking and control systems, and contingency response. Examples include NSS-27-G, paras 4.140–4.145, PDF pp. 99–100; NSS-40-T, paras 3.44 and 6.1–6.5, PDF pp. 29 and 136–137; NSS-25-G, paras 4.57–4.58, PDF pp. 34–35; NSS-32-T, paras 3.18–3.20, PDF pp. 19–20; and relevant passages in NSS-44-T, NSS-46-T, and NSS-39-T.

These passages identify possible SDA populations, not automatic SDA designations. Whether a particular system is an SDA still depends on the function it supports, its dependencies, and the consequences that could result from its compromise.

Tables and figures are where honesty matters most

Tables quickly became the most persistent parsing problem, a table expresses relationships between rows, columns, headings and cells. PDF extraction often preserves the words but loses those relationships. It can return every cell in sequence without reliably indicating which row or column each value belonged to.

Continued tables may repeat their headings. Cell labels can resemble paragraph numbers. A paragraph immediately following a table can be absorbed into the table record. Some annexes contain curricula or report outlines that are visibly tabular but have no useful table metadata in the text layer.

The parser handles table boundaries conservatively and recognises several outline-style blocks as synthetic tables. It preserves the raw extracted text and relevant page range, leaving uncertain row and column relationships unresolved.

For a simple table, the extracted text may be enough. Where the relationship between a row and column affects the meaning, the page image must be checked.

For figures, the record contains the identifier, caption and page reference. Interpreting the diagram still requires inspection of the page image. Someone asked why I did not have another LLM interpret the figure. That would have reintroduced exactly the kind of probabilistic judgement I was trying to keep out of the parser 😉

Building in reasons not to trust it

A tool used for high-consequence material should make its weaknesses visible. Each run produces manifests, source checksums, a parser version, a run identifier and QA reports alongside the knowledge files. Automated checks look for problems such as:

• requirement markers left inside ordinary paragraphs;
• footnote bodies contaminating substantive text;
• page headers or footers that leaked into records;
• wrapped headings that were not rejoined;
• contents-page entries mistaken for body content;
• table cells misidentified as paragraphs;
• status labels inconsistent with document regions;
• suspicious encoding or damaged characters;
• duplicate record identifiers; and
• disagreement between the records and the series manifest.

These checks are alarms rather than certifications of perfect parsing. An empty report means only that the deterministic tests found none of the failure patterns they currently recognise.

What the tool cannot promise

The parser has limits that I’ve already described and there are still a substantial number of parsing failures, particularly in the Safety Standards Series where I have much less experience with the historical formatting conventions (should a Schedule be treated the same as an Appendix?). I mostly built this for my own use across the NSS.

Using the parser output in a Custom GPT or NotebookLM adds another layer of limitations. A structural boundary may be missed. A damaged text layer may alter a symbol whose exact value matters. A complex table may lose the relationship between a heading and a cell. A figure caption cannot substitute for the figure. A low-confidence footnote boundary may require manual review. Metadata inference can also be wrong, which is why the parser supports overrides and records its sources.

The retrieval system may fail to surface a relevant record or may retrieve a passage without enough neighbouring context. The LLM may misunderstand a label, combine passages too aggressively or produce a citation that does not support every part of its sentence.

For important interpretations, decisions or exact quotations, the answer must be checked against the official PDF. The parser and LLM are best used to help readers locate and compare material before they return to the official publication for verification.

What I learned

Building the parser changed the question I was asking. I began by wondering how to put a collection of IAEA publications in front of an LLM. The harder problem was deciding how much of the publication, conveyed visually, I could possibly reconstruct from the PDF. I tried to find a practical compromise, but many of these problems will ultimately require full OCR, human interpretation or, ideally, structured output from the IAEA’s publishing system.

The closest analogy is a library in which every book has been emptied into one pile of pages. An LLM may be remarkably good at reading those pages, but someone still has to restore the page numbers, chapter dividers, catalogue records and shelf labels.

At least that is what I was trying to do. You can find the output captured in Custom GPTs here:

• Nuclear Security Series Assistant
• Nuclear Safety Standards Assistant

I am going to keep playing with it, but there are already too many edge cases for a side project. The irony is that I will probably let AI do more of the coding for a tool designed to stop AI from getting things wrong. My role may soon be to just stay on good terms with our machine overlords.