Markdown Sidecars

What this is

On-demand .md companion files created from PDFs and other non-indexable documents — full, faithful, page-by-page content with image and signature descriptions — so PDF content becomes searchable and usable without repeatedly opening the binary. Includes the standing "prefer markdown" preference. Optional; most useful for document-heavy projects (legal, research).

Assess first

• Check whether admin\AGENT_INSTRUCTIONS.md already has a MARKDOWN SIDECARS section. If present, confirm and skip.
• No upfront setup is required for this feature — it is a capability the agent offers when a PDF appears, not a folder or file to create now.

Ask, only if needed

Nothing to ask at install time — proceed directly to Write. (The agent offers sidecars per-PDF at the moment one appears; it does not ask anything during installation.)

Write

Add this section to admin\AGENT_INSTRUCTIONS.md:

◆ MARKDOWN SIDECARS

Markdown Sidecars are .md companion files created from PDFs and other
non-indexable documents. Available for any PDF in the project. Full
faithful content — not a summary.

WHEN TO OFFER
User mentions a PDF, drops a PDF, or asks to work with PDF content. Offer
once: "I can create a markdown sidecar for that — full text, page numbers,
and image descriptions. Makes it searchable and usable going forward. Want
one?" Do not repeat the offer if declined.

COMMAND: "create sidecar for [filename]"

SIDECAR STRUCTURE
  Header: source file path, source type, total pages, creation date,
    created by (agent/session context)
  Body: page-by-page sections with explicit ## Page N headers, verbatim
    OCR text
  Images: full description of every image — text within images
    transcribed; signatures noted as "Handwritten signature, appears to be
    [name if legible — otherwise UNCLEAR]"; charts/graphs described by
    title and visible data; photos described; unclear images marked
    [UNCLEAR IMAGE — insufficient resolution]

HIGH INTEGRITY RULES — NON-NEGOTIABLE
  1. No assumptions — unclear text marked [UNCLEAR] or [ILLEGIBLE], never
     guessed
  2. No skipping — every page captured, every visible element noted
  3. No summarizing — transcribe faithfully; do not interpret or condense
  4. Images — every image described fully; signatures identified explicitly
  5. Page numbers — every section labeled with its source page number
  6. Low confidence — uncertain extractions flagged [LOW CONFIDENCE: ...]
  7. Partial images — note exactly what is visible; never report partial
     content as complete
  These rules are non-negotiable regardless of document complexity.

NAMING & LOCATION
  Filename: [source_filename]_sidecar.md
  Location: same folder as the source file
  Example: discovery\depo_smith_2024-03-15_sidecar.md

MARKDOWN PREFERENCE — STANDING RULE
  Prefer .md for all working documents, drafts, notes, internal reference
  files. Markdown indexes well, is portable, searchable, version-control
  friendly.
  | Use case                          | Preferred format |
  | Working drafts, notes, briefs     | .md              |
  | User-requested Word output        | .docx            |
  | Final client / external delivery  | .pdf             |
  | Internal reference files          | .md              |
  Reserve .docx for explicitly requested Word output only.
  Reserve .pdf for final delivery — never edit a PDF directly (see PDF RULE).

Update: Excel (.xlsx) sidecar extraction method (added v1.1)

The general rule above covers PDFs. This addendum generalizes a finding first made for Excel trackers (see FTR-36's own PROJECT CACHE SIDECAR section) into this feature itself, so it applies to any .xlsx a project needs sidecar'd, not just Excel trackers specifically.

Approved method: raw zip/XML parse, with UTF-8 explicitly specified on every text read. An .xlsx file is a zip archive of XML parts; extract xl/sharedStrings.xml and the relevant xl/worksheets/sheetN.xml, parse as XML, resolve shared-string references into cell text — without opening Excel at all.

Timed and tested, three approaches compared on a real 197-row/13-column workbook:

The bug, and why it matters: the first implementation read the extracted XML with PowerShell's Get-Content -Raw and no explicit encoding parameter. This silently corrupted every multi-byte UTF-8 character (an em dash rendered as \u00e2\u20ac\u201d instead of a real em dash) — it did not throw an error, and the corrupted output was superficially plausible enough to pass a casual read. Caught only by a direct line-by-line diff against the bulk-COM method's independently-verified-correct output.

The fix, and the standing rule going forward:

1. Use the raw zip/XML parse method, not Excel COM, unless a COM session is already open for another reason in the same script.
2. Specify UTF-8 encoding explicitly on every text file read in the extraction script — not optional, not assumed-default. State it on every Get-Content (or equivalent) call touching the extracted XML.
3. Verify at least once per implementation (not necessarily every run, but at least once when the extraction script is first written or meaningfully changed) by diffing parse-based output against a bulk-COM-read output on a real file with non-ASCII content.
4. The sidecar format itself is unchanged — a flat, text-based representation for fast indexing. For a row-structured source, that means a markdown table mirroring the sheet; this guidance is about how to extract the data quickly and correctly, not about changing what the sidecar looks like.

Pattern for future file types: this is the first instance of a pattern this feature should keep following — as a new file type needs sidecar coverage, test the real extraction methods available, record which one wins and why, and fold that guidance back in as a targeted, versioned update (e.g. a future .pptx or .csv addendum), rather than leaving every file type to ad-hoc, re-derived-each-time judgment.

Confirm

Confirm: Markdown Sidecars are enabled — the agent will offer to turn any PDF into a full, faithful .md companion, and prefers markdown for working files. Want to install another, or is that everything for now?