Multimodal Prompting Guide: Better Prompts for AI Images, PDFs, Audio, and Video

Overview

Multimodal AI can inspect visual, document, audio, and video inputs alongside written instructions. The same basic prompt-engineering principles still apply: define the objective, provide relevant context, explain what counts as a useful answer, and ask the model to distinguish evidence from inference. However, each medium introduces different failure points.

An image may be low-resolution or cropped. A PDF may contain scanned pages, complex tables, or text in an unusual reading order. Audio can include overlapping speakers, background noise, or unfamiliar terminology. Video adds a time dimension: an answer may depend on what happens before, during, or after a particular moment.

A strong multimodal prompt therefore has five parts:

1. Input identification: state what the file or media contains and which pages, frames, regions, or time ranges matter.
2. Task: describe the question or transformation in direct language.
3. Evidence rules: ask the model to quote, locate, describe, or label the evidence it uses.
4. Uncertainty handling: require it to say when content is unreadable, missing, ambiguous, or inferred.
5. Output format: specify headings, fields, tables, timestamps, confidence labels, or machine-readable output.

These principles are useful whether you are testing a consumer AI tool, building an LLM application, or designing an internal AI workflow. For production systems, combine the prompt with input validation, access controls, logging, and human review. A prompt cannot compensate for a missing page, corrupted file, or unsupported format.

Checklist by scenario

1. Analysing images

For image inspection, tell the model what to look for and what not to assume. If location matters, identify a region such as “top-right panel” or “the second label from the left”. Ask for a description before an interpretation when accuracy matters.

Analyse the attached image for [specific purpose].

1. Describe only the visible elements relevant to the task.
2. Extract any readable text exactly, preserving line breaks where practical.
3. Identify objects, labels, relationships, or visual changes that support your answer.
4. Separate direct observations from interpretations.
5. Mark any text, detail, or area that is unclear rather than guessing.
6. Return the result as:
   - Observations
   - Extracted text
   - Interpretation
   - Uncertainty and areas to verify

For AI image prompts used to create or edit images, include the subject, composition, setting, lighting, style, aspect ratio, exclusions, and any reference image instructions. Keep the most important constraints near the beginning and state which reference elements must be preserved.

2. Prompting AI to analyse PDFs

Begin by explaining the document’s purpose and the decision you need to make. Mention relevant page ranges, headings, tables, appendices, or definitions. If you need extraction rather than a summary, say so explicitly.

Review the attached PDF, “[document name]”, to answer [business or research question].

- Use pages [range] first, then consult other pages only when needed.
- Extract relevant claims, dates, figures, definitions, and limitations.
- For every important point, provide the page number and section heading when available.
- Reproduce table values carefully and retain the original units.
- Distinguish the document’s statements from your own synthesis.
- If a page is scanned, incomplete, or difficult to read, flag it.
- Do not fill gaps with assumptions.

Return:
1. Short answer
2. Evidence table: claim | page | supporting text or table | uncertainty
3. Open questions

When a PDF is part of a retrieval workflow, preserve page or chunk references in the application’s data model. A useful answer without traceable evidence is difficult to audit. See the practical RAG tutorial for the wider retrieval pattern and the RAG evaluation guide for checking retrieval and answer quality.

3. Interpreting charts and diagrams

Charts need a more constrained prompt than a general image. State whether you want values, trends, comparisons, anomalies, or a critique of the visualisation. Ask the model not to estimate values that cannot be read reliably.

Inspect the attached chart.

Identify the chart type, title, axis labels, units, legend, date range, and displayed series. Then:
- report only values that are labelled or can be read clearly;
- describe the main trend and notable comparisons;
- identify possible outliers without explaining their cause unless the chart provides evidence;
- state whether the scale, missing data, or design limits interpretation;
- flag any value that is approximate.

Return a table followed by a concise interpretation.

For technical diagrams, ask for components, connections, sequence, and labels separately. This reduces the chance that a plausible interpretation is presented as a confirmed design fact.

4. Transcribing and analysing audio

Audio prompts should define the desired transcript and the analysis separately. Specify speaker labels, timestamps, filler-word treatment, redaction requirements, language or vocabulary, and whether uncertain words should be marked.

Transcribe the attached audio for internal review.

- Add timestamps at speaker changes and at least every [interval].
- Label speakers as Speaker 1, Speaker 2, and so on unless names are explicitly provided.
- Preserve technical terms and numbers carefully.
- Mark uncertain words as [unclear] rather than inventing text.
- Do not infer private details that are not stated.

After the transcript, provide:
1. Decisions
2. Action items with owner and deadline when explicitly stated
3. Open questions
4. Statements requiring human verification

For meetings, keep the original transcript separate from the summary. A summary should not silently correct, combine, or reinterpret what was said. The AI meeting notes workflow guide covers prompts and review steps for this use case.

5. Analysing video

Video analysis benefits from explicit time boundaries. Ask for events in chronological order and request timestamps for every material observation. If the task involves safety, compliance, or quality control, define the observable criteria rather than asking whether the video “looks acceptable”.

Review the video from [start time] to [end time] for [specific task].

Create a chronological event log with:
- timestamp or time range;
- visible action or spoken content;
- relevant person, object, or screen area;
- evidence supporting the observation;
- uncertainty or obstruction.

Then compare the observations with these criteria: [list criteria].
Do not infer intent, identity, or cause unless the video provides direct evidence. Mark events that require manual review.

What to double-check

Before relying on a multimodal answer, run through this input-quality checklist:

• File support: confirm that the tool accepts the selected format and can process the relevant content type. Capabilities, file limits, and interface features vary and may change.
• Readability: check resolution, contrast, orientation, clipping, noise, compression, and whether all pages or frames were included.
• Scope: identify the exact page range, image region, speaker, time range, or chart series to prevent irrelevant analysis.
• Context: provide domain terms, abbreviations, units, expected audience, and the decision the output will support.
• Evidence: require page numbers, timestamps, quoted text, coordinates, or an observation label where traceability matters.
• Output: define a stable schema if the result will feed another tool. For example, request JSON fields for observation, evidence, uncertainty, and review_required.
• Privacy: remove unnecessary personal, confidential, or commercially sensitive information before uploading media, and follow the rules applicable to your organisation.

Run a small representative test before processing a large batch. Include difficult examples, not just clean files. Record the prompt, model or tool configuration, input version, output, and reviewer decision so that changes can be compared. A prompt testing framework can help turn this into a repeatable evaluation process.

Common mistakes

• Using one generic instruction for every medium: images, PDFs, audio, and video need different evidence and uncertainty rules.
• Asking for conclusions without observations: require the model to describe the relevant evidence first.
• Omitting locations: “What does this document say?” is less auditable than “Which pages support this claim?”
• Ignoring unreadable content: explicitly instruct the model to flag missing, blurred, obscured, or inaudible material.
• Trusting extracted numbers automatically: verify units, decimal points, column alignment, axis scales, and transcription of dates.
• Mixing extraction and judgement: use separate stages for transcription, structured extraction, interpretation, and approval.
• Failing to define a human checkpoint: high-impact decisions should have a reviewer who can inspect the original media.
• Changing tools without retesting: a new model, interface, context limit, or file parser can alter results even when the prompt is unchanged.

When to revisit

Return to this checklist whenever you process a new media type, receive files from a different source, or notice a decline in review accuracy. Revisit it before seasonal planning cycles if your team regularly handles reports, recordings, product images, or operational footage. It is also worth reviewing after a model, tool, file limit, supported format, prompt, or downstream workflow changes.

For a practical refresh, take one recent example and ask:

1. Did the system receive every page, frame, region, and audio segment required?
2. Can a reviewer trace each important conclusion back to evidence?
3. Were uncertainty and unreadable content reported clearly?
4. Did the output follow the required structure without hidden assumptions?
5. Which failure should the next prompt or validation step prevent?

Update the relevant template, add the example to your test set, and compare the revised result with the previous version. For larger workflows, connect the multimodal step to explicit guardrails and review states; the AI agent workflow guide provides a useful framework for multi-step designs. The goal is not a longer prompt. It is a clearer chain from input, to evidence, to output, to accountable action.