Multimodal Vision AI Architecture: Bounding Boxes, Spatial Grounding & Document OCR
Visual reasoning in 2026 has advanced far beyond descriptive captions. Learn how to architect spatial object detection, parse complex financial graphs, extract dense tabular invoices, and calculate image token budgets with high-detail tiling.
[ymin, xmin, ymax, xmax] alongside every extracted entity. Spatial grounding forces the attention mechanism to physically locate items before generating claims.
Cursor AI & Claude 3.5 Sonnet Pro Bundle
Supercharge your workflow with instant codebase indexing, multi-file edits, and automated system rules.
1. Normalized 2D Bounding Box Protocol
Modern frontier vision models (Claude 3.5 Sonnet, Gemini 1.5 Pro, GPT-4o) understand normalized integer grids from 0 to 1000. Here is how to format spatial detection requests:
{
"detected_elements": [
{
"label": "total_due_amount",
"box_2d": [785, 620, 815, 890],
"extracted_value": "$14,850.00",
"confidence": 0.99
},
{
"label": "signature_line",
"box_2d": [890, 150, 960, 420],
"is_signed": true
}
]
}2. Image Tiling Economics & Resolution Calibration
Passing images to vision endpoints incurs input token costs calculated via dynamic tiling algorithms:
| Image Resolution | Tiles Created (512x512) | Total Input Tokens | Estimated API Cost |
|---|---|---|---|
| 512 x 512 (Thumbnail) | 1 Tile | 255 tokens | $0.0006 |
| 1920 x 1080 (Full HD) | 4 Tiles | 765 tokens | $0.0019 |
| 3840 x 2160 (4K Document) | 12 Tiles | 2,125 tokens | $0.0053 |
Frequently Asked Questions
Should I downscale high-resolution scans before sending to the API?
For dense documents with small 8pt text, do NOT downscale below 1500px width. Downscaling introduces anti-aliasing blur that causes number hallucination (e.g. confusing 8 and 3).
Can vision models read multi-page PDFs directly?
Yes. Modern endpoints accept PDF files by automatically converting each page into an image tile array. For large 50-page documents, convert only relevant pages to avoid token exhaustion.
Cursor AI & Claude 3.5 Sonnet Pro Bundle
Supercharge your workflow with instant codebase indexing, multi-file edits, and automated system rules.