Multi-Modal Model Interaction & Context Fusion

Extending the Four Layers to Vision, Document Scans & Audio

Vision Patch Tiling Math & Spatial Coordinate Grounding

export function calculateVisionTokens(width: number, height: number, detail: 'low' | 'high'): number {
  if (detail === 'low') return 85;
  
  // Step 1: Scale to fit within 2048x2048 box
  let [w, h] = fitWithinBox(width, height, 2048, 2048);
  // Step 2: Scale shortest side to 768px
  [w, h] = fitShortestSide(w, h, 768);
  
  // Step 3: Count 512x512 tiles
  const tilesX = Math.ceil(w / 512);
  const tilesY = Math.ceil(h / 512);
  return 85 + (tilesX * tilesY * 170);
}

const SpatialExtractionSchema = z.strictObject({
  vendorName: z.string(),
  invoiceTotal: z.number().positive(),
  // Normalized coordinates [0, 1000]
  totalBoundingBox: z.tuple([
    z.number().min(0).max(1000), // ymin
    z.number().min(0).max(1000), // xmin
    z.number().min(0).max(1000), // ymax
    z.number().min(0).max(1000), // xmax
  ]),
});

// Layer 2 Preprocessing Pipeline
export async function prepareScanForContext(rawImageBuffer: Buffer): Promise<Buffer> {
  return await sharp(rawImageBuffer)
    .rotate() // Auto-orient via EXIF
    .deskew() // Straighten tilted scans
    .linear(1.2, -10) // Boost contrast
    .resize({ width: 1536, withoutEnlargement: true })
    .toBuffer();
}

export function verifyInvoiceExtraction(extraction: InvoicePayload): VerificationResult {
  // 1. Math reconciliation
  const calculatedSum = extraction.lineItems.reduce((acc, item) => acc + item.price, 0);
  if (Math.abs(calculatedSum - extraction.invoiceTotal) > 0.01) {
    return { passed: false, reason: `Math mismatch: items sum to ${calculatedSum} but total is ${extraction.invoiceTotal}` };
  }
  // 2. Spatial bounding check
  if (extraction.totalBoundingBox[0] < 500) {
    return { passed: false, reason: 'Total bounding box must be in bottom half of invoice' };
  }
  return { passed: true };
}

Specify visual quadrant focus (e.g. "top-right vendor header") and request normalized bounding boxes [ymin, xmin, ymax, xmax].

High-res image tiling: 85 base tokens + 170 tokens per 512x512 patch (e.g. 2048x1536 = ~1,105 tokens).

Validate PDF MIME types, sanitize Base64 streams, and pass temporary pre-signed S3 URLs rather than raw buffers.

Tabular schema reconciliation (line item sum == total) and bounding box IoU validation.