Turn form images into fillable fields

A developer used Gemini to map form fields in JPEG and PNG images to boxes, making it easier to automate repetitive form completion.

Done with Geminiยท Advancedยท Apr 2025

What they did
The developer had a friend who wanted image-based forms filled automatically, so they combined Gemini with earlier OmniParser work. To make the system practical, they assembled a labeled set of IRS and OPM forms where each field was marked clearly. Gemini then reviewed a form image, identified the fields it contained, and paired those fields with bounding boxes.
How it went
They described it as a workable way to handle boring form-filling tasks, especially in manual supply-chain work. The thread did not include accuracy figures, timing, or specific breakdowns.
Tips
This approach depends on having your own annotated examples first; prompting alone was not enough. For similar tasks, pause before any final export or submission so a person can check the detected fields and box placement.

Try it yourself with Gemini

Build a small workflow that takes a form image from [file path], finds each labeled input area, and matches every field to its bounding box. Use the details in [my info file] to populate the form, then create a completed copy at [output path]. Before you save, submit, or send anything, show me the detected fields and box locations for review and wait for my approval.

Fill in the [brackets]. Agents can make mistakes, so review anything before it is sent, paid or deleted.

Read the original on reddit.com โ†—

Get 3 agent uses like this every morning

With the day's top AI news. Free, and one click to unsubscribe.