We Rebuilt the Camera: Live Receipt Capture in ReceiptIQ 2.3.0

The old camera in ReceiptIQ was the standard iPhone camera with a shutter button. It worked, in the sense that it took a photo. What it did not do was help: a receipt on a dark table, held at a slight angle, photographed with a little motion blur, is exactly the input the text recognition likes least. Version 2.3.0 replaces it with a camera that looks for the receipt while you aim.

What it does

• Finds the paper. An outline follows the receipt in the live preview, using Apple's document segmentation on every frame.
• Waits until it is worth taking. The outline stays amber while the receipt is cut off, tilted, too dark, glaring or moving. It turns green when the paper is flat, sharp, well lit and inside the frame.
• Takes the photo itself once it has been green for a moment, then crops to the paper and straightens it. Long receipts are handled; you are asked to move back rather than fail silently.
• Sends a cleaner image to text recognition. Table, hand and shadow are gone before the scan starts.

The classic camera is one switch away in Settings if you prefer it, and importing from the photo library is unchanged.

Six rounds on a real phone

We test on real devices before shipping, and this feature needed six rounds. Each one found something the simulator could not:

1. The outline was drawn ninety degrees off. The conversion Apple provides expects coordinates from the sensor's landscape orientation, while the frames we analyse are already rotated to portrait. We now map the points by hand.
2. The app crashed after the first capture. The still was the sensor's maximum, 48 megapixels, and straightening that image ran the phone out of memory. Stills are now capped at 12 megapixels, which is more than any receipt needs.
3. Every image was three times larger than we thought. A rendering helper was drawing at the screen's pixel density, so a "2400 pixel" image was really 7200 pixels wide. This had been true for years and nobody noticed; it explained a good deal of the app's memory use.
4. Tall receipts were squeezed. A cap on the longest side turned a tall, narrow crop into a 700-pixel-wide strip and text recognition lost seven of eleven prices. The cap is now a pixel budget instead, so a tall crop keeps its width.
5. Stored images were unreadable. A wrong scale value made saved receipts 226 pixels wide. Fixed, and stored images are now about a third of their previous size at higher legibility.
6. The crop shaved the last digit off every price. The outline sat a few pixels inside the paper's right edge, and "¥1,900" became "¥1,9". The parser then built a receipt that added up and was wrong. The crop now keeps a wider margin on the sides.

That last one is the kind of bug we care about most: nothing crashed, the sum matched, and the answer was wrong. It was caught by scanning a real receipt and comparing it with the paper, which is the only test that catches that class of error.

What it is not

It does not stitch multiple frames together, and it does not try to read the receipt live. It takes one good photo instead of one mediocre one. Faded thermal paper and handwriting are still hard for any camera. If the outline never turns green, the shutter button still works.

Version 2.3.0 is on the App Store now.