Choosing ISP IP: connect sensor requirements to pixel throughput

Define the sensor-to-output boundary

An ISP converts sensor data into a form usable by the next imaging stage. Start with a named sensor mode: active dimensions, frame rate, RAW bit depth, color-filter arrangement, exposure modes and embedded metadata. Then specify the required output format and consumers: display, video encoder or machine-vision algorithm. A camera image that looks good on a monitor may not be the best input for measurement or detection.

MIPI CSI-2 specifies image transport from sensors to application processors. Treat the CSI-2 receiver and PHY, ISP processing and sensor-control software as separate integration responsibilities unless the quoted deliverables explicitly include them. “RAW input supported” does not establish that the complete camera interface is included.

Sources: 1

Calculate pixels per second before choosing parallelism

For an illustrative UHD input of 3840 × 2160 at 60 fps, active throughput is 497.664 million pixels/s. At RAW12, the tightly packed active-pixel payload is 5.971968 Gb/s, or 0.746496 GB/s. This excludes transport overhead and timing gaps. If memory stores each sample in a 16-bit container, one full-frame write instead carries 0.995328 GB/s before alignment and bus overhead.

A hypothetical four-pixel-per-cycle pipeline at 150 MHz offers 600 million pixels/s, leaving only 17.1% of its nominal capacity beyond that active load. Validate stalls, boundary handling, clock-domain crossings and the instantaneous sensor rate. “Supports 4K” is incomplete without frame rate, enabled functions, parallelism and an operating-corner timing result.

Separate feature presence from a validated operating mode

Map the RAW, RGB and YUV stages and identify optional bypasses. For HDR, ask whether exposures arrive as separate frames, interleaved lines or a sensor-specific layout, and where merging occurs. For temporal noise reduction, request history-buffer size, traffic and behavior on scene cuts. Do not combine the maximum throughput of one configuration with the feature list of another.

Specify tuning deliverables alongside RTL: parameter documentation, sensor/lens calibration, reference images, control APIs and ownership of auto-exposure/white-balance integration. Test simultaneous streams with the actual shared-memory budget. Select the pipeline that meets your image-quality, latency and integration targets together.

Bring these to your IP evaluation

• Provide sensor mode, RAW layout and output format.
• State pixels/s, memory packing and simultaneous stream count.
• Request timing and quality evidence for the exact enabled pipeline.