Increasing the Repeatability of Your Sharpness Tests

Owned by Lexi Irwin
Last updated: Apr 26, 2022
7 min read
By Robert Sumner
With contributions from Ranga Burada, Henry Koren, Brienna Rogers and Norman Koren

Consistency is a fundamental aspect of successful image quality testing. Each component in your system may contribute to variation in test results. For tasks such as pass/fail testing, the primary goal is to identify the variation due to the component and ignore the variation due to noise. Being able to accurately replicate test results with variability limited to 1-5% will give you a more accurate description of how your product will perform.

Since Imatest makes measurements directly from the image pixels, any source that adds noise to the image can affect measurements. A primary source of noise in images is electronic sensor noise. Photon shot noise also contributes significantly in low-light situations. Other systemic sources of measurement variability, such as autofocus hysteresis, will not be addressed in this post.  

In order to reduce variation in your sharpness results and increase test repeatability, you should take steps to decrease the amount of noise in your image.

Here are 5 tips to limit noise in your test results:

1. Maximize your samples

Since most sources of noise are independent across both different exposures and pixel locations within an exposure, their influence can be effectively canceled by averaging multiple samples.

In order to exploit the temporal aspect of noise, you can combine multiple images of the same scene using the “Combine files for signal averaging” option when selecting multiple image files for analysis using Imatest’s Fixed Modules. This trick works for all analyses in Imatest.

Similarly, for analyses based on Regions of Interest (ROI) which are scale-invariant, such as the preferred MTF-measuring slanted edge technique, increasing the area of the ROI around the slanted edge you analyze in an image increases the number of independent random samples used. In general, you want to select as large a window around a slanted edge as possible given the chart and staying within the desired region of the image field (since MTF generally varies around the field). This trick does not work for tests that have a fixed feature size in an image, such as Siemens-stars and hyperbolic wedges.

2. Ensure adequate signal level via chart contrast (but not too much)

For sharpness measurements, the signal that you want to measure is related to the amount of contrast in the image. The more contrast you have, the higher the signal to noise ratio (SNR) and the less effect noise will have on results. There can be too much of a good thing, however: as the pixels in your sensor reach the point of saturation, clipping, or non-linear response regions, unrealistic increases in sharpness will occur.  The ISO 12233:2014 standard specifies a test chart edge printed at 4:1 contrast in order to prevent this saturation for most systems.

The slanted edge signal level is intimately tied to the contrast of the light and dark sides of the edge. There is an optimal range of contrast for slanted edges that you should try to achieve in order to obtain reliable results. This partially involves choosing a chart appropriate for your test setup.

Increasing the contrast in the outer regions of a test chart, which are most impacted by shading, is one technique for increasing signal level. When ordering charts from Imatest, you can request customizations such as this as appropriate.

3. Be aware of the effects of processing an image

Some devices tested with Imatest can produce raw images that have not been processed by software after capture. In such cases, Imatest can provide accurate measurements of the combined system of lens and sensor. Whenever a camera device processes an image prior to input to Imatest the effects of that processing can be observed, studied, and understood, but cannot be ignored.

When an image is converted to an 8-bit (24-bit color) JPEG from a higher bit-depth sensor, noise increases slightly due to quantization. The noise increase can be worse (“banding” can appear) if extensive image manipulation (dodging and burning) is required. It is often best to convert to 16-bit (48-bit color) files. Processing also often includes sharpening, which can increase the relative power of noise at higher frequencies.

A final caveat on processed images is that many consumer cameras (especially mobile device cameras) use non-linear noise reduction (such as bilateral filtering), which may smooth out images noise on slanted edge targets but also reduce texture detail. (Side note: Averaging multiple images as suggested above will not work when non-linear processing such as this is involved.) In such a case, slanted edge measurements may not tell the whole story of sharpness, and a random (texture analysis) chart may be more appropriate.

4. Ensure a better exposure

Lower light environments often necessitate higher ISO speeds in order to get a good exposure, which leads to increased sensor noise and variation. Ensuring a good photographic exposure can reduce both photon shot noise and the relative effect of sensor noise in an image. The two primary ways of increasing exposure values (though be careful to keep the light areas below the saturation level of your sensor, as mentioned above!) are:

  1. Increase the amount of light reflected by the chart by increasing the brightness of your light source

  2. Increasing the exposure time to gather more light, as long as the camera and target are both stationary

5. Select a repeatable measurement

The shape of an MTF curve gets perturbed in the presence of noise. It is often impractical to compare two full MTF curves or to include one in a report, so engineers often reduce the information about the curve to one or two summary metrics. These are meant to convey the most important information about the curve in a single number. Common examples are:

  • MTF10, MTF30, and MTF50: the frequency values at which the MTF curve reaches 10%, 30%, and 50% of its normalized (DC) value respectively

  • MTF50P: the frequency value at which the MTF curve reaches 50 of its maximum value (which can be greater than the value of 1 found in DC if sharpening is present)

  • MTF at ¼ and ½ Nyquist: The MTF value at one half and one quarter the Nyquist sampling rate (0.125 and 0.25 cycles/pixel, respectively).

  • MTF Area: The area under the MTF curve from DC to 0.5 cycles/pixel, usually with the curve normalized to peak at a value of 1. (Less common.)

These are illustrated below on a synthetic, noise-free MTF curve example. The MTF Area value is the integral of the light red region under the curve.

The value of each of these metrics will change slightly for each different realization of noise (i.e., each photograph you take of the slanted edge), but some metric values tend to be less stable (have more variance) in the presence of noise. It is important to make sure you are using a metric that embodies the MTF characteristic you care about but is also repeatable considering the amount of noise you might expect to encounter.

Shown below is a set of 10 different MTF curves calculated (using Imatest’s SFR module) from a set of simulated slanted-edge images. Our simulation process involved generating a slanted edge at 5 degrees (bilinear interpolation), applying a Gaussian blur kernel, adding white gaussian pixel-wise noise (a different instance per curve below), and applying sharpening using an unsharp masking technique. Overlaid on the family of MTF curves are boxplots (handy plots that succinctly represent the important statistics of an entire population, in this case, 100 simulations using the above process) corresponding to the different summary metrics.

The most important aspect of these boxes for this discussion is the length of each of them, which represents how much variance each metric shows over the population of noisy edge images. (Lengths of vertically oriented metrics have been compensated for different axis scales in this image to allow visual comparison with horizontally plotted ones.) Note that MTF50 and MTF50P have smaller amounts of variance than the similarly common MTF30, MTF10. MTF at ½ and ¼ Nyquist vary in a different scale since they have different units than the previously mentioned metrics, with the latter being significantly more affected by noise. MTF Area has the least variance, though this is also on a different scale and has a different relationship to sharpness. (We will study the suitability of MTF Area, which is a very promising though not commonly used metric, in the future post.)

The plots below further show how the different MTF metrics vary at different levels of sharpening and noise. The standard deviation of each, σmetric, is calculated for each metric at each noise level over 100 random instances. The original simulated slanted edge test image was valued between [0, 255] and with 4:1 contrast.

The above figures display the expected general trends of increasing variability for all metrics, both with increasing noise and increasing levels of sharpening. Interestingly, the ordering of the metrics by variability is essentially constant across all levels of noise and sharpening. Another interesting note is that MTF10 and MTF at ½ Nyquist are especially sensitive to sharpening- their variability jumps the most when sharpening is applied. These two metrics are also generally the most variable overall, while MTF Area is the most consistent.

When choosing a metric to use to report the sharpness of an imaging system, it is important to keep in mind how susceptible reported values are to variation due to random noise. By using a more stable summary metric value, you can ensure repeatability of your results in future tests.