Automating CPIQ analysis Using Imatest IT and Python

The Python interface to Imatest IT provides a simple means of invoking Imatest’s tests. This post will show how Imatest runs can be automated, then the results of those tests can be collected and easily processed. For this example, we will run five Imatest modules across three light levels, then extract CPIQ quality loss metrics.

Development Setup

Image Collection

For each device, we capture the following set of images of different test charts at different light levels:   (Note: full details will be available with the publication of the standard in March of 2017)

  • DeviceX_eSFR_5000KLED_1000lux.JPG

  • DeviceX_eSFR_TL84_100lux.JPG

  • DeviceX_eSFR_Tung_10lux.JPG

  • DeviceX_Dot_5000KLED_1000lux.JPG

  • DeviceX_Dot_TL84_100lux.JPG

  • DeviceX_Dot_Tung_10lux.JPG

  • DeviceX_SG_5000KLED_1000lux.JPG

  • DeviceX_SG_TL84_100lux.JPG

  • DeviceX_SG_Tung_10lux.JPG

  • DeviceX_Coins_5000KLED_1000lux.JPG

  • DeviceX_Coins_TL84_100lux.JPG

  • DeviceX_Coins_Tung_10lux.JPG

  • DeviceX_Unif_100lux.JPG

Running All The Tests

Once we have these images collected in a folder named “DeviceX”, we define our lighting conditions, initialize the Imatest IT library and run our processing function on this and a number of other devices:

led5k = "5000KLED_1000lux"               # Light Levels tl84 = "TL84_100lux" tung = "Tung_10lux" light_conditions = [led5k, tl84, tung] imatest = ImatestLibrary() phones = ['DeviceX',                     # Just for example...           'DeviceY',           'DeviceZ'] for phone in phones:     calc_phone(phone)   imatest.terminate_library()

The main processing script invokes the five modules, collects their data outputs and saves them to a JSON file:

def calc_phone(phone):     print('Calculating Metrics for '+phone+ ' *****************************')     uniformity_results = uniformity(phone) # Uniformity     multitest_results  = multitest(phone) # Multitest     dotpattern_results = dotpattern(phone) # Dot Pattern     esfriso_results    = esfriso(phone) # eSFR ISO     random_results     = random(phone) # Random       # Output data to disk     file_name = os.path.join(images_dir, phone, phone + '.json')     with open(file_name, 'w') as outfile:         full_data = {'multitest'  : multitest_results,                      'dotpattern' : dotpattern_results,                      'esfriso' : esfriso_results,                      'random' : random_results,                      'uniformity' : uniformity_results}         json.dump(full_data, outfile, indent=4, separators=(',',': '))       return full_data

We call the Imatest eSFR ISO module for each specified lighting condition:

# eSFR ISO (ISO 12233:2014) is used to test Visual Noise (VN) # and Spatial Frequency Response (SFR) def esfriso(base):     global images_dir, ini_file, light_conditions, op_mode     output = {}     for light_source in light_conditions:         print('Testing ' + base + ' eSFR ISO at ' + light_source)         input_file = os.path.join(images_dir, base, base+'_eSFR_'+light_source+'.jpg')         if os.path.exists(input_file) != True:             raise Exception('File ' + input_file + 'not found!')         print('opening ' + input_file)         result = imatest.esfriso_json(input_file=input_file, root_dir=root_dir,                                       op_mode=op_mode, ini_file=ini_file)         data = json.loads(result)         output[light_source] = data['esfrisoResults'];   return output

On the fly INI file changes

The most difficult part of this processing is to manually update region selections for the calls to Multitest, which does not yet have automatic region detection for the Colorchecker SG target (coming soon…).

In this case, we are we replacing the [sg] roi, nwid_save, and nht_save keys with the values stored in this multicharts-rois.json file:

We use this script to modify INI files to dynamically insert different settings:

Imatest INIs are case sensitive. Make sure any INI manipulator will preserve the case of sections and keys.

These functions are then called using the values loaded from the JSON file inside of our multicharts function call:

Extracting Scores

Finally, we extract Key Performance Indicators (KPI), which are all in perceptual quality loss (QL) in units of just noticeable differences (JND).  This script is a bit long for this post, so view the source on Github if you are interested./

Here is the resulting set of calculated metrics in JSON format:

Once test procedures and combined score calculations are finalized by the IEEE Conformity Assessment Program, these outputs will be combined into a simple camera phone rating score.

The full python source code for this example is posted on GitHub.

To add these automation capabilities to your existing Imatest Master license, upgrade to Imatest Ultimate. You can also request a free 30-day trial of Imatest IT to see how this works for you before you purchase.