Automating CPIQ analysis Using Imatest IT and Python
- Henry Koren
- Lexi Irwin
The Python interface to Imatest IT provides a simple means of invoking Imatest’s tests. This post will show how Imatest runs can be automated, then the results of those tests can be collected and easily processed. For this example, we will run five Imatest modules across three light levels, then extract CPIQ quality loss metrics.
Development Setup
Windows 10 (or Ubuntu Linux)
Python 2.7 (Python 3.3 and 3.4 also supported)
Image Collection
For each device, we capture the following set of images of different test charts at different light levels: (Note: full details will be available with the publication of the standard in March of 2017)
DeviceX_eSFR_5000KLED_1000lux.JPG
DeviceX_eSFR_TL84_100lux.JPG
DeviceX_eSFR_Tung_10lux.JPG
DeviceX_Dot_5000KLED_1000lux.JPG
DeviceX_Dot_TL84_100lux.JPG
DeviceX_Dot_Tung_10lux.JPG
DeviceX_SG_5000KLED_1000lux.JPG
DeviceX_SG_TL84_100lux.JPG
DeviceX_SG_Tung_10lux.JPG
DeviceX_Coins_5000KLED_1000lux.JPG
DeviceX_Coins_TL84_100lux.JPG
DeviceX_Coins_Tung_10lux.JPG
DeviceX_Unif_100lux.JPG
Running All The Tests
Once we have these images collected in a folder named “DeviceX”, we define our lighting conditions, initialize the Imatest IT library and run our processing function on this and a number of other devices:
led5k = "5000KLED_1000lux" # Light Levels
tl84 = "TL84_100lux"
tung = "Tung_10lux"
light_conditions = [led5k, tl84, tung]
imatest = ImatestLibrary()
phones = ['DeviceX', # Just for example...
'DeviceY',
'DeviceZ']
for phone in phones:
calc_phone(phone)
imatest.terminate_library() |
The main processing script invokes the five modules, collects their data outputs and saves them to a JSON file:
def calc_phone(phone):
print('Calculating Metrics for '+phone+ ' *****************************')
uniformity_results = uniformity(phone) # Uniformity
multitest_results = multitest(phone) # Multitest
dotpattern_results = dotpattern(phone) # Dot Pattern
esfriso_results = esfriso(phone) # eSFR ISO
random_results = random(phone) # Random
# Output data to disk
file_name = os.path.join(images_dir, phone, phone + '.json')
with open(file_name, 'w') as outfile:
full_data = {'multitest' : multitest_results,
'dotpattern' : dotpattern_results,
'esfriso' : esfriso_results,
'random' : random_results,
'uniformity' : uniformity_results}
json.dump(full_data, outfile, indent=4, separators=(',',': '))
return full_data |
We call the Imatest eSFR ISO module for each specified lighting condition:
# eSFR ISO (ISO 12233:2014) is used to test Visual Noise (VN)
# and Spatial Frequency Response (SFR)
def esfriso(base):
global images_dir, ini_file, light_conditions, op_mode
output = {}
for light_source in light_conditions:
print('Testing ' + base + ' eSFR ISO at ' + light_source)
input_file = os.path.join(images_dir, base, base+'_eSFR_'+light_source+'.jpg')
if os.path.exists(input_file) != True:
raise Exception('File ' + input_file + 'not found!')
print('opening ' + input_file)
result = imatest.esfriso_json(input_file=input_file, root_dir=root_dir,
op_mode=op_mode, ini_file=ini_file)
data = json.loads(result)
output[light_source] = data['esfrisoResults'];
return output |
On the fly INI file changes
The most difficult part of this processing is to manually update region selections for the calls to Multitest, which does not yet have automatic region detection for the Colorchecker SG target (coming soon…).
In this case, we are we replacing the [sg] roi, nwid_save, and nht_save keys with the values stored in this multicharts-rois.json file:
We use this script to modify INI files to dynamically insert different settings:
Imatest INIs are case sensitive. Make sure any INI manipulator will preserve the case of sections and keys.
These functions are then called using the values loaded from the JSON file inside of our multicharts function call:
Extracting Scores
Finally, we extract Key Performance Indicators (KPI), which are all in perceptual quality loss (QL) in units of just noticeable differences (JND). This script is a bit long for this post, so view the source on Github if you are interested./
Here is the resulting set of calculated metrics in JSON format:
Once test procedures and combined score calculations are finalized by the IEEE Conformity Assessment Program, these outputs will be combined into a simple camera phone rating score.
The full python source code for this example is posted on GitHub.
To add these automation capabilities to your existing Imatest Master license, upgrade to Imatest Ultimate. You can also request a free 30-day trial of Imatest IT to see how this works for you before you purchase.