Color & density reference files in CSV or CGATS format

Open

Color reference files read by Imatest can be CSV or CGATS format containing L*a*b* color and luminance reference data.

The illuminant used to create L*a*b* reference files is usually D50. The most widely used color spaces (sRGB and Adobe RGB) have D65 white points. A Bradford Transform is applied to convert the D50 file data to D65 color spaces. It is applied to spectrophotometer measurements for individual charts.

Color Reference files

Color CSV reference files files consist of the number of lines that the test chart has patches, with L*, a*, b* values on each line separated by spaces, commas (,), or semicolons (;). Example (first 3 lines of 24):

38.08, 12.09, 14.39
66.38, 13.22, 17.14
51.06, 0.38, -22.06
…

If you have an Excel .CSV file with extra rows or columns, you can easily edit it Excel by selecting the key region (3 columns, 24 rows), copying it to a new file, and saving it in .CSV format. L*a*b* data is preferred to xyY data below because it is independent of white point color temperature, hence less error-prone. 

Row vs. Column major

Patch order: Reference files read by Imatest should be in row-major order. This means that the order for an m x n (m rows x n columns) chart is {R1C1, R1C2, …, R1Cn, R2C1, R2C2, …, R2Cn, …, RmC1, RmC1, …, RmCn}, i.e., patch data for each row is entered before proceeding to the next.

But some reference files available on the internet are in column-major order (R1C1, R2C1, …, RmCqn R1C2 R2C2, …, RnC2, …, R1Cn R2Cn, …, RmCn). They must be transformed into row-major order to be read correctly into Imatest. We show a method for doing this below.

Using measured values as a reference: Color reference files can also be derived from measured values from a camera you determine to be a “gold standard” reference by running Color/Tone Interactive, then pressing File, Save L*a*b* results as CSV reference (making sure to check CSV reference file of L*a*b* results…). This can be useful for manufacturing testing because the camera under test is compared with the attainable values of a golden sample rather than unattainable ideal values.

The CGATS file format is also supported.

Color/Tone Interactive General mxn charts (4×12 shown on the left). Any chart with an m x n grid can be analyzed. You must specify the grid size (click Settings, ISO speed & mxn chart settings to set the number of rows (m) and columns (n). You must also enter a CSV or CGATS file with reference patch settings (typically L*, a*, b* values, one set per line). If you have a spectrophotometer you can create a custom reference file (CGATS format) using techniques described in Measuring test chart patches.

Obtaining the color reference

In Color/Tone Interactive you can perform a number of operations that relate to the reference file using the File dropdown menu, shown on the right.

The operation of interest here is Copy L*a*b* reference to clipboard. This shows the currently selected reference, whether it’s a standard or custom file. You can also save it as a named file for future use as a reference— perhaps for a “gold standard”. Here are the first three values for the post-Nov 2014 X-Rite Colorchecker default values.

37.31 13.39 14.58
64.37 18.05 17.05
49.62 -1.162 -22.16
…

These are close, but not identical, to the values shown above, which were derived from a different source.

Density Reference Files

Density CSV reference files are much simpler, they contain one optical density value per line, going from least-dense (smallest) patch, up to the most dense patch.  

Imatest will provide you with a density reference file when you purchase a 36-patch dynamic range or contrast resolution chart. You should always use this reference file while performing dynamic range measurements with multicharts or color-tone instead of using the default values.

Note that Copy density reference to clipboard is available in the File dropdown menu on the right.

To load in your density reference file into Imatest:

1. Load your image(s) into Imatest

2. Start your analysis

3. Proceed through settings & ROI selection

4. In the interactive results window, select Density CSV file (one value per line) in the Ref dropdown, as shown in the image below. This will open a browser for you to select your reference file.

Color reference files, color temperature, and color spaces

The color reference for a test chart contains CIELAB (device-independent L*a*b*) values for each patch of the chart. It can be

• obtained from default values built into Imatest or

• read from a file, typically in CSV or CGATS format. The file can contain default values supplied by the chart manufacturer or values individually measured with a spectrophotometer.

Questions frequently arise regarding the color temperature used to (1) measure the patches for the reference file, (2) measure camera color accuracy from a chart image, and (sometimes forgotten), and (3) built into the color space used by the camera under test. (All interchangeable files have a color space associated with them, implicitly or explicitly.)

Patches are measured with a light source with a known color temperature, typically 5000K (D50). Color spaces have a illuminant white point. sRGB (the widely used internet Windows standard) has a 6500K (D65) illuminant white point and a D50 (~5000K) ambient (viewing environment)  white point. Wikipedia does little to resolve the confusion, but the 6500K illuminant white point is the chief property of the file. A Bradford transform is typically applied to convert the 5000K measurement to the 6500K color space illuminant white point.

A frequent question is, “Is a special reference file needed for measuring charts with illuminants different from the color space white point (typically 6500K)?”  The question is important because Imatest measurements (and indeed almost all camera operation) can be made with a wide variety of color temperatures, ranging from 2400K for incandescent lights to >10000K for some daylight conditions. The quick answer is “No”.

The standard reference should be used. The raw image is often transformed in the camera by an algorithm that estimates the illuminant color temperature, then applied a Color Correction Matrix (CCM) to the image to white balance and correct the colors. A CCM can also be applied to an uncorrected image by Imatest.

Converting a reference file from column-major to row-major

To convert reference files from column-major order (frequently found in reference files on the internet) to row-major (the order used by Imatest), download the file and open it in Excel. Reference files often include extra data. Here is an example from the Image Engineering TE262 Universal Test Target file, used for evaluating scanners. The color pattern in the large UTT chart is shown on the right. (There are two of these in some printed charts.)

Here is a portion of the reference file, opened in Excel. We want to extract cells B7 through D33 (m = 9 rows, n = 3 columns in column-major order). In this case, we are using the default (not individually measured) data, shown with the blue background.

To obtain the data-only, without the extra annotation, select the required cells (B7:D33), and copy them to the clipboard. Then paste into a new blank worksheet. The first 10 lines (of 27) will look like

Now comes the tricky part. Add numbers to the fourth column indicating how you’d like to sort the data to convert to column to row-major. Line 1 doesn’t change, so add 1. Line 2 will go into Row 2, Column 1, which is line 4 in row-major. Line 3 will go into Row 3, column 1, which is line 7 in row-major. Continue in increments of n = 3 for the 9 rows. Then line 10 will go into Row1, Column 2, etc. The first 14 rows (of 27) will be

Because of the simple repetition in column 4 (mostly increments of 3 columns), it’s a lot easier than copying and pasting lines. More generally, for an m row x n column image, column 4 should contain {1, k+1, 2k+1, …, (m-1)n+1, 2, k+2, 2k+2, (m-1)n+2, …, n, k+n, 2k+n, …, (m-1)n+n = mn}.

Now Select the full 27 lines and 4 columns, right-click, and press Sort > Custom sort. This opens the Sort window. Choose Sort by Column D, then press OK. This sorts the data so that column D is in sequence, 1, 2, …, 27.

Now select columns 1-3 (A1:C27), copy them, paste into a blank worksheet, and save as a csv or txt file. You have a column-major reference field. . Here is an example from Color/Tone Interactive.

See Also

• Color/Tone Interactive – Reference File 

• Colorcheck Reference File (legacy module: Color/Tone is recommended for new work)

• Measuring Test Chart Patches with a Spectrophotometer for details about how to create a reference file using measurements from an instrument.