Generating test cases

Test cases for VC-2 encoders and decoders are generated by the vc2-test-case-generator program. Below we’ll walk through the process of defining the capabilities of our codec in a ‘codec features’ file and then generating the test cases accordingly.

Defining codec features

In order to generate an appropriate set of test cases for a VC-2 encoder or decoder we must first enumerate the capabilities of that codec in a codec features file. This file must contain a table, in Comma Separated Values (CSV) format enumerating codec configurations to be tested. An example table is given below:

name

hd_lossy

hd_fragmented

hd_lossless

custom_8_bit_rgb

# (11.2.1)

level

unconstrained

unconstrained

unconstrained

unconstrained

profile

high_quality

high_quality

high_quality

high_quality

# (11.1)

base_video_format

hd1080p_50

hd1080p_50

hd1080p_50

hd1080p_50

picture_coding_mode

pictures_are_frames

pictures_are_frames

pictures_are_frames

pictures_are_frames

# (11.4.3)

frame_width

default

default

default

640

frame_height

default

default

default

480

# (11.4.4)

color_diff_format_index

default

default

default

color_4_4_4

# (11.4.5)

source_sampling

default

default

default

progressive

top_field_first

default

default

default

TRUE

# (11.4.6)

frame_rate_numer

default

default

default

25

frame_rate_denom

default

default

default

1

# (11.4.7)

pixel_aspect_ratio_numer

default

default

default

1

pixel_aspect_ratio_denom

default

default

default

1

# (11.4.8)

clean_width

default

default

default

640

clean_height

default

default

default

480

left_offset

default

default

default

0

top_offset

default

default

default

0

# (11.4.9)

luma_offset

default

default

default

0

luma_excursion

default

default

default

255

color_diff_offset

default

default

default

0

color_diff_excursion

default

default

default

255

# (11.4.10)

color_primaries_index

default

default

default

hdtv

color_matrix_index

default

default

default

rgb

transfer_function_index

default

default

default

tv_gamma

# (12.4.1) and (12.4.4.1)

wavelet_index

le_gall_5_3

le_gall_5_3

le_gall_5_3

le_gall_5_3

wavelet_index_ho

le_gall_5_3

le_gall_5_3

le_gall_5_3

le_gall_5_3

dwt_depth

2

2

2

2

dwt_depth_ho

0

0

0

0

# (12.4.5.2)

slices_x

240

240

240

80

slices_y

135

135

135

60

# Slice size, etc.

lossless

FALSE

FALSE

TRUE

FALSE

picture_bytes

1296000

1296000

230400

fragment_slice_count

0

8

0

0

# (12.4.5.3)

quantization_matrix

default

default

default

4 2 2 0 4 4 2

Download CSV

The first row should provide a unique name for each codec configuration for which test cases are to be generated with the left-most cell containing the text name. The remaining rows specify the parameters which define the codec configurations. Empty rows and rows whose first column starts with a # are ignored (i.e. treated as comments).

The following parameters must be given for each codec configuration.

level

Integer or alias. The VC-2 level number (see annex (C.3)) to report in the bit stream. If not 0 (the ‘unconstrained’ level), the remaining parameters below must be set to values compatible with this level, otherwise the test case generator will reject the codec configuration.

The following aliases can be used in place of an integer for additional readability.

Level

Alias

0

 unconstrained

1

 sub_sd

2

 sd

3

 hd

4

 d_cinema_2k

5

 d_cinema_4k

6

 uhdtv_4k

7

 uhdtv_8k

64

 progressive_hd_over_sdi

65

 hd_over_sd_sdi

66

 uhd_over_hd_sdi
profile

Integer or alias. The VC-2 profile number (see annex (C.2)). 0 for ‘low-delay’ or 3 for ‘high quality’.

The following aliases can be used in place of an integer for additional readability.

Profile

Alias

0

 low_delay

3

 high_quality
picture_coding_mode

Integer or alias. The picture coding mode to use (see section (11.5)).

The following aliases can be used in place of an integer for additional readability.

Picture Coding Mode

Alias

0

 pictures_are_frames

1

 pictures_are_fields
base_video_format

Integer or alias. The base video format index to use (see section (11.3) and annex (B)).

The following aliases can be used in place of an integer for additional readability.

Base Video Format

Alias

0

 custom_format

1

 qsif525

2

 qcif

3

 sif525

4

 cif

5

 foursif525

6

 foursif

7

 sd_480i_60

8

 sd576i_50

9

 hd720p_60

10

 hd720p_50

11

 hd1080i_60

12

 hd1080i_50

13

 hd1080p_60

14

 hd1080p_50

15

 dc2k

16

 dc4k

17

 uhdtv4k_60

18

 uhdtv4k_50

19

 uhdtv8k_60

20

 uhdtv8k_50

21

 hd1080p_24

22

 sd_pro486

Note

This parameter is provided as a convenient way of setting the video parameters below to common sets of options. This parameter, and the options below, do not directly define how the video format is encoded in bitstreams – this will be determined automatically by the test case generator. In fact, the test case generator will produce test cases with several different encodings when possible.

frame_width and frame_height

Integer or default. The dimensions of frames (not pictures) of video (see section (11.4.3)). If default, uses the dimensions specified by the base_video_format.

color_diff_format_index

Integer, alias or default. The color difference subsampling mode to use (see section (11.4.4)). If default, uses the mode specified by the base_video_format.

The following aliases can be used in place of an integer for additional readability.

Index

Alias

0

 color_4_4_4

1

 color_4_2_2

2

 color_4_2_0
source_sampling

Integer, alias or default. The scan format to use (see section (11.4.5)). If default, uses the mode specified by the base_video_format.

The following aliases can be used in place of an integer for additional readability.

Index

Alias

0

 progressive

1

 interlaced

Note

This parameter is used as metadata only. It should not be confused with the picture_coding_mode parameter which determines whether each picture in a sequence contains a whole frame or a field of video.

top_field_first

TRUE, FALSE or default. Indicates, for interlaced formats, whether the earlier field in a sequence contains the top field of a frame (TRUE) or bottom field (FALSE) (see section (11.4.5)). If default, uses the mode specified by the base_video_format.

frame_rate_numer and frame_rate_denom

Integers or default. The frame rate (see section (11.4.6)). If default, uses the mode specified by the base_video_format.

pixel_aspect_ratio_numer and pixel_aspect_ratio_denom

Integers or default. The pixel aspect ratio (see section (11.4.7)). If default, uses the mode specified by the base_video_format.

clean_width, clean_height, left_offset and top_offset

Integers or default. The clean area (see section (11.4.8)). If default, uses the mode specified by the base_video_format.

luma_offset, luma_excursion, color_diff_offset and color_diff_excursion

Integers or default. The luma and color difference picture component signal ranges (see section (11.4.9)). If default, uses the mode specified by the base_video_format.

color_primaries_index, color_matrix_index and transfer_function_index

Integers, aliases or default. color specification options (see section (11.4.10)). If default, uses the mode specified by the base_video_format.

The following aliases can be used in place of an integer for additional readability.

Color Primaries Index

Alias

0

 hdtv

1

 sdtv_525

2

 sdtv_625

3

 d_cinema

4

 uhdtv

Color Matrix Index

Alias

0

 hdtv

1

 sdtv

2

 reversible

3

 rgb

4

 uhdtv

Transfer Function Index

Alias

0

 tv_gamma

1

 extended_gamut

2

 linear

3

 d_cinema

4

 perceptual_quantizer

5

 hybrid_log_gamma
wavelet_index and wavelet_index_ho

Integers or aliases. Wavelet transform types to use vertically and horizontally, respectively (see sections (11.4.1) and (11.4.4.1)). For symmetric transforms, these values must be the same.

The following aliases can be used in place of an integer for additional readability.

Index

Alias

0

 deslauriers_dubuc_9_7

1

 le_gall_5_3

2

 deslauriers_dubuc_13_7

3

 haar_no_shift

4

 haar_with_shift

5

 fidelity

6

 daubechies_9_7
dwt_depth and dwt_depth_ho

Integers. Wavelet transform depths to use for both dimensions and horizontally only, respectively (see sections (11.4.1) and (11.4.4.1)). For symmetric transforms, dwt_depth_ho must be 0.

slices_x and slices_y

Integers. The number of horizontal and vertical picture slices to use (see section (12.4.5.2)).

lossless

Boolean. If FALSE, test cases will be generated for a constant bit rate (lossy) codec. If TRUE test cases will be generated for a lossless (variable bit rate) codec. Lossless mode is only supported by the high quality profile.

picture_bytes

Integer or blank. The number of bytes to use to encode the slices in each picture. Must be an integer when lossless is FALSE and blank when lossless is TRUE.

For the low delay profile, this sets the slice_bytes_numerator and slice_bytes_denominator values used by the stream (see section (13.5.3.2)) to the value picture_bytes divided by the number of slices per picture.

For the high quality profile, when lossless is FALSE, slices are assigned sizes using the same formula as used for the low delay profile. When lossless is TRUE, slices are sized as small as possible for the data they hold.

Note

This value only accounts for picture slice data, i.e. the data read by the slice pseudocode function in section (13.5.3). It does not take into account other stream overheads (e.g. sequence headers and transform parameters). As such the resulting stream will have a slightly higher bit rate than picture_bytes bytes per picture.

fragment_slice_count

Integer.

If zero, non-fragmented picture coding is used: each picture will be coded as a single picture parse data unit.

If greater than zero, fragmented picture mode will be used (see section (14)). Pictures will be coded as several fragment parse data units containing at most fragment_slice_count slices each.

quantization_matrix

List of space-separated integers or default. Specifies the quantization matrix to be used.

If default, the default quantisation matrix for the wavelet transform specified by wavelet_index, wavelet_index_ho, dwt_depth and dwt_depth_ho will be used (see annexe (D.2)).

If a list of space separated integers are provided defining a quantisation matrix, these will be used instead and encoded as a custom quantisation matrix in the stream (see (12.4.5.3)).

Quantisation matrix values, if provided, should be given in the same order they would appear in the stream as defined by the quant_matrix pseudocode function (12.4.5.3). For example for a transform with dwt_depth = 1 and dwt_depth_ho = 2, the following value:

0 1 2 3 4 5

Describes the following quantization matrix:

{
    0: {"L": 0},
    1: {"H": 1},
    2: {"H": 2},
    3: {"HL": 3, "LH": 4, "HH": 5},
}

If a non default value is given, the majority (though not all) generated test cases will use the supplied quantization matrix (with the custom_quant_matrix flag set (12.4.5.3)).

Generating test cases

Once a codec features CSV has been created, with columns covering the major operating modes of the codec to be tested, the vc2-test-case-generator command can be used to generate test cases.

In the simplest case, the command should be provided with the filename of your codec features CSV:

$ vc2-test-case-generator path/to/codec_features.csv

By default, a test_cases directory will be created into which the test cases are written. This can be changed using the --output <path> argument. The --verbose option can be used to keep track of progress.

If only test cases for an encoder are required, the --encoder-only option can be given. Alternatively if only decoder test cases are needed --decoder-only can be used. By default, test cases are generated for both encoders and decoders.

Before any test cases are generated, the test case generator internally generates and then validates a simple test stream for each column of the codec features table. This step ensures that the codec features specified are not in conflict with themselves or the VC-2 standard. If this step fails, an error message is produced indicating the problem and test case generation is aborted.

If you are using a wavelet transform combination or depth for which a default quantization matrices are not provided in the VC-2 specification (see annexe (D.2)), the test case generator will produce the following warning:

WARNING:root:No static analysis available for the wavelet used by codec '<name>'. Signal range test cases cannot be generated.

See Generating static wavelet filter analyses for instructions on this specific case.

Warning messages are otherwise only produced for degenerate codec configurations. It is very unlikely a useful codec configuration will result in a warning. If any are produced, check the values in your codec features CSV if warnings are encountered.

Test case generation typically requires several hours, depending on the codec feature sets provided.

Note

The slow runtime performance of the VC-2 conformance software is an unfortunate side effect of it being based on the pseudocode published in the VC-2 specification. This design gives a high degree of confidence that it is consistent with the specification at the cost of slow execution.

Parallel test case generation

To speed up test case generation on multi-core systems, independent test cases can be generated in parallel. To do this, the --parallel argument is used. Instead of generating test cases, when --parallel is used, the test case generator will print a series of commands which can be executed in parallel to generate the test cases, for example using GNU Parallel:

$ # Write test case generation commands to 'commands.txt'
$ vc2-test-case-generator path/to/codec_features.csv --parallel > commands.txt

$ # Run test case generation in parallel using GNU Parallel
$ parallel -a commands.txt

Warning

Some test cases require relatively large quantities of RAM during test case generation. You might need to reduce the number of commands run in parallel if your system runs out of memory. If you’re using GNU parallel, the -j N argument can be used to set the number of parallel jobs to N (with the default being however many CPU cores are available).

Directory structure

The test case generator produces a directory structure as outlined below:

  • test_cases/
    • <codec feature set name>/
      • decoder/ – Test VC-2 bitstreams for decoders.

        • <test-case-name>.vc2 – VC-2 bitstream to be decoded.

        • <test-case-name>_metadata.json – Optional metadata file provided for some tests

        • <test-case-name>_expected/ – Reference decoding of the bitstream.

          • picture_<N>.raw

          • picture_<N>.json

      • encoder/ – Test raw video streams for encoders.

        • <test-case-name>_metadata.json – Optional metadata file provided for some tests

        • <test-case-name>/ – Raw video to be encoded

          • picture_<N>.raw

          • picture_<N>.json

The testing procedures for decoders and encoders are described in the next two sections: