.. _guide-generating-test-cases: Generating test cases ===================== Test cases for VC-2 encoders and decoders are generated by the :ref:`vc2-test-case-generator` program. Below we'll walk through the process of defining the capabilities of our codec in a 'codec features' file and then generating the test cases accordingly. .. _codec-features: Defining codec features ----------------------- In order to generate an appropriate set of test cases for a VC-2 encoder or decoder we must first enumerate the capabilities of that codec in a codec features file. This file must contain a table, in Comma Separated Values (CSV) format enumerating codec configurations to be tested. An example table is given below: .. csv-table:: :file: ../_static/user_guide/sample_codec_features.csv .. only:: html `Download CSV <../_static/user_guide/sample_codec_features.csv>`_ .. only:: latex A CSV version of this table can be `downloaded here `_. The first row should provide a unique name for each codec configuration for which test cases are to be generated with the left-most cell containing the text ``name``. The remaining rows specify the parameters which define the codec configurations. Empty rows and rows whose first column starts with a ``#`` are ignored (i.e. treated as comments). The following parameters must be given for each codec configuration. ``level`` Integer or alias. The VC-2 level number (see annex (C.3)) to report in the bit stream. If not ``0`` (the 'unconstrained' level), the remaining parameters below must be set to values compatible with this level, otherwise the test case generator will reject the codec configuration. The following aliases can be used in place of an integer for additional readability. .. enum-value-table:: vc2_data_tables.Levels :value-heading: Level :name-heading: Alias ``profile`` Integer or alias. The VC-2 profile number (see annex (C.2)). ``0`` for 'low-delay' or ``3`` for 'high quality'. The following aliases can be used in place of an integer for additional readability. .. enum-value-table:: vc2_data_tables.Profiles :value-heading: Profile :name-heading: Alias ``picture_coding_mode`` Integer or alias. The picture coding mode to use (see section (11.5)). The following aliases can be used in place of an integer for additional readability. .. enum-value-table:: vc2_data_tables.PictureCodingModes :value-heading: Picture Coding Mode :name-heading: Alias ``base_video_format`` Integer or alias. The base video format index to use (see section (11.3) and annex (B)). The following aliases can be used in place of an integer for additional readability. .. enum-value-table:: vc2_data_tables.BaseVideoFormats :value-heading: Base Video Format :name-heading: Alias .. note:: This parameter is provided as a convenient way of setting the video parameters below to common sets of options. This parameter, and the options below, do not directly define *how* the video format is encoded in bitstreams -- this will be determined automatically by the test case generator. In fact, the test case generator will produce test cases with several different encodings when possible. ``frame_width`` and ``frame_height`` Integer or ``default``. The dimensions of frames (not pictures) of video (see section (11.4.3)). If ``default``, uses the dimensions specified by the ``base_video_format``. ``color_diff_format_index`` Integer, alias or ``default``. The color difference subsampling mode to use (see section (11.4.4)). If ``default``, uses the mode specified by the ``base_video_format``. The following aliases can be used in place of an integer for additional readability. .. enum-value-table:: vc2_data_tables.ColorDifferenceSamplingFormats :value-heading: Index :name-heading: Alias ``source_sampling`` Integer, alias or ``default``. The scan format to use (see section (11.4.5)). If ``default``, uses the mode specified by the ``base_video_format``. The following aliases can be used in place of an integer for additional readability. .. enum-value-table:: vc2_data_tables.SourceSamplingModes :value-heading: Index :name-heading: Alias .. note:: This parameter is used as metadata only. It should not be confused with the ``picture_coding_mode`` parameter which determines whether each picture in a sequence contains a whole frame or a field of video. ``top_field_first`` ``TRUE``, ``FALSE`` or ``default``. Indicates, for interlaced formats, whether the earlier field in a sequence contains the top field of a frame (``TRUE``) or bottom field (``FALSE``) (see section (11.4.5)). If ``default``, uses the mode specified by the ``base_video_format``. ``frame_rate_numer`` and ``frame_rate_denom`` Integers or ``default``. The frame rate (see section (11.4.6)). If ``default``, uses the mode specified by the ``base_video_format``. ``pixel_aspect_ratio_numer`` and ``pixel_aspect_ratio_denom`` Integers or ``default``. The pixel aspect ratio (see section (11.4.7)). If ``default``, uses the mode specified by the ``base_video_format``. ``clean_width``, ``clean_height``, ``left_offset`` and ``top_offset`` Integers or ``default``. The clean area (see section (11.4.8)). If ``default``, uses the mode specified by the ``base_video_format``. ``luma_offset``, ``luma_excursion``, ``color_diff_offset`` and ``color_diff_excursion`` Integers or ``default``. The luma and color difference picture component signal ranges (see section (11.4.9)). If ``default``, uses the mode specified by the ``base_video_format``. ``color_primaries_index``, ``color_matrix_index`` and ``transfer_function_index`` Integers, aliases or ``default``. color specification options (see section (11.4.10)). If ``default``, uses the mode specified by the ``base_video_format``. The following aliases can be used in place of an integer for additional readability. .. enum-value-table:: vc2_data_tables.PresetColorPrimaries :value-heading: Color Primaries Index :name-heading: Alias .. enum-value-table:: vc2_data_tables.PresetColorMatrices :value-heading: Color Matrix Index :name-heading: Alias .. enum-value-table:: vc2_data_tables.PresetTransferFunctions :value-heading: Transfer Function Index :name-heading: Alias ``wavelet_index`` and ``wavelet_index_ho`` Integers or aliases. Wavelet transform types to use vertically and horizontally, respectively (see sections (11.4.1) and (11.4.4.1)). For symmetric transforms, these values must be the same. The following aliases can be used in place of an integer for additional readability. .. enum-value-table:: vc2_data_tables.WaveletFilters :value-heading: Index :name-heading: Alias ``dwt_depth`` and ``dwt_depth_ho`` Integers. Wavelet transform depths to use for both dimensions and horizontally only, respectively (see sections (11.4.1) and (11.4.4.1)). For symmetric transforms, ``dwt_depth_ho`` must be ``0``. ``slices_x`` and ``slices_y`` Integers. The number of horizontal and vertical picture slices to use (see section (12.4.5.2)). ``lossless`` Boolean. If ``FALSE``, test cases will be generated for a constant bit rate (lossy) codec. If ``TRUE`` test cases will be generated for a lossless (variable bit rate) codec. Lossless mode is only supported by the high quality profile. ``picture_bytes`` Integer or blank. The number of bytes to use to encode the slices in each picture. Must be an integer when ``lossless`` is ``FALSE`` and blank when ``lossless`` is ``TRUE``. For the low delay profile, this sets the ``slice_bytes_numerator`` and ``slice_bytes_denominator`` values used by the stream (see section (13.5.3.2)) to the value ``picture_bytes`` divided by the number of slices per picture. For the high quality profile, when ``lossless`` is ``FALSE``, slices are assigned sizes using the same formula as used for the low delay profile. When ``lossless`` is ``TRUE``, slices are sized as small as possible for the data they hold. .. note:: This value only accounts for picture slice data, i.e. the data read by the ``slice`` pseudocode function in section (13.5.3). It does not take into account other stream overheads (e.g. sequence headers and transform parameters). As such the resulting stream will have a slightly higher bit rate than ``picture_bytes`` bytes per picture. ``fragment_slice_count`` Integer. If zero, non-fragmented picture coding is used: each picture will be coded as a single picture parse data unit. If greater than zero, fragmented picture mode will be used (see section (14)). Pictures will be coded as several fragment parse data units containing at most ``fragment_slice_count`` slices each. ``quantization_matrix`` List of space-separated integers or ``default``. Specifies the quantization matrix to be used. If ``default``, the default quantisation matrix for the wavelet transform specified by ``wavelet_index``, ``wavelet_index_ho``, ``dwt_depth`` and ``dwt_depth_ho`` will be used (see annexe (D.2)). If a list of space separated integers are provided defining a quantisation matrix, these will be used instead and encoded as a custom quantisation matrix in the stream (see (12.4.5.3)). Quantisation matrix values, if provided, should be given in the same order they would appear in the stream as defined by the ``quant_matrix`` pseudocode function (12.4.5.3). For example for a transform with dwt_depth = 1 and dwt_depth_ho = 2, the following value:: 0 1 2 3 4 5 Describes the following quantization matrix:: { 0: {"L": 0}, 1: {"H": 1}, 2: {"H": 2}, 3: {"HL": 3, "LH": 4, "HH": 5}, } If a non ``default`` value is given, the majority (though not all) generated test cases will use the supplied quantization matrix (with the ``custom_quant_matrix`` flag set (12.4.5.3)). Generating test cases --------------------- Once a codec features CSV has been created, with columns covering the major operating modes of the codec to be tested, the :ref:`vc2-test-case-generator` command can be used to generate test cases. In the simplest case, the command should be provided with the filename of your codec features CSV:: $ vc2-test-case-generator path/to/codec_features.csv By default, a ``test_cases`` directory will be created into which the test cases are written. This can be changed using the ``--output `` argument. The ``--verbose`` option can be used to keep track of progress. If only test cases for an encoder are required, the ``--encoder-only`` option can be given. Alternatively if only decoder test cases are needed ``--decoder-only`` can be used. By default, test cases are generated for both encoders and decoders. Before any test cases are generated, the test case generator internally generates and then validates a simple test stream for each column of the codec features table. This step ensures that the codec features specified are not in conflict with themselves or the VC-2 standard. If this step fails, an error message is produced indicating the problem and test case generation is aborted. If you are using a wavelet transform combination or depth for which a default quantization matrices are not provided in the VC-2 specification (see annexe (D.2)), the test case generator will produce the following warning:: WARNING:root:No static analysis available for the wavelet used by codec ''. Signal range test cases cannot be generated. See :ref:`generating-static-analyses` for instructions on this specific case. Warning messages are otherwise only produced for degenerate codec configurations. It is very unlikely a useful codec configuration will result in a warning. If any are produced, check the values in your codec features CSV if warnings are encountered. Test case generation typically requires several hours, depending on the codec feature sets provided. .. note:: The slow runtime performance of the VC-2 conformance software is an unfortunate side effect of it being based on the pseudocode published in the VC-2 specification. This design gives a high degree of confidence that it is consistent with the specification at the cost of slow execution. Parallel test case generation ----------------------------- To speed up test case generation on multi-core systems, independent test cases can be generated in parallel. To do this, the ``--parallel`` argument is used. Instead of generating test cases, when ``--parallel`` is used, the test case generator will print a series of commands which can be executed in parallel to generate the test cases, for example using `GNU Parallel `_:: $ # Write test case generation commands to 'commands.txt' $ vc2-test-case-generator path/to/codec_features.csv --parallel > commands.txt $ # Run test case generation in parallel using GNU Parallel $ parallel -a commands.txt .. warning:: Some test cases require relatively large quantities of RAM during test case generation. You might need to reduce the number of commands run in parallel if your system runs out of memory. If you're using GNU parallel, the ``-j N`` argument can be used to set the number of parallel jobs to ``N`` (with the default being however many CPU cores are available). Directory structure ------------------- The test case generator produces a directory structure as outlined below: * ``test_cases/`` * ``/`` * ``decoder/`` -- Test VC-2 bitstreams for decoders. * ``.vc2`` -- VC-2 bitstream to be decoded. * ``_metadata.json`` -- Optional metadata file provided for some tests * ``_expected/`` -- Reference decoding of the bitstream. * ``picture_.raw`` * ``picture_.json`` * ``encoder/`` -- Test raw video streams for encoders. * ``_metadata.json`` -- Optional metadata file provided for some tests * ``/`` -- Raw video to be encoded * ``picture_.raw`` * ``picture_.json`` The testing procedures for decoders and encoders are described in the next two sections: * :ref:`guide-decoder-testing` * :ref:`guide-encoder-testing`