Introduction

In this guide we’ll walk through the steps involved in testing a VC-2 implementation for conformance to SMPTE ST 2042-family of specifications.

Testing procedure overview

The conformance testing processes for decoders and encoders is outlined below.

Decoders are tested using a series of test bitstreams (generated using vc2-test-case-generator). The pictures produced by the decoder are then compared against reference decodings. If the decoded pictures are bit-for-bit identical (as checked by vc2-picture-compare), and the decoder did not crash, the decoder passes the test.

Encoders are tested using a series of test pictures (generated using vc2-test-case-generator). The encoded bitstreams are then fed to a bitstream validator (vc2-bitstream-validator) which simultaneously validates the bitstream against the specification and decodes the encoded pictures. If the bitstream is free from technical errors, the decoded pictures are then compared with the input pictures (both visually and using vc2-picture-compare). If the decoded pictures are sufficiently similar to the inputs, the encoder passes the test.

Guide outline

In Conformance Software Installation we will walk through the process of installing the VC-2 conformance software on your computer.

In Video file format the simple planar raw video file format used by the conformance software is introduced. You are responsible for converting between this format and the format natively accepted by the codec under test.

In Generating test cases we will use the vc2-test-case-generator tool to generate a set of test pictures and bitstreams. Because VC-2 supports such a wide variety of video formats and coding behaviours, test cases are generated on demand to suit the particular features of a codec under test.

In VC-2 decoder conformance testing procedure and VC-2 encoder conformance testing procedure we describe how the test pictures and bitstreams should be processed by the codec under test. We also describe the procedures for verifying that the codecs behaved as expected. Additionally, in Testing additional bitstreams’ conformance we explain how bitstreams produced outside of the conformance testing procedures can also be tested for conformance.

In Codec debugging suggestions we provide some advice on how to approach the problem of debugging failing tests.

Finally, in Conformance test limtations some of the limitations of the conformance test cases and procedures are enumerated.

So, lets move on to Conformance Software Installation…