One of the main issues I encountered when developing Showout is the need for intra-frame encodings, seen in the various HAP formats as well as Resolume’s DXV format.
Intra-frame formats are preferred for VJ applications because frames can be displayed without having to decode the surrounding frames, this improves scrubbing performance.
Initially I started off with an in-app HAP Q encoder, this was relatively simple to implement due to the fact that FFmpeg contains one. I felt as though to be competitive in the current market it would be best for me to work on implementing the latest 8-bit HAP version, HAP R.
The workflow I started using was transcoding clips in TouchDesigner since it was one of the few options for exporting in HAP R, and even then, the export was limited to 1280x1280 for the non-commercial version. The external workflow is cumbersome when preparing show-ready clips because another program is needed, it is a new tool requiring learning, and TouchDesigner does not support Linux.
Although FFmpeg does not include a HAP R encoder, Richard Geldreich has implemented a handful of BC7 encoders.
The first encoder I tried was bc7enc and on my sample clip the complete preparation took 205.10 seconds, I knew there were improvements that could be made so I tried several encoders and configurations:
- bc7enc
- bc7e_scalar: veryfast, basic, slow
- bc7f: partial, non-analytical
- ISPC bc7e: ultrafast, veryfast, fast, basic, slow, veryslow
The benchmark results are below.
| Clip: worst-frame luma PSNR (dB) / encoding time (ms/frame) | bc7enc | bc7e_scalar veryfast | bc7e_scalar basic | bc7e_scalar slow | bc7f partial | bc7f non-analytical |
|---|---|---|---|---|---|---|
| archimedes | 47.77 / 57 | 47.75 / 83 | 48.09 / 126 | 49.85 / 296 | 45.28 / 7.0 | 45.47 / 25 |
| worm parade | 44.91 / 66 | 44.74 / 88 | 45.05 / 135 | 47.71 / 339 | 43.51 / 9.6 | 43.79 / 26 |
| membrane | 42.82 / 67 | 42.53 / 89 | 42.93 / 133 | 44.77 / 337 | 40.20 / 10.3 | 40.32 / 26 |
| firegrid | 44.17 / 53 | 43.90 / 81 | 44.23 / 117 | 47.23 / 276 | 42.13 / 5.6 | 42.41 / 25 |
| meta_build | 47.57 / 72 | 47.62 / 94 | 48.00 / 136 | 48.91 / 355 | 44.13 / 9.3 | 44.35 / 26 |
During testing, it was found that bc7f was the fastest encoder. I was initially pleased with this result until comparing the RGB PSNR and luma PSNR, where I found the luma quality to have degraded in a meaningful way I felt was not suitable for Showout.
The one I finally decided on was bc7e at its perceptual Basic level, which utilizes SIMD and is built for four SIMD instruction sets: SSE2, SSE4, AVX, and AVX2. This requires ISPC to compile though which required an additional build step.
The Basic setting seemed to be the right compromise of speed while retaining or improving luma quality.
HAP R frames store an alpha value for every pixel and one issue I encountered with the bc7e encoder (as well as bc7e_scalar) was that some pixels of fully opaque frames were getting decoded with an alpha value of 254 instead of 255, making them not fully opaque. This issue was a result of BC7’s mode 6 sharing the p-bit between color and alpha and alpha could only decode to 255 when that bit is 1. This issue has been raised here and has been open since May 2021.
The p-bit was forced to 1 for opaque mode-6 blocks because of this issue.
| Clip: worst-frame luma PSNR (dB) / encoding time (ms/frame) | bc7enc | ISPC bc7e Basic, patched |
|---|---|---|
| archimedes | 47.77 / 61 | 48.03 / 26 |
| worm parade | 44.91 / 69 | 45.03 / 25 |
| membrane | 42.82 / 70 | 42.92 / 25 |
| firegrid | 44.17 / 56 | 44.22 / 26 |
| meta_build | 47.57 / 81 | 47.94 / 27 |
| test-vj | 49.28 / 55 | 49.67 / 23 |
After deciding on the bc7e encoder, encoding and decoding the source clip still ran sequentially. To improve this, encoding and decoding were run in parallel as opposed to sequentially. This resulted in a final prep time for the sample clip of 42.91 seconds on a Windows laptop (yes, a Windows version is being built) with an Intel Core i7-11800H (8C/16T) CPU and an NVIDIA RTX 3050 Ti Laptop GPU.
A 45 second synthetic clip was used for these benchmarks.
| Configuration: 45-second, 1080p60 sample, 2,700 frames | Complete preparation time |
|---|---|
| HAP R: original scalar bc7enc | 205.10 seconds |
| HAP R: ISPC bc7e Basic, sequential | 53.04 seconds |
| HAP R: same sequential build, rerun | 63.73 seconds |
| HAP R: ISPC bc7e Basic, overlapping decode and encode | 42.91 seconds |
| HAP Q baseline | 29.84 seconds |
A new Prepare button is added to each clip in the Sources dock with a toggle at the top of the dock to switch between encoding with HAP Q and HAP R. Based on my testing, HAP R gives 5.0–8.4 dB higher worst-frame luma PSNR over HAP Q but produces 16–31% larger compressed video frames.
For someone that is low on disk space or wants to transcode a large directory with a small amount of time, HAP Q would be a more reasonable choice.
6 real VJ clips were used for this benchmark, where every 30th frame was sampled for 10 frames per clip, resulting in 60 frames total.
| Clip | HAP Q: shader worst-frame luma / RGB PSNR (dB) | HAP R Basic: shader worst-frame luma / RGB PSNR (dB) | Luma gain |
|---|---|---|---|
| archimedes | 42.06 / 39.33 | 47.65 / 40.76 | +5.6 dB |
| worm parade | 39.88 / 37.72 | 44.86 / 39.07 | +5.0 dB |
| membrane | 36.45 / 34.86 | 42.81 / 35.79 | +6.4 dB |
| firegrid | 38.30 / 37.37 | 44.02 / 38.20 | +5.7 dB |
| meta_build | 41.04 / 38.34 | 47.64 / 39.54 | +6.6 dB |
| test-vj | 40.68 / 40.64 | 49.07 / 48.54 | +8.4 dB |
The original files stay unchanged while the transcoded copies are placed in a prepared folder in the clip’s directory.
If clips are already set in cells and the clip is prepared to HAP Q/HAP R in the Sources dock, the “Use prepared” button will replace that cell’s clip with the prepared version.
The hope for this feature is that Showout users will spend less time preparing their media and more time rehearsing for shows. This feature is available in the latest version of the app.
Review the other changes in the new version in the release notes.
Feedback on this new feature is appreciated and can be sent to support@showout.io.