assorted hackery

A while back I got an Elgato Cam Link 4K, mostly as a reverse engineering target since it's based around the ECP5 FPGA (which is well-supported by open source tools) and it has USB3 & HDMI interfaces that would be interesting to play with. However, it works rather well in its intended form - as an HDMI capture device - so I've just been using it as-is so far.

I ran into problems recently when trying to use it for videoconferencing. While it works just fine in OBS, it would not work in Discord/Jitsi/Zoom/etc.: either showing a blank output or not giving an option to use the device at all.

Update (2021-07-20):

This issue has now been patched in the Linux kernel & backported to stable releases. The fix is included in the following releases:

4.4.276
4.9.276
4.14.240
4.19.198
5.4.134
5.10.52
5.12.19
5.13.4
5.14-rc1

A simple workaround

After a bunch of searching around, I found this workaround. This creates a dummy v4l2 (Video for Linux, 2) device:

$ sudo modprobe v4l2loopback devices=1 exclusive_caps=1

Then it uses ffmpeg to stream from the Cam Link to the dummy device:

$ ffmpeg -f v4l2 -input_format yuyv422 -video_size 1920x1080 -i /dev/video0 \
         -pix_fmt yuyv422 -codec copy -f v4l2 /dev/video1

A not-so-simple solution?

Problem solved! We could just stop here, but I didn't want to have to run that every time I wanted to use a webcam. Also, I didn't want to waste CPU for ffmpeg to shunt the data around, so let's dig a bit further.

The crucial part of this workaround seems to be the ability to force a particular pixel format, which is not an option available in the videoconferencing applications I'm trying to use.

Using v4l2-ctl I could check which format options are available:

$ v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'YUYV'
	Name        : YUYV 4:2:2
		Size: Discrete 1920x1080
			Interval: Discrete 0.020s (50.000 fps)

	Index       : 1
	Type        : Video Capture
	Pixel Format: 'NV12'
	Name        : Y/CbCr 4:2:0
		Size: Discrete 1920x1080
			Interval: Discrete 0.020s (50.000 fps)

	Index       : 2
	Type        : Video Capture
	Pixel Format: 'YU12'
	Name        : Planar YUV 4:2:0
		Size: Discrete 1920x1080
			Interval: Discrete 0.020s (50.000 fps)

I could also run Discord, check which format is in use, and see that it has picked 'YUV 4:2:0', not 'YUYV 4:2:2':

$ v4l2-ctl -d /dev/video0 --get-fmt-video
Format Video Capture:
	Width/Height      : 1920/1080
	Pixel Format      : 'YU12'
	Field             : None
	Bytes per Line    : 1920
	Size Image        : 4147200
	Colorspace        : sRGB
	Transfer Function : Default (maps to sRGB)
	YCbCr/HSV Encoding: Default (maps to ITU-R 601)
	Quantization      : Default (maps to Limited Range)
	Flags             :

I couldn't find any option in V4L to force selection of 'YUYV 4:2:2', so instead I started looking into patching the Cam Link firmware so that it would not offer any other formats.

Fortunately, several people have done some great work on reverse engineering this device already. In particular, Kate Temkin has already dumped the firmware & written some neat tools for experimenting with it, which are in her camlink-re repository.

A quick look for interesting strings in the firmware shows that one of the pixel formats is there in plaintext:

$ hexdump -C camlink_4k_spi_flash.bin | grep NV12 -C5
0000aef0  00 00 0e 02 00 05 10 24  01 03 cd 00 83 00 03 01  |.......$........|
0000af00  00 00 01 00 00 00 1b 24  04 01 01 59 55 59 32 00  |.......$...YUY2.|
0000af10  00 10 00 80 00 00 aa 00  38 9b 71 10 01 00 00 00  |........8.q.....|
0000af20  00 1e 24 05 01 01 80 02  e0 01 00 00 94 11 00 00  |..$.............|
0000af30  94 11 00 60 09 00 0a 8b  02 00 01 0a 8b 02 00 06  |...`............|
0000af40  24 0d 01 01 01 1b 24 04  02 01 4e 56 31 32 00 00  |$.....$...NV12..|
0000af50  10 00 80 00 00 aa 00 38  9b 71 0c 01 00 00 00 00  |.......8.q......|
0000af60  1e 24 05 01 01 80 07 38  04 00 00 a7 76 00 00 a7  |.$.....8....v...|
0000af70  76 00 48 3f 00 0a 8b 02  00 01 0a 8b 02 00 06 24  |v.H?...........$|
0000af80  0d 01 01 01 1b 24 04 03  01 49 34 32 30 00 00 10  |.....$...I420...|
0000af90  00 80 00 00 aa 00 38 9b  71 0c 01 00 00 00 00 1e  |......8.q.......|
--
0000b100  cd 00 83 00 03 01 00 00  01 00 00 00 1b 24 04 01  |.............$..|
0000b110  01 59 55 59 32 00 00 10  00 80 00 00 aa 00 38 9b  |.YUY2.........8.|
0000b120  71 10 01 00 00 00 00 1e  24 05 01 01 80 07 38 04  |q.......$.....8.|
0000b130  00 00 a7 76 00 00 a7 76  00 48 3f 00 0a 8b 02 00  |...v...v.H?.....|
0000b140  01 0a 8b 02 00 06 24 0d  01 01 01 1b 24 04 02 01  |......$.....$...|
0000b150  4e 56 31 32 00 00 10 00  80 00 00 aa 00 38 9b 71  |NV12.........8.q|
0000b160  0c 01 00 00 00 00 1e 24  05 01 01 80 07 38 04 00  |.......$.....8..|
0000b170  00 a7 76 00 00 a7 76 00  48 3f 00 0a 8b 02 00 01  |..v...v.H?......|
0000b180  0a 8b 02 00 06 24 0d 01  01 01 1b 24 04 03 01 49  |.....$.....$...I|
0000b190  34 32 30 00 00 10 00 80  00 00 aa 00 38 9b 71 0c  |420.........8.q.|
0000b1a0  01 00 00 00 00 1e 24 05  01 01 80 07 38 04 00 00  |......$.....8...|

YUY2 and I420 look interesting too...

Next up is to learn a bit more about how the device presents these different format options. I could've gone and read the USB Video Class (UVC) spec here, but I found it quicker to go look at the Linux source for the UVC driver.

In uvc_driver.c there is a long list of "UVC format descriptors", and a few interesting ones pop out:

{
	.name		= "YUV 4:2:2 (YUYV)",
	.guid		= UVC_GUID_FORMAT_YUY2,
	.fcc		= V4L2_PIX_FMT_YUYV,
},

{
	.name		= "YUV 4:2:0 (NV12)",
	.guid		= UVC_GUID_FORMAT_NV12,
	.fcc		= V4L2_PIX_FMT_NV12,
},

{
	.name		= "YUV 4:2:0 (I420)",
	.guid		= UVC_GUID_FORMAT_I420,
	.fcc		= V4L2_PIX_FMT_YUV420,
},

Further down, there is a function that takes a 16-byte guid, and searches the format descriptor list to find it:

static struct uvc_format_desc *uvc_format_by_guid(const u8 guid[16])
{
	unsigned int len = ARRAY_SIZE(uvc_fmts);
	unsigned int i;

	for (i = 0; i < len; ++i) {
		if (memcmp(guid, uvc_fmts[i].guid, 16) == 0)
			return &uvc_fmts[i];
	}

	return NULL;
}

This function is used by uvc_parse_format to match the guid if the format type is UVC_VS_FORMAT_UNCOMPRESSED or UVC_VS_FORMAT_FRAME_BASED (some unrelated code removed for clarity, indicated by ...):

static int uvc_parse_format(struct uvc_device *dev,
	struct uvc_streaming *streaming, struct uvc_format *format,
	u32 **intervals, unsigned char *buffer, int buflen)
{
...

	format->type = buffer[2];
	format->index = buffer[3];

	switch (buffer[2]) {
	case UVC_VS_FORMAT_UNCOMPRESSED:
	case UVC_VS_FORMAT_FRAME_BASED:
...
		/* Find the format descriptor from its GUID. */
		fmtdesc = uvc_format_by_guid(&buffer[5]);
...
	default:
		uvc_trace(UVC_TRACE_DESCR, "device %d videostreaming "
		       "interface %d unsupported format %u\n",
		       dev->udev->devnum, alts->desc.bInterfaceNumber,
		       buffer[2]);
		return -EINVAL;
	}

uvcvideo.h and video.h have the definitions of some of the macros used above:

#define UVC_GUID_FORMAT_YUY2 \
	{ 'Y',  'U',  'Y',  '2', 0x00, 0x00, 0x10, 0x00, \
	 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71}
#define UVC_GUID_FORMAT_NV12 \
	{ 'N',  'V',  '1',  '2', 0x00, 0x00, 0x10, 0x00, \
	 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71}
#define UVC_GUID_FORMAT_I420 \
	{ 'I',  '4',  '2',  '0', 0x00, 0x00, 0x10, 0x00, \
	 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71}

#define UVC_VS_UNDEFINED				0x00
#define UVC_VS_FORMAT_UNCOMPRESSED			0x04
#define UVC_VS_FORMAT_FRAME_BASED			0x10

Putting all this information together: in the firmware I'd expect to see a format byte of either 0x04 or 0x10, followed by 2 bytes, and then followed by one of the GUID sequences. Looking back at some of the hexdump above, that's exactly what's in the firmware:

04 01 01 59 55 59 32 00 00 10 00 80 00 00 aa 00 38 9b 71
^^       ^^ ^^ ^^ ^^
format   Y  U  Y  2

So, if I overwrite the format to 0x00 (undefined), the kernel driver should fall through to the default switch case and ignore the "unsupported" format. I went through and created a patched firmware file with any instance of NV12 and I420 formats disabled in this way. Now I just needed a way to flash it.

The USB interface/microcontroller used on the Cam Link is the Cypress FX3. Conveniently, it reads its firmware from an SPI flash IC on every boot and, if it doesn't read a valid image it will drop into a USB bootloader. I could pop open the case and short out some data pins on the flash IC while plugging in the device:

This would force it into the bootloader:

[26808.088901] usb 1-6: new high-speed USB device number 21 using xhci_hcd
[26808.237617] usb 1-6: New USB device found, idVendor=04b4, idProduct=00f3
[26808.237622] usb 1-6: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[26808.237627] usb 1-6: Product: WestBridge 
[26808.237630] usb 1-6: Manufacturer: Cypress
[26808.237633] usb 1-6: SerialNumber: 0000000004BE

Now I could use fx3load from the usb-tools pyfwup project to test out my patched firmware:

$ fx3load camlink_4k_spi_flash_patched.bin
Target found!
Programming 4194304 bytes...
Programming complete!                                                           

Running newly-programmed application.

and woohoo! It no longer has the problematic formats

$ v4l2-ctl --list-formats-ext -d /dev/video1 
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'YUYV'
	Name        : YUYV 4:2:2
		Size: Discrete 1920x1080
			Interval: Discrete 0.020s (50.000 fps)

and it works!

Making it permanent

Since I loaded this through the FX3 bootloader, it was only loaded into RAM and would be lost on re-plugging the device - it still needs to be written to the SPI flash IC. Fortuately, Kate's camlink-re repo from above has more than just firmware dumps: it also has some firmware and host software for poking at the device in various ways, including writing the SPI flash. After putting the device back into the bootloader I could load up the exploration firmware and use the camlink application to check it out & re-flash it:

$ fx3load firmware/exploration.img 
Target found!
Programming 101892 bytes...
Programming complete!                                                           

Running newly-programmed application.

$ ./camlink scan
Detected an FPGA with IDCODE: 41111043.

$ ./camlink flash ../factory_fw/camlink_4k_spi_flash_patched.bin 
Flashing the SPI flash with 4194304 bytes.
                                                                                
Done.

I unplugged the device, plugged it back in, and... it didn't work - straight back to the bootloader. Damn.

Oh well, not everything's simple. I thought about it a bit, did some more research, and came up with three ways I thought it could be going wrong:

There's probably a checksum somewhere? My modifications worked just fine when loading to RAM, but maybe there's a checksum baked into the image that's only used when loading from SPI flash.
The flashing process wasn't working? Kate's exploration firmware is explicitly marked as "work-in-progress" and "not really ready for use" - I should expect some sharp edges.
Maybe the flash images are tied to a specific device? I edited Kate's flash dump and tried to re-flash my device with it. I'm really regretting not taking a backup of my own one at this point...

The first one is easiest to check, I went and actually read some documentation. Cypress' AN76405 document has details of the FX3 boot options, and section 7.4 has details of the boot image format including a checksum at the end:

It also has some example code for validating the checksum:

I grabbed the code, fixed it up a bit, and generated the correct checksum for my patched image. I re-flashed it and... still back to the bootloader.

Onto the next option - was it flashing correctly? I could check by reading back the image using the camlink application & comparing it against what I flashed:

$ ./camlink dump dump.img
$ diff dump.img ../factory_fw/camlink_4k_spi_flash_patched.bin 
Binary files dump.img and ../factory_fw/camlink_4k_spi_flash_patched.bin differ

Looking closely at the differences shows some interesting patterns. First, most of the file matches - there are only small differences in the areas that changed. Pictured below is the changed serial number:

Comparing bits between the two images shows that all '0' bits match between the two images, but dump.img is missing some '1' bits - this gives a hint about what might be going wrong.

>>> bin(0x30003300020005000000020041) # dump.img
'0b110000000000000011001100000000000000100000000000000101000000000000000000000000000000100000000001000001'
>>> bin(0x32003300420037004500330043) # camlink_4k_spi_flash_patched.img
'0b110010000000000011001100000000010000100000000000110111000000000100010100000000001100110000000001000011'

Flash memory is programmed in two stages: first, it is erased which sets all bits in a sector to '1'. Next, individual bits are set back to '0' to program in the correct data. Since all the '0's are correct but '1's are missing, this suggests that the erase step may not be working correctly.

I soldered some bodge wires on to the device so I could look on an oscilloscope and eventually noticed that no erase commands were ever happening on the SPI bus. I dug into the camlink code and found a subtle bug that was causing it to never send the erase commands at all. A couple of patches later and now flashing & readback matched up correctly:

$ ./camlink flash ../factory_fw/camlink_4k_spi_flash_patched.bin 
Flashing the SPI flash with 4194304 bytes.
                                                                                
Done.
$ ./camlink dump dump.img
$ diff dump.img ../factory_fw/camlink_4k_spi_flash_patched.bin 
$ # no output means they match!

and now the device boots again!

[84760.124434] usb 2-6: new SuperSpeed USB device number 48 using xhci_hcd
[84760.145329] usb 2-6: LPM exit latency is zeroed, disabling LPM.
[84760.146062] usb 2-6: New USB device found, idVendor=0fd9, idProduct=0066
[84760.146071] usb 2-6: New USB device strings: Mfr=1, Product=2, SerialNumber=4
[84760.146078] usb 2-6: Product: Cam Link 4K
[84760.146087] usb 2-6: Manufacturer: Elgato
[84760.146092] usb 2-6: SerialNumber: 00023B7E3C000
[84760.160030] uvcvideo: Found UVC 1.10 device Cam Link 4K (0fd9:0066)
[84760.174855] uvcvideo 2-6:1.0: Entity type for entity Processing 2 was not initialized!
[84760.174863] uvcvideo 2-6:1.0: Entity type for entity Input 1 was not initialized!
[84760.175392] input: Cam Link 4K: Cam Link 4K as /devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6:1.0/input/input44
[84760.179833] hid-generic 0003:0FD9:0066.0019: hiddev2,hidraw8: USB HID v1.01 Device [Elgato Cam Link 4K] on usb-0000:00:14.0-6/input2

Success!

For more on the reverse engineering effort around this device, there's a page on the apertus wiki.

Patching my Cam Link 4K to play nicer on Linux

Update (2021-07-20):

A simple workaround

A not-so-simple solution?

Making it permanent