Xorg was starting before udev could finish the monitor command. This prevented the X server from having access to the input devices. For some reason, any subsequent starts would be fine.
Adding a small delay to the X server start and triggering a request for device events from the kernel after a delay seems to fix this issue.
The current supervisor.d configs are configured specifically for a particular supervisor version / installation.
This decouples the configuration so we can use it on any distro.