Xorg was starting before udev could finish the monitor command. This prevented the X server from having access to the input devices. For some reason, any subsequent starts would be fine.
Adding a small delay to the X server start and triggering a request for device events from the kernel after a delay seems to fix this issue.
