Drop the event list semaphore and only use the wait queue and the list
to synchronize queue access.  Break out of a poll or read whenever
the device is disconnected.

Signed-off-by: Kristian Høgsberg <krh@redhat.com>
Signed-off-by: Stefan Richter <stefanr@s5r6.in-berlin.de>