USB: fix usb-serial regression

This patch reverts d9a7ecac since it
breaks drivers that need to access the ->port[] array in shutdown
(most of them).

Signed-Off: Jim Radford <radford@blackbean.org>
Acked-by: Mark Lord <mlord@pobox.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>