This is now our strategy for everything except a speed of 0 (which means
autonegotiate); and a speed of 1 Gbps (which requires autoneg anyway).

Not all cards allow ethtool to "fix" speeds; but some (still) require
it.  For instance, this commit when applied to an Intel X710 card that
should be at 10Gbps has no affect; apparently ethtool cannot fix speeds
for that card and/or driver.  On the other hand, Mellanox 10/25Gbps
cards sometimes require the speed to be manually set (i.e., if they are
directly connected to each other via a layer1 switch).  Even on those
cards, it's not really setting it; it's just hinting to the autoneg
process which speed you really want (and I'd guess that is true for all
modern high-speed Ethernet chips).

Anyway, we don't know when to force the speed set/suggest or not unless
we track more data at the server side, so the current strategy is to
always attempt to set/suggest the speed we want, and fallback to setting
autoneg if we fail.  Note that sometimes ethtool returns successfully
even if settings fail (I'm looking at you, Intel X710), so this strategy
is already sort of doomed to failure (but it hasn't made anything not
work!).  If it causes problems for any cards/drivers/speed combinations,
we'll revisit this, obviously.