>If the latter, I suppose we could instead say our destination is where we calculate the planet will be four years from now. Except that my travel time was instantaneous, so again either I've arrived too early and need to wait around for four years, or I jumped 4 years into the future (at which point that's not really FTL travel, just kind of stepping outside of time into some nether state for four years).

This doesn't make sense. If you have a wormhole teleporter, and teleport to where that 4ly-away planet is in your observation, it won't be there, since you saw it 4 years ago and it's moved. This seems fairly obvious.

Now, if you observe its motion and predict where it's traveled in the 4 years between your observation (now) and when the photons you saw started from that planet (4y ago, relative to your current position in spacetime), and set your wormhole teleporter to take you there instead, it should take you to the planet's current position. (Hopefully your calculations were accurate and you don't teleport into the planet...) I don't see why you think you'd arrive too early, or jump into the future. Your friend who stays behind would need to wait 4 years to see you arrive at the planet in your super-telescope, but that's just because it takes light that long to arrive.

I don't see how this particular thought experiment necessitates time travel.