Thinking about that spooky-handling high-end bike...
I rode my wife's track bike last Sunday and had to stop and ponder what the heck was coming loose after less than a lap. I felt unstable, wobbly, like it was about to go out of control. Nothing was loose, so I persisted and got used to it. The difference in 'handling' was all in my position. I was slightly lower and more forward than on my bike. I couldn't get out of the saddle at all - it felt like I would fall if I did. To fix this 'handling' problem would involve firstly raising the saddle and secondly pushing it back, or perhaps changing the stem length. And testing it after every alteration, one change at a time.
My wife is entirely happy with her track bike as it is set up and it looked OK to me as well, with my short legs and all, but shifting my weight forward even slightly changed the whole feel of the thing. My point is that spooky handling can be a combination of many things, not just wheels or frame but position as well. Stem length is a big one in my book, and look at how you are distributing your weight overall, between front and rear.
Just a thought.