So was his Tele, and Stevie Ray Vaughan's #1 Strat.
Fenders were designed to become Partscasters when the neck wears out. Based on that, I don't consider any value lost on a non-collectors grade Fender that has had the neck swapped for another of equal or lesser value.
Basically I'm not saying I'd pay full price for a Custom Shop Tele with a Baja neck, but I would pay full price for an American Special with a Baja neck.
The waters really get muddied when using genuine Fender parts like the neck I used for mine. It has a Squier body but if it had a Fender body would it be a Fender?