I've long been interested in this festival, but been scared away by the apparently huge crowds and traffic.

Can anyone compare to Port Credit's Southside Shuffle?
I have been to both many times. Burlington's is mostly free (IIRC the Shuffle used to be free).
In Burlington access from the highway is more convenient, and parking is well organised with shuttles from large lots.
There is a shuttle from the GO train station so you can also get on the GO in Mississauga.
You can usually find street parking within a 15 minute walk also. Try west of Maple St., or come down Guelph Line, go west on New St. and start looking north of New once you're west of Drury Lane. Walk west on New to Brant and enjoy the street performances on Brant.
The crowds do get large, but we usually can find seating in the beer area, so who cares?