Performance has always been a bit of a black art in SharePoint, with so many core files being dropped onto landing pages it's not uncommon to have a 500kb payload taking over 10 seconds to load.. of course for most corporate websites this is simply not good enough.
When building the RNIB website we spent a lot of time getting the page payload lower, and the caching model tweaked so that the pages were loading more responsively. We were mostly successful but there were plenty of challenges.
First off was the ScripResource files (used by Ajax). They were only really used for editing (the rich text editor we used from Telerik). Putting those references into a SharePoint "EditPanel" seemed to sort that one out (so the files were only ever being loaded if the page was in Edit Mode .. which was never the case for anonymous users.
The caching profiles were also a big boon, enabling BLOB caching for the script files and static images, and also tuning the SharePoint site collection cache profiles so that they worked best for anonymous users.
We also used a bunch of other 3rd party tools, including the awesome Fiddler2 and YSlow for FireFox.
There has also been a very good article recently from the Microsoft SharePoint Team Blog describing how they went about optimising the sharepoint.microsoft.com website. Its a good read and brings up some lesser known optimisation techniques.
If you have any tips on performance optimisation I'd love to hear it.