I have owned the 4, 4S and now the 5.
My opinion on them are as follow.
4:
It has the best battery life (when the battery is new). This is due to it being a single core CPU but it is the slowest of the three in terms of launching apps. The antenna design requires a case to protect your from dropping calls (unless you can hold it properly) and the camera is decent but no where near the 4S and 5. This is a very solid phone still. My wife won't give hers up.
4S:
The best compromise between the 4 and 5. Speedy, incredible camera, and battery life is very serviceable (when the battery is new). Apps can launch anywhere from 2 to 3 seconds faster than the 4. Antenna problem was fixed. I almost didn't upgrade to the 5 since I am thinking about leaving iOS development. The phone is simply that good.
5:
Fastest of the 3. Benchmarks for CPU performance show it to be four times faster than the 4 and twice as fast as the 4S. Camera as good as the 4S. Larger screen allows me to display more text from websites and other text based apps that have updated to the 5 new screen size. No antenna issues so far and LTE (when available) is fast. Light weight is probably the biggest thing I notice about it. I could blind pick up each of the phones and tell you the 5 from either the 4 and 4S with no issues. That much of a difference.
But the thing that sold me on it, IE pushed me to buy it even though my 4S was perfect for my needs, was the optical clarity and brightness of the new screen. The color saturation is better and more vibrant. The brightness can now be set high enough to overcome direct sunlight, a first in iPhone history. BUT it does come at a price. At night, at the lowest brightness setting, it doesn't get as dark as the 4S and 4. So using it in bed has to be done in such a way that the 5 doesn't wake my wife up. 4 and 4S didn't have that issue very often or at all.
Again, just my opinion after using the 4 from March 31st 2011 till Oct 14th 2011, the 4S fro Oct 14th 2011 till Oct 13th 2012 and the 5 from Oct 13th till today. And yes, it is odd that I know those dates off the top of my head.