There certainly are differences. The iPhone 5 came out in September 2012, whereas the SE was made available last year in March. In the meantime, Apple didn't keep the iPhone the way it was back then. They tried getting it close in performance and features to the iPhone 6s.
The SE has an A9 chip with 64 bit architecture, with 2 GB of RAM, the 5 has an A6 chip with 32 bit architecture, with 1 GB RAM. That's one of the major differences, and this means the SE way faster than the 5.
The cameras are also different. While you can use a camera with 12 MP on the iPhone SE, it's 8 MP on the iPhone 5, and the SE has features that were not yet available for the 5.
The newer iPhone has a Touch ID fingerprint sensor, which the older one has not.
Battery life is better for the iPhone SE, and also connectivity is better and faster than on the iPhone 5.
Hope that helps.