Welcome to iPhone Forums, chris333!
The built in apps, those that you don't need, can be hidden in a folder. Delete every other app you don't need. If you just intend to use the iPhone as MP3 player, you don't need email accounts in Settings - Mails, Contacts, Calendars.
If you have music purchased in iTunes, you'll need the Music app, and your Apple ID in Settings - iTunes & App Store.
If you use iCloud, disable Mail, Contacts, Calendars, Notes, and everything else you find there, except for Backups, if you prefer to backup your device through iCloud.
Make sure the songs are stored on your iPhone, or else you'd need an internet connection to listen to them.