I have never transfers files via Bluetooth...but I'm very certain, you need to be jailbroken first.
Once jailbroken...there are a few tweaks for this specific function.
There is also a tweaks called "SendAny" that allows you to send "any" type of file via iMessage. You must also have "ReceiveAny" installed...as well as the user you are sending it to.