Try this. It's the same process for any downgrade. Follow all the steps carefully and you shouldn't have any problems.
http://www.iphoneforums.net/forum/iphone-4-hacking-17/how-do-downgrade-your-iphone-4-3-a-9736/
Heres the link for 4.1 for the iPhone 4 direct from apple...
iPhone3,1_4.1_8B117_Restore.ipsw