If it's not in the contract, then it's just your word against his; however, you did make a deal, so if the dealer gets mad at you, I wouldn't necessarily blame him. On the other hand, he's making money either way on the financing. Still, if you agreed to his offer, you are backing out---not saying you're a bad person or anything, but just throwing that out there so that you might understand if the dealer gets grumpy on you.
Obviously if there is a cancellation clause then the warranty company is okay with it. No doubt the dealer can offer any qualified buyer the lower interest rate.