There are lots of errors that can be blamed on the difference between written and spoken English (also true in Spanish, and I would assume, in most languages). These differences come mostly from dropping syllables or truncating words and can be blamed on one thing: laziness.
After all, if people know what you mean, why take longer to say it? Who wouldn't rather go to the gym than to the gymnasium? When you're really hungry, why say "I have got to eat something" when you could say, "I gotta eat somethin'"?
Now, don't worry—I'm not going to insist that you should speak the way you should write. Part of what's great about language is the ability to play around with it. However, sometimes this can lead to bad grammar—and it does, often, in the case of "of" versus "have."
In spoken English, have is sometimes shortened to sound like of:
He should have stopped at beer six.
He should 'ave --> He should of --> He should of stopped at beer six.
Now, hearing it sounds fine because your brain can deal with these nuances of pronunciation. This particular nuance, however, is tricky because the shortened version is also a word in English. Using "of" when the situation requires "have" weakens your writing and, for those of your readers who know the rules, destroys your respectability.
So, how do you know when this is happening?
Well, replace "of" with "have" and if the sentence still makes sense, then it should be "have."
I had twenty of those mini cookies yesterday. --> I had twenty have those mini cookies yesterday.
"Have" does not make sense here...it should stay "of."
I could of been a millionaire! --> I could have been a millionaire!
"Have" definitely makes sense here...change the "of" to "have" for a grammatically correct sentence.