Has to do with who is doing what.
Implication comes from imply. You imply something, suggest it. It's active, moving the thought along. However, infer is something you do and is passive.
imply -- hint, insinuate
infer -- assume
Mr. Raleigh implied (insinuated, gave the implication) that I was a bank robber. I inferred (assumed from his statement) he had called the police and I would be getting a visit from them soon.
Imply THAT.
Infer FROM.
All better now? (Don't feel badly. I struggled with that once upon a time and read and reread until I understood the different connotations.)