A feeling that something is not quite right, or awry.
There's another phrase similar to this one that goes 'this smells fishy.' Both expressions mean something feels wrong, but the person is unable to discern exactly what it might be.
For centuries, the word smell has been used in a figurative sense to define someone who, because of their intuition, perceives something to be off. For example, Shakespeae, a popular English poet, wrote: "Do you smell a fault?" in the play King Lear, which is believed to have been written between 1603 to 1606.
According to the Facts on File Encyclopedia of Word and Phrase Origins, written by Robert Hendrickson, he mentions that the origins for this idiom are uncertain, however, he explains a theory behind it might possibly involve a cat being able to smell a rat nearby, despite not having sight of it. The idea, I assume, is that after picking up a rat's scent, a cat perceives that something is not quite right, but without having vision of the rat, the cat is unable to confirm its feelings.
Anyways, the phrase with its modern meaning was used as early as June 1851 in the County Courts Chronicle newspaper:
"Two other cases the witness mentioned, in the first of which he alleged that the judge, in reference to
an insufficiency of evidence said, 'I smell a rat; I don't believe the defendant or her witness.'"
References: Facts on File Encyclopedia of Word and Phrase Origins, by Robert Hendrickson
Note: The origins for most phrases and sayings cannot be said with a certainty. What's provided are theories that may be plausible to how a phrase originated, but not necessarily so.
In addition, quotes that contain a particular phrase may be taken from old newspapers, poems, or books that were written centuries ago, but this by no means confirms that the phrase originates from said newspapers, poems, or books. In all likelihood, if an expression is being used in a newspaper, it's probably already a well known saying and is from an older time.
* Jake smelled a rat at his company when some of his private documents about a new proccessing chip for computers went missing; he suspected it was one of his employees.