I consider both interpretations possible, and I'd rely on context to make sense of it.
Originally Posted by hippokrene
Adverbs to the rescue:
Meaning 1.: She gave a polite, if not even friendly, smile.
Meaning 2.: She gave a polite, if not exactly friendly, smile.
I suppose that this constellation of words has no clear literal meaning on its own, because in many contexts both meanings express the same communication. Um, that sounds complicated. Let me try again.
First, there's semantics, what the words actually mean. "It's cold in here," is a statement about temperature. Then, there's pragmatics, what the words are supposed to communicate. "It's cold in here, don't you think?" could well be an invitation to close the window.
Now, it's entirely possible that two different literal interpretations of the words could result in the same practical effect.
Here's an example I found in the BYU-BNC (the British National Corpus):
Ants stumble into the pit by accident and, because it is constructed of friable soil or sand and has steep sides, they have to struggle to get out. The task is made more difficult, if not impossible, by the ant-lion.
I could write an essay about estimation vs. knowledge, the semantics of difficult vs. impossible. The formulation "if not impossible", however, here has the pragmatical function of forestalling such a discussion. The section is about the preying behaviour of the ant lion; the odds of survival for the ant are a side point, and the phrasing suggests this. Whether it achieves the point by escalation (more difficult --> impossible!!!) or by a cautious disclaimer (well, not impossible, but - yes! - more difficult) doesn't really matter. We'll get the point of the text.
Note that your sentence is different, in that the interpretation has an actual effect on how a reader imagines the smile. Is it an ambiguous smile that could - maybe - be friendly? Or is it a polite smile, which is better than nothing, although the writer would prefer it to be friendly?
Glancing through the corpus examples, for most of them it's not important to resolve the ambiguity. They function much like the ant-lion section above. It does make a difference, but the difference it makes is de-emphasised by the text, so that people rarely bother to ask the question. [It's a British Corpus; I haven't tried any of the American Corpora at the same page, as I don't know their syntax and I'm too lazy to find out, today.]
So my - unhelpful - answer: it could mean both. Context matters.