hope that helped

The phrase "let's say" is an idiom used to introduce hypotheticals: "Let's say it starts raining this afternoon. Will you still want to go to the park?" The idea is that you're agreeing to both (or all) assume that whatever follows is true in order to further explore the consequences of it.

Mr Amber's usage is a further refinement of that - rather than admitting to wanting sherry to drink, he's saying that if he did, hypothetically, want something, it would be sherry - it's considered unfashionable in posh society to directly admit to actually wanting something.