See, to me, it's absolutely fine. If you're having a conversation with me, you have no expectation of privacy against me, and me making an audio recording is no different to me getting out a pen and paper and making a written record of what we say - the only difference being an audio recording is less fallible.

Now, publishing that recording is another matter, and might be some sort of offence. The recording can only be for personal use. For courts in the UK, the way it is handled is that you simply write a transcript of the recording and submit that as a statement in evidence - this way, the court doesn't waste their time bickering over the sound of a word on the recording, and instead maintains its position of deciding an argument between two parties, and the added advantage is that the other party doesn't even get to listen to the recording itself.