You are completely right. This needs to be done right, otherwise it can get annoying and users will leave.
To this time I haven’t really got any negative comments/feedback about the captcha, so I let it be, but as now I see someone has an issue with it. It increased in priority in my to-do list.