The answer is that it depends nearly entirely on your hosting provider and their reputation. The solutions you’ve mentioned make it dead simple to properly configure a mail server if you have self-hosting experience.
I have never had issues with mail delivery or receiving using these setups, personally. However, if you are using a budget VPS host; or even worse, if you are doing it from a residential ISP; you will almost certainly encounter issues.
For that reason it’s usually not worth it for people to do. But there is nothing about email that magically makes it much harder than self-hosting anything else. It’s about the same as other self-hosted services.
Some previous discussion I’ve found for reference: