I think this isn’t the right way to think about it.
By default, the security measures of a Linux desktop and Linux phone are likely compatible.
But the usage and attack surface of a Linux phone is so much larger. The phone has become our new daily computer. It’s always on, always near us, and is connecting to networks all the time, and with cellular capability. Different threat model than desktop Linux as the trust boundaries have grown.
Which this is the scary part of Linux as a phone.