No , those file are not be permanently deleted. Usually on deleting files your system just forgets path to that stored data. So told like testdisk can find and recover this data, unless such space hasn’t been overwritten by other data.
So if you feel you don’t want people to be able to recover your data, you should use Disk encryption on your OS as mentioned on PG recommendation page Recommended Encryption Software: VeraCrypt, Cryptomator, PicoCrypt, and OpenPGP - Privacy Guides.
Also keep a good user password to access your machine. If you have a good password
i don’t think you need to delete things on drive everytime. unless its a public computer. Then i would suggest not store any sensitive data on it at all, Or just create a separate user profile and encrypt home directory if on linux
Also refer to this thread of mine for secure data erase ShredOS and other Disk erase tools
I am not sure about the USB drive part. You can use a veracrypt container to store sensitive files on it.
Also in case of a threat from state actor secure data erase may not be sufficient to prevent advanced recovery tools in recovering your data, still disk encryption should help a lot.