I’ve considered this and it seems like a pain in the ass.
I believe the implementation used is HMAC challenge-response. This is a problem because yubikeys only have two slots, only one can be used for challenge-response.
There’s also another implementation called HSM but I haven’t looked into it.
The reason I hesitate doing this is because it relies on things that are out of my control. It relies on system time being synced with the key, for example. There’s a lot of moving parts, and I don’t trust it enough.
My preferred implementation is a static password. With a yubikey you can configure up to two static passwords (they share the same slots with challenge-response, but you can have one of each or two static passwords only).
Bonus tip, use a short, memorised password / phrase in addition to the static password. This way it’s two-factor (something you have, and something you know).