It’s expected that you’ll take a performance hit as the toggle says:
Reduces performance but makes V8 more resistant to attacks.
That being said, it also shouldn’t cripple your browser by any means. For example, using Speedometer 3.0, my score drops from ~30 to ~25 with the V8 optimiser disabled. Obviously, it will depend a lot on the hardware you’re using. If your browser becomes unbearably slow, I would allow sites to use the V8 optimiser (since an unusable browser won’t do you much good anyways). Alternatively, you can enable it on a per-site basis if only certain sites perform poorly.