I just tested both /wiki and a gem purchase via PayPal on my Windows 8 x64 machine and both opened IE10 on my desktop without issue. I tried it with IE closed and also already running – when it was running it opened new tabs (as expected). I did an in-place upgrade from Win7 to Win8 on my machine and that does influence some of the default behaviors.
You might try configuring IE 10 to open the desktop version instead of the metro version. You can do that by running IE 10 (desktop) and choosing Settings (gear icon), Internet options, and then Programs. In the top of this dialog, change “Choose how you open links” to “Always in Internet Explorer in the desktop”. The side effect of that setting is that things like mail and whatnot will now launch in the desktop version of IE 10, which you may or may not want. At least this gives you a starting point to try to troubleshoot and then you can decide what option works best for you.