I had some trouble with enabling 128-bit WEP encryption on OpenWRT Kamikaze 7.09 (kernel 2.4). I'd run iwconfig and see "Encryption key:<too big>" but could not figure out how the key could be too big as I entered into "/etc/config/wireless" file, under "key" option, a 26-digit hexadecimal number as the WEP key.
As to be expected someone else ran into the same problem and discussed it in OpenWRT Forum and references a link with the solution. The solution did not work for me, though.
So I followed the instructions here to use WPA instead of WEP. It's working great. Don't know why WEP wouldn't work.
Afterthought: The router I am using is Linksys WRT54GL v1.1. I didn't try WPA2.