I ran 180's back in the day as well, but on the 20/24", you're mostly standing and sprinting from corner to corner. Good leverage for out of the gate and accelerating quickly out of the turns, but I dunno if I'd want to be spinning those long cranks while sitting in the saddle for long stretches when you want to get a good ankle whip going. Just food for thought, as I've pondered this question as well. YMMV.
My guess is the knee pain is the extreme flexion/extension at the top (12/6 o'clock) of the stroke due to the longer cranks, if you don't have the bone length. Again, prolly not too much of a deal while standing but the saddle time is what kills it.
You ran a gear other than 44x16?
btw, I'm ~5' 10" /* edit - and run 175mm on my 26/29" bikes */