I think you're confused here, although I may have not made it clear enough.

X is not yet installed, nor are the nVidia X drivers, hence the paragraph that X are getting installed tonight. :P

Having said that though it was pretty silly of me to try using nvidiafb considering that I will be using the official nVidia drivers. And as mentioned about I am using vesafb. I should mention that this is an AMD64 system, so vesa-tng isn't an option.