I knew Inventor could load quicker than this, so I kept messing with my settings. Finally, I stumbled upon something that seemed to help.
When you hit the 'File>Open' option in Inventor, click on the options button.

The default open options will come up.

The other option is to load will all components suppressed. This will load the assembly the quickest. You can then right click on selected assemblies and components to load them up individually. This can really help when assemblies get really large, and you only need to work on a small section.

That's it for now! I'm hoping to write up another section on creating your own Load on Demand views soon. Until then. Happy Halloween!