If I wasn't using an DI container, I wouldn't have to reference
EntityFramework library in my MVC3 app
Even when using a DI container, you don't have to let your MVC3 project reference Entity Framework, but you (implicitly) choose to do this by implementing the Composition Root (the startup path where you compose your object graphs) inside your MVC3 project. If you are very strict about protecting your architectural boundaries using assemblies, you can move your presentation logic to a different project.
When you move all MVC related logic (controllers, etc) from the startup project to a class library, it allows this presentation layer assembly to stay disconnected from the rest of the application. Your web application project itself will become a very thin shell with the required startup logic. The web application project will be the Composition Root that references all other assemblies.
Extracting the presentation logic to a class library can complicate things when working with MVC. It will be harder to wire everything up, since controllers are not in the startup project (while views, images, CSS files, must likely stay in the startup project). This is probably doable but will take more time to set up.
Because of the downsides I generally advice to just keep the Composition Root in the web project. Many developers don’t want their MVC assembly to depend on the DAL assembly, but that should not be a problem. Don't forget that assemblies are a deployment artifact; you split code into multiple assemblies to allow code to be deployed separately. An architectural layer on the other hand is a logical artifact. It's very well possible (and common) to have multiple layers in the same assembly.
In this case you'll end up having the Composition Root (layer) and the Presentation Layer in the same web application project (thus in the same assembly). And even though that assembly references the assembly containing the DAL, the Presentation Layer still does not reference the DAL—this is a big distinction.
Of course, when you do this, you're losing the ability for the compiler to check this architectural rule at compile time. But most architectural rules actually can't be checked by the compiler. In case you're afraid your team won't follow the architectural rules, I'd advise introducing code reviews, which is an important practice to increase code quality, consistency and improve the skills of a team. You can also use tools like NDepend (which is commercial), which help you verifying your architectural rules. When you integrate NDepend with your build process, it can warn you when somebody checked code in that violates such architectural rule.
You can read a more elaborate discussion on how the Composition Root works in chapter 4 of my book Dependency Injection, Principles, Practices, Patterns.