Here's a concrete example of why the preprocessor matters deeply in this discussion.

If we build from source, we have to handle the (all too common) situation where a given library has an arbitrary set of build inputs (-D flags, compiler flags, etc).  If those aren't compatible, that'll be a problem.

If we force them to all be the same, then we have to worry about providing a mechanism to specify which flags (from an unbounded list) are required / compatible with a given library.  That's hard to extract from source.

If we instead allow each library to be built from different compiler invocations, we risk ODR.  Keep in mind that even something as simple as an assert() in a header is an ODR violation if we don't enforce consistent build flags: building that header with one definition of NDEBUG in one TU results in a different definition than one built without NDEBUG in another TU.  

In all honesty, the preprocessor and consistent build invocation is possibly the hard part.  If we come to agreement on those points, relying on existing package management solutions probably suffices.