Given that I expose the Lazy<T> in a public property. Should I avoid such a design if lazy-initialization might throw an exogenous exception? 
The exception would be thrown when accessing the Value property. And property getters should not throw exceptions. On the other hand, the Lazy<T> is crafted and well documented in that area of passing initialization exceptions to the Value property getter.
                        
It's better in this case to expose a method and let the laziness be an implementation detail.
Don't implement a public property using lazy initialization in case this could throw an exception, for the same reasons as given in the property design guideline.