You can do a partial clone without blobs like this:
git clone --filter=blob:none
This will make Git lazily download blobs when needed (when a checkout or diff is done for instance).
According to my tests the blobs remain downloaded afterward.
This means that if you were to walk in a lot of commits you would download a lot of blobs. Is it possible to ask Git to remove from its cache all blobs that are not currently in use (that are not needed to checkout the current commit)?
The equivalent with Git LFS would be git lfs prune.