We are experiencing this issue for a user that has read access to the configuration list. It is happening on a vLookup to a folder in a document library. The user can access the document library folder.
The dev tools has one 404 error, but that is for their profile photo (Request URL: https://graph.microsoft.com/v1.0/users/redacted/photo/$value). Everything else is 200.