As per the comment under this answer, references were introduced primarily to support operator overloading which quotes Bjarne Stroustrup:
References were introduced primarily to support operator overloading. C passes every function argument by value, and where passing an object by value would be inefficient or inappropriate the user can pass a pointer. This strategy doesn’t work where operator overloading is used. In that case, notational convenience is essential so that a user cannot be expected to insert address− of operators if the objects are large.
Which implies that operator overloading can't work with pointer. But it doesn't clearly explain why operator overloading with pointers can't work. Why wouldn't operator overloading work for pointers?
IMO where references are used, pointers can also be used in its place.