As others have touched on, there are other sources of information. Soil and ice cores have been mentioned, but old ships logs extend the record of directly measured recent air and water temperatures, and atmospheric pressures, for close to 200 years. Almost every scientific discipline has people involved in modelling historical temperatures for geological time spans. These include geologists, botanists, marine, terrestrial and atmospheric biologists, bacteriologists and my personal favourite, one I met is a paleopalynologist. All of these disciplines create models which, while not as accurate as direct measurement, can be calibrated reasonably accurately. One model by itself probably doesn't mean much, but so many models, based on different parameters, nearly all showing the same thing, are a pretty sound indication of what's happening.