During the last glacial period, the Patagonian Ice Sheet grew along the southern Andes, leaving marks on the landscape showing its former extents and timing. We use paleoclimate and ice sheet models to replicate its glacial history. We find that errors in the model-based ice sheet are likely induced by imprecise reconstructions of air temperature due to poorly resolved Andean topography in global climate models, while a fitting regional climate history is only captured by local sediment records.