Deep learning has revolutionized fields ranging from image recognition to natural language processing, but its theoretical foundations remain an open frontier. In this colloquium, we will explore how tools and concepts from physics provide a powerful lens for understanding the mechanisms and behavior of deep neural networks, as well as lead to practical engineering advances.