Why Don’t Satellites in Low Earth Orbit Fall Straight Down?

I understand that gravity is still strong in low Earth orbit, so I’m puzzled by the common explanation that satellites are “falling around” Earth. If gravity is pulling them down, why don’t they simply drop back through the atmosphere?

Is their sideways speed the whole explanation, or do orbital height and atmospheric drag matter just as much? A clear everyday explanation of how low Earth orbit works would help me connect the physics to what satellites actually do.