Here are some ideas to make caves harder.
Self explanitory. The ground could shake, and you have some time to go up and get to safety. If you hit by a cave in, you die. Only happens on a couple floors at a time, not the whole mine. Also, cave-ins happen more often and are bigger the lower down you are.
When mining rocks, you can rarely uncover lava, which pours out, deals damage, gets rid of items on the floor. Maybe, kills you unless you have some sort of heat-resistant armor, when armor crafting is done? also, if you have a heat-resistant bucket, or consistantly cool a bucket with water, you could take lava up to the town, and use it for somthing. (Smelting high-tier metals, pouring on people, fuel)
Natural traps, kinda like in terraria, randomly spawned in. There could be spike traps on the floor, arrow traps, lava traps, things like that. If you see it, you might be able to harvest materials from it? (like harvest poison arrows from an arrow trap)