Skyrim has a pretty rudimentary stealth system - you equip a bow or a dagger, crouch behind someone, and one-shot them. If the body is found, nothing happens. You can even waltz in front of guards and avoid being caught, or just slot a bucket on someone's head. But modder Monitor144 has overhauled it, giving it much-needed depth and challenge.
There are a few major features to Book of Shadows with several of their own quirks - you have smoke bombs, animated masks, warhorns, sentries, and takedowns. To start with, if you're caught committing any crimes with a mask on, you accumulate a "Mask bounty", separate from your normal bounty. However, if you get caught pulling the mask down, both bounties will merge. You can craft four different colour variations - black, blue, brown, and red.
Smoke bombs meanwhile cloud enemy vision, even stunning them by making them cough. Everyone in the cloud will also be forced to move at the speed of walking. These are useful for quick getaways as you can cause mass confusion, rather than running around like a headless chicken as the random encounter thieves do in cities. These smoke bombs also come with colour variations much like the masks - red, green, blue, orange, grey, purple, yellow, and black.
Sentries are a little more complicated. These are NPCs that carry around warhorns they can sound to alert other enemies, essentially ringing an alarm bell to let everyone know, "Hey, there's a criminal doing some bad criminal shenanigans here". Before you know it, reinforcements will zone in on your location, punishing you more severely for getting caught. You can't just kill a sentry after they fire off the horn and slip into the shadows. Or at least, it'll be much more difficult.
However, you can pickpocket the horn off them beforehand to avoid it being sounded, and, as pictured above, you can easily make oeut sentries as the warhorn is equipped to their belt. If pickpocketing isn't your thing, though, you can always assassinate, choke, or stagger them to prevent it from being sounded. They are found among the Thalmor, Imperials, Stormcloaks, and bandits of Skyrim.
Finally, the mod adds takedowns, which are sneak attacks from behind that you can use when unarmed. It "Repurposes killmove behaviour and uses behaviour input to detect [takedowns]", though it doesn't work on essential NPCs and its currently in the experimental phase.