Kill someone on the Gold Coast and you'll get a letter inviting you to a guild of likeminded murderers; take out an innocent in Cyrodiil and you'll wake up to a sketchy figure in black robes, waiting in your bedroom, and if you steal the Dark Brotherhood's contract in Skyrim, they'll kidnap you in your sleep. All lead to an invitation, but what if you decline? Not in-game - ignoring it and moving on with your life - but lore-wise, as in what would happen to any character? That's what Reddit has been discussing on r/teslore.
One theory suggests that you yourself would be killed unless the Night Mother suggested otherwise. Or, in Astrid's case - who has abandoned the Night Mother and other Dark Brotherhood traditions - she'd simply kill you out of her need to be in control.
As it turns out, a lorebook titled A True History of the Night Mother has some answers that corroborate that idea. This book was written by someone who "witnessed the customs of [the] dark brotherhood back when [the] night mother was alive," zeclem_ said. "He was given that privilege under the condition of keeping it an absolute secret. He didn't, since we have the book, and editors note of the book says he got murdered". Clearly, the Dark Brotherhood don't expect you to spill their secrets out of fear, but if you do, they'll kill you.
Others point out that, in Oblivion's case, it's unlikely anyone contacted by the Dark Brotherhood would be in danger if they ignored them, given that you're approached by an agent in robes while you sleep. No information is given about hideouts, members, targets, or anything else, meaning you'd have nothing to hand into the guards or to use against the Brotherhood.
Not to mention, everyone knows about the Brotherhood. For a shadow-y assassin organisation, they aren't very hidden, so unless you had any vital information that could completely ruin them, you're not much of a threat. In summary, the likely answer is, Astrid would off you, but the traditional Dark Brotherhood wouldn't be all that fussed if you said no.