Yes. Only the cardset admin can do it, obviously. It's a link from the cardset's front page; e.g. this page for the Conversation set. Be careful, as it's completely irreversible; even I can't get the data back if you change your mind. (For a less drastic approach, you could consider editing the cardset's settings so only admins can see it. That'll remove it from the recent activity link for everyone except you.)