The goal is to give online creators an alternative to either blocking all crawlers or allowing them all in for just a handful of referrals.