Spider control
More actions
This is a standard proposed by Orion Moonclaw to unify the behavior of public folder spiders/indexers such as RedX or Resdex.
While there has historically been some controversy over the development of such tools, mostly around their opt-out nature, this standard aims to be neutral and a net-positive for both opt-in and opt-out indexes by avoiding fragmentation and spreading awareness of the existence of control mechanisms.
User Guide
If you want your folder and all of its subfolders to be excluded from indexing, simply place an item inside it that has a slot on it named "scscdisallow". If you then want to explicitly allow a subfolder within that you can place an item with the "scscallow" rule on it.
You can use "scscdisallowthis" to disallow only the item that has the slot on it, this can be useful if you don't want the item to show up even if other people save it inside their public folders.
If you own a folder that contains things that are not allowed in public sessions as per Guidelines such as adult content, consider using the "scscunsafe" rule over a disallow, as that allows it to still be indexed by crawlers that intend to show adult content.
Rule application
The spider control protocol operates on the level of inventory paths. Every record in an inventory has one, for example "Inventory\My Public\Things". For the purposes of this protocol the path also includes the name of the record as the last segment, and from now on they will be referred to as "full paths", while paths that don't include the name will be referred to as the "parent paths". All paths are scoped to the record owner.
Rules apply recursively to the record with the specified path and all of its children, and the most specific rule always takes precedent (for instance having a folder with an allow rule inside a folder with an exclude rule will make that folder indexable while its parent remains hidden).
In situations where rules would conflict with each other, the more restrictive rule wins.
Rules
Rules are parsed from record tags with a magic prefix of "scsc" followed by the instruction word (with no spaces). For objects this can be achieved simply by creating a slot under it with the correct name, such as "scscdisallow" which will become a tag upon saving.
| instruction (including prefix) | description |
|---|---|
| scscallow | Allows its parent path. |
| scscdisallow | Disallows its parent path. |
| scscallowthis | Allows its full path. |
| scscdisallowthis | Disallows its full path. |
| scscunsafe | Marks its parent path as unsafe. |
| scscunsafethis | Marks its full path unsafe. |
| scscsafe | Marks its parent path as safe. |
| scscsafethis | Marks its full path as safe. |
To be considered compliant with the standard an indexer MUST respect all disallow rules and allow overriding them via more specific allow rules. An opt-in indexer MAY choose to treat all paths as disallowed unless an allow instruction is present.
Safe/unsafe rules are used to make it easier to provide a SafeSearch-style feature, for instance a folder with content that is forbidden in public sessions according to the Guidelines SHOULD be marked unsafe.
Legacy compatiblity
For compatibility with the standard first implemented by RedX, any record named ".noindex" MUST be treated as a disallow rule.
Implementation status
| Name | Type | Status | Notes |
|---|---|---|---|
| RedX | opt-out | Legacy-only | |
| Resdex | opt-out | Compliant |