Skip to content

chore(flax): update flax for improved robots.txt

Ben Whitmore requested to merge update-flax-20240305 into main

To use this change on Flax.

Previously, only index.html was disallowed, and this was always disallowed. This meant:

  1. Theoretically, if someone linked to any of the draft pages from an external public website, a crawler could follow that link and end up crawling the entire draft site.
  2. In non-draft node, a crawler was disallowed from crawling the listing of articles, which would probably prevent it indexing the site.

This change fixes both issues.

Merge request reports