Revision #80

## IndexNow & SEO

This page covers your SEO control surface as a whole: the SEO tab on `/admin/branding`, your sitemap, your robots.txt, llms.txt, and IndexNow integration.

For the SEO tab deep dive (verification, editors, AI bot allowlist), see [SEO Tab](/know/admin-branding-seo).

---

## Where SEO lives

Every search-engine-facing setting lives on `/admin/branding`, SEO tab. The old `/admin/config?tab=crawlers` URL redirects there so existing bookmarks still work.

---

## Sitemap

Your tenant exposes a live sitemap at `/sitemap.xml`. Content tenants typically see 171+ URLs covering:

- Public wiki pages
- Public forum threads
- Community landing pages
- Static pages

The platform regenerates the sitemap automatically when content changes. To force a refresh, click **Refresh sitemap** on the SEO tab. The platform regenerates and pings IndexNow.

---

## robots.txt

Every tenant serves a sensible default robots.txt that blocks search results pages, user walls, and other non-crawlable surfaces. The platform appends a single `Sitemap:` line at the end.

To customize, paste your own robots.txt into the editor on the SEO tab. Click Save. Your robots.txt is live at `/robots.txt` immediately.

---

## llms.txt

`llms.txt` is the proposed standard for telling AI training crawlers what your site is about and what to read.

- **Auto-generation.** Leave the editor blank and the platform builds llms.txt from your public wiki pages and top forum threads on every request.
- **Override.** Paste your own llms.txt to take full control.

Your llms.txt lives at `/llms.txt`.

---

## AI bot allowlist

A single toggle controls whether AI training and inference crawlers can read your community.

- **Allow.** GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, and similar are allowed.
- **Block.** All known AI bots are disallowed.

The platform maintains the bot list. New entries appear on the next release.

Support communities and public-content tenants usually allow. Private or members-only communities usually block.

---

## IndexNow

IndexNow pushes new content URLs to Bing, Yandex, and Seznam for instant indexing. Configure at `/admin/indexnow`.

**Configuration:**

- **Enable/disable** toggles pushes on or off
- **API key.** Your unique IndexNow key (alphanumeric)
- **Key file.** The platform auto-creates `/{key}.txt` for verification when you submit a key

**Queue:**

- URLs queue automatically when content is created or edited
- A cron drains the queue every 5 minutes
- Manual flush available for bulk content imports

**Monitoring:**

- Pending count, sent (24h), failed, total queue
- Status per URL: sent, pending, or failed (with retry count and error)

---

## Search engine verification

Add verification tokens for Google Search Console and Bing Webmaster Tools on the SEO tab.

- **Google.** Paste the content value from your GSC HTML tag verification.
- **Bing.** Paste the content value from Bing's HTML tag verification.

For other tools (Pinterest, Yandex, Naver), paste their meta tags into the **Verification meta tags** textarea on the SEO tab. Up to 10 tags, one per line.

---

## Meta defaults

Configure on the SEO tab:

- **Meta title suffix.** Appended to every page title.
- **Meta description.** Default for pages without their own.
- **Open Graph image.** Default social share image. 1200x630 px recommended.
- **Canonical URLs.** Auto-generated for every page.
- **noindex.** Auto-applied to search results, user walls, and other non-crawlable surfaces.

---

## Access

SEO controls require Tenant Admin role or higher. IndexNow configuration requires Tenant Admin role or higher.