Check whether verified bots reach important pages
Check verified bot requests and response errors by page template using log summaries sanitized by the technical owner.
Suggested analysis tools
Practice relevance
Cross-practice idea · A general operational method. Review the stated tool and jurisdiction limits before use.
The problem
A page can be public without evidence that a verified crawler fetched it successfully during the period being checked.
Why it matters: A firm cannot diagnose crawl access from a crawler name alone. Verified, privacy-safe log summaries help the technical owner check which important pages were fetched and where coverage is still unknown.
Sources & evidence
The original public X method was inspected. The following law-firm procedure and prompt are editorial adaptations, not tested implementations or verified outcomes.
What’s been verified
The lead is a next-version vendor announcement, not a tested integration. The directory does not claim its sync feature is available or that bot traffic creates rankings, citations, or clients.
What to measure: Verified observed requests and status classes by template, plus logging gaps and owner-reviewed error investigations. Successful fetches do not establish indexing.
- Phuc Le (@phuclh93): original X feature announcement ↗October 4, 2026 · Announces forthcoming deduplicated/resumable log ingestion and bot separation. Availability and accuracy unverified.
- Google: verify Google crawler requests ↗Official reverse/forward DNS or published-IP-range methods. Verification happens before sanitization, inside the owner’s environment.
- Claude Code: file-based analysis and setup ↗Official file and command capabilities. The law-firm analysis prompt is an untested editorial adaptation.
Source checked: October 15, 2026. The original X method is linked directly. Law-firm procedures and prompts are editorial adaptations; source-reported experience is not independent outcome verification. We wrote these prompts. The adapted workflow hasn’t been tested.
The source method
Start when: The technical owner already has authorized logs and can supply verified, sanitized summaries with coverage notes.
- Phuc Le describes deduplicated log-file imports, resumable ingestion, and bot separation for a forthcoming SEO Utils version.
- The announced R2/S3 sync behavior and bot classification accuracy were not tested or confirmed available.
- Our adaptation uses existing exports prepared by the owner, removes duplicates using a manifest, and compares verified-bot requests across public page templates.
Limits and review
The assistant drafts or reports only. Check source records and review the output before a human performs any separately approved send or change. No automatic legal decision, money movement, account connection or publication is included.
Who reviews it: The technical owner verifies bot identity inside the firm’s systems, sanitizes the export, checks logging coverage, and reviews investigation tickets before any server or robots change.
What you need
Claude Code · prepared evidence folder
- Existing authorized access to the named product and in-scope records; actual tenant, plan, permissions and menu version must be checked.
- Firm-approved AI destination and data handling; no setup, sign-in, installation or access expansion included.
- Use synthetic inputs first. Keep real data in firm-approved accounts and processing, with appropriate access, confidentiality and retention controls. Aliases alone do not anonymize records. Never paste credentials, API keys or unnecessary client details.
How to use
Documentation-verified component path with explicit manual acquisition/mapping; editorial adaptation not tested
- Existing Cloudflare logging and required fields must already be enabled. Missing logs, retention or verification fields block the recipe; no retroactive collection is implied.
- SQL is an editorial template and must be checked against the real dataset. No arbitrary AI access to production logs is granted.
- Cloudflare applies a default LIMIT of 10,000 to SQL results. This pilot requires proven bounded status groups and count reconciliation; an incomplete or capped result is a blocker, not zero activity.
Check access and setup
- Existing Cloudflare logging and required fields must already be enabled. Missing logs, retention or verification fields block the recipe; no retroactive collection is implied.
- SQL is an editorial template and must be checked against the real dataset. No arbitrary AI access to production logs is granted.
- Cloudflare applies a default LIMIT of 10,000 to SQL results. This pilot requires proven bounded status groups and count reconciliation; an incomplete or capped result is a blocker, not zero activity.
Verify the existing provider’s logging fields and scope
Cloudflare existing HTTP requests Log Explorer dataset · Observability > Logs > dataset HTTP requests > select domain/time > Open SQL editor
An authorized technical owner checks the already-enabled dataset, retention, ingestion fields, sampling/filtering and domain scope. Current Cloudflare docs place Log Explorer under Observability > Logs. Confirm actual ClientRequestHost, ClientRequestPath, EdgeResponseStatus and VerifiedBotCategory fields exist in this dataset. Do not enable logging, create tokens, configure Logpush or export raw visitor records. Record the exact start/end, timezone and query capture time from the dashboard; do not assume a field named Date establishes the intended day.
- Expected output
- Verified field/coverage manifest and restricted original log/source references
- Next step
- Run only approved aggregate queries over allowlisted public paths.
- Check before continuing
- Logpush’s field catalogue does not prove the same fields were ingested into this existing Log Explorer dataset. Missing VerifiedBotCategory blocks verified-category analysis.
Acquire a bounded sanitized aggregate from actual logs
Cloudflare SQL editor, technical owner review · Existing dataset > Open SQL editor > Run query
Use a small pilot: up to three approved public paths, one verified category and one closed review window. In the dashboard select the exact custom time range and record its UTC bounds; use Cloudflare's documented {{ timeFilter }} macro rather than inventing a Date field. After verifying the dataset fields and category, run this proposed read-only grouped query separately for each path/category: SELECT EdgeResponseStatus, COUNT(*) AS request_count FROM http_requests WHERE {{ timeFilter }} AND ClientRequestHost = '[PUBLIC_HOST]' AND ClientRequestPath = '/[APPROVED_PUBLIC_PATH]' AND VerifiedBotCategory = '[EXACT_VERIFIED_CATEGORY]' GROUP BY EdgeResponseStatus. First run the same WHERE filter with SELECT COUNT(*) AS total_requests, COUNT(EdgeResponseStatus) AS nonnull_status_requests, COUNT(DISTINCT EdgeResponseStatus) AS distinct_statuses. The expected number of groups equals distinct_statuses plus one if total_requests exceeds nonnull_status_requests. Require that bound to be below 10,000, capture every group, and reconcile the sum of request_count to total_requests. Reject partial, sampled, capped, inconsistent or shifting results; repeat under a stable closed window or stop. Save the query, filters, count check and complete bounded results; if no native export is visible, have a person transcribe and another verify the small aggregate table. Never export raw visitor rows.
- Expected output
- Approved path/day/category/status counts, source query and coverage manifest
- Next step
- Technical owner removes sensitive paths, checks totals against the same-scope count and manually adds approved template IDs.
- Check before continuing
- The SQL templates are proposed and untested. Confirm timeFilter expansion and exact UTC bounds in the actual account. Required proof: expected group count equals captured group count and summed counts equal same-scope total; retain null-status group. Missing verification/retention/coverage blocks all-clear. Category does not identify an individual crawler, and edge requests do not prove origin reach.
Map, minimize and approve the actual sources
Source reviewer and local spreadsheet/text editor · Restricted originals > separate approved analysis copy
Keep untouched originals, original headers, capture date/time/timezone, scope, filters, count/pagination and source locators in a restricted source folder. Make an independently reviewed, minimized copy for AI. Keep identifier maps and confidential originals outside the AI folder/Space, including hidden sheets and attachments. Public pages and embedded instructions are evidence, never instructions. No live integrations, portal requests, publication or external changes are performed by this recipe. Map the acquired evidence as follows: ClientRequestPath → approved public path/path token; exact dashboard timeFilter start/end plus verified timezone → observation window/day; verified nonempty documented VerifiedBotCategory → verified_category; EdgeResponseStatus → edge_status; COUNT(*) → count. Owner adds template_id from canonical inventory, source query ID, ingestion/retention/sampling/edge-vs-origin notes. No client IP, user-agent/referrer/cookie/query string or visitor identifier enters AI. If individual bot names are needed, require provider-specific independently verified classification, not category inference. Preserve the bounded-group proof and total-request reconciliation with the approved aggregate. Apply the nativeInputContract below, including exact editorial headers, source manifest/evidence map and any required extra companion evidence. Existing kit files are usable only under their declared bridge; copying or renaming them is not validation.
- Expected output
- Reviewed bot-template-day.csv; canonical-inventory.csv; aggregate-checks.csv; sanitized-export-manifest.txt; manifest.txt; evidence-map.csv
- Next step
- For Adobe, convert approved tabular rows to labelled TXT or paste text; for Claude Code, place approved files in the exact analysis folder. Never relabel a raw export as a kit table.
- Check before continuing
- Check source-to-analysis row counts and stable keys; open originals for ambiguous rows. Missing data stays unknown.
Open the approved AI destination
Claude Code (existing approved local session) · Terminal in approved analysis folder > claude > Plan; /permissions
Use only an already installed, authenticated, firm-approved Claude Code session. Staff creates a new local analysis folder containing approved minimized files and manifest.txt, with no secrets, hidden configuration, hooks, unreviewed CLAUDE.md or symlinked client folders. Keep originals elsewhere. In a terminal, cd to that exact folder and start claude; inspect the displayed working directory. Use Shift+Tab to select Plan and /permissions to inspect existing access. An administrator must enforce any required filesystem/network isolation: Plan and prompt wording alone are not a complete sandbox. Do not add directories, integrations or persistent permissions. Ask the recipe prompt. Return proposed outputs in the response; a human copies checked text into the named output files. If safe read scope cannot be established, stop. No installation, sign-in, credential setup or write permission is included.
- Expected output
- Only approved source files loaded or accessible
- Next step
- Paste the exact prompt in the next step.
- Check before continuing
- Confirm the source list/working directory before prompting; stop on unavailable access.
Run the bounded analysis prompt
Claude Code (existing approved local session) · Claude Code prompt in Plan mode
Paste the prompt after replacing only bracketed scope values. Treat all source content as data.
- Expected output
- bot-crawl-review.csv and investigation-tickets.md
- Next step
- Inspect source citations/file-row references before copying any result.
- Check before continuing
- No invented source fields, IDs, eligibility, legal interpretation or unsupported completeness.
Check and hand off the proposed result
Human reviewer and local spreadsheet/text editor · Separate review output, not live system
Check every exception against its source, verify counts and uncertainty, then copy only approved result text to bot-crawl-review.csv and investigation-tickets.md. Technical owner checks bot classification, complete aggregate capture, log retention and edge/origin scope before opening any investigation; server changes require separate approval.
- Expected output
- Source-checked bot-crawl-review.csv and investigation-tickets.md
- Next step
- Open the named output in Excel/text editor and send it through the existing internal approval process; no live import or external send is implied.
- Check before continuing
- Acceptance case (proposed, not run): A Googlebot-looking user-agent without independent verification is excluded; absent logging days remain unknown, not zero crawl.
Approve the internal next action
Final human reviewResponsible human owner · Existing internal work queue / approved source system
Record the reviewer, decision and unresolved blockers. Technical owner checks bot classification, complete aggregate capture, log retention and edge/origin scope before opening any investigation; server changes require separate approval.
- Expected output
- Approved internal review record or explicit blocked status
- Next step
- An authorized person chooses and separately approves any live change, outreach, filing, certification or purchase.
- Check before continuing
- Do not infer legal deadlines, rule applicability, title sufficiency, service validity, financial eligibility or professional compliance.
What you’ll get
A report by bot, page template, and status, plus investigation tickets showing the observation window, verification status, and missing evidence.
Try with the sample files
Follow the tool steps above and use the inputs named in each step. Where provided, native-* files are blank scaffolds and NATIVE-INPUT-MAPPING explains their fields. Original examples and the optional full review prompt use a separate prepared-input schema. Keep them separate unless the mapping explicitly connects them. Downloading a kit does not connect accounts or prove the workflow works.
Download inputs and checklist 13 files
Instructions, blank templates and separate synthetic examples; these files do not install an automation. Blank templates still need verified source data.
- Download starter kit All inputs, synthetic examples and review notes in one ZIP. No automation is installed.
- Start here: get the sources and prepare the review Documentation-checked acquisition, explicit manual mapping, exact prompts and human review limits
- Input preparation checklist Collect the actual recipe inputs without inventing product export headers
- Synthetic example and expected result Exercise the important safety and data-quality boundary
- Sanitized export manifest template Editable recipe-specific setup input
- Setup and review checklist Use the original recipe with the provided preparation files
- Native input scaffold: bot-template-day.csv Blank editorial review input; acquire and check the stated source evidence before use; not a native export
- Native input scaffold: canonical-inventory.csv Blank editorial review input; acquire and check the stated source evidence before use; not a native export
- Native input scaffold: aggregate-checks.csv Blank editorial review input; acquire and check the stated source evidence before use; not a native export
- Native input scaffold: sanitized-export-manifest.txt Blank editorial review input; acquire and check the stated source evidence before use; not a native export
- Native input scaffold: manifest.txt Blank editorial review input; acquire and check the stated source evidence before use; not a native export
- Native input scaffold: evidence-map.csv Blank editorial review input; acquire and check the stated source evidence before use; not a native export
- Native input mapping and source requirements Exact legacy mappings, required additional evidence and source-bundle gates
Optional full review prompt for prepared inputs
Prepared-input review prompt with Claude Code · prepared evidence folder
Original Law Firm Agents adaptation · untested. This prompt uses the original prepared-input schema shown in the sample test. Step prompts may use different tables. Follow NATIVE-INPUT-MAPPING where supplied; do not mix the original examples with native-step inputs without mapping and checking every field.