TrueSignal crawler

If you found this page from a User-Agent string in your server logs or your WAF dashboard, this page tells you what the crawler is, how it behaves, and how to allow or block it.

What it is

TrueSignal operates a crawler, ts-crawler, that reads a business's own public website to gather facts about that business — how long it has been operating, what services it offers, where it works — so that the business can confirm those facts rather than retype them.

It reads only pages that are publicly available on the site, and every fact it records cites the page it was read from.

How to identify it

Every request it makes carries this User-Agent, verbatim:

ts-crawler/0.1 (+https://usetruesignal.com/crawler; contact hello@usetruesignal.com)

It sends no other User-Agent. It does not present itself as a browser, and it does not rotate or vary this string.

What it does

• Respects robots.txt. It fetches robots.txt before anything else and does not request a path that is disallowed for it. A site-wide disallow ends the crawl with no pages fetched.
• Paces its requests. At least a quarter of a second between requests to the same host, and no more than four connections open to that host at once.
• Stops. It reads a bounded number of pages per site — typically a few dozen — and then stops. A crawl is a single pass, not a continuous re-crawl, and the result is cached for several days.

What it does not do

• It does not work around bot mitigation. No disguised User-Agent, no rotating identities, no headless browser imitating a person, no retries designed to get past a rule. If a site refuses us — a 403, a challenge page, a block page — we record the refusal and stop.
• It does not log in or submit forms. It does not create accounts, post data, or reach anything behind authentication.
• It does not retain raw page HTML. The HTML is parsed and discarded inside the request that fetched it. What is kept is the extracted text and the facts read from it, each with the URL it came from, for about a week — after which it is deleted.
• It does not render JavaScript. Content that arrives by script is simply not read.

If you want to allow it

Allowlist by User-Agent. In most WAF and bot-mitigation products this is a rule that skips or bypasses the challenge when the User-Agent contains:

ts-crawler

We are not on the verified-bot lists that Cloudflare, Akamai and similar vendors ship by default, so if your site challenges unrecognized agents, an explicit rule is what lets us through.

If you would rather block it

Add this to your robots.txt:

User-agent: ts-crawler
Disallow: /

It is read on every crawl and honored. Blocking the User-Agent at your edge works too, and we will record it as a refusal and stop rather than try another route.

Contact

If the crawler is causing a problem on your site, if you want it stopped immediately, or if you just want to know what it read, email us:

hello@usetruesignal.com

A person reads that address.