Web scraper details
Japeto Bot is a web scraper designed to support our chatbot platform. This page explains what our scraper does, why you’re seeing it in your logs, and how to manage it.
Why are we scraping your website?
Chatbot link previews
Chatbots built by our customers sometimes contain links in the chatbot’s website. When we detect links in the chatbot’s response, we visit the link to generate a preview of the site’s content, such as the title, description and an example image.
We store the results of this visit for one day, so we visit each URL a maximum of once per day.
Building chatbots
When you build a chatbot using our service, you can optionally use our scraping service to automatically generate chatbot content by looking at your website.
We try to visit every page on your site, download the content, and use AI tools to turn this into chatbot content.
This only happens at the start of the chatbot creation process, and only with your permission.
Japeto.ai quote calculator
The quote calculator on our website can generate an estimate of how much we would charge to redevelop your website. This is based on the number of pages on your current site, so the bot visits your website’s sitemap to calculate the number of pages.
This only happens if you use our quote calculator and we do not proactively scrape websites to generate quotes.
What happens to the scraped data?
Page data collected for chatbot link previews are cached for one day and are deleted via a daily task, so they are kept for a maximum of two days.
Data scraped using the chatbot build process is stored for the time that you have a chatbot active with us, and this data is deleted along with the chatbot.
Scraped data to build your website quote using the quote calculator is only used to generated the quote and is never persistently stored.
User agent
Our bot will always identify itself using the below user agent string. Please note that the version may change over time.
Mozilla/5.0 (compatible; JapetoBot/1.0; +http://www.japeto.ai/japeto-bot) Managing access to our bot
You can disallow our bot from scraping your website using the following directive. We will also obey blanket directives not to crawl your website.
User-agent: JapetoBot
Disallow: /
We use cloud service providers for our bot using a variety of IP addresses which may be used by other cloud service customers, so please do not block our bot using IP address rules.