Google rolled out a new crawler named Google-CloudVertexBot and this bot aims to crawl for clients of a commercial nature who use Vertex AI.

The Google-CloudVertexBot added to Google’s crawler documentation serves to crawl sites only at the request of the site owners. This crawler also differentiates itself from other listed crawlers in Search Central that work with advertising or in by Google Search.

Google-CloudVertexBot Crawler: More on this crawler

It has been stated that in Vertex AI Agent Builder, different types of data stores have been contained. Furthermore, Google says a data store can contain a single data type. There are around six types of data and two types of site crawling. Advanced Website Indexing and Baisc Website Indexing are the two types of site crawling in Google-CloudVertexBot.

Furthermore, the Changelog also states that the Google-CloudVertexBot has been listed among Google crawlers to assist website owners in identifying ‘new crawler traffic’. The Google documentation also hints that crawlers do not index public websites. However, the changelog says otherwise. It states that website owners will be able to identify new crawler traffic.

Besides the new crawler introduction, Google recently launched its August 2024 core updates. The August 2024 Core updates are meant to foster functional content from smaller and independent publishers. The update has been stated to take a full month to roll out fully. It is targeted towards improving the search result quality by allowing people to look at ‘useful’ content.

Suggested:

Google August 2024 Core Update Has Been Released.

Enhancing SEO with Cross-Domain Canonicals: Google’s New Guidelines.

The post Google-CloudVertexBot: New Crawler by Google appeared first on Tricky Enough.

Google analyst, Gary Illyes made the alert in the most recent episode of Google’s Search Off The Record. Ilyes has stated in the podcast that the use of parameters may cause never-ending URLs for one page, which may create issues regarding site crawling.

URL Parameters: Limitless URLs and What It Entails

The statement made about URL parameters mostly relates to e-commerce platforms and websites. This is an issue that is quite common in e-commerce sites that employ URL parameters for various purposes such as filtering, tracking and sorting the products. The results as told by Ilyes turn ‘much more complicated’.

In the episode of the podcast, Ilyes states that while developers could technically add limitless URLs, the server simply ignores the URLs that do not change the reaction.

Ilyes has mentioned that URL parameters cause an infinite amount of URLs all on one page which could cause issues for crawlers in search engines. Problems arise with the use of these parameters including indexing, crawl budgeting, and issues relating to the site’s structure.

Crawling is a technical term used to describe the automatic access of a website and retrieval of data through software. Google also employ crawlers to collect various inputs such as images, videos and texts found on the internet. Google then analyses the input and stores it in Google’s index which acts as a sizeable database.

Suggested:

Enhancing SEO with Cross-Domain Canonicals: Google’s New Guidelines.

Country Code Domains Confirmed to Perform Better in Search Results.

The post URL Parameters Cautioned to Cause Crawl Problems appeared first on Tricky Enough.

Although this is generally a minor issue, it is a serious one for SEOs and publishers that rely on the documentation to implement firewall rules.

A website may unintentionally stop a valid Google crawler if the proper data is not noted.

Google Inspection Instrument

The documentation section on the Google Inspection Tool contains the error. This significant crawler is dispatched to a website in response to two requests.

Search Console’s URL inspection feature

Google’s system reacts with the Google Inspection Tool crawler when a user wishes to check in the search console whether a webpage is indexed or to request indexing.

Detailed results test

This test determines whether structured data is accurate and meets the criteria for a rich result, an improved search result. A specific crawler will be prompted to retrieve the webpage and examine the structured data if this test is used to prompt it.

The issue with Crawler User Agent Typo Error

Websites that are behind a paywall yet whitelist particular robots. The Google InspectionTool user agent may run into difficulty because of this.

Incorrect user agent identification can also cause issues if the CMS must utilize robots.txt. Or a robot’s meta directive to prevent Google from seeing pages it shouldn’t be looking at.

To prevent bots from indexing certain sites. Some forum content management systems delete links to things like the user signup page, user profiles, and the search feature.

If you or a client are adding Google’s crawlers to a whitelist or preventing crawlers. From accessing particular webpages, be sure to update the relevant robots.txt, meta robots directives, or CMS code. Compare the current version with the previous one (via Internet Archive Wayback Machine).

Suggested:

Google Declares That its Sites are “Not Ideal For SEO Purposes”.

Google’s Guidelines For Paid Guest Posts.

The post Google Corrects a Typo in The Documentation For The Crawler appeared first on Tricky Enough.