Managing the internets agentic middlemen

Date:
Tue, 05 May 2026 09:45:29 +0000

Description:
AI bots reshape digital discovery, forcing brands to rethink visibility, control, and competitive positioning online.

FULL STORY ======================================================================Copy link Facebook X Whatsapp Reddit Pinterest Flipboard Threads Email Share this article 0 Join the conversation Follow us Add us as a preferred source on Google Newsletter Subscribe to our newsletter AI bots are no longer on the fringes of the internet. They are integral actors that are reshaping traffic patterns and digital discovery. Far beyond simple crawlers, todays bots can execute complex, high-volume interactions at scale.

In the retail sector, AI bots are increasingly interacting with e-commerce websites as large language model providers expand their use of crawlers to gather and retrieve information. These bots can access product descriptions, pricing and availability data that may later be surfaced and prioritized through AI-powered search and assistant tools. Article continues below You
may like AI is redefining product discovery, structured data, and trust critical for visibility Why agentic AI and unified commerce will define ecommerce in 2026 Are Google's days numbered? Study finds AI bots are
becoming more 'search-like' - and it's affecting how brands are being seen online Richard Meeus Social Links Navigation

EMEA Director of Security Technology and Strategy at Akamai. As AI-driven discovery becomes more common, retailers are beginning to consider how this automated access could directly shape product visibility, customer journeys and competitive positioning.

However, with automated agents increasingly sitting between brands and consumers, the implications for performance, security and visibility cannot
be ignored. These agents often mimic real users, making it harder to distinguish legitimate automated traffic from malicious activity.

As the volume of AI-driven requests grows, organizations must also manage the strain this places on IT infrastructure while improving visibility into machine-to-machine interactions. The shift towards an agentic commercial landscape As the public grows more aware of AIs potential, bot traffic patterns have shifted from simple data scraping to sophisticated interaction. In the second half of 2025, a surge in AI fetchers and search crawlers signaled a shift toward everyday AI reliance. Are you a pro? Subscribe to our newsletter Sign up to the TechRadar Pro newsletter to get all the top news, opinion, features and guidance your business needs to succeed! Contact me
with news and offers from other Future brands Receive email from us on behalf of our trusted partners or sponsors By submitting your information you agree to the Terms & Conditions and Privacy Policy and are aged 16 or over.

By Q4, momentum had already pivoted toward AI training crawlers, highlighting a rapid ecosystem evolution as providers optimized model architectures to
meet shifting consumer demands.

This shift reveals the next phase of competition: the race to build
autonomous agents capable of executing entire transactions. For enterprises, this creates immense operational pressure, as they inadvertently provide the data fuel for models that could ultimately disintermediate them.

To manage this, businesses are increasingly adopting bot mitigation strategies, weighing the trade-offs between blocking bots to protect intellectual property and allowing them access to drive discovery. Effective mitigation now requires deep visibility into bot behavior. What to read next AIpowered commerce is here how ready are you for autonomous buying? AI and bots now account for more internet traffic than humans, report finds Agentic Search Optimization reshapes brand visibility in AI search

The right approach depends on whether an agent is driving commerce or siphoning away engagement.

However, this transition will not be seamless. AI agents are already disengaging from retailers that fail to provide high-precision, structured data . Moving forward, agentic commerce will only flourish where clear frameworks of trust and permission are established.

Businesses must treat these agents as a new class of participant, requiring explicit verification, differentiated access, and specific rules of
engagement to safely usher in the era of autonomous transactions. Visibility in the agentic age As the presence of AI bots accelerated throughout 2025, many organizations responded cautiously, limiting or blocking access while they assessed the implications.

But as AI assistants and agentic tools increasingly shape how users discover information, products, and services, the conversation is shifting from simple control to strategic visibility.

Rather than navigating directly to websites, users are increasingly relying
on AI systems to summarize options, compare products, and surface recommendations on their behalf.

This changes the mechanics of discovery. It is no longer enough for organizations to be indexed. They must also be interpretable, prioritized,
and usable within AI-generated outputs.

As a result, visibility is becoming less about driving traffic and more about influencing how and whether a brand appears in machine-mediated decisions.

This introduces a new tension between control and access, as organizations weigh the risks of exposing content to automated agents against the
commercial reality that, without that access, they may simply not appear at all.

In this environment, competition is no longer confined to search rankings,
but takes place within the answers and actions generated by AI systems themselves.

The next phase of digital discovery will be defined by how effectively organizations make their content accessible to AI systems.

Generative Engine Optimisation (GEO) focuses on ensuring content can be interpreted and used by generative AI models, while Answer Engine
Optimisation (AEO) ensures key information is structured so it can be
surfaced directly within AI-generated responses.

Together, these approaches represent the evolution of search optimization for an AI-mediated web.

At the same time, organizations will need clearer governance over automated traffic. Rather than blanket policies that either allow or block bots entirely, businesses must develop more nuanced approaches that distinguish trusted AI crawlers from other forms of automated activity while maintaining control over valuable digital assets.

As 2026 continues to unfold, organizations that treat AI bot traffic as strategic infrastructure, rather than background noise, will be best positioned to compete and be chosen in an internet increasingly navigated by machines. We've featured the best AI tools. This article was produced as part of TechRadar Pro Perspectives , our channel to feature the best and brightest minds in the technology industry today.

The views expressed here are those of the author and are not necessarily those of TechRadarPro or Future plc. If you are interested in contributing find out more here: https://www.techradar.com/pro/perspectives-how-to-submit



======================================================================
Link to news story: https://www.techradar.com/pro/managing-the-internets-agentic-middlemen


--- Mystic BBS v1.12 A49 (Linux/64)
* Origin: tqwNet Technology News (1337:1/100)