Unlike a search engine, which interprets a page’s meaning, the Sitelman simply announces existence and hierarchy. In doing so, it exerts immense power. By assigning a priority of 0.9 to a “Support” page and 0.3 to a “Legal Notice” page, the Sitelman shapes corporate priorities. By excluding pages with noindex tags from the sitemap entirely, it performs a kind of digital erasure—not deletion, but un-mapping .
Enter the first Sitelmen. These were human information architects and webmasters who manually crafted sitemap.html pages. They were the cartographers of the early web, listing every major section of a site in a hierarchical bullet-point list. The term "Sitelman" began as internal slang at early search engines like AltaVista and WebCrawler, describing the engineer responsible for ensuring a site’s structure could be fully indexed. It was a low-level but critical job: if the Sitelman failed, the search engine’s spider would wander aimlessly, never finding the hidden gems buried four clicks deep. sitelm
In the early days of the Web, sites were small. A personal homepage on GeoCities or a university faculty page might consist of a handful of HTML files linked together in a linear chain. Navigation was intuitive because scale was limited. But as the web exploded with the advent of e-commerce, news portals, and user-generated content, a problem emerged: . Unlike a search engine, which interprets a page’s
The future Sitelman will be an AI agent itself: a crawler that not only lists pages but also infers relationships, clusters content by latent topic, and presents a dynamic, multi-perspective map of a digital property. It will ask not just “What pages exist?” but “What conceptual territories are here, and how do they overlap?” By excluding pages with noindex tags from the