HACKER Q&A
📣 oriettaxx

Disappearing URLs: Issues with Institutional Websites in Google Search


In the past 20 days, my colleagues and I have noticed that many URLs from institutional websites, primarily those of public administrations like town halls, have disappeared from Google search results.

According to Google Search Console, these pages show the message: "Page is not indexed: Crawled - currently not indexed."

We have made adjustments to our robots.txt files and are currently waiting for results, but we still have many uncertainties. Has anyone else experienced something similar recently?

  👤 oriettaxx Accepted Answer ✓
Could it be that, due to the need to all have the same "design/theme", google interpret them as duplicates?

I also see that

rel="canonical"

in https://developers.google.com/search/docs/crawling-indexing/... (which is the first time I hear about): is it kind of necessary?

👤 bityard
It's been my experience that Google doesn't like to show results for pages that don't have anything for Google to sell, advertise, or track users with. It would make sense that they wouldn't bother indexing the as well.

My guess is that Google will happily index the page if you create a Google Analytics account and add their JavaScript snippet to all of the pages on your site.

👤 solardev
I wonder if civil servants are covering their asses and trying to mass erase stuff before the new administration comes in, maybe?