How are bots finding my (brand new) site?

Question

I have just put a site live & I can basically guarantee no back-links point to my site and I, the creator, cannot conjure up any google search to return it at all (it does not appear to be indexed - I have done no SEO whatsoever). It does have DNS records, just no way to find it from a search engine or crawl for links to it.However, I can tell from the logs that bots are already probing for vulnerabilities.How on earth do bots even find such a target?

dmlittle · Accepted Answer

They might be keying off SSL certs created in the past few days/hours. For example, this cert[1] for www.krsnalite.com was created a few minutes ago.[1] https://crt.sh/?id=3120549998

gregjor · Answer

How do ants find a piece of fruit I left on the table?How do Jehovah&rsquo;s Witnesses find me no matter where I live?Finite address space, lots of time for bots to go door to door.

terrycody · Answer

There is a domain pool which contains all domains in the world registered and many people/companies in the world will scrape the list 24/7 to filter sth to meet their own needs.
Except this, there are a lot of possible reasons you may expose your website to a bot:
1) if you choose wordpress, when your site is a alive, site will Ping some services. 2) when your site is live, your site's IP is fixed, thus may get scanned somehow. 3) when you register your domain, your provider know your domain, the name can expose to somewhere. 4) if your domain name happen to be a "reused name", your website is exposed.
There are many many possible ways I don't even know, but in a severe case, people will receive spam email less than 1 hour after they bought their domain, you guess it.
Nothing is hidden today.

gvb · Answer

The bots scan the entire public IP address space for open ports.

elliottinvent · Answer

Another possibility to throw into the mix is that the domain is now in a public zone file, like the one ICANN[1], Nominet and others provide:1. https://czds.icann.org/

thephyber · Answer

Brute force IP crawling (this is how Shodan.io works).
SSL Cert transparency logs.
Remember that bot nets and other malware allow the type of scale that you wouldn't expect. There is a big advantage for the first mover in hacking. I wouldn't be surprised if nation states or cyber mafias paid bounties to lots of contractors for identified vulnerable boxes to use as intermediate proxies (although there are so many I'm not sure if bounties would pay well).

achairapart · Answer

Also take a look at things like browser extensions. It's not uncommon that they phone home every URL you visit, then crawl them for whatever reason.

mattmanser · Answer

They watch for new DNS entries and then ping it.It's the reason why Google have their own DNS registrar, they get told about all new websites.

ryanmccullagh · Answer

IP addresses are recycled.

How are bots finding my (brand new) site?

They might be keying off SSL certs created in the past few days/hours. For example, this cert[1] for www.krsnalite.com was created a few minutes ago.
[1] https://crt.sh/?id=3120549998

How do ants find a piece of fruit I left on the table?
How do Jehovah’s Witnesses find me no matter where I live?
Finite address space, lots of time for bots to go door to door.

The bots scan the entire public IP address space for open ports.

Another possibility to throw into the mix is that the domain is now in a public zone file, like the one ICANN[1], Nominet and others provide:
1. https://czds.icann.org/

Also take a look at things like browser extensions. It's not uncommon that they phone home every URL you visit, then crawl them for whatever reason.

They watch for new DNS entries and then ping it.
It's the reason why Google have their own DNS registrar, they get told about all new websites.

IP addresses are recycled.