How difficult is a search engine MVP that just works?

Question

what it is, i mean, what's stopping you from making the next google search?Tf-Idf? No. Just kidding. Infromation retrieval, text mining, ML/DL ?! what is going on with this field ! Every other resource seems outdated? What is the state of art ?Reading some of these posts : https://boyter.org/2010/08/build-vector-space-search-engine-python/https://www.dr-josiah.com/2010/07/building-search-engine-using-redis-and.htmlhttps://stevenloria.com/tf-idf/https://stories.algolia.com/a-search-engine-in-css-b5ec4e902e97

a3camero · Accepted Answer

I&rsquo;ve done this over the last year on a tiny scale for my own needs: gorillafind.com. It&rsquo;s from scratch and just for government sites to sidestep some of the challenges (but so far only has 50 sites). The cost per site is around $1/mo for crawling, indexing, converting file formats and then serving up results. It&rsquo;s difficult but not impossible and very educational. If you&rsquo;d like to hear more about doing it yourself and some of the challenges feel free to email me with the contact info on the site. My system isn&rsquo;t open source but I&rsquo;m more than happy to chat about the research I&rsquo;ve done and how you can make one.I&rsquo;d start off with not doing state of the art because it&rsquo;s overkill for an &ldquo;MVP&rdquo;. And if you don&rsquo;t need proper browser rendering of pages, there&rsquo;s open source crawlers out there like Nutch that might work. If you&rsquo;re making one yourself, the outdated academic papers and presentations by search companies are a good resource as the basic ideas of crawling and indexing haven&rsquo;t changed too much (even if ranking and other components have changed a lot). A search engine is really a set of related components and there are many examples out there to use as inspiration for your MVP.

ploika · Answer

Have you read this post?https://danluu.com/sounds-easy/It describes some of the various difficulties in building the next Google Search, much better than I could.

bluu00 · Answer

And to add, i am not looking for a "how to learn"?Neither it is some site-wide search que.And not a business model as well.Not a "privacy first but no results" search.Something that works.

ioli · Answer

I don't know how feasible it is, but couldn't you use ElasticSearch ???

techdragon · Answer

I built a &ldquo;search engine&rdquo; but unfortunately my approach lacked any implicit ranking mechanism.TLDR: my regex search engine needs result ranking to be more useful before I consider showing it to other humans.

How difficult is a search engine MVP that just works?

Have you read this post?
https://danluu.com/sounds-easy/
It describes some of the various difficulties in building the next Google Search, much better than I could.

And to add, i am not looking for a "how to learn"?
Neither it is some site-wide search que.
And not a business model as well.
Not a "privacy first but no results" search.
Something that works.

I don't know how feasible it is, but couldn't you use ElasticSearch ???

I built a “search engine” but unfortunately my approach lacked any implicit ranking mechanism.
TLDR: my regex search engine needs result ranking to be more useful before I consider showing it to other humans.