Published August 10, 202534 min read A while back, I decided to undertake a project to challenge myself: build a web search engine from scratch. Aside from the fun deep dive opportunity, there were two motivators: Search engines seemed to be getting worse, with more SEO spam and less relevant quality content. Transformer-based text embedding models were taking off and showing amazing natural comprehension of language. A simple question I had was: why couldn't a search engine always result in top...