You Should Probably Pay Attention to Tokenizers
Last week I was helping a friend of mine to get one of his new apps off the ground. I can’t speak much about it at the moment,
other than like most apps nowadays it has some AI sprinkled over it. Ok, maybe a bit maybe more just a bit – depends on
the way you look at it, I suppose.
There is a Retrieval-augmented generation (RAG) hiding somewhere in most of the AI apps. RAG is still all the RAGe – it even has
its own Wikipedia page now! I’m not sure if anyone is tracking
how fast a term reaches th...
Read more at cybernetist.com