I want to build a small site which acts as a broad, searchable FAQ for a certain topic.
Consider I have the FAQ:
What is the approximate mass of Earth?
It's 5.9722 × 10^24 kilograms, wow!
I want the user to have a chance at finding this FAQ by asking How heavy is our planet
Looking at this basically, the two similar questions have only one shared word, "is", which is an extremely common word. So using something really simple like word comparison or even stemming/lemmatization alone won't help.
On the very other end of the spectrum, a search engine's AI feature can interpret this effectively, rephrase the question and give a similar answer. So, what strategies are are in-between these two extremes?
-
A few people will be adding questions to the site regularly.
-
If possible, no external services, just self-hosting on an affordable server.
-
Simpler and lighter solutions are preferred.
Are any of the features in OpenSearch (ElasticSearch/Lucene fork) able to do this? Is it overkill?
Since the site will have new questions to match regularly, will a solution require the repeated, wasteful retraining of NLP models to to create weights? Or is training so efficient for small-scale text datasets that it's responsible and reasonable to do on a cheap low-end server?
edit: Just spitballing here, I could try a solution which does the bulk work at insert-time rather than runtime, by asking a general pre-trained language model to rephrase the question many different ways, or generate keywords, then use those responses to generate tags for a basic keyword search to match. This would avoid making a heavy search function or retraining any model on the server.
Example result:
GPT-4o mini
Here’s a list of synonyms for the keywords in "What is the approximate mass of Earth?" formatted as an array of strings:
json
[
"weight",
"heaviness",
"bulk",
"load",
"volume",
"estimated",
"rough",
"approximal",
"near",
"close to",
"planet Earth",
"the globe",
"the world",
"Terra",
"our planet"
]
What Kind of database do you use and how do you search in it? Do you do your tokenisation manually or do you use a service for that?
If you happen to hand craft it, you should consider adding a field to your faq table that includes those variation phrases which you also use to select/find an answer to your FAQ.
The site hasn't been designed yet, so I can use whatever database is most appropriate. I'm only familiar with SQL and MongoDB so far, but this project shouldn't be a complicated design so I'm open to alternatives.
I can definitely add a field for that, and I suppose I could use even a lightweight langauge model (or hypothetically even a static thesaurus dataset) to populate it automatically, allowing for manual overrides by a user in case it suggests something irrelevant.