- let us say I want to index every blog ever listed on HN
- should be a small subset of the 400 billion pages out there on the internet no?
- First I need to gather data, what do you use to load so many webpages rapidly? asyncio with aiohttp in python? are there better options?
- how do you handle proxies? rotation? are there libraries you recommend for this?
- what about pages that use cloudflare? or block your request or present a captcha or a challenge of some kind?
- what are the filetypes you collect? only html or media as well?
- where and in what format do you store all these collected files? flat file storage? duckdb? postgres? hstore? something else?
- what is the frequency at which you refresh each page? once a day? once a week? something else?
- what kind of pre-processing do you use on the collected data? remove extra spaces? special characters? some kind of complex regex pipeline? LLM?
- how do you match the incoming query with processed data? simple text matching? regex? vector embedding match? something else?
Others could be answered by examining the source code, as it is open source.
It's good etiquette to check existing documentation and resources before badgering OSS project teams with a lengthy list of questions.
- let us say I want to index every blog ever listed on HN
- should be a small subset of the 400 billion pages out there on the internet no?
- First I need to gather data, what do you use to load so many webpages rapidly? asyncio with aiohttp in python? are there better options?
- how do you handle proxies? rotation? are there libraries you recommend for this?
- what about pages that use cloudflare? or block your request or present a captcha or a challenge of some kind?
- what are the filetypes you collect? only html or media as well?
- where and in what format do you store all these collected files? flat file storage? duckdb? postgres? hstore? something else?
- what is the frequency at which you refresh each page? once a day? once a week? something else?
- what kind of pre-processing do you use on the collected data? remove extra spaces? special characters? some kind of complex regex pipeline? LLM?
- how do you match the incoming query with processed data? simple text matching? regex? vector embedding match? something else?