What is the easiest way to scrape websites today?

Question

With the rise of large language models (LLMs), what&rsquo;s the easiest way to scrape the web today? Additionally, what&rsquo;s the simplest method to automate interactions with a website?

fumanchu36 · Accepted Answer

The easiest way to scrape websites today in my opinion also features a simple way to automate interactions with websites. The scraping browser integrates with the 3 major headless browsers (Selenium, Playwright, Puppeteer) and lets you interact with your target site&rsquo;s HTML in a multi-step manner to extract the data you need. Try it here https://bit.ly/3STY9ON

eimrine · Answer

Maybe some headless browser and a lot of proxies? I don't see how neural networks affect downloading some webpages.

gbertb · Answer

check out https://spider.cloud - llm-friendly markdown support, crawl dozens of urls in secs, proxy and headless chrome etc

JSDevOps · Answer

If you are relying on scraping you are doing it wrong

throwaway211 · Answer

LLMs lower the bar for being correct.You don't need to scrape sites if you dictate your truth.Therefore read some psychology papers about how people create their realities. Then make stuff up; what gets believed is going to be mostly correct in your audience's reality, which should feed back into your model. The cost of getting stuff wrong is profit if you exploit it. Let your audience filter do it for you, and who are you to dictate what's correct for someone else anyway? Make it sound catchy and convincing for a variety of tastes. Generate a few short video creators with different personas like Street Fighter characters from yesteryear that viewers can project themselves on to.