HACKER Q&A
📣 iceonmars

What tools do you use for web scraping?


I mostly use python. I have tried Scrapy, it's nice but I think it has much overhead for some simple tasks and it's not much easy to learn imo. What tools do you use? Also is there any api to get js rendered content of the pages? I can use headless chrome but I want to know if there's a more lightweight way (despite ajax calls). And my last question, what do you use to overcome captchas? Thanks


  👤 pinewurst Accepted Answer ✓
Did you try Beautiful Soup? I found that super easy to use.