Whenever you do house cleaning, you may be pursuing different goals such as finding your keys, socks, wallet, yesterday’s day or satisfying your mom. You’re parsing the littered with lumber room trying to find something like keys and have quick access to them next morning.

Apart from keys, you may be interested in other useful things that you harvest from the messy flat. You’re simply tired of the chaos in your house, and you want everything to be assorted, neat-looking and clean. Usually, your scraping tools are hands, broom, mop or vacuum if you’re really into automation.

While you clean, you can stumble upon some “Oh, here you are, I’ve been looking for it for ages” things and do a quick personal “price comparison” for these things. Maybe you even want to see if the old but your most favorite t-shirt can compete with the new one that you bought last week.

Sometimes, you can find out that the leg of your sofa is broken or this lamp on the table is barely functioning and also this shaky chair is already on the deathbed. So, you’re kind of looking for vulnerabilities in your house that you can fix.

This is what web scraping looks like in real life.

It can be a little bit more complex though, but it doesn’t stop people from doing it, because it’s all about collecting data that they can use for the variety of purposes.

Usually, web scraping can be done for:

  • Ranking pages;
  • Accessibility and vulnerability checks;
  • Extracting data from SERPs (keywords, rankings, etc.);
  • Collecting website data (products, prices, ratings, reviews, etc.);
  • Other purposes.

Web scraping challenges


The most high-level challenges with scraping web and especially e-commerce websites are in processing big datasets on large scales. For example, parsing one million products requires a lot of resources to complete successfully such as time, money, servers, proxies, etc.

A real-life room can be messy, that’s for sure, but usually, a pair of hands and enthusiastic spirit are enough to deal with the task, crawling a bunch of dirty socks. Scraping thousands or even millions of products on Amazon is different.

Usually, when you send a lot of queries to Amazon website you’ll face two main obstacles – you need to either solve captcha or your IP will simply get banned. The problem with parsing Amazon smoothly and effectively is that it has to be automatic and should avoid such brick walls.

When you crawl through the trash in your house, your wife will not send you captcha to continue, of course, if you’re not crawling her wardrobe. In this case, you’ll get killed on the spot.

This is similar to what Amazon does. Amazon doesn’t like their website to be crawled non-stop. You can only do that by sending queries with some time in between, or your scraper will hit captcha.

Is Scraping Amazon Data really that painful?

According to NYC Data Science Academy’s article, it’s very complicated to extract the products from Amazon categories because most of the scraper’s queries return no results even though they show HTTP response code of 200 which stands for “ok” status, meaning that the server gave a response.

In his research, Tom Hunter was trying to answer the question “Can you predict product sales by extracting Amazon product listing data?”. He says that he was quite successful getting some data that was enough to make some predictions and forecasting, however, it’s far from being compelling.

When you crawl Amazon products, your scraper will struggle with different key attributes and features that you should include to upgrade your scraper as you crawl or when something changes on the Amazon side. You must be able to pause and continue the process while not getting back to the start if you crawl without any API.

Another issue with getting Big Data from Amazon is that you need to deal with different layouts, for example, top-level categories, people also look for, sponsored products, etc.


The data that you extract may change depending on the layout, attributes, headers, etc. Also, there are tones of the same products from different vendors, and you need to be getting the correct ASIN and other information that may be useful for making better business decisions.

As you crawl these massive amounts of data, you need to store all this data somewhere. So, getting a database to save and access this data is necessary.

All in all, the biggest issues are IP blocks and captcha thing that you need to solve to have your crawling run smoothly. There a few solutions on the web for it. The first one is a using proxy. The problem with buying a lot of proxy servers is that all their IPs have already been spotted and blocked from everywhere. And only a small amount of them do actually work.

The main issues of crawling Amazon yourself:

  • Captcha and IP blocks;
  • You need to upgrade the scraper;
  • Different layouts, attributes and features of pages;
  • You should have many VPNs or proxies;
  • You need a database;
  • Legal issues (sometimes your persistent crawling may upset website owners);
  • Structuring data is difficult;
  • Other issues.

Solutions to scraping Amazon Data

If you’re building your own scraper, the solution is to have a lot of money and undying desire to solve any upcoming challenges to complete the process successfully and luckily automate things in the future. However, not everybody has development skills. And definitely not everybody wants to build their own scraping tools, simply because it’s out of their targeted niche and it also requires a lot of manual work first.

Some may only need to do competitors analysis, prices comparison, product sale forecastings, product URLs, reviews and ratings, etc.

Before creating anything yourself, I would recommend searching for already-existing solutions on the web which are APIs.

While there are a lot of comprehensive API guidelines explaining what APIs are and how they work, we will not dig into it today. However, what’s worth mentioning is that Amazon has its own official API which deals with all the above-mentioned issues but does it more effectively and smoothly.

However, it’s proven that some people are not always satisfied with official APIs and have an interest in looking for third-party APIs either because they want a simpler interface or they want to build their own software.


While some developers like scraping for sport or profit, business owners, especially retail and e-commerce businesses need to crawl Amazon to make prices comparison, forecasting product sales, estimating competition rate, etc. Creating your own scraper is a time-consuming, challenging process and is only for the most enthusiastic enthusiasts out there.

On the other hand, if you’re a decision-maker, you may be interested in APIs to hit your goals directly. Usually, APIs are doing pretty much the same work and solve the same issues, however, more effectively. While APIs may or may not always be cheaper than building your own scraper, they can definitely save time and nerves.