The Difference Between Crawling, Indexing & Rendering (Non-Techie)

There’s no need to struggle with the differences between crawling, rendering, and indexing. Whether you’re trying to learn, or you need to explain to a client, your boss, or a new employee, here is how I explain them to clients.

Think of a baby who just learned to crawl.  You set them down at one side of the room and their favorite treat at the other.  Baby will take off from one side of the room to the other.

• Crawling is the baby going across the room to explore the other side and find their treat.
• Indexing are the things that get stuck to the baby like dust, loose carpet, and room debris.
• Rendering is what baby gets on their hands and then shows you what was found.

Search engine spiders do this too.  You put them into your website and they begin exploring.  They crawl your pages, index the ones you specify should be indexed (and sometimes ones they shouldn’t), and display what they can understand (render) when it is relevant.  And you can control the experience the spider has.  Lets go back to the baby example.

You line up some baby gates in a pattern like a maze to challenge the babies brain and thinking skills. At the end of the maze you have baby’s favorite treat and you’ve shown it to them. Now baby has an incentive to get there.  Along the way baby discovers new paths and eventually either quits trying to get their treat, or finds it and displays it to you by eating it.

• The path baby takes is crawling. (This is controlled by saying allow and disallow in robots.txt)
• The new locations and dead ends are pages baby begins to remember. (Meta robots tell spiders what to index and follow, and what to not index or follow)
• The treat at the end is finding the goal, a relevant page to answer a searcher’s question.

There’s one layer deeper, and that is rendering.  Maybe you show baby there are multiple treats like toys, snacks, and other things they like.  Now baby wants to go back into the maze.  Some are under see-through containers that are too heavy to lift, others are able to be seen but hard to get, and the final one at the end of the maze is out in the open.

• If baby can see the reward but cannot access it, it cannot display it to you.  Therefore it never gets used or exposed.
• If baby can access the reward but it takes effort, baby may look for another option that is easier to access. When you have excess code, slow load times, content buried under ads and fluff, search engines might choose an equal answer or page that displays it more efficiently.
• If baby can get the reward quickly and enjoy it without effort, chances are it will go for and engage with this option because it is the easiest one.  This type of page will normally win the top rankings and get the traffic.

And that’s a simple and non-technical way to look at the differences between crawling, indexing, and rendering for SEO.