Technical SEO is the process of increasing profit, visibility and optimisation of your website. Good tech SEO improves your chances of getting on the first page of Google when someone searches for something you sell or that you provide information about.
Technical SEO involves optimising the processes of crawling, understanding, and indexing your pages to make it easier for search engines to access your content.
What is Technical SEO?
Technical SEO has always been around. However, technical SEO has become its own beast next to content and link building over the last five to seven years.
With Technical SEO these are the big must-haves that I will talk through:
- Crawling,
- Indexing,
- Rendering,
- SSL Certificates,
- Migrations,
- Content Optimisation.
Crawling
Crawling is a discovery process where search engines send out little teams of robots, also known as crawlers or spiders, to find new and updated content. This content can vary from images, blog posts, landing pages, or a PDF, but no matter what the content is, it is discovered with the help of links.
Crawlers start by fetching some web pages, and then they follow links on those webpages to find new URLs, hopping along paths of links. The XML sitemap is also used to discover content that website owners want to be indexed. Search engines like Google discover new content and add it to an index called Caffeine.
What is Caffeine?
Caffeine is a web indexing system that helps Google crawl data more efficiently. First released in June 2010, Caffeine allows the search engine to collect data and index it within seconds, meaning that fresher information is available from a broader range of sites.
Before introducing Caffeine, Google’s search index comprised of multiple levels, some of which were updated more quickly than others. Caffeine now provides 50% fresher results for web searches than the last index, meaning that your content can be crawled and indexed shortly after you publish it.
When and Why was Caffeine Introduced?
Caffeine was introduced by Google in 2009 and released on 8th June 2010. This update was so huge that Google offered months of what they called the “Developer Preview”. This preview gave SEO professionals and developers early access to deep dive and see if there were any issues with it.
When Google originally built the index back in 1998, there were only 2.4 million websites and 188 million people on the internet. By 2009, there were 100 times that amount, at a vast 238 million websites and 1.8 billion people online. The old index couldn’t handle that amount of data, so they needed to upgrade. That’s why Caffeine was designed.
Robots.txt file
A Robots.txt file tells search engines where they can and can’t go on your site. Google may crawl pages that they haven’t yet indexed.
It is important to understand that the robots.txt file should not be used to keep a page out of Google. The search engine may still crawl and index a page that you have disallowed in your robots.txt file if it is linked to from other places. This means if you link to the page from somewhere else on your website, or if someone else links to it from their website, then it may still be crawled and indexed.
The only way to keep a page out of Google’s index is to noindex the page using meta robots or by password protecting the page.
There’s also a crawl-delay directive that you can use in robots.txt. This is something that many crawlers support and lets you set how often they can crawl pages. However, Google doesn’t really like this, and for Google, you will have to manage your crawl rate in Google Search Console.
Check out this guide and flowchart, to help guide you understand how to determine how URLs could be removed from Google.
What Does Crawl Rate Mean?
Crawl rate refers to how many requests per second the Googlebot makes to your site when crawled.
To see what pages have been crawled by Google, the best tool to use is the Google Search Console Crawl Stats Report. This report gives you loads of information about how they’re crawling your website.
Suppose you want to see the crawl activity on your website. In that case, you’ll need access to your server logs, this is quite advanced, but if your hosting has a control panel, like cPanel, you should have access to raw logs. You’ll then need some aggregators like Awstats, Webalizer, and Screaming Frog. These tools allow you to analyse the data, parse and analyse the server log files, to produce HTML reports.
Indexing
Once the search engine has processed each of the pages it has crawled, it will compile a massive index of all the words it sees and their location on each page.
This extracted content is then stored, organised, and interpreted by the search engine’s algorithm. It does this to measure the importance and compares that to other similar pages.
It is essential to ensure that all pages that you want to show up in search results have been indexed. One of the ways to check this is to go to Google Search and search for: site:yourwebsite.com (remember to replace yourwebsite.com with your domain).
However, using this search operator is not an entirely reliable way to see if your content is indexed, as it may show some URLs that Google knows about but has not indexed (which could be for a variety of reasons). The only way to see with 100% accuracy what URLs are in Google’s index is to use Google Search Console. The URL inspection feature of GSC is helpful here too.
Rendering
Rendering is where the ‘spiders’ or Google bots retrieve your pages, run your code and then assess your content to further understand the layout and structure of your site.
Google will then use all information collected by the rendering process to rank the quality of your site and the value of your content compared to other sites and what people are searching for on Google.
Every webpage has two states, in which rendering occurs in between them. The two states are the initial HTML and then the rendered HTML. The initial HTML occurs first, and it’s a response to the server. It links to resources like Javascript, CSS, and images that need to build the page.
A Rendered HTML is more widely known as the DOM, which stands for Document Object Model. Every webpage has a DOM, and it presents the original HTML, but then it will display any changes that have been made from Javascript. You can view the DOM in the browser’s developer tools and then under the Console tab.
To increase your chances of ranking on Google, you have to make sure your website is indexable. But to be indexed, you have to be rendered. If Google can’t render the content, it is harder for Google to understand or evaluate your site.
SSL Certificates
SSL stands for Secure Sockets Layer; they are a security measure for your website. They consist of small data files installed onto your web server, and this then activates a padlock that will allow a secure connection from a web server to the browser.
SSL Certificates recently became free, but some websites still don’t use them despite the many benefits. Google has emphasised this, and SSL certificates are known to improve the visibility of your website. If you now have an SSL Certificate, it is seen as a positive factor and that your website is ‘trustworthy”.
Having an SSL Certificate and HTTPS connection is a basic necessity for ranking on SEO and a must-have for your site.
Migrations
A Migration is a process of making significant changes to your website, which in turn should help its online visibility. Migrations include changes like the domain name and extension, content, design, server, or more.
Migrations play a crucial role in SEO and the overall marketing effort; a mishandled migration can have massive effects on your organic traffic. Different migrations are dependent on the nature of the changes. Various types of migrations can be seen in the image below:
You may come across different terminology when doing your research, such as website relaunch/change/transfer, which refer to similar projects. The truth is that they are all interchangeable when using them with site migration since they cover the same aspects.
It is essential to differentiate between the different types of website migrations as they come in all shapes and sizes. The six most common are:
- Hybrid Migrations – This refers to, for example, a website migration that requires multiple changes at once, such as structural changes, content changes, or a platform change.
- Structural Changes – This is a vital aspect of your SEO, as the structure or architecture of your website will show Google which pages of your site are most important, typically through the use of a logical page structure with additional subcategories.
- Replatforming – Replatorming a website is where you move the current CMS (content management system) to a new CMS. It’s about re-positioning your website, app, and entire digital presence.
- Site Move Migrations – This is where you transfer your search engine ranking, authority, and indexing signals to reflect significant changes to your website or URL structure.
- Content Migrations – A process of moving information stored on a CMS to a new system. You would likely be moving your content onto a new system.
- Site Redesign – This is a detailed process of revamping your website, including updating content, refreshing layouts, and improving navigation; this should help with better conversions and site performance.
Content Optimisation
We all want to get our content to the largest audience. We can do this by ensuring the right keywords are present, adding meta descriptions and title tags while also making sure you have the relevant links and more.
How Do You Do It?
There are many ways to optimise your content, from optimisation of media, including videos, on-site optimisation, and even tweaking your text.
One method would be when creating different sections in your text, instead of changing your text to bold, actually make them into headers. You can choose many different headings, from H1 to H6, and you should use them in hierarchical order. This helps send signals to the search engine and can move you up in the search engine rankings.
Another way is optimising your title tag. Title tags tell the search engines about the content on your web page. It would look something like this: “<head><title>Example Title</title></head>”. You want it to be around 50-60 characters long with a format similar to “Primary Keyword – Secondary Keyword | Brand Name”.
Here is an example:
Optimising your images is also another way of optimising your content to its best potential. Google can’t read pictures themselves, but there are ways you can help them “read” your images.
Alt Text is fundamental in helping search engines know what they’re all about. So putting a short description with keywords relating to that image in the “Alt Text” section of your CMS will help you and your content in the long run.
Don’t overdo your keywords here, though; that is called “Keyword Stuffing”, which can negatively affect your rankings on Google.
Why is it Important?
Optimising your content is important because, without it, your content is a lot less likely to rank or be seen by your audience. Creating the correct content and optimising it for search should go hand in hand.
You want the content to look attractive to your audience; that’s creating your content, but then you want to make it appealing for the algorithms as well; that’s content optimisation.
Summary
This article was just scratching the surface of Technical SEO, but hopefully, it has helped you understand the basics and beginnings of tech SEO. There is much to learn, and it is really up to you how advanced you want to go.
Make sure to ask lots of questions, and don’t be afraid to do your own research and learn as much as you can. I would suggest going and checking out my website, as I have Guides on Yoast SEO and even a data studio dashboard if you’re looking for ways to visualise your data, such as Using Search Console Performance.
There are many ways you can broaden your technical SEO knowledge, but if you’re looking for consultancy, training or ongoing help, I would be more than happy to help you along your tech SEO journey.
