Any online presence needs to first be seen, then assessed, before being appropriately ranked by the search engines. What this means is that when Google’s crawlers come looking for your site, it needs to be doing everything it can to get the best possible rating. In this blog we’re going to look at all the things you can do to get as high as you can up the search rankings. Let’s get going!
Making resources fully accessible
How Google's crawlers do their work:
Google’s objective is to index a site in its entirety, and it does so by sending out bots (or crawlers) that have a good route around and gather as much information as they can before they can grade each page. So far, so good; however, the bots aren’t humans, and they only have a limited capacity to read and understand what they are looking at. For example, there are elements of web pages they struggle to accurately assess and other assets that may not be fully accessible. In such cases, the bots will report back with an incomplete assessment of a page as they have incomplete data to work from.
The assets and resources the bots have the most trouble with are things like CSS files, images, and complex coding that potentially remain hidden from Google. The repercussions of this are that you might have lots of fancy images and snazzy coding on your site that you would logically assume is helping to boost your SEO, but in reality Google might not know it's even there. Potentially even worse, in the case of a blocked CSS file, Google’s bots may not be able to interpret the layout and navigation of your site, and actually rank it lower because of it, leading to reduced visibility and ultimately less organic traffic heading towards your site!
Can the Google bots see everything they need to?
Here’s a selection of detailed examples depicting instances where resources are blocked from the Google bots:
- Specific Content: If you have dynamic content such as JavaScript embedded somewhere in a page on your site, Google might not be able to fully access them.
- Granting permission: Google requires permission from servers to look at some files, so if your permissions aren’t up together then the Google bots might not be allowed to access them.
- Robots.txt config: Robots.txt controls crawling, and if it is not configured correctly, access might be restricted for the Google bots.
Four steps that will help fix your accessibility issues
Step 1 > Conduct a full site audit
We always recommend a full site audit before any changes are made to your site. ‘You don’t know what you don’t know’ is a very appropriate adage when it comes to website performance and SEO. There’s very helpful tools out there such as Google’s URL Inspection tool that will give you insights into how Google views and assesses your site as it currently stands. This should highlight the areas that are hidden and even recommend steps to take to remedy the situation.
Step 2 > Assess your Robots.txt
The robots.txt file associated with your website needs to be set correctly to allow access by the bots. It is true that some assets can be set to block in order to reduce the overall budget, but all the important resources should be configured to allow assessment, and this includes JavaScript and images.
Step 3 > Getting your Sitemap in order
The humble Sitemaps is the secret to ensuring Google sees the most important content on your website. Over time sitemaps can become a bit neglected and new pages etc aren’t always added. Remember that Google looks to your Sitemap for guidance, be sure it gets it when it does so.
Step 4 > Fully Readable Content
If you have dynamic content on any webpage it must be made accessible for the search engines. Server-side rendering (SSR) can be employed to allow visibility of dynamic content, plus there are other pre-rendering tools out there that can help with correct indexing.
Your Google Indexing Questions Answered
1. I don’t know what a ‘robots.txt’ file does?
A robots.txt file is a text file that informs and instructs search engine crawl bots how to find and access the various elements of a website. If used incorrectly it can actually block Google from assessing important information and even web pages, and that seriously harms SEO. Correct configuration of the robots.txt file is a fairly technical exercise so its important to get professional help from a skilled SEO provider such as Web Choice
2. Is there any way to tell if anything is hidden from Google that shouldn’t be?
There are various tools that you can easily find online such as the URL Inspection tool from Google's Search Console. A tool like this will highlight any problems and may give you hints and tips as to how to repair any issues.
3. I’m looking at my sitemap but I don’t know if it’s correct?
Your sitemap should have listed all the important pages that your website is composed of. Pages such as homepage, main subpages pages, and a contact page are hugely important for SEO, so make sure they feature on your sitemap. Once again, an SEO specialist will be able to help check your sitemap’s performance Web Choice
4. I’ve have CSS and JavaScript content on my site, is that affecting SEO?
If your site has CSS and JavaScript files that are blocked somehow, this can indeed affect SEO as Google can reduce your ranking. It can also have an impact on UX (the user experience) so it’s really important you get active content unblocked.
5.How should I implement dynamic content correctly?
Ensure all dynamic content is readily accessible by the search engine crawl bots by implementing server-side rendering, and by using pre-rendering tools. These measures will help Google to view and index your site properly so that your search engine rankings aren’t negatively affected.
For help with all the issues we’ve listed here you can assess further help and advice from a trusted web development company / web development agency, or SEO specialist. Don’t leave SEO to chance as your business could end up slipping down the page rankings which can be very costly to fix and take a long time to bounce back up the page again.
For help and advice from a local web development agency that has your best interests at heart, get in touch with the team at Web Choice right now.
