Hello friends! at times during automation, we are required to fetch all the links present on a webpage. Also, this is one of the most frequent requirements of web-scrapping. In this tutorial, we will learn to fetch all the links present on a webpage by using tagname locator.
If you have a basic understanding of HTML, you must be aware of the fact that all hyperlinks are of type anchor tag or ‘a’.
<a href="selenium-introduction.html">Selenium Introduction</a>
Content
How to fetch all the links on a webpage?
- Navigate to the desired webpage
- Get list of WebElements with tagname ‘a’ using driver.findElements()-
List<WebElement> allLinks = driver.findElements(By.tagName(“a”)); - Traverse through the list using for-each loop
- Print the link text using getText() along with its address using getAttribute(“href”)
System.out.println(link.getText() + ” – ” + link.getAttribute(“href”));
Sample Code to Scrape Links
package seleniumTutorials;
import java.util.List;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
public class GetAllLinks {
public static void main(String[] args){
WebDriver driver = new FirefoxDriver();
//Launching sample website
driver.get("https://artoftesting.com/sampleSiteForSelenium");
driver.manage().window().maximize();
//Get list of web-elements with tagName - a
List<WebElement> allLinks = driver.findElements(By.tagName("a"));
//Traversing through the list and printing its text along with link address
for(WebElement link:allLinks){
System.out.println(link.getText() + " - " + link.getAttribute("href"));
}
//Commenting driver.quit() for user to easily verify the links
//driver.quit();
}
}
Kuldeep is the founder and lead author of ArtOfTesting. He is skilled in test automation, performance testing, big data, and CI-CD. He brings his decade of experience to his current role where he is dedicated to educating the QA professionals. You can connect with him on LinkedIn.
Hi Kuldeep, in the above sample code, if I want to print the below format
System.out.println(index value of the link +”:”+ linktext), how can i do it?
Hi Chandra,
Try below lines for expected result
for(int i=0;i<allLinks.size();i++){
System.out.println(i +" : "+ allLinks.get(i).getText());
}
Thank you so much.
List jobLinks =driver.findElements(By.tagName(“loc”));
List texts = new ArrayList();
for (WebElement element : jobLinks) {
texts.add(element.getText());
}
for(String text:texts) {
What if I want to click and open the link
There are different ways to do this. You can store all the links and then navigate to each URL.
List jobLinks =driver.findElements(By.tagName(“loc”));
List texts = new ArrayList();
for (WebElement element : jobLinks) {
texts.add(element.getText());
}
for(String text:texts) {
if I want handle one of the link so which code is work
How to validate title of each link after link get opened in new tab
How do I ignore random links and only want to get certain links from a div?
You have to write a more precise locator instead of just the ‘anchor’ tag.
how to check for 404 in footer ,and also how to naviagte back from other domain to main domain
Thankyou so much , its very clear and easy to understand..