Using Sikuli with Selenium Java for Image-Based Automation

Automating web applications with Selenium is now a standard in testing. However, Selenium has its limitations—especially when it comes to interacting with non-HTML elements like images, flash components, custom dialogs, or legacy Java applets. That’s where Sikuli comes into play. Sikuli is an image recognition automation tool that lets you automate anything you see on the screen using screenshots.

In this blog, we’ll explore how Sikuli integrates with Selenium in Java to create a powerful image-based automation framework.


What is Sikuli?

Sikuli is an open-source visual automation tool that uses image recognition to automate graphical user interfaces (GUIs). Instead of locating web elements using IDs or XPath (as in Selenium), Sikuli identifies elements based on screenshots.

You provide an image of a button or text field, and Sikuli searches for that image on the screen and interacts with it—just like a human would.


Why Combine Sikuli with Selenium?

While Selenium is excellent for automating browser-based UI elements, it cannot handle:

Captcha-based elements

Native OS file dialogs (like file upload/download windows)

Flash or Silverlight components

Images, charts, and non-DOM elements

Sikuli fills this gap by interacting with screen elements based on visual cues. When combined, Selenium handles DOM-based automation and Sikuli handles image-based operations.


Setting Up Sikuli with Selenium in Java

Step 1: Add Sikuli to Your Project

Download the SikuliX API from https://raiman.github.io/SikuliX1/ and include the JAR files in your Java project. If you’re using Maven, you might need to manually add the dependency.


Step 2: Import Packages

java


import org.openqa.selenium.WebDriver;

import org.openqa.selenium.chrome.ChromeDriver;

import org.sikuli.script.Pattern;

import org.sikuli.script.Screen;


Step 3: Write a Test Using Selenium + Sikuli

java


public class SeleniumSikuliTest {

    public static void main(String[] args) throws Exception {

        System.setProperty("webdriver.chrome.driver", "path/to/chromedriver");

        WebDriver driver = new ChromeDriver();


        driver.get("https://example.com/upload");


        // Trigger file upload button (Selenium)

        driver.findElement(By.id("uploadBtn")).click();


        // Wait for native dialog and handle it with Sikuli

        Screen screen = new Screen();

        Pattern fileInput = new Pattern("C:\\images\\file_input.png");

        Pattern openButton = new Pattern("C:\\images\\open_button.png");


        screen.wait(fileInput, 10);

        screen.type(fileInput, "C:\\path\\to\\file.txt");

        screen.click(openButton);

    }

}

In this example, Selenium clicks the upload button, and Sikuli interacts with the OS-level file dialog using stored screenshots.


Best Practices

Use high-resolution, clear screenshots for accurate detection.

Store images in a separate folder and name them descriptively.

Combine wait() with timeouts to prevent unnecessary test failures.

Use Sikuli only when Selenium cannot handle the task.


Limitations

Sikuli scripts may break with screen resolution or UI changes.

It’s dependent on the foreground screen—multi-tasking during execution may interfere.

Slower than DOM-based interactions.


Conclusion

By combining Selenium and Sikuli, testers can automate both web-based and GUI-based components seamlessly. This hybrid approach is especially valuable for applications involving native OS interactions or legacy systems. With proper setup and clear image patterns, Sikuli adds an extra layer of power to your automation toolkit.


 
Learn Selenium with Java Training

Read More: Capturing Network Traffic Using BrowserMob Proxy with Selenium Java
Read More: Using Robot Class in Selenium Java for Keyboard and Mouse Actions
Read More: Automating Drag and Drop Actions in Selenium WebDriver Java




Visit Our IHUB Talent Institute Hyderabad
Get Direction 

Comments

Post a Comment

Popular posts from this blog

How to Use Tosca's Test Configuration Parameters

Tosca by Tricentis is a powerful model-based test automation tool widely used in enterprise environments for end-to-end testing. One of its most useful features is Test Configuration Parameters (TCPs), which provide flexibility and control in managing test execution. TCPs enable testers to create reusable test cases by passing dynamic values without hardcoding them. This significantly enhances test maintenance, scalability, and efficiency. In this blog, we’ll explore what Test Configuration Parameters are, how to create and use them, and the best practices to follow when working with TCPs in Tosca. What Are Test Configuration Parameters? Test Configuration Parameters are custom-defined key-value pairs that can be associated with test cases, test case folders, or execution lists in Tosca. They help manage configuration data such as environment URLs, browser types, credentials, or any other variable data that may differ across test runs. By using TCPs, you can avoid duplicating test case...
Read more

Using Hibernate ORM for Fullstack Java Data Management

In the world of fullstack Java development, managing data efficiently and seamlessly across the application is critical. One of the most powerful tools in the Java ecosystem for handling database operations is Hibernate ORM (Object-Relational Mapping). Hibernate simplifies the interaction between Java applications and relational databases, reducing boilerplate code and improving maintainability. This blog explores how Hibernate ORM works, why it’s widely used, and how it fits into the data management strategy of a fullstack Java application. What is Hibernate ORM? Hibernate ORM is an open-source Java framework that provides a powerful and flexible mechanism for mapping Java objects to database tables. It abstracts the low-level JDBC code, allowing developers to work with high-level object-oriented concepts. Instead of writing raw SQL queries, Hibernate allows developers to define database interactions using simple Java classes (also known as POJOs) and annotations or XML configurations...
Read more

Creating a Test Execution Report with Charts in Playwright

Modern test automation isn't just about running tests—it's also about presenting clear, visual feedback on test results. When working with Playwright, one of the most powerful end-to-end testing tools for modern web apps, generating detailed execution reports can help your team track progress, identify issues quickly, and make data-driven decisions. A visual report with charts brings your testing efforts to life. In this blog, we’ll walk through how to create a test execution report with charts using Playwright’s built-in capabilities along with third-party tools for enhanced visualization. Why Visual Reports Matter Plain text logs and console outputs are hard to parse at scale. A good report should answer questions like: How many tests passed or failed? What’s the overall success rate? Which tests are flaky or slow? How have test results changed over time? Visualizing this data with charts—like pie charts, bar graphs, or timelines—helps QA teams and stakeholders better underst...
Read more