Get the most out of the framework you are using

To start things off, here is a common question you've probably heard or asked yourself: "What is the best programming language to learn?"

Now, here is the same question in automation testing: "What is the best framework for automation testing?"

You might have already guessed the answer: there is no best framework for testing (same goes for programming language). Let me rephrase this answer, to make the point more clear:

You are an essential part of the automation framework you chose. If you're a rookie programmer or you just started exploring automation testing, your tests will probably suck, and even if Jon Skeet himself wrote a magnificent testing framework just for you, it wouldn't make your tests suck any less until you deepen your understanding, grow and develop yourself. This is your primary objective.

Harsh truths aside, whatever framework you choose, you can only bring out the most out of it when you know what you are doing. You need to expand your knowledge about the framework you will use. This is the secondary objective. Don't mix these up! Don't go watching 40+ hour tutorials about selenium testing just yet.

Don't stress about picking the right framework. Your project will not fall apart if you pick "the wrong one". It is great if you are knowledgeable enough to make informed decisions about which framework to pick, but its just a tool and you shouldn't waste too much time on this decision. There is no wrong framework. There just might be a different one which will make development easier when dealing with certain situations.

This is why I wrote about you, the actual QA engineer, in the last article. This is why I wrote about the mindset you need to have to write good automation tests. In this article, I will write more about automation testing fundamentals, and give examples written in C# using the Atata framework. Why, you ask? Atata is written with fluent style, chainable methods, and has fluent assertions. Tests written with fluent style are almost like comprehensive sentences that can be understood by anyone.

Why do we even need frameworks, can't we just use selenium directly?

Yes, you can use selenium directly, but selenium was first written in 2009 and frontend design has evolved much since then. Web design philosophy and frontend components evolve all the time. Developers use them as pieces while building the frontend puzzle, so when testing, you want to use a modern framework that is in sync with current trends of development. It should expose a bunch of methods and switches which you can easily use to have full control of frontend components which are currently used everywhere.

Using selenium directly in this scenario is like multiplying 8-digit numbers by hand and refusing to use a calculator. Here is an even better example:

1. Use selenium to select the email and password inputs, fill them out, and press the sign-in button - this should be easy, just native selenium commands: sendKeys() and click()
   
   
2. When the sign-in button is clicked, a loading indicator will spin until some process finishes, and then the homepage loads - suddenly you have to use WebDriverWait, maybe you decide to wrap it somehow into some nice function, but a new junior programmer will have to spend significant time looking at this test to figure out what it does exactly.

Current frameworks are written so that this behavior is achieved by a few lines of code. They make your tests shorter! They make your tests easy to read, and I will explain why this is important.

Make your tests atomic

Atomic is just a fancy word for saying that the test should have a single responsibility and test a single feature. You can tell a test is atomic when:

• The test only has one assertion or two assertions at most. Because sometimes we need another assertion to make sure our state is correct.
• Atomic tests have very few UI interactions and they’re only on a maximum of two screens. In rare cases, an atomic test might navigate through 3 screens (although I’d like to see this example)

Example of an atomic test in Atata:

public void Login_Should_Work()
{
    //Arrange
    Go.To<LoginPage>()
    //Act
        .Email.Set("[email protected]")
        .Password.Set("password123")
        .SignIn.ClickAndGo()
    //Assert
        .Title.Should.Equal("Homepage");
}

We open the login page, enter the required information and click sign in, and then verify that the homepage is loaded by checking if the title is "Homepage". Only two screens are involved, the login screen and the homepage.

Here is a bad version of the same test:

public void Login_Should_Work()
{
    //Arrange
    Go.To<LoginPage>()
    //Act
        .SignIn.Click()
    //Assert
        .Username.ValidationMessage.Should.Equal("Username is required.")
        .Password.ValidationMessage.Should.Equal("Password is required.")
    //Act
        .Username.Set("administrator")
        .Password.Set("password123")
        .SignIn.ClickAndGo()
    //Assert
        .Title.Should.Equal("Homepage");
}

You can see there are 2 Acts and 2 Asserts, so you can figure out this test has two responsibilities:

1. checking if the validation is working properly
2. checking if sign-in works correctly

It should be refactored into two tests, and this is a great example of another fundamental concept.

Atomic tests tell you exactly what is not working when they fail. They make debugging easier, test execution quicker and test results more reliable.

Take a look at the last example. If Login_ShouldWork() fails, it doesn't necessarily mean that login doesn't work:

1. it could be that the validation message isn't visible.
2. it might mean validation message text changed.
3. it could be that sign-in is not working correctly.

Write tests so that when they fail, you know exactly what went wrong. If the test Login_Should_Work() fails it should mean that login doesn't work, and if Login_Should_Validate() fails it should mean that validation doesn't work. When a test fails, you will need to run it again multiple times to debug and figure out what is going on. Long tests with multiple responsibilities need a long time to execute and debug.

Move complex logic into the framework layer

Framework wraps around Selenium and exposes methods to make your job easier. Whoever made the framework, thought long and hard about common problems they had to solve over and over again in Selenium, and writing methods to simplify this process. Your job is to use these methods in Act part of testing and keep tests short and atomic.

Here is an example of how we use Atata to deal with the loading indicator after sign-in button is clicked.

Login page object:

public class LoginPage : Page<LoginPage>
{
    [FindByName("username")]
    public TextInput<_> Username { get; private set; }

    [FindByName("password")]
    public PasswordInput<_> Password { get; private set; }

    [FindByName("sign-in")]
    [WaitForElement(WaitBy.Class, "loading-indicator", Until.VisibleThenMissingOrHidden, on: TriggerEvents.AfterClick)]
    public Button<HomePage, _> SignIn { get; private set; }
}

Actual test:

public void Login_Should_Work()
{
    //Arrange
    Go.To<LoginPage>()
    //Act
        .Username.Set("[email protected]")
        .Password.Set("password123")
        .SignIn.ClickAndGo()
    //Assert
        .Title.Should.Equal("Homepage");
}        

The test remains the same, and the logic for loading indicator appearing and disappearing is handled by the frameworks attribute [WaitForElement]

What you need to take from this article is that tests need to be short, atomic and independent of each other. If you write them this way, you can optimize execution later on and execute them in parallel. Learn about the framework you are using and it will help you write shorter tests (you will move complicated logic into the framework layer and use methods that the framework exposes). To make them atomic you will need to separate responsibilities of the feature you are testing and make sure that each test only verifies a single responsibility.

I will talk more about project architecture, best practices when using Atata and how to make tests independent of each other in the following articles.