Black-box testing: how to make the most of specification and requirements

Although there are a number of different techniques used in software testing, we can group them into 3 main categories:

• black-box techniques – based on the product specification
• white-box techniques – based on the product structure
• experience-based techniques – relying on the tester's experience, skills, and intuition.

As you probably already expect, in this article, I’m going to focus on a couple of black-box techniques.

A short introduction to black-box testing techniques

First, let’s make clear what is black-box testing by discussing some distinctive features of the black-box techniques:

• usually based on specification – testers rely on project documentation, which requires specification or user stories.
• relating to its internal structure – it's not necessary for the tester to know the structure of the component or system that will be tested.
• system behavior testing – tester is checking the behavior of the system when it is subjected to inputs (without analysis of what happens between inputs and outputs).
• models are created during the creation process – the models are a simplification of the behavior of the real system. They can be state diagrams, decision tables, or short descriptions – and they do not have to be formally documented.
• used in functional and non-functional testing – e.g. load testing.

Today, I will discuss in the following black-box testing techniques:

• Equivalence Partitioning
• Boundary Value Analysis
• Decision Table Testing
• State Transition Testing
• Use Case Testing

Equivalence Partitioning

Let’s start with Equivalence Partitioning (EP), also known as Equivalence Class Partitioning, which is used to reduce the number of test cases. To employ this testing method efficiently, the tester needs to partition input values (these can be also internal values or time-related values) on sets of values that are processed in the same manner by the application. It’s crucial to remember that value sets can’t overlap and that both valid and invalid values are considered. Then, the tester selects one representative value from each partition, assuming that it covers all elements of a selected value's class.

Test level: any test level

When can this black-box testing technique be used?

Equivalence Partitioning can be used during the testing process, when all the values within a given set of values are handled in the same way and when testers can uniquely extract the values for the valid and invalid partitions.

Values that can be tested this way belong in the following categories:

• Values ​​in a continuous range. For example, passengers aged 3 to 18 pay half the price of the ticket. Range for the valid partition will be 3 <= x <= 18 and the representative valid value can be 10.
• Discrete values. For example, a weekday such as "Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday"

It's a good idea to use EP with boundary value analysis because this way, the tester extends the test values with the weight at the edges of the partitions. This, in turn, lets the QA check whether the boundaries are set correctly.

Equivalence Partitioning example: set password of length 8-12 and not in the top 100 common password

The team develops an application with a user registration feature granting access to other features of the app. As part of this functionality, the client provided password requirements for acceptance criteria:

• password should contain from 8 to 12 characters,
• the application won't accept passwords from the list of 100 most popular passwords.

In case of this functionality, we can use 2 different cases of equivalence partitioning:

• password length,
• passwords from the top 100 and from outside the set.

Password lengths

The first test should determine what valid and invalid sets comprise when it comes to the password’s length.

• Valid value: password length range between 8 and 12
• Invalid values: password length less than 8 or password length greater than 12

In this example, the tester has 3 partitions: one valid and two invalid. Now, their task is to choose representative values.

• Too short password – 4 characters long password (invalid) like Pozn
• Password with correct length – 10 characters long password (valid) like Poznań2020
• Too long password – 16 characters long password (invalid) like PoznańMerix62020

Passwords from the top 100 and from outside the set*

• Valid value: password different than any of the 100 most common passwords
• Invalid values: password listed among 100 most common passwords

In this example, the tester has 2 partitions: one valid and one invalid. Now, they need to choose representative values.

• Password different than any of the 100 most common passwords (valid) like Poznań2020
• Password listed among 100 most common passwords (invalid) like dragon

*Please note that in good functional testing, we often come across what is popularly considered the security domain.

Types of defects

Dealing with this black-box testing technique, we can come across functional defects in the handling of various data values. It may happen that applications treat valid data as an invalid value and vice versa. For instance, when the user enters 2000 as the year of birth, the application doesn't accept this value but it accepts 2030.

Boundary Value Analysis

The next black-box testing technique – which I’ve already mentioned in the previous paragraph, by the way – is Boundary Value Analysis (BVA). This method is often used with EP, whereby boundary values are found on the edges of the equivalence partitions. When the QA wants to use the BVA technique, they need to know the minimal increment of values that will be tested. Depending on the risk assessment, the tester can follow one of two approaches:

• two value testing – the tester uses 2 values for each boundary: the first one is the boundary value on the boundary whereas the second is one is just outside the boundary with the smallest possible boundary (below or above the boundary, depending on whether the upper or the bottom boundary is tested). Let’s take an example of the age of majority which equals or is above 18 in Poland. In this case, the increment equals 1 so the two test values for the bottom boundary will be 17 and 18.
• three value testing – the tester uses 3 values: the value just below the boundary value, the boundary value, and the value just above the boundary. Once again, we can refer to the age of majority in Poland. The three test values for the bottom boundary will be 17,18, and 19.

Test level: any test level

When can this black-box testing technique be used?

Boundary Value Analysis is applicable when data can be organized into equivalence partitions – which is necessary to determine the boundaries. To give you a practical example, this black-box testing technique can be used in the following events:

• A range of numbers, for example <10, 20>, where the lower limit is 10 and the upper limit is 20.
• Numeric attributes of non-numeric variables, such as the length of the password. Another example is that of determining by the application whether the result of a test is within the norm, below or above it. Such is the case with the concentration of red blood cells determined by the blood test and comparison of the result with the norm.
• Loops – for instance, when logging into the bank, the user's account is blocked after entering an incorrect password for the third time in a row.
• Stored data structures – an example would be the size of a table, such as a table listing role permissions where the number of roles is limited.
• Physical objects (including memory) – for example, in-app memory limitation per user.
• Time-determined activities – an example may be a temporary restriction of access after entering the correct password several times.

Boundary Value Analysis example: consent to participate in the cosplay competition

Imagine the client orders an application for an upcoming comic convention. One of the functionalities concerns the registration of users to the cosplay competition. According to the regulations, you must be at least 13 years old to participate in the competition, however, participants under 18 must submit the consent of a parent or a legal guardian to participate. Developers handed over the functionality for testing regarding the requirement for consent for minors and now we need to test it.

First, we have to set the age range for which consent is required: <13, 18)

As you can see, the interval is left-closed, right-open, because we are allowed to participate in competition when the participant is 13 years old and until the participant turns 18, the parent's consent is required.

If, after a risk assessment, we decide to test using two values:

• for the bottom boundary, we should choose the values: 12 and 13,
• for the top boundary, we should choose the values: 17 and 18.

If the risk is considered to be high, then select:

• 12, 13, and 14 for the bottom boundary,
• 17, 18, and 19 for the top boundary.

Types of defects

Boundary Value Analysis may come in handy when it comes to identifying the following types of defects:

• excessive boundaries,
• incorrect setting or incorrect processing of limit values (less than, greater than) e.g. when the application should allow shopping for users 18+ but it allows users to shop only at the age of 18.
• non-functional load limit defects, e.g. if a system supports 2000 concurrent users but the application crashes when 10 users start using it.

Decision Table Testing

Decision tables make it easier for the tester to check combinations of input conditions. It is usually an easy way to present all data or essential combinations in a transparent form that facilitates their design, programming, and testing.

The size of the decision table depends on the number of input conditions. In the case when the inputs can only accept true/false values, the number of combinations is 2ⁿ - whereby ⁿ is a number of input conditions. This means that when we deal with a large number of conditions, the table can turn out too big to be tested. Then, the tester should consider the possibility of reducing the table and carefully analyze the risks arising from doing this.

Test level: generally used for integration, system, and acceptance test levels. May be used for testing components that are responsible for a set of decision logic.

When can this black-box testing technique be used?

Decision tables can be used in integration, system, acceptance, or modular tests, where decision logic occurs. They’re also applicable when requirements are written in the form of flow diagrams or business rule tables.

Decision Table Testing example: permanent discounts in the online bookstore

Let’s imagine an online bookstore offering two permanent discounts:

• customers buying at least 5 books will pay 50% for the cheapest one,
• customers placing an order for a minimum of 100 Euro will be for free delivery.

Possible test cases for this discount functionality will look like this:

1. Fewer than 5 books are ordered and the order value is less than 100 Euro.
2. 5 or more books are ordered and the order amount value less than 100 Euro.
3. Less than 5 books are ordered but the order value is at least 100 Euro.
4. 5 or more books are ordered and the value of the order is at least 100 Euro.

The QA can present these cases using an array:

For more information about the decision tables, you can read my previous article Decision table: How to use it in software testing?. You’ll find there instructions on how to create a decision table or reduce it when this table is over-complicated, as well as get to know more use cases for this technique.

Types of defects

This black-box testing technique detects any defects related to the processing of the inputs’ combination, such as:

• incorrect processing with certain combinations,
• unexpected results,
• documentation defects, during the process of creating an array come out all omissions and contradictions,
• unsupported or incorrectly handled conditions.

State Transition Testing

State transition testing tests the ability to change state in the software as well as the ability to enter and exit this state. During creating test cases for this black-box testing technique, QAs should take into account events that trigger state changes. The tester can present state transitions by using diagrams with present correct transitions or state tables that contain both valid and invalid transitions.

Test level: any test level

When can this black-box testing technique be used?

As the name suggests, State Transition Testing is used to test any kind of software that has defined states and events capable of causing transitions between those states. Here are some examples of software where this black-box testing technique would prove particularly useful:

• embedded software, e.g. washing machine: each subsequent stage of washing, if they are not to be skipped, can occur only in a strictly defined order: start-> pre-wash -> washing -> rinsing -> spin -> stop.
• web application, e.g. onboarding a new user: when a user logs in for the first time, they go through successive pages of onboarding one after the other.
• transactional systems, e.g. banking transactions: states from the start of the transfer to the moment of money appearing in the default account (with the option of rejecting the transfer).
• control systems, e.g. traffic light: where transfer from red to green is not allowed and passing through the yellow light is required by software green<-->yellow<-->red.

State Transition Testing example: user status

The application can be used only by registered and active users. When new users sign up, a new account is created with “pending” status, and the activation link is sent to the user. When they click the activation link, the user's account status is changed to “active” and the user can deactivate their account now. When a user has an inactive account, they can activate the account again.

Admin can block active or inactive users. Admin can also unblock the user's account – and when the account gets unblocked, the status of the user changes to active. Regardless of the user's status, the user can be removed by admin.

The QA’s task is to test changes in the account’s status. To make testing easier, the tester can create a diagram showing the transition between states.

S1 - pending
S2 - active
S3 - inactive
S4 - blocked
S5 - no user in DB

Types of defects

Any defects captured by this technique relate to state changes, transitions between states and their deficiencies, or data transfer between these states. Some examples include:

• incorrect event/action type or value,
• incorrect initial state,
• lack of possibility to reach some exit state or enter required states,
• unnecessary states,
• inability to execute a valid transition correctly,
• ability to execute invalid transitions,
• documentation defects such as omissions or contradictions.

Now, how about some practical knowledge? Performing State Transition tests, a tester can identify the following bugs:

• the ability to go from shopping cart to payment without going through the choice of order pick up or order confirmation,
• additional double pages for the user,
• a situation whereby the customer entered an active 10% discount code in the cart and proceeds to the order summary – but there is no information about the discount in the order summary (it shows only the value of the order before the discount),
• the ability to go from shopping cart to payment without going through the choice of shipping method or order confirmation,
• no option to log into the application due to the lack of possibility to activate a new account after registration because the activation step is missing – when users have pending status after registration.

Use Case Testing

Use cases are based on a scenario or a description of the application/software operation that imitates the use of the component or system. In this black-box testing technique, we need to define interactions between an actor (human users, external hardware, other components, or other systems) and a component or a system that responds to the actor and behaves in a defined way.

Test level: any test level

When can this black-box testing technique be used?

Use Case Testing proves helpful in the following situations:

• system and acceptance tests – checking if all customer requirements have been implemented,
• integration tests – checking whether the system or its components behave as described in the use cases. Also, seeing if the new function works well with the rest of the system and there are no conflicts between components.
• performance tests – when load and performance requirements are specified in use cases.

Use Case Testing example: the customer wants to use the 10% discount for clothes

In the online shop, sellers can send a customer a one-time coupon with a time limit. The tester should check if the discount works properly.

System: online-shop application
Actor: shop’s customer

*S - system

Types of defects

Testers using this black-box testing technique can come across all kinds of defects associated with incorrect processing of the given condition or an invalid scenario processing. Lack of alternative path support as well as incorrectly or weakly implemented error reporting can be spotted as well.

Final thoughts on black-box testing

Of course, the five methods discussed above are by no means all black-box testing techniques there are. The ISTQB Test Analyst 2019 syllabus takes into account the existence of at least two others: Classification Tree Technique and Pairwise Testing. I haven’t included them in this guide as I'm still exploring them – so it’s likely once I become familiar with these two, I’ll come back with well-presented examples of Classification Tree and Pairwise Testing.

Ready to fulfill your true testing potential? Join our QA team!