Rust Testing Mastery: From Basics to Best Practices

This article provides a comprehensive overview of testing in Rust, delving into its built-in test framework, common patterns, useful crates, best practices, advanced testing, and benchmarking.

Unit Tests

Structure

Unit tests are small tests that focus on a specific component or function in isolation. In Rust, you can define unit tests in the same file as the code they test, usually within a mod tests block marked with #[cfg(test)].

The #[cfg(test)] attribute ensures that the annotated code is only compiled when running tests, not when building the library or binary.

fn add(a: i32, b: i32) -> i32 { 
    a + b 
} 
 
#[cfg(test)] 
mod tests { 
    use super::*; 
    #[test] 
    fn test_add() { 
        assert_eq!(add(2, 3), 5); 
    } 
}

Running the Tests

When you run cargo test in a Rust project, it compiles and runs the test suite. The output of this command can provide valuable feedback about the state of your project's tests. Let's break down the components of the output you typically encounter.

As each test runs, you’ll see its name printed to the terminal followed by “…” indicating it’s currently running.

Example:

running 5 tests 
test tests::test1 ... ok 
test tests::test2 ... ok 
test tests::test3 ... FAILED 
test tests::test4 ... ok 
test tests::test5 ... ok

Test Summary

After all tests have been run, cargo test provides a summary:

test result: FAILED. 4 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s

This summary provides the following details:

• Overall test result (either OK or FAILED).
• The number of tests that passed.
• The number of tests that failed.
• The number of tests ignored (tests with #[ignore] attribute).
• The number of benchmark tests (measured).
• The number of tests filtered out (if you run cargo test with arguments to run specific tests).
• The time taken for the entire suite to run.

Failed Test Details

If any tests fail, cargo test will then provide detailed output for each failed test. This includes:

• The name of the test.
• The location of the test in your source code (file and line number).
• The specific error or assertion that failed, along with any error messages.

Example:

---- tests::test3 stdout ---- 
thread 'tests::test3' panicked at 'assertion failed: `(left == right)` 
  left: `3`, 
 right: `4`', src/lib.rs:15:13

This tells you that the test tests::test3 failed because it expected a value of 3 to equal 4, and this assertion was made in the file src/lib.rs on line 15.

Additional Flags and Outputs

• --verbose or -v: Running cargo test --verbose will give more detailed output, including the exact commands invoked by Cargo and the output from those commands.
• --nocapture: By default, cargo test captures any output printed to the console (e.g., via println!). If you want to see this output, you can use the --nocapture flag.

Integration Tests

While unit tests focus on individual components, integration tests ensure multiple components work together properly. Integration tests are typically placed in the tests directory at the top level of your crate.

For example, given a tests/integration_test.rs:

use your_crate_name::function_or_module; 
 
#[test] 
fn test_integration() { 
    // Your test code here... 
}

Run the tests with cargo test and Cargo will include both unit and integration tests.

Doc-tests

Rust supports “documentation tests” or “doc-tests”. These are tests written in your documentation comments. The idea is to ensure that code examples in your documentation are always correct.

/// Adds two numbers together. 
/// 
/// # Examples 
/// 
/// ``` 
/// let result = your_crate_name::add(2, 3); 
/// assert_eq!(result, 5); 
/// ``` 
pub fn add(a: i32, b: i32) -> i32 { 
    a + b 
}

When you run cargo test, it will also test your documentation examples.

Handling Test Failures

The primary macros for asserting conditions in Rust tests are assert!, assert_eq!, and assert_ne!.

• assert! checks a boolean condition.
• assert_eq! and assert_ne! check for equality and inequality, respectively.

When an assertion fails, the test will fail, and Rust will display a diagnostic message.

Test Attributes

Rust provides several attributes to modify test behaviour:

• #[ignore]: Marks a test to be ignored by default. Useful for long-running tests. Use cargo test -- --ignored to run these tests.
• #[should_panic]: Marks a test that should panic to be considered successful.

Third-party Test Libraries

While Rust’s built-in test framework is robust, some developers prefer third-party libraries for extended features. Popular choices include:

• proptest: Property-based testing similar to Haskell's QuickCheck.
• mockall: A powerful mocking library for Rust.
• criterion: A benchmarking library that integrates with Rust's test system.

Let’s have a quick look at each of them.

The proptest Crate

The proptest is a property-based testing tool that enables developers to write tests that automatically generate test cases. Instead of manually writing individual unit tests, you specify the properties the system should have and proptest produce test cases that verify those properties.

What is Property-Based Testing?

Before delving into proptest, it's crucial to understand property-based testing. Unlike traditional unit testing where you write specific examples of inputs to test, in property-based testing, you define properties that your code should satisfy. The testing tool then generates a variety of random test cases to check if the properties hold.

Getting Started with proptest

To start using proptest, you need to include it in your Cargo.toml:

[dependencies] 
proptest = "1.0"

Also, don’t forget to include the proptest_derive crate if you want to derive strategies for your custom types.

proptest-derive = "1.0"

Basic Usage

Let’s consider an example where we want to test the property of string reversal:

use proptest::prelude::*; 
 
proptest! { 
    #[test] 
    fn test_string_reversal(s: String) { 
        let reversed = s.chars().rev().collect::<String>(); 
        assert_eq!(s, reversed.chars().rev().collect::<String>()); 
    } 
}

In this test, we’re checking that reversing a string twice results in the original string. The proptest! macro generates random strings and runs the test for each string.

Using Custom Strategies

Sometimes, you may want to restrict the input space or use custom input generation strategies:

use proptest::prelude::*; 
use proptest::collection::vec; 
 
proptest! { 
    #[test] 
    fn test_list_properties(ref lst in vec(i32::ANY, 1..100)) { 
        // Test that reversing a list twice gives the original list 
        let reversed = lst.iter().rev().cloned().collect::<Vec<_>>(); 
        assert_eq!(*lst, reversed.iter().rev().cloned().collect::<Vec<_>>()); 
    } 
}

In the above example, we’re testing lists of integers, ensuring they have a length between 1 and 100.

Shrinking

One of the powerful features of proptest is its ability to "shrink" failing test cases to simpler ones that still cause the test to fail. This makes it easier to diagnose issues when they arise.

For instance, if the test test_list_properties failed on a list of 50 random integers, proptest would attempt to find a smaller list that still causes the test to fail, making the cause of the problem easier to pinpoint.

The mockall Rust Crate

What is mockall?

mockall is a flexible and powerful mocking framework for Rust. It allows developers to easily mock interfaces (traits in Rust) without creating manual implementations. With mockall, one can automatically generate mock objects, specify return values, and even set expectations about how functions should be called.

Getting Started

To start using mockall, add it to your Cargo.toml:

[dev-dependencies] 
mockall = "0.10"

Basic Usage

Define a trait and mock it

use mockall::mock; 
 
trait Greeter { 
    fn greet(&self, name: &str) -> String; 
} 
mock!{ 
    Greeter, 
    fn greet(&self, name: &str) -> String; 
}

The mock! macro generates a MockGreeter struct that can be used in tests.

Using the mock in tests

To utilize the mock, you set expectations and then use it as if it were a real implementation.

#[test] 
fn test_greeting() { 
    let mut mock = MockGreeter::new(); 
 
    mock.expect_greet() 
        .with(mockall::predicate::eq("Alice")) 
        .times(1) 
        .returning(|_| "Hello, Alice".to_string()); 
    assert_eq!("Hello, Alice", mock.greet("Alice")); 
}

Here, we tell the mock to expect a call to greet with "Alice" as the parameter. If any other value is passed or the method is called more or fewer times than specified, the test will panic.

Advanced Features

Mocking methods with generic parameters

mockall supports mocking methods with generic parameters. Let's expand our Greeter trait:

trait Greeter { 
    fn greet<T: Into<String>>(&self, name: T) -> String; 
} 
 
mock!{ 
    Greeter, 
    fn greet<T: Into<String>>(&self, name: T) -> String; 
}

Using sequences

Sequences allow you to set up ordered expectations:

use mockall::Sequence; 
 
#[test] 
fn test_greeting_sequence() { 
    let mut seq = Sequence::new(); 
    let mut mock = MockGreeter::new(); 
    mock.expect_greet() 
        .with(mockall::predicate::eq("Alice")) 
        .times(1) 
        .in_sequence(&mut seq) 
        .returning(|_| "Hello, Alice".to_string()); 
    mock.expect_greet() 
        .with(mockall::predicate::eq("Bob")) 
        .times(1) 
        .in_sequence(&mut seq) 
        .returning(|_| "Hello, Bob".to_string()); 
    assert_eq!("Hello, Alice", mock.greet("Alice")); 
    assert_eq!("Hello, Bob", mock.greet("Bob")); 
}

In this example, the test will panic if mock.greet("Bob") is called before mock.greet("Alice").

The Criterion Library for Benchmarking in Rust

Criterion.rs is a statistics-driven micro-benchmarking tool. Unlike some benchmarking tools which simply time how long a piece of code takes to run, Criterion.rs provides a rich set of statistical analyses of the results. This helps you understand how long the benchmark took to run, how much variance there was between runs, whether the results are statistically significant, and more.

Setting Up Criterion

To start using Criterion.rs in your Rust project:

Add it as a development dependency in your Cargo.toml:

[dev-dependencies] criterion = "0.3"

Configure your project to use Criterion.rs by adding or modifying your .cargo/config.toml file:

[build] bench = false [profile.release] lto = true panic = "abort" [profile.bench] lto = true panic = "abort"

Writing Benchmarks

Place your benchmarks in the benches directory of your project. Here's a simple benchmark:

use criterion::{criterion_group, criterion_main, Criterion}; 
 
fn fibonacci(n: u64) -> u64 { 
    match n { 
        0 => 1, 
        1 => 1, 
        n => fibonacci(n-1) + fibonacci(n-2), 
    } 
} 
fn benchmark(c: &mut Criterion) { 
    c.bench_function("fibonacci 20", |b| b.iter(|| fibonacci(20))); 
} 
 
criterion_group!(benches, benchmark); 
criterion_main!(benches);

In the example above, c.bench_function takes a name for the benchmark ("fibonacci 20") and a closure which tells Criterion.rs how to run the benchmark.

Running Benchmarks

To run benchmarks, use:

cargo bench

Criterion will then perform multiple runs of each benchmark, providing statistics and charts detailing the performance of the tested functions.

Analyzing Results

Criterion provides detailed reports that can show:

• Average Time: The mean execution time across iterations.
• Variance: The variance and standard deviation, indicate how much individual iterations differ from the mean.
• Regression Analysis: Helps identify if code changes have led to statistically significant changes in performance.

Moreover, Criterion.rs can plot the distributions of execution times, helping visualize how the results spread out.

Comparing Benchmarks

Criterion can compare benchmarks across different versions of your code. After you make changes to your codebase and rerun cargo bench, Criterion will compare the new results with the previous ones and report on performance regressions or improvements.

Advanced Features

• Parameterized Benchmarks: You can run benchmarks with different input values. This is useful to see how performance scales with input size or to benchmark against various test cases.
• Throughput Measurement: If your function processes data, like compressing/decompressing bytes, Criterion can measure throughput in bytes per second.
• Setup and Teardown: For benchmarks requiring setup (like populating a data structure) or teardown, Criterion supports separate setup and teardown functions that don’t count towards the benchmark time.
• Custom Plots: You can generate custom plots using the gnuplot integration.

Advanced Testing Patterns in Rust

While the basics of Rust testing are straightforward, some more advanced patterns and practices can help create efficient and comprehensive tests.

Parameterized Tests

Instead of writing multiple tests that are essentially the same except for the input data, you can use parameterized tests to run the same test logic with different input values. Although Rust does not have a built-in mechanism for parameterized tests, you can use loops and macros for similar effects.

For instance, using a macro:

macro_rules! param_test { 
    ($name:ident, $value:expr, $expected:expr) => { 
        #[test] 
        fn $name() { 
            assert_eq!($value, $expected); 
        } 
    }; 
} 
 
param_test!(test_add_1_1, super::add(1, 1), 2); 
param_test!(test_add_2_3, super::add(2, 3), 5);

Error Scenarios

While the #[should_panic] attribute is helpful, sometimes you want more granularity. Consider using the Result<T, E> return type for tests, which allows you to use the ? operator and handle errors explicitly.

#[test] 
fn test_error_scenario() -> Result<(), String> { 
    let result = some_function_that_can_fail()?; 
    assert_eq!(result, expected_value); 
    Ok(()) 
}

Continuous Integration (CI)

Automating your testing process with CI is crucial. Platforms like GitHub Actions, GitLab CI, and Travis CI integrate well with Rust projects. Adding a configuration file to your repository can automate building, testing, and sometimes deploying your Rust application.

Functional vs. Non-functional Testing

While we’ve covered the importance of unit and integration testing (functional testing), Rust also allows you to perform non-functional tests. These assess performance, usability, and other non-functional requirements:

• Performance Testing: Using libraries like criterion, you can determine how long your functions take to run, how they handle large datasets, or how they perform under stress.
• Memory Safety: Rust’s ownership system ensures memory safety without a garbage collector, making it unique among programming languages. Tools such as valgrind can still be employed to detect memory leaks and unwanted behaviours.
• Usability Testing: This involves human testers and isn’t specific to Rust. However, when developing Rust libraries or tools, consider getting user feedback about documentation clarity, API intuitiveness, and overall user experience.

Regression Testing

Ensuring that new code changes don’t introduce bugs in existing functionalities. Automated testing, particularly in a CI/CD pipeline, can frequently check the codebase to detect and prevent regressions. Using version control systems (like Git) alongside CI tools ensures that any breaking change can be quickly identified and rectified.

Test Driven Development (TDD) with Rust

Rust’s integrated testing tools make it well-suited for Test Driven Development:

1. Write a Failing Test: Begin by writing a test for the new functionality, which will fail because the functionality still needs to be implemented.
2. Write the Code: Implement the minimum code required to pass the test.
3. Refactor: Refine the code without altering its behaviour, ensuring tests pass. This ensures maintainability and efficiency.

By iterating through these steps, developers can ensure they’re writing functional and thoroughly tested code from the outset.

Code Coverage

It’s useful to know what proportion of your code is covered by tests. Tools like tarpaulin or kcov can provide coverage metrics. While striving for 100% coverage is only sometimes practical or necessary, these tools can identify parts of your codebase that might benefit from additional testing.

Tips for Effective Testing

1. Keep Tests Small and Focused: Each test should verify a single behaviour or concept. This makes it easier to pinpoint issues when a test fails.
2. Test Boundary Conditions: Rust’s type system helps prevent many common errors, but you should still test edge cases, such as overflow conditions or potential division by zero.
3. Stay Updated: The Rust ecosystem is vibrant and continually evolving. Regularly check for updates to your dependencies and the Rust compiler. This ensures you benefit from the latest optimizations and safety checks.

Check out more articles about Rust in my Rust Programming Library!

Stay tuned, and happy coding!

Visit my Blog for more articles, news, and software engineering stuff!

Follow me on Medium, LinkedIn, and Twitter.

All the best,

CTO | Tech Lead | Senior Software Engineer | Cloud Solutions Architect | Rust 🦀 | Golang | Java | ML AI & Statistics | Web3 & Blockchain