Verification Viewpoints: Shift Left with Sanity Testing

Shift left is a commonly used industry phrase meaning ‘test early and test often’. The goal is to catch bugs as early in the development cycle as possible since earlier bug detection results in huge productivity gains. In the semiconductor industry, there are a couple of ways we can shift left:

1. Catch as many bugs as possible by the earliest possible team. Meaning by the time the verification team has completed its work all bugs have been found that were visible at the verification team’s scope. This is also referred to as improving the quality.  
2. Catch bugs as quickly as possible when they are introduced. This means ideally, as soon as a bug gets submitted, it is detected through some mechanism such as sanity testing.

Often the focus for verification teams is the first of these two because each time a bug makes it past a team (from verification to the lab to the customer), the cost associated with finding and fixing the bug rises exponentially. However, the second way is also important as it increases the productivity of both the design and verification teams.

In this post I’ll discuss ‘sanity testing’ - why verification teams should run sanity tests frequently and how doing so can help teams catch bugs earlier and improve productivity.

So how often should a sanity test run? The answer is as often as reasonably possible. Some teams will set up a cron job to run periodically (every 1-4 hours is typical). My preferred approach is to launch a sanity with each submission. My team typically sets up sanity testing using an open-source tool called Jenkins. Whenever a new check-in happens, Jenkins is able to sync out a workspace and launch a test. It takes care of the workspace management and results areas of cleanup for you.

In terms of which submissions will result in a sanity test launch, the target should be anything that can change how the test is built or run. Primarily this will be changes to either your design or testbench files (recompile), but it can also include collateral used at runtime like memory initialization files, firmware code, test vectors, etc. Lastly, even a change to the scripts used to compile and/or run your tests should trigger a sanity test, as changes there can also affect the results of your tests.

Why is it so important to catch bugs quickly through sanity testing? The longer a bug sits undetected in your code base (design or testbench), the harder it is to narrow down exactly what caused the failure.  Additionally, it is common for multiple people on the team to experience the same failure and they may try to debug/resolve the issue independently, causing duplicated effort. Lastly, if you don’t know which change caused the failure immediately you have fewer options on how to resolve it – for example if a failure is causing productivity issues for the whole team and you know exactly which check-in caused it, you may choose to roll back the check-in until the issue can be debugged.

Sanity testing is an important component to improve productivity for design and verification teams. It does so by running a small but representative sample of tests on each submission to catch issues as early as possible.