We developed TESA, a novel tool that enables a comprehensive and automatic evaluation of code-coverage metrics on an arbitrary set of npm packages. TESA runs on GraalVM and is built on top of NodeProf. Our tool evaluates and assembles test suites for any package, including the original tests from the package developers, located in the package’s development repository.
As the original tests often yield only limited coverage, it is beneficial to find additional third-party tests as a complement or alternative. We propose a novel methodology to automatically extend the code coverage of a target package by treating the tests of third-party packages that rely on the target package (called dependents) as tests for the target package itself. By executing the tests of the dependents (called dependent tests) the resulting workload can exercise the code of the target package, thus behaving as if the tests were written for the target package itself. We implement our test-coverage extension methodology in TESA. To the best of our knowledge, TESA is the first framework for the automatic assembly of test suites fully leveraging the abundance of code available in the npm registry. Thus, TESA assembles a new test suite for arbitrary packages that combines two complementary sets of tests, i.e., the original tests and the dependent tests.
TESA provides a compaction algorithm that produces an optimal compacted test suite, i.e., containing only those tests necessary to maximizing code coverage while minimizing the total test execution time. Finally, in addition to extending code coverage, TESA enables the automatic execution of arbitrary dynamic program analyses (DPAs) on the testing code (including both original and dependent tests) of any set of npm packages, allowing the automated execution of large-scale DPAs “in the wild”. Thanks to TESA, we run a state-of-the-art DPA for identifying performance problems in npm packages. Our results demonstrate that dependent tests allow finding performance problems that cannot be detected with the original tests of a package, further confirming the benefits of extending code coverage by exploiting tests written for dependents.
A research paper describing this work has been accepted at AST’21 . We are actively working on releasing TESA as open-source software.
 Haiyang Sun, Andrea Rosà, Daniele Bonetta, Walter Binder: Automatically Assessing and Extending Code Coverage for NPM Packages. AST 2021: 40-49 [pdf][video][slides]