I love working with .NET Core on the command line (CLI) and Visual Studio Code. Until recently getting code coverage metrics for your .NET Core projects had required using Visual Studio or a 3rd party paid tool. With the introduction of Coverlet you can now generate code coverage metrics on the command line, and the further process the collected data into reports using Report Generator.
Interested? Read on and I'll explain how and provide link a sample project at the end of this post.
Unit Testing .NET Core with Xunit on the Console
Using Xunit to unit test your .NET Core code is no different than unit testing your .NET Framework code. It still helps to write your code in a manner that is conducive to testing - think dependency injection, loose coupling, etc. For this post I'll assume you're already familiar with unit testing and XUnit, if you're not read up on that first and come back to this post when you're ready to find some test coverage metrics.
You don't have to use Xunit for this, but it's the testing framework I like, and it isn't always as well documented for this kind of thing so it is the tool I'm using.
Simple Code Coverage With .NET Core CLI
What is Coverlet
I mentioned Coverlet at the start of this article, you're probably wondering what it is? Coverlet is a code coverage framework for .NET, with support for line, branch and method coverage. It can be used both as a Global Tool, or installed into a .NET Core project as a Nuget package. I'll show you how to use it both ways in this post. If you'd like to, read more about Coverlet on the project's GitHub
Using Coverlet with Your .NET Core Project
The first way I'll show to generate code coverage metrics is to add Coverlet to your test project. To add the nuget package to your project run the following command - dotnet add package coverlet.msbuild from the cosole and you should be all set. If you're curious my example project uses v2.6.3.
After you've added the Coverlet package, be sure to perform a dotnet restore and/or dotnet build to make sure everything worked ok, then you are ready to run your tests and collect the coverage metrics!
Generating Code Coverage Metrics the Easy Way
If you want to run your tests get metrics on it as easily as possible, here's the command you want to run:
dotnet test /p:CollectCoverage=true
This command will run your unit tests, and collect the coverage metrics which get stored in the current working directory in a new file "coverage.json". If you look at the coverage.json file it likely won't make any sense, don't worry we'll do something with it in a bit.
Generating Code Coverage Metrics the More Descriptive Way
The previous example showed you how to quickly get some metrics from your tests. If you're like me you probably were left with questions like how to I specify where to store the output, how to I generate the metrics in another format, and so on. Good news, you can supply more parameters to dotnet test to specify such things!
dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:Exclude="[xunit*]\*" /p:CoverletOutput="./TestResults/"
When you run the command you'll see something like this:
PS c:\source\test-coverage-sample-code\src> dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:Exclude="[xunit*]\*" /p:CoverletOutput="./TestResults/"
Test run for c:\source\test-coverage-sample-code\src\SampleApi.Test\bin\Debug\netcoreapp2.2\SampleApi.Test.dll(.NETCoreApp,Version=v2.2)
Microsoft (R) Test Execution Command Line Tool Version 16.1.1
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
Test Run Successful.
Total tests: 2
Passed: 2
Total time: 2.7664 Seconds
Calculating coverage result...
Generating report '.\TestResults\coverage.cobertura.xml'
+-----------+-------+--------+--------+
| Module | Line | Branch | Method |
+-----------+-------+--------+--------+
| SampleApi | 15.9% | 0% | 18.18% |
+-----------+-------+--------+--------+
+---------+-------+--------+--------+
| | Line | Branch | Method |
+---------+-------+--------+--------+
| Total | 15.9% | 0% | 18.18% |
+---------+-------+--------+--------+
| Average | 15.9% | 0% | 18.18% |
+---------+-------+--------+--------+
In the command above we're specifying to use the cobertura format with the /p:CoverletOutputFormat=cobertura parameter, and the /p:CoverletOutput="./TestResults/ tells dotnet test & Coverlet to store the resulting metrics in a folder called "TestResults".
One last parameter that comes in handy is /p:Exclude. This allows us to filter out code that we don't want included in code coverage metrics. Filtering can be applied by namespace, assembly, and more in this manner /p:Exclude=[Assembly-Filter]Type-Filter.
This is a pretty high level overview of some of the options you can supply to Coverlet, if you want to get more details on what is possible, check out the Coverlet documentation.
Turn .NET Core Coverage Metrics into a Report
If the metrics that were written out to the console in the previous step weren't enough for you then read on because the reporting is about to get A LOT better!
What is ReportGenerator
ReportGenerator is another tool I briefly mentioned, it generates easy-to-read coverage reports in various formats. The reports generated can show total coverage from a project, file, and namespace level, and also allow you to drill into the source code and visualize which lines have been covered. If you'd like to, read more about ReportGenerator on the project's GitHub
Here's a Sample Code Coverage Report from a .NET Core Project
Preparing To Generate Reports
The first way to generate a report that I'll cover uses a "per-project tool", essentially this is a Nuget package that is added to the project in a slightly different way. To add ReportGenerator as a tool, you will need to manually add this code to your test project file:
<ItemGroup>
<DotNetCliToolReference Include="dotnet-reportgenerator-cli" Version="4.2.10" />
</ItemGroup>
Before you go on, do a quick check of the version number, you likely want the latest version number if you have no reason to use a lower one. Find the current & past versions of ReportGenerator here. You may want to change the version number being used, but other than that you can save your project file and then run a dotnet restore to finish setting up the ReportGenerator tool.
One word of warning, DO NOT use dotnet add package... to install ReportGenerator, this will not allow it to work as a tool on the CLI.
Simple Report Generation
Now that the test project has ReportGenerator installed, we're ready to actually make something useful from the code coverage metrics we've collected. From your console run the following (from your test project's directory):
dotnet reportgenerator "-reports:TestResults\coverage.cobertura.xml" "-targetdir:TestResults\html" -reporttypes:HTML;
This command assumes that you've got your coverage data store in the sub-directory TestResults, and that you've got a coverage.cobertura.xml file (you will if you've been following along). The other two parameters specify the target directory for the results to be stored, and the report type format - HTML.
Running the above command will produce the following:
PS c:\source\test-coverage-sample-code\src\SampleApi.Test> dotnet reportgenerator "-reports:TestResults\coverage.cobertura.xml" "-targetdir:TestResults\html" -reporttypes:HTML;
Arguments
-reports:TestResults\coverage.cobertura.xml
-targetdir:TestResults\html
-reporttypes:HTML
Loading report 'c:\source\test-coverage-sample-code\src\SampleApi.Test\TestResults\coverage.cobertura.xml' 1/1
Preprocessing report
Initiating parser for Cobertura
Current Assembly: SampleApi
Coverage report parsing took 0.1 seconds
Initializing report builders for report types: HTML
Analyzing 3 classes
Creating report 1/3 (Assembly: SampleApi, Class: SampleApi.Controllers.SampleController)
Writing report file 'SampleApi_SampleController.htm'
Creating report 2/3 (Assembly: SampleApi, Class: SampleApi.Program)
Writing report file 'SampleApi_Program.htm'
Creating report 3/3 (Assembly: SampleApi, Class: SampleApi.Startup)
Writing report file 'SampleApi_Startup.htm'
Creating summary
Writing report file 'TestResults\html\index.htm'
Report generation took 0.5 seconds
Per-project tools are pretty useful, and great for build servers. However, they are only available in the context of the project that adds them as a reference. Using them outside of the project won't work because the command cannot be found - you may have seen this with ReportGenerator if you tried to run it from outside the Test project's directory.
If you're interested in a way to set up your local system so that you don't need to worry about installing more dependencies or where you're at in the file system in relation to the test project, read on to find out how to set up Coverlet and ReportGenerator as Global Tools.
Introducing .NET Core Global Tools
The concept of ".NET Core Global Tools" was introduced in .NET Core 2.1 as a feature of the .NET Core CLI. Global Tools are essentially cross-platform console tools that you can create and distribute with .NET Core & Nuget. I will use two different Global Tools to produce some code coverage reports similar to how we did earlier in the post. If you're interested in other tools that are available here's a good list of some Global Tools.
Getting Started With the Coverlet Global Tool
Before you can use the Coverlet Global Tool, you must install it. That can be done by going to your command line (I prefer Powershell) and running:
dotnet tool install --global coverlet.console
After installing Coverlet, you can run dotnet tool list -g and verify it was installed, as well as view any other Global Tools you may already have installed.
Running the Coverlet Global Tool
Once you've installed Coverlet, you can now run the command to generate the coverage metrics. Here's a sample of what that command looks like:
coverlet .\SampleApi.Test\bin\Debug\netcoreapp2.2\SampleApi.dll --target "dotnet" --targetargs "test --no-build" --exclude "[xunit*]\*" --output "./coverage-reports/"
This command is generating coverage metrics based on the "SampleApi.dll". Overall it looks pretty similar to what we were doing before, except now we supply the path to the SampleApi.dll. We're still excluding the Xunit code as we don't need to include that in our metrics, we then output the results to the "coverage-reports" folder.
Note the Global Tool method doesn't require adding the coverlet.msbuild nuget package to the project. So if you're interested in the code coverage of your project, but don't want to commit to the package, this isn't a bad way to go.
The output from running Coverlet as a Global Tool is more or less the same as the per-project tool's output, so I'll skip including it here.
Setting Up the ReportGenerator Global Tool
Setting up the ReportGenerator Global Tool is just as straightforward as setting up the Coverlet Global Tool was.
dotnet tool install --global dotnet-reportgenerator-globaltool
Running the ReportGenerator Global Tool
After you've installed the ReportGenerator Global Tool, you can run it using this command:
reportgenerator "-reports:coverage-reports\coverage.cobertura.xml" "-targetdir:coverage-reports\html" -reporttypes:HTML;
Be sure that all of the command is on one line or it may not work correctly - don't try to break it up into multiple lines, wrapping is fine.
Conclusion
In this blog post I've shown you a few different ways to run code coverage and report generation. If you only care about running code coverage for a single project, you probably want to go the "per-project tool" route (adding the nuget package / CLI Tool Reference). Likewise if you think you may want to run code coverage on a build server, you probably want to go with the packages installed via your .csproj file.
If you will want to run code coverage on your local dev across many projects you may want to install the global tools. Your install can be the global tools alone, or you can install both the per-project tooling as well as the global tools and be prepared for your local dev environment as well as your build server.
Finally, as promised here is a small sample project where I've set up the a sample API, and sample API test project to demonstrate what was mentioned here in this blog post.