Build Analytics



 
build.png One of the areas ripe for monitoring is your development build - it is important to get notifications for build failures plus there is a slew of information that is generated by a typical build process. Leveraging Wolfpack's plugin architecture it's been an easy task to create a set of plugins that allow you to monitor your TeamCity build status and extract information from reports generated by tools run during the build process such as NCover, SpecFlow and StoryQ. It is also a simple task to further extend this range of plugins to extract statistics from other tools and track the build state from other build systems such as TFS, CruiseControl etc.

The Build Analytics feature works by first setting up a Health Check to track the build state. As a new build is detected a result message is generated (this contains the state of the build, pass or fail) and this is passed to all the Publishers you have configured (as per normal wolfpack behavior). However Build Analytics provides some special Publishers - these Publishers will only trigger when a result message from a specific source arrives (your build check) and their function is to parse a build tool report file (eg: ncover output), extract the important information from it and then publish this data as further Health Check result messages. This approach means you can tailor which parsers to run once the build has completed - your build might run StoryQ but not NCover, no problem - just enable the parsers that you need! You can set up as many build checks & parsers as required.

The final piece of this feature is to ensure we can easily visualise the extracted report information - the Wolfpack Geckoboard Data Service allows us to connect Wolfpack directly into Geckoboard to quickly display your build statistics.

Build Systems supported include...
  • TeamCity
Build Tools supported include...
  • NCover summary report (html), extracting...
    • Branch, Symbol and Method coverage %
    • Cyclomatic complexity
  • SpecFlow xml report extracting...
    • Success rate %
    • Total number of specifications
    • Number of Successes, failures, pending and ignored
  • StoryQ xml report extracting...
    • The number for each result category
  • Other stats...
    • Build duration

Walk Thru

These are the steps required to connect your TeamCity build to Geckoboard to expose stats from your NCover & SpecFlow output.

1. The Config\buildanalytics.castle.config file contains four component entries...we are going to configure the "CoverageBuildNotificationCheckConfig" one to detect new builds. Note: you can create as many of these build notifications as required - eg: 1 per build that you want to know the status of or extract the stats from. Just remember to also create a new binding for it (copy the "CoverageBuildNotificationBindingConfig" component).
    <component id="CoverageBuildNotificationCheckConfig" lifestyle="singleton" 
               type="Wolfpack.Contrib.BuildAnalytics.Checks.TeamCityBuildNotificationCheckConfig, Wolfpack.Contrib.BuildAnalytics">
      <parameters>
        <FriendlyId>CoverageBuildNotification</FriendlyId>
        <Enabled>false</Enabled>
        <BuildServerUrl></BuildServerUrl>
        <UserId></UserId>
        <Password></Password>
        <UseSsl>false</UseSsl>
        <TrustDuffSslCertificate>false</TrustDuffSslCertificate>
        <ProjectName></ProjectName>
        <ConfigurationName></ConfigurationName>
      </parameters>
    </component>
2. Update the Enabled property to true

3. Set the BuildServerUrl to your TeamCity project url, usually something like http://buildserver:8080

4. Set the UserId and Password properties to your TeamCity credentials

5. If your server uses SSL then set the UseSsl property to true, also if the certificate is self-certified then also set the TrustDuffSslCertificate property to true

6. Set the ProjectName and ConfigurationName property to the TeamCity project name and build configuration name respectively.

7. We need to enable the build parsers, one for the NCover report and one for the SpecFlow report.

8. In the Config\buildparser.castle.config file set up two components - one will handle parsing the NCover report and the other the SpecFlow report. An important field is the TargetHealthCheckName property. The value here must match the FriendlyId property of the build notification component configured in step 1. TeamCity artifacts are zip files so the SpecFlow and NCover reports will be extracted from these then parsed. The ZipFileTemplate property should be set to the path to the respective zip file - however this path usually will contain the build number so the {buildid} token should be used and the actual build id value (included in the build notification result) will be substituted. The ReportFileTemplate property allows you to specify the filename (including path) within the zip.

NOTE If your build output is NOT a zip archive then leave the ZipFileTemplate property blank and just set the full path (including the {buildid} token if required) to the uncompressed report file in the ReportFileTemplate property.
    <component id="SpecFlowParserConfig" lifestyle="singleton" 
               type="Wolfpack.Contrib.BuildAnalytics.Parsers.SpecFlowHtmlReportParserConfig, Wolfpack.Contrib.BuildAnalytics">
      <parameters>
        <FriendlyId>SpecFlowParser</FriendlyId>
        <Enabled>true</Enabled>
        <TargetHealthCheckName>CoverageBuildNotification</TargetHealthCheckName>
        <ZipFileTemplate>c:\builds\{buildid}\acceptance.zip</ZipFileTemplate>
        <ReportFileTemplate>AcceptanceReport.html</ReportFileTemplate>
      </parameters>
    </component>

    <component id="NCoverParserConfig" lifestyle="singleton" 
               type="Wolfpack.Contrib.BuildAnalytics.Parsers.NCoverHtmlSummaryReportParserConfig, Wolfpack.Contrib.BuildAnalytics">
      <parameters>
        <FriendlyId>NCoverParser</FriendlyId>
        <Enabled>true</Enabled>
        <TargetHealthCheckName>CoverageBuildNotification</TargetHealthCheckName>
        <ZipFileTemplate>c:\builds\{buildid}\coverage.zip</ZipFileTemplate>
        <ReportFileTemplate>coveragesummary.html</ReportFileTemplate>
      </parameters>
    </component>

9. We now need to expose this information to Geckoboard...we will need to enable a storage publisher and the Geckoboard Data Service activity. To enable a storage publisher open the Config\publisher.castle.config file and set the Enabled property to true on either the SQLitePublisher or SqlServerPublisher component.

10. Enable the Geckoboard Data Service by setting these properties in the activity.castle.config file. Set the Enabled to true on the GeckoboardDataServiceActivityConfig component and on the GeckoboardDataServiceConfig component set the DataProvider property to match your publisher choice in step 9 - use values "SQLite" or "SqlServer".
    <component id="GeckoboardDataServiceActivityConfig"
				   lifestyle="singleton"
				   type="Wolfpack.Core.Geckoboard.GeckoboardDataServiceActivityConfig, Wolfpack.Core">
      <parameters>
        <Enabled>true</Enabled>
        <!-- This must be the service implementation class name including the namespace -->        
        <ServiceImplementation>Wolfpack.Core.Geckoboard.GeckoboardDataService, Wolfpack.Core</ServiceImplementation>
        <!-- The base address that the service is available at
             Currently Geckoboard only supports Http at port 80 -->
        <Uri>http://localhost/Geckoboard</Uri>
        <!-- An ApiKey is set up on each Geckoboard widget. Use the same one and 
             set this value to it to ensure only requests with this ApiKey will be 
             returned your monitoring data. Omit this property or leave blank to disable 
             the ApiKey security -->
        <ApiKey></ApiKey>
      </parameters>
    </component>
    
        <component id="GeckoboardDataServiceConfig"
				       lifestyle="singleton"
				       type="Wolfpack.Core.Geckoboard.GeckoboardDataServiceConfig, Wolfpack.Core">
          <parameters>
            <!-- Possible values are SQLite, SqlServer -->
            <DataProvider>SQLite</DataProvider>
          </parameters>
        </component>
11. (Re)start Wolfpack to pick up these config changes.

12. Connect Wolfpack to Geckoboard to display your build stats. Remember to ensure internet connectivity to the server running Wolfpack - ensure firewalls and port forwarding are configured. The simplest way to view your NCover and SpecFlow numbers is using Geckoboard Number Comparison widgets (piechart and geck-o-meter widgets soon to be supported). The comparison widget will display the last value with a +/-% compared to the previous build value. The url to enter into the Geckoboard widget to display the NCover branch coverage % number is...

http://hostname/geckoboard/comparison/sites/sitea/CoverageBuildNotification-NCover?tag=branch

Other supported stats are selected by replacing the tag parameter with...
  • symbol
  • method
  • complexity (for cyclomatic complexity figure)

The magic ingredient here is the CoverageBuildNotification value highlighted - this must be the same value that we originally used in the build notification component FriendlyId property back in step 1 and also used in step 8 as the TargetHealthCheckName property value. If you set up further build analytics then ensure that this name is unique and used throughout the notification, parser and geckoboard url locations.

The url to display the SpecFlow success rate % is...

http://hostname/geckoboard/comparison/sites/sitea/CoverageBuildNotification-SpecFlow?tag=SuccessRate

Other supported stats are selected by replacing the tag parameter with...
  • Total
  • Success
  • Failures
  • Pending
  • Ignored

The remaining Geckoboard widget properties that must be set are...
  • Widget type = Custom
  • Feed format = JSON
  • Request Type = POST

You can also visualise these stats with a "Geck-o-meter" widget - the url format is, (for NCover branch coverage %)

http://hostname/geckoboard/geckometer/CoverageBuildNotification-NCover?tag=branch

buildanalytics-meter.png

Example Dashboard

Here is an example dashboard from a project of mine - the line chart is the duration (in seconds) of the coverage build (teamcity build configuration that runs the code coverage) and the remaining numbers are the current values of the coverage and specflow reports (with the % delta from the last build).

buildanalytics.png

Bonus Stuff

Since the build notification appears as a regular Health Check result it can leverage all the same publishers including Growl - this means you can get desktop notifications when your build completes - even better forward these on to your smartphone so you are always up to date with your build status!

The other bonus is that the duration of the build is also reported in the result, the ResultCount property contains the build duration in seconds. This means you could plot this with the Geckoboard Linechart widget to get an instant indicator of your build performance. The url for the build duration linechart is,

http://hostname/geckoboard/linechart/CoverageBuildNotification/success

Just replace the CoverageBuildNotification value with the name of your build (also in step 1 & 8)

Last edited Oct 26, 2011 at 4:12 PM by jimbobdog, version 31