How to use change tracking to measure performance velocity

As a DevOps engineer, it is crucial to integrate observability into your CI/CD pipelines to ensure you can confidently deploy your applications. With New Relic change tracking, you can do just that—visualize your deployments to validate your changes, deliver them faster, and capture different events across your pipeline. 

In this blog post, you’ll learn how to send change tracking events on New Relic entities using two different approaches: the New Relic CLI and GraphQL. You can use the New Relic CLI to automate and implement observability directly into your pipelines, and the New Relic GraphQL API offers a second approach for engineers. You’ll also learn three bonus ways of implementing change tracking markers with GitHub Actions, Jenkins, and Ansible.

How can change tracking help you?

Change tracking should be an integral step in your CI/CD pipelines. By integrating markers into your pipelines, you’ll see how your changes affect your deployed application and any related services also instrumented in New Relic. Change tracking covers your whole CI/CD estate, ensuring you can deploy, validate, and improve your DevOps performance. 

Recording change tracking markers is a simple process, benefiting you with a correlation of errors, logs, anomalies, and incidents across your deployed applications and related services. The next image shows what change tracking markers look like in the New Relic APM capability. Change tracking markers are easily identifiable, represented as circles, across all your charts in New Relic.

When you select a marker in New Relic, you can see deployment details which contain valuable insights into each change you track, including faceted errors, log attribute trends, related issues and anomalies, and calculated impacts on key signals.

Check out the next screenshot: the layout is simple and customizable to your needs.

In the highlighted rectangle, you can see three key metrics, related errors, logs, and alerts across your change. You can compare your golden signals from before and after deployment to see if any improvements or fixes have been applied successfully. Just remember, the more you can decorate your change tracker markers with metadata, the more meaningful and helpful the deployment details in New Relic will be.

Prerequisites for deploying change tracking markers

To follow along with this blog post, you’ll need these prerequisites:

• A New Relic account: If you don't have one yet, you can sign up for free. Your free account includes up to 100 GB/month of data ingest, one full-platform user, and unlimited basic users.
• An app to deploy markers: You can use this Change Tracking App demo if you don't have one. (If you use this demo app, you must install Node.js.)
• The New Relic CLI: You can get installation instructions from newrelic-cli on GitHub or the New Relic CLI documentation.
• A New Relic user key: You can get your user keys from https://one.newrelic.com/api-keys.

How to create a user key

Important: If you don’t have an existing user key that you can use with GraphQL, GitHub Actions, Ansible, or Jenkins, you’ll need to create a new user key. Follow these instructions to create a new user key, and use the next images for reference:

1. Open to your API Keys in New Relic. 
2. Select Create a key.
3. Within the form:
   • Select the account the key needs to be created in.
   • Set key type to User.
   • Name the key.
   • Select Create a key to submit the form.

Once the key is created, copy it by selecting the icon with the three dots next to the key, then selecting Copy key.

Option 1: GraphQL for change tracking

In simple terms, GraphQL is a query language that makes it easier for you to query, update, or delete data from your New Relic account. You can leverage GraphQL in your automation, CI/CD pipelines, or scripts. New Relic offers a GraphQL playground called the NerdGraph explorer that you can use to either build your queries for automation, or you can use the UI directly.

The next image shows an example of how the NerdGraph explorer can help you build a GraphQL query to create a change tracking marker. You’ll walk through this process in detail in the next section.

Step 1: Get your entity GUID.

Go to New Relic and follow the instructions shown the screenshot below to get your entity GUID:

1. Select APM & Services.
2. Select your application from the list.
3. Select the tag icon to see metadata, manage tags, and assign teams.
4. Find and copy the value for Entity guid.

All of your entities in New Relic—browser monitoring, infrastructure monitoring, mobile monitoring, and more—have entity GUIDs. You can find them in the same way.

Step 2: Create a change tracking marker with the NerdGraph explorer for GraphQL.

1. Open the NerdGraph explorer.
2. Select or Create a new user key from the dropdown menu.
3. Create a mutation in the the query builder in the left column by expanding changeTracking and then selecting changeTrackingCreateDeployment and  deployment!.
4. Within deployment!, select description, entityGuid!, user, and version!, and fill out the details, including your entity GUID from the previous step. Optionally, you can select other fields to add additional information to your marker. The fields with ! are required.
5. Select the Play button.
6. You’ll know that you’ve successfully created a marker if you see similar output to the next image.

Step 3: Go to the New Relic platform to see your marker.

1. Select APM & Services.
2. Select your application from the list.
3. See the marker in the charts.
4. Select a marker to open the deployment details page to see more information about the most recent deployment.

As this example shows, adding markers in New Relic with the NerdGraph explorer is a simple process. The next example will demonstrate how to do it using the New Relic CLI.

Option 2: newrelic-cli for change tracking

Newrelic-cli is the officially supported command line interface for New Relic. The CLI consolidates some of the tools New Relic offers for managing resources. You can leverage it in your automation, to make GraphQL calls, to manage API access, to create deployment markers, and more. 

Before following along with the next set of instructions, make sure you have installed the New Relic CLI and have your user key.

Step 1: Create a user key.

Follow the instructions outlined in the prerequisites to create a new user key in New Relic.

Step 2: Set up your environment.

Next, export the necessary environment variables:

• Export your User Key: export NEW_RELIC_API_KEY=<your_personal_api_key>
• Export your License Key: export NEW_RELIC_LICENSE_KEY=<your_license_key>
  • You can get your license key in New Relic.
• If your account is in the EU, set your region: export NEW_RELIC_REGION="EU"

Step 3: Test the CLI.

Run either of these next commands in your terminal to locate your application and test that the CLI is configured correctly.

• newrelic entity search searches for all entities in New Relic.
• Add the -d flag to return all entities within the specified domain, for example INFRA, APM, MOBILE.
• Use the --name <your app name> flag to search by your application’s name.

The next image shows the output you should see if the CLI is working as expected.

Step 4: Create a marker.

From the terminal output, locate the guid of your chosen application (for example, infrastructure or mobile). Then use it in the next command to create a marker:

You can add more options to the command like in the example with the NerdGraph explorer, following the same --name value syntax. 

The next image shows an example of what is returned when you successfully create a marker.

Step 5: Check out your marker in New Relic.

Navigate to the summary page of your application within New Relic. You should see the change tracking marker you just created in the charts. You can select the marker to see the deployment details page for more  information about the change.

Change tracking plugins for GitHub Actions, Ansible, and Jenkins

Integrating change tracking into your CI/CD pipelines is crucial in detecting and resolving potential deployment issues. For example, sometimes your pipeline might indicate that a deployment is successful, and when you traverse through your app, everything looks fine. Because of this, you might not immediately realize you've introduced a bug. With change tracking, New Relic correlates any changes in your app, whether throughput, error rate, or increased response times, helping you to identify any unexpected deployment issues faster.

Let's examine additional examples of triggering deployment markers from various CI/CD tools, such as Ansible, GitHub Actions, and Jenkins. You’ll learn how to use pre-built modules for these tools that New Relic offers to make integrating change tracking that much faster. After you implement these integrations, your CI/CD pipeline will generate markers for your entities.

Integrate with GitHub Actions

1. Create GitHub secrets in your repository with the following environment variables:

• NEW_RELIC_API_KEY: User key from the previous steps.
• NEW_RELIC_DEPLOYMENT_ENTITY_GUID: New Relic entity GUID to create the marker on.

2. Open a new or existing workflow file. Within jobs, add a newrelic job with the following steps:

3. After you add and push your workflow file to your remote repo, check your actions tab in GitHub and expand the details to verify that everything is correct.

4. Find the marker in New Relic. This confirms that your marker has been successfully sent from your GitHub Actions deployment. 

For more information on setting up change tracking with GitHub Actions, see:

• Track changes with GitHub Actions in the New Relic documentation. 
• New Relic Application Deployment Marker in the GitHub documentation.

Integrate with Ansible

To track deployments with markers in Ansible, you can create markers using the New Relic change tracking module. The process is simple: you’ll add the module from the nr-ansible-modules GitHub repo to your Ansible modules folder and define a new task. Note the ansible module provided is experimental and open source and does not come with New Relic support. Pull requests and issues are welcome. 

1. Get your New Relic user key.

Grab an existing user key, or follow the steps outlined in the prerequisites to create a new user key.

2. Set deployment markers in your Ansible playbook.

You must add the newrelic_ct module to your Ansible modules folder and reference it in your ansible.cfg.

3. Create a change tracking task.

• Define a new task with a meaningful name. For example, Send Change Tracking Event.
• Within the task, define the newrelic_ct module with the parameters shown in the next code snippet. Make sure to update all parameter values with your own information.

3. Execute your playbook.

Run ansible-playbook playbook.yml -v.

If the status code is 200, the change tracking event was successfully sent.

4. Check your app in New Relic to confirm that you see the deployment marker.

For more information on setting up change tracking with Ansible Playbooks, see the GitHub repo nr-ansible-modules.

Integrate with Jenkins

1. Set deployment markers in Jenkins.

Download the deployment marker plugin and either manually install it or install the plugin with the download link.

2. Install the plugin with the download Link in Jenkins.

Add the download link to the URL field, like the next image, and select Deploy.

3. Create the credentials.

Now you need to create credentials for your plugin. You can use a quickstart in New Relic. To find it, open New Relic, select Add data, search for “change tracking” and select Jenkins. Then follow along with the guidance to install the quickstart. 

Or, you can create the credentials manually in Jenkins, as shown in the next image.

• For Kind, select Username with password.
• Leave the checkbox for Treat username as secret unselected.
• For Password, add your user API key from New Relic.
• Leave the input for ID blank.

After the credentials are created, copy the auto-generated ID. The next image shows an example of what your ID should look like.

4. Add the Pipeline stage.

Add the stage shown in the next code snippet to your pipeline script. Make sure you update all values with your environment variables.

5. Push your pipeline script to your remote repository.

6. Check your Jenkins build logs to confirm that the changes to your pipeline are successful.

7. Check your app in New Relic to confirm that you see the deployment marker.

For more information on setting up change tracking with Jenkins, see Track changes using Jenkins in the New Relic documentation.