In today's complex application landscape, logs serve as a valuable source of information for monitoring and troubleshooting. They provide insights into your system's behavior and enable you to understand how your applications perform. However, managing logs can quickly become overwhelming without the right tools and processes. That's where a log management platform like New Relic comes in.
New Relic offers robust log management capabilities that connect your log data with the rest of your application and infrastructure data. These capabilities give you a holistic view of your system and reduce mean time to resolution (MTTR). However, given the complex nature of systems involving multiple log sources, configurations and tools, ensuring accurate log forwarding to New Relic can be challenging.
This blog will provide the top five tips to ensure your logs forward correctly to New Relic. By following these tips, you'll:
- Confirm your network access to New Relic domains
- Verify that you have selected a supported log forwarder
- Send troubleshooting logs from the log forwarder for further debugging
- Access designated log files and services to confirm correct permission levels
- Check that you are correctly parsing your logs to avoid errors and truncation
Tip #1: Ensure network access to New Relic domains
To ensure proper functionality of New Relic, it is essential to establish network connectivity between your environment and New Relic domains. Since New Relic is a Software as a Service (SaaS) based offering, your environment must be able to communicate with multiple domains over the Internet. Add the domains listed in this document to your ‘allow’ list to guarantee this connectivity. These domains are essential for New Relic to perform optimally, and blocking them can lead to issues with product features or prevent pages from loading entirely.
You can perform the following ping test to verify your connection to New Relic domains.
ping collector.newrelic.com
Executing the above command will help confirm the connectivity between your system and the New Relic collector domain.
Please note that if you are using a proxy, it is necessary to include your proxy variables as part of your infrastructure agent configuration settings. This will ensure that your logs are properly forwarded to New Relic.
Important! If you encounter any issues while trying to connect to New Relic, it is most likely due to network connectivity problems. Consult with your network team for potential workarounds, or seek additional whitelisting approvals.
Tip #2: Use supported log forwarders
While using the New Relic infrastructure agent is one way to manage logs, there are other options. If you don't need the host metrics that infrastructure log forwarding provides or already have a log forwarder in place, you can choose from other supported methods to forward your logs to New Relic. New Relic provides a comprehensive list of log forwarders, including those for popular cloud platforms such as AWS, Azure, and GCP.
Before you explore alternative log forwarding options, let's first understand how log forwarding works.
How log forwarding works
The following diagram illustrates the process of log forwarding:
Now that you have a visual understanding of log forwarding let's consider some specific methods you can use to forward your logs to New Relic.
Logstash plugin for log forwarding
If your log data is already monitored by Logstash, you can use our Logstash plugin to forward and enrich your log data in New Relic. By forwarding your Logstash logs to New Relic, you can take advantage of enhanced log management capabilities. This includes collecting, processing, exploring, query, and setting alerts on your log data more efficiently. Install and configure the Logstash plugin for sending logs to New Relic.
Logs API
If the existing New Relic log forwarding solutions don’t fulfill your requirements, you can use the Log API to directly send log data to New Relic via an HTTP endpoint. The platform is built around the four fundamental telemetry data types: metrics, events, logs, and traces, which are necessary for effective system monitoring. These data types are exposed via an API, often called "MELT" in the observability industry.
Here is a simple Python example for sending logs to the New Relic Logs API.
import json
import requests
# Endpoint for New Relic Log API
url = "https://log-api.newrelic.com/log/v1"
# Headers to include the content-type and Api-Key
headers = {
'Content-type': 'application/json',
'Api-Key': 'INGEST_KEY'
}
# Loading the JSON file
with open('./sampleLogs.json') as file:
sampleLogs = json.load(file)
# Post request to the url with the sampleLogs data and headers
response = requests.post(url, json=sampleLogs, headers=headers)You can also use curl to send logs to New Relic Logs API. Below is a simple example of forwarding logs to New Relic Logs API using curl.
curl -X POST https://log-api.newrelic.com/log/v1 \
-H "Content-Type: application/json" \
-H "Api-Key: <YOUR_LICENSE_KEY>" \
-H "Accept: */*" \
-d '{
"message": "Logs Example",
"logtype": "accesslogs",
"service": "production-service",
"hostname": "production.example.com"
}'Follow our documentation to learn more about how to use the New Relic Log API to forward your logs.
Important! If you encounter any issues when sending logs to New Relic using supported log forwarders, you are likely using an unsupported approach. In such cases, it's recommended to review the log forwarding solutions provided by New Relic and choose a supported method that meets your requirements.
Tip #3: Send troubleshooting logs
To help troubleshoot issues with log forwarding to New Relic, you can configure the agent to send its logs to New Relic. This provides valuable insights into the log forwarding process and helps identify potential issues. To enable this feature, you need to edit your newrelic-infra.yml configuration file and add a configuration snippet to enable log forwarding to New Relic.
To send troubleshooting logs to New Relic via infrastructure agent, follow these steps:
- Edit your
newrelic-infra.ymlconfiguration file. - Add the configuration snippet below to enable log forwarding to New Relic, as seen here.
log:
level: trace # Recommended: Helps with troubleshooting
forward: true # Enables sending logs to New Relic
format: json # Recommended: Enable agent logging in JSON format
stdout: false # On Windows and systems that don't use `systemd` or where `journald` is inaccessible- Restart the agent by following the instructions to apply the new settings.
By following these steps, you configure the infrastructure agent to send its logs to New Relic. The above configuration snippet enables log forwarding to New Relic, sets the log level to trace for troubleshooting purposes, specifies the log format as `JSON`, and disables the stdout output in specific system environments.
If you can send troubleshooting logs successfully, it indicates that the Infrastructure agent is communicating with New Relic without issues. However, remember that trace logging generates an enormous amount of data quickly. To avoid high disk usage and data ingestion, you should set the log level to `info` or lower once you have generated logs.
If you cannot see troubleshooting logs in New Relic, you will need to look into the underlying causes of the issue. This may include networking problems, firewalls or permission issues.
Verify infrastructure agent connectivity
Linux: You can use the following command to check if the New Relic Infrastructure agent is running successfully.
# list and check the td-agent process
ps -ef | grep -i td-agentWindows: You can check whether the New Relic Infrastructure service runs in Windows Services for Microsoft Windows.
Address permission issues
If you are experiencing issues with the New Relic agent, it is possible that you have permission issues preventing it from running successfully. The following resources provide additional information to help you resolve any permission-related issues.
Linux: By default, the agent runs and installs as root. You can also select privileged or unprivileged run modes.
Windows: The agent must be installed from an Administrator account and requires Administrator privileges to run.
macOS: The agent can be installed from any user account.
In addition, you can leverage the NrIntegrationError event provided by New Relic to gain more visibility into potential issues related to data ingestion, limits, and feature configuration. This event is designed to capture issues arising from exceeding data ingest and query limits, configuration errors, and malformed data.
Troubleshoot with NrIntegrationError Event
New Relic provides the NrIntegrationError event, which offers visibility into potential data ingestion, limits, and feature configuration issues. Use the following NRQL query to examine the structure of the NrIntegrationError event and see its attached attributes.
FROM NrIntegrationError SELECT keyset()The image below shows an example of issues captured by the NrIntegrationError event.
Important! As mentioned earlier, if you encounter any issues with sending troubleshooting logs to New Relic, you likely have networking issues preventing the Logs from being sent, or you do not have the necessary system permissions to run the New Relic Infrastructure agent.
Tip #4: Access the designated log file and service
When working with log files, it's important to ensure that the log file you're attempting to use is accessible and has the necessary permissions. In some cases, the log file may have been created when the agent was running as root, resulting in the nri-agent user to lose write permissions.
To check the necessary permissions and ensure that you have access to the log file, you can use the ls -l command in Linux.
ls -l /path/to/fileThe above command provides detailed information about the permissions and ownership of the file.
Once you have identified any issues with file permissions, you can consider implementing the following options: :
Option 1: Change the owner of the log file: By changing the owner of the log file, you can ensure that the appropriate user has the necessary permissions to access and modify the file.
Option 2: In the /etc/newrelic-infra.yml configuration file, modify the log_file entry to specify a log file location that allows the nri-agent user to write to it. Our installation scripts create the /var/log/newrelic-infra/ folder for this purpose, and we recommend using the same. For example:
log_file: /var/log/newrelic-infra/newrelic-infra.logWhen encountering issues with the agent not running or logs being inaccessible on Microsoft Windows, it is often caused by antivirus or security products installed on the system. These products can sometimes interfere with the proper functioning of the agent and prevent access to essential log files. It is important to check the settings of these products and ensure that proper exceptions have been added to allow the agent to run and logs to be accessed. Additionally, reviewing any recent updates or changes made to the security software may be helpful to determine if they may be causing the issue.
Important! If you encounter any issues when accessing the designated log file and service, you will likely not have the proper permissions and access. Check with the team to ensure you have all the correct configurations and approvals.
Tip #5: Implement appropriate log parsing
In New Relic, logs are often presented in JSON format, recommended over plain text. JSON log messages are automatically parsed into key-value pairs, reducing the likelihood of unexpected issues.
Below is an example of a log successfully parsed in New Relic.
When logging in plain text, the lack of structure can pose challenges when parsing log data. Extracting helpful information from plain-text logs becomes difficult and may even result in parsing errors. On the other hand, JSON is a structured data format that provides a consistent structure for log data. Using JSON as an output format, log messages are automatically converted into key-value pairs, making extracting useful information from logs easier.
When logs don’t appear correctly in New Relic, there are usually two reasons.
Reason one: invalid JSON format
An invalid JSON format is one possible reason for logs not appearing correctly in New Relic. The log data must be in valid JSON format to ensure proper parsing and prevent truncation. Otherwise, it will be stored as a string and truncated if it exceeds the character limit. Therefore, verifying that the log data is in valid JSON format before forwarding it to New Relic is crucial.
Workaround
If you encounter issues with log parsing, knowing that a wealth of information is available in the technical community regarding parsing logs is helpful. You can find many examples provided by FluentBit, New Relic, or other resources commonly available to help. By leveraging this knowledge, you can ensure your logs are correctly parsed into New Relic.
Reason two: “stringified” JSON content
Another potential reason for logs not appearing correctly in New Relic is the presence of "stringified" JSON content. This occurs when valid JSON content is converted into a string format using escape characters, resulting in it being evaluated as a string rather than JSON. This means the content will be truncated to 4,096 characters before being assessed as JSON. This truncation will produce an invalid JSON, storing the data as a string. Additionally, if the JSON contains arrays, they will be flattened and stored as unparsed strings making it challenging to extract and analyze the data within the arrays.
Workaround
To avoid issues related to "stringified" JSON content and the flattening of arrays, send the data in their original JSON format without converting them to a string. This allows the content to be parsed correctly even if it exceeds the character limit.
Important! If you encounter any issues related to log data formatting and parsing, likely, you do not have the correct log format or the logs are not being correctly parsed into New Relic. Explore the data structure of your logs and use the appropriate log parsers.
Conclusion
To summarize, we have covered five tips to make sure your logs are correctly forwarded to New Relic, allowing you to gain valuable insights into your application behavior and quickly identify potential issues. The information provided in this post is designed to help you understand the basics of log forwarding and guide you in troubleshooting and verifying your log forwarding configurations. By utilizing these tips, you can easily ensure that your logs are working correctly and being sent to New Relic.
Next steps
To further explore the topics discussed in this blog post, you can refer to the following documentation for more information:
The views expressed on this blog are those of the author and do not necessarily reflect the views of New Relic. Any solutions offered by the author are environment-specific and not part of the commercial solutions or support offered by New Relic. Please join us exclusively at the Explorers Hub (discuss.newrelic.com) for questions and support related to this blog post. This blog may contain links to content on third-party sites. By providing such links, New Relic does not adopt, guarantee, approve or endorse the information, views or products available on such sites.
