guide

1.Configure the .NET App to Expose Metrics

Goal: Ensure the .NET application is instrumented to expose metrics over an endpoint.
Step 1: Install Required OpenTelemetry Packages
Run the following commands in your .NET application directory:

dotnet add package OpenTelemetry.Exporter.PrometheusAspNetCore
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Instrumentation.SqlClient


Step 2: Modify Program.cs to Enable Metrics Exposure
Add the following code to Program.cs (for .NET 6+):

using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

// Define OpenTelemetry resources
builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics =>
    {
        metrics.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("MyDotNetApp"))
               .AddAspNetCoreInstrumentation()
               .AddHttpClientInstrumentation()
               .AddSqlClientInstrumentation()
               .AddRuntimeInstrumentation()
               .AddProcessInstrumentation()
               .AddPrometheusExporter(); // Enables Prometheus endpoint
    });

var app = builder.Build();

// Map Prometheus metrics to an endpoint
app.UseOpenTelemetryPrometheusScrapingEndpoint();

app.MapGet("/", () => "Hello World!");
app.Run();

Step 3: Verify Metrics Endpoint
Run the .NET app:
dotnet run
Open a browser or use curl to check if metrics are being exposed:
curl http://localhost:9464/metrics
(Port 9464 is the default for Prometheus metrics in OpenTelemetry.)
If you see a list of metrics, your app is correctly exposing metrics! 


2.Configure the OpenTelemetry Collector to Collect Metrics

Goal: Set up the Otel Collector to scrape metrics from the .NET app and forward them to Splunk.

Step 1: Install & Run the OpenTelemetry Collector
If using Docker, run:

docker run --name otel-collector -p 4317:4317 -p 4318:4318 -p 9464:9464 \
  -v $(pwd)/otel-collector-config.yaml:/etc/otelcol-config.yaml \
  --rm otel/opentelemetry-collector:latest --config /etc/otelcol-config.yaml
(Ensure you have a valid otel-collector-config.yaml file.)

Step 2: Create otel-collector-config.yaml
Create an otel-collector-config.yaml file to scrape the .NET app’s metrics and export them to Splunk:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "dotnet_app"
          scrape_interval: 5s
          static_configs:
            - targets: ["host.docker.internal:9464"]  # Change to "localhost:9464" if running locally

exporters:
  otlphttp:
    endpoint: "https://ingest.us0.signalfx.com/v2/datapoint"
    headers:
      X-SF-Token: "<YOUR_SPLUNK_ACCESS_TOKEN>"

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [otlphttp]
(Replace <YOUR_SPLUNK_ACCESS_TOKEN> with your actual Splunk Observability Cloud token.)

Step 3: Start the OpenTelemetry Collector
If using Docker, restart the collector:
docker restart otel-collector
If running manually, start it with:
otelcol --config=otel-collector-config.yaml

At this point, the OpenTelemetry Collector should be scraping metrics from the .NET app and preparing to send them to Splunk! 

3.Send Metrics to Splunk Observability Cloud

Goal: Ensure that metrics from the Otel Collector are being ingested into Splunk.
Step 1: Verify the OpenTelemetry Collector Logs
Run:

docker logs otel-collector | grep "error"
If there are no errors, the Otel Collector is running fine.

Step 2: Check Metrics in Splunk Observability Cloud
Log in to Splunk Observability Cloud.
Navigate to Metrics → Metric Finder.
Search for your app’s metrics, e.g.:
metric_name: dotnet_app_http_requests_total
If metrics appear, the setup is working correctly! 

Step 3: Set Up a Dashboard in Splunk
Go to Dashboards → Create Dashboard.
Click Add Chart → Select Time Series.
Query your metric:
metric_name: dotnet_app_http_requests_total
Click Save to create the chart.




Method 2 Using Nuggets :


1. Install the Splunk OpenTelemetry .NET NuGet Package
The NuGet package method is the easiest and recommended for instrumenting your application, especially if you control the build process but not the machine where the application is running.

Step 1: Install the NuGet Package
Open your terminal/command prompt.

Navigate to the root directory of your .NET application where the .csproj file is located.


cd <path_to_your_application>
Run the following command to add the Splunk OpenTelemetry Auto Instrumentation NuGet package to your project:


dotnet add <project_name>.csproj package Splunk.OpenTelemetry.AutoInstrumentation --prerelease
Replace <project_name>.csproj with the name of your actual project file.

Step 2: Handle Missing Instrumentation Packages
If the build fails and prompts you to add missing instrumentation packages, you can either:

Add the missing instrumentation package manually.

Or, you can skip the instrumentation of specific packages by adding them to the SkippedInstrumentation property in your .csproj file. For example:

xml
Copy
<PropertyGroup>
  <SkippedInstrumentations>MongoDB.Driver.Core;StackExchange.Redis</SkippedInstrumentations>
</PropertyGroup>
Alternatively, you can do this through the terminal:


dotnet build -p:SkippedInstrumentations=StackExchange.Redis%3BMongoDB.Driver.Core
Step 3: Build the Application
After installing the NuGet package and handling any missing packages, build the application.


dotnet build
Step 4: Run the Instrumented Application
After building your application, you’ll find launch scripts generated in the output folder.

For Windows, use the splunk-launch.cmd script.
For Linux, use the splunk-launch.sh script.
For example, on Windows:


splunk-launch.cmd <application_executable>
Or using the .NET CLI:


splunk-launch.cmd dotnet <your_application.dll>
Make sure to replace <application_executable> with the name of your executable or the appropriate .dll file.

Step 5: Verify Instrumentation
(Optional) If you want to verify that the instrumentation is working and view telemetry data in your console, you can add the OTEL_*_EXPORTER environment variables:

For tracing:


set OTEL_TRACES_EXPORTER=otlp,console
For metrics:


set OTEL_METRICS_EXPORTER=otlp,console
For logs:

set OTEL_LOGS_EXPORTER=otlp,console
Once you run your application, you should start seeing telemetry data in your console and in Splunk Observability Cloud (Splunk O11y Cloud).





3. Manual Installation of Splunk OpenTelemetry .NET
If you prefer to manually install the Splunk OpenTelemetry .NET instrumentation, follow these steps.

Step 1: Download the Splunk OpenTelemetry .NET Distribution
Download the required installation files from the official GitHub Releases page:

Windows: Download Windows PowerShell module
Linux: Download Linux installation script
Import the PowerShell Module (for Windows):

powershell
Copy
$module_url = "https://github.com/signalfx/splunk-otel-dotnet/releases/latest/download/Splunk.OTel.DotNet.psm1"
$download_path = Join-Path $env:temp "Splunk.OTel.DotNet.psm1"
Invoke-WebRequest -Uri $module_url -OutFile $download_path
Import-Module $download_path
Step 2: Install OpenTelemetry for .NET
Run the install command in PowerShell to install OpenTelemetry.

powershell
Copy
Install-OpenTelemetryCore
Step 3: Register the Instrumentation
You need to register your application type (e.g., .NET, ASP.NET, Windows service). For example:

For a .NET application:

powershell
Copy
Register-OpenTelemetryForCurrentSession -OTelServiceName "MyAppService"
For ASP.NET applications running on IIS:

powershell
Copy
Register-OpenTelemetryForIIS
You can also add a service version if required:

powershell
Copy
$env:OTEL_RESOURCE_ATTRIBUTES='deployment.environment=prod,service.version=1.0'
Step 4: Run Your Application
Once the instrumentation is set up, run your application as usual. The telemetry data will be collected and sent to Splunk Observability Cloud.

Step 5: Verify Instrumentation
As with the NuGet method, you can verify the instrumentation by checking the Splunk Observability Cloud to see if your application's data (traces, metrics, logs) appears.

3. Optional Configuration
You can configure advanced settings to customize the instrumentation based on your needs.

Change trace propagation formats.
Change the endpoint URLs.
Add custom attributes to spans (e.g., for more detailed trace information).
For more advanced configuration options, refer to the Advanced Configuration page of the Splunk OpenTelemetry .NET documentation.

4. Sending Data to Splunk Observability Cloud Directly
By default, telemetry data is sent to the local OpenTelemetry Collector. However, if you want to send data directly to Splunk Observability Cloud, you can set the following environment variables:

Windows:

powershell
Copy
$env:SPLUNK_ACCESS_TOKEN=<access_token>
$env:SPLUNK_REALM=<realm>
Linux:

bash
Copy
export SPLUNK_ACCESS_TOKEN=<access_token>
export SPLUNK_REALM=<realm>
Replace <access_token> with your Splunk API token and <realm> with your Splunk realm (e.g., us1, eu1).