Monitor C++ Application

Prerequisite

C++ application should be compiled with OpenTelemetry libraries.

Installation of Otel Forwarder

note

The below given steps are applicable only for Linux VM.

Follow the below steps to install Otel data forwarder:

1. Click the below link and download the file.

   https://raw.githubusercontent.com/snappyflow/apm-agent/master/otel-forwarder-install.sh

2. Run the following commands.

   chmod +x otel-forwarder-install.sh 

   sudo ./otel-forwarder-install.sh 

3. Allow port 9595 to receive data from instrumented C++ application. The port accepts only HTTPS communication. The forwarder endpoint will be by default https://0.0.0.0:9595/otel-trace-service/export which relys on basic authentication with username and password.

Modify Authentication

The authentication can be changed in two methods:

• During installation phase
• Through environment file

During installation phase

You the change the username and password while installing the Otel forwarder as shown in the below command.

./otel-forwarder-install.sh --env   "username=<-username->, password=<-password>"

Through environment file

You can add or modify the existing username and password post installation of Otel forwarder. Add the below lines to the env.conf file located at /opt/sfagent/otel-trace-data-forwarder.

username=<username>
password=<password>

Opentelemetry Instrumentation of C++ Application

The instrumentation of a C++ application involves making code modifications and establishing links with the OpenTelemetry libraries.

Follow the below steps:

1. Set the below environment variables in shell.

   SF_FORWARDER_URL=https://<InstanceIP>:9595/otel-service/export
   SF_FORWARDER_AUTH_USER=otelforwarderuser
   SF_FORWARDER_AUTH_PASS=Forwarder-SnappyFlow-Agent@7$
   SF_TARGET_PROJECT_NAME=<target-sf-projectName>
   SF_TARGET_APP_NAME=<target-sf-appName>
   SF_TARGET_PROFILE_KEY=<target-sfProfileKey>

   InstanceIP: IP address of the device where Sf Otel forwarder is installed.

   target-sf-projectName: The project name under which monitored data will be displayed in SnappyFlow.

   target-sf-appName: The project name under which monitored data will be displayed in SnappyFlow.

   target-sfProfileKey = Profile key from the SnappyFlow.

2. Create a trace_common.h file. This header file includes all the required libraries. It has initialization and cleanup method for both trace and logger. Add the code give below to the trace_common.h file. Change the SERVICE_NAME with real time value.

   ////This file contains code for initialization and clean-up method for trace and logger.
   #include "opentelemetry/exporters/otlp/otlp_http_exporter_factory.h"
   #include "opentelemetry/sdk/logs/batch_log_record_processor_factory.h"
   #include "opentelemetry/sdk/logs/batch_log_record_processor.h"
   #include "opentelemetry/exporters/otlp/otlp_http_exporter_options.h"
   #include "opentelemetry/context/propagation/global_propagator.h"
   #include "opentelemetry/context/propagation/text_map_propagator.h"
   #include "opentelemetry/exporters/ostream/span_exporter_factory.h"
   #include "opentelemetry/nostd/shared_ptr.h"
   #include "opentelemetry/sdk/trace/simple_processor_factory.h"
   #include "opentelemetry/sdk/trace/tracer_context_factory.h"
   #include "opentelemetry/sdk/trace/tracer_provider_factory.h"
   #include "opentelemetry/trace/propagation/http_trace_context.h"
   #include "opentelemetry/trace/provider.h"
   
   #include <cstring>
   #include <iostream>
   #include <vector>
   #include <fstream>
   #include <list>
   
   #define SERVICE_NAME “manage-employee”
   
   using namespace std;
   namespace nostd    = opentelemetry::nostd;
   namespace otlp     = opentelemetry::exporter::otlp;
   namespace sdktrace = opentelemetry::sdk::trace;
   namespace resource = opentelemetry::sdk::resource;
   namespace trace_sdk      = opentelemetry::sdk::trace;
   namespace logs_sdk  = opentelemetry::sdk::logs;
   namespace logs      = opentelemetry::logs;
   
   opentelemetry::nostd::shared_ptr<logs::Logger> get_logger()
   {
     auto provider = logs::Provider::GetLoggerProvider();
     return provider->GetLogger(SERVICE_NAME, SERVICE_NAME);
   }
   opentelemetry::nostd::shared_ptr<opentelemetry::trace::Tracer> get_tracer()
   {
     auto provider = opentelemetry::trace::Provider::GetTracerProvider();
     return provider->GetTracer(SERVICE_NAME);
   }
   
   typedef unsigned char uchar;
   static const std::string b = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";//=
   std::string base64_encode(std::string &in) {
       std::string out;
   
       int val=0, valb=-6;
       for (uchar c : in) {
           val = (val<<8) + c;
           valb += 8;
           while (valb>=0) {
               out.push_back(b[(val>>valb)&0x3F]);
               valb-=6;
           }
       }
       if (valb>-6) out.push_back(b[((val<<8)>>(valb+8))&0x3F]);
       while (out.size()%4) out.push_back('=');
       return out;
   }
   
   
   namespace
   {
     template <typename T>
     class HttpTextMapCarrier : public opentelemetry::context::propagation::TextMapCarrier
     {
     public:
       HttpTextMapCarrier<T>(T &headers) : headers_(headers) {}
       HttpTextMapCarrier() = default;
       virtual nostd::string_view Get(nostd::string_view key) const noexcept override
       {
         std::string key_to_compare = key.data();
         if (key == opentelemetry::trace::propagation::kTraceParent)
         {
           key_to_compare = "Traceparent";
         }
         else if (key == opentelemetry::trace::propagation::kTraceState)
         {
           key_to_compare = "Tracestate";
         }
         auto it = headers_.find(key_to_compare);
         if (it != headers_.end())
         {
           return it->second;
         }
         return "";
       }
   
       virtual void Set(nostd::string_view key, nostd::string_view value) noexcept override
       {
         headers_.insert(std::pair<std::string, std::string>(std::string(key), std::string(value)));
       }
   
       T headers_;
     };
   
   
     void initTracer(char *exporter_url, char *exporter_auth_user, char *exporter_auth_pass, 
                     char *sf_project_name, char *sf_app_name, char *sf_profile_id)
     {
       otlp::OtlpHttpExporterOptions opts;
       std::string url(exporter_url);
       std::string exporter_trace_endpoint = url + "/trace";
       
       opts.url = exporter_trace_endpoint;
       opts.content_type  = otlp::HttpRequestContentType::kBinary;
         
          
       std::string username(exporter_auth_user);
       std::string password(exporter_auth_pass);
       std::string authString = username + ":" + password;
       std::string encodedAuthString = base64_encode(authString);
   
       opts.http_headers = {{"Authorization", "Basic "+encodedAuthString}};
       
   
       char hostname[256];
       gethostname(hostname, sizeof(hostname));
         
       std::ifstream osRelease("/etc/os-release");
       std::string line;
       std::string os_type;
       std::string os_description;
       if (osRelease.is_open()) {
           while (getline(osRelease, line)) {
               if (line.substr(0, 8) == "ID_LIKE=") 
               {
                   os_type = line.substr(8);
               } 
               else if (line.substr(0, 12) == "PRETTY_NAME=") 
               {
                   os_description= line.substr(12);
               }
           }
           osRelease.close();
       }
       resource::ResourceAttributes resource_attributes = {
           {"service.name", SERVICE_NAME }, 
           {"service.version", "1.0.1"} ,
           {"host.name", hostname},
           {"os.type", os_type},
           {"os.description", os_description},
           {"snappyflow/projectname", sf_project_name},
           {"snappyflow/appname", sf_app_name},
           {"snappyflow/profilekey", sf_profile_id}
       };
       auto resource = resource::Resource::Create(resource_attributes);
   
       auto exporter = otlp::OtlpHttpExporterFactory::Create(opts);
       trace_sdk::BatchSpanProcessorOptions batchSpanOpts;
       batchSpanOpts.max_queue_size = 2048;
       batchSpanOpts.max_export_batch_size = 512;
       auto processor = trace_sdk::BatchSpanProcessorFactory::Create(std::move(exporter), batchSpanOpts);
   
       std::shared_ptr<opentelemetry::trace::TracerProvider> provider =
           trace_sdk::TracerProviderFactory::Create(std::move(processor), resource);
   
       trace_api::Provider::SetTracerProvider(provider);
   
       opentelemetry::context::propagation::GlobalTextMapPropagator::SetGlobalPropagator(
           opentelemetry::nostd::shared_ptr<opentelemetry::context::propagation::TextMapPropagator>(
               new opentelemetry::trace::propagation::HttpTraceContext()));
     }
     
     void initLogger(char *exporter_url, char *exporter_auth_user, char *exporter_auth_pass, 
                     char *sf_project_name, char *sf_app_name, char *sf_profile_id)
     {
       std::string url(exporter_url);
       std::string exporter_log_endpoint = url + "/log";
       
       otlp::OtlpHttpLogRecordExporterOptions logger_opts;
       
       logger_opts.url = exporter_log_endpoint;
       logger_opts.content_type  = otlp::HttpRequestContentType::kBinary;
   
       std::string username(exporter_auth_user);
       std::string password(exporter_auth_pass);
       std::string authString = username + ":" + password;
       std::string encodedAuthString = base64_encode(authString);
   
       logger_opts.http_headers = {{"Authorization", "Basic "+encodedAuthString}};
         // opts.insecure  = true;
       resource::ResourceAttributes resource_attributes = {
           {"service.name", "manage-employee"}, 
           {"service.version", "1.0.1"} ,
           {"snappyflow/projectname", sf_project_name},
           {"snappyflow/appname", sf_app_name},
           {"snappyflow/profilekey", sf_profile_id}
       };
       auto resource = resource::Resource::Create(resource_attributes);
       // Create OTLP exporter instance
       auto exporter  = otlp::OtlpHttpLogRecordExporterFactory::Create(logger_opts);
   
       logs_sdk::BatchLogRecordProcessorOptions  logProcesorOpts;
       logProcesorOpts.max_queue_size = 2048;
       logProcesorOpts.max_export_batch_size = 512;
   
       auto processor = logs_sdk::BatchLogRecordProcessorFactory::Create(std::move(exporter), logProcesorOpts);
       // auto processor = logs_sdk::SimpleLogRecordProcessorFactory::Create(std::move(exporter));
       std::shared_ptr<logs::LoggerProvider> provider =
           logs_sdk::LoggerProviderFactory::Create(std::move(processor), resource);
   
       opentelemetry::logs::Provider::SetLoggerProvider(provider);
     }
   
     void CleanupTracer()
     {
       std::shared_ptr<opentelemetry::trace::TracerProvider> none;
       trace_api::Provider::SetTracerProvider(none);
     }
     void CleanupLogger()
     {
       std::shared_ptr<logs::LoggerProvider> none;
       opentelemetry::logs::Provider::SetLoggerProvider(none);
     }
   }

3. In each of the application file make below changes.

   a. Include below headers.

   #include "opentelemetry/exporters/otlp/otlp_http_exporter_factory.h"
   #include "opentelemetry/exporters/otlp/otlp_http_exporter_options.h"
   #include "opentelemetry/sdk/trace/batch_span_processor_factory.h"
   #include "opentelemetry/sdk/trace/batch_span_processor_options.h"
   #include "opentelemetry/sdk/trace/tracer_provider_factory.h"
   #include "opentelemetry/trace/propagation/http_trace_context.h"
   #include "opentelemetry/trace/provider.h"
   #include "opentelemetry/exporters/ostream/span_exporter_factory.h"
   #include "opentelemetry/sdk/trace/simple_processor_factory.h"
   #include "opentelemetry/sdk/trace/tracer_provider_factory.h"
   #include "opentelemetry/trace/provider.h"
   #include "opentelemetry/trace/semantic_conventions.h"
   #include "opentelemetry/ext/http/client/http_client_factory.h"
   #include "opentelemetry/ext/http/common/url_parser.h"
   #include "opentelemetry/trace/context.h"
   #include "opentelemetry/exporters/otlp/otlp_http_log_record_exporter_factory.h"
   #include "opentelemetry/exporters/otlp/otlp_http_log_record_exporter_options.h"
   #include "opentelemetry/logs/provider.h"
   #include "opentelemetry/sdk/common/global_log_handler.h"
   #include "opentelemetry/sdk/logs/logger_provider_factory.h"
   #include "opentelemetry/sdk/logs/simple_log_record_processor_factory.h"
   #include "tracer_common.h"

   b. Create Namespace.

   using namespace opentelemetry::trace;
   namespace trace_api      = opentelemetry::trace;
   namespace trace_sdk      = opentelemetry::sdk::trace;
   namespace trace_exporter = opentelemetry::exporter::trace;
   namespace context     = opentelemetry::context;
   namespace otel_http_client = opentelemetry::ext::http::client;

   c. Add the below lines in the main function to read the environment variables and initialize the tracer and logger.

   char* forwarder_url = std::getenv("SF_FORWARDER_URL");
   if (forwarder_url == nullptr) {
      std::cerr << "SF_FORWARDER_URL is not set; exiting;" << std::endl;
      return -1;
    }
   char* forwarder_auth_user = std::getenv("SF_FORWARDER_AUTH_USER");
   if (forwarder_auth_user == nullptr) {
       std::cerr << "SF_FORWARDER_AUTH_USER is not set; exiting;" << std::endl;
       return -1;
   }
   
   char* forwarder_auth_pass = std::getenv("SF_FORWARDER_AUTH_PASS");
   if (forwarder_auth_pass == nullptr) {
       std::cerr << "SF_FORWARDER_AUTH_PASS is not set; exiting;" << std::endl;
       return -1;
   }
       
   char* sf_target_project_name = std::getenv("SF_TARGET_PROJECT_NAME");
   if (sf_target_project_name == nullptr) {
       std::cerr << "SF_TARGET_PROJECT_NAME is not set; exiting;" << std::endl;
       return -1;
   }
   char* sf_target_app_name = std::getenv("SF_TARGET_APP_NAME");
   if (sf_target_app_name == nullptr) {
       std::cerr << "SF_TARGET_APP_NAME is not set; exiting;" << std::endl;
       return -1;
   }
   char* sf_target_profile_key = std::getenv("SF_TARGET_PROFILE_KEY");
   if (sf_target_profile_key == nullptr) {
      std::cerr << "SF_TARGET_PROFILE_KEY is not set; exiting;" << std::endl;
      return -1;
   }
   initTracer(forwarder_url, forwarder_auth_user, forwarder_auth_pass, 
             sf_target_project_name, sf_target_app_name, sf_target_profile_key);
   initLogger(forwarder_url, forwarder_auth_user, forwarder_auth_pass, 
              sf_target_project_name, sf_target_app_name, sf_target_profile_key);
   

   d. Add below code according to your application requirement. This is used to start, initialize, and end the span. It also initializes the logger.

   
   //start the span and initialize it. 
   StartSpanOptions options;
   options.kind = SpanKind::kInternal;
   auto span =  get_tracer()->StartSpan("details  of the function", options);
   auto scope = get_tracer()->WithActiveSpan(span);
       
   //get logger and set the logger context to context of the span
   auto logger      = get_logger();
   auto ctx         = span->GetContext();
   
   //depending on the type of function use the appropriate attributes
   //set the attributes of the span. This example is for a sql query.
   span->SetAttribute(SemanticConventions::kDbName, "employeeDB");
   span->SetAttribute("db.type", "sqlite");
   span->SetAttribute("db.action", "query");
   span->SetAttribute("db.instance", "HDD-Optimized-v1");
   span->SetAttribute(SemanticConventions::kDbUser, "sabari");
   span->SetAttribute(SemanticConventions::kDbStatement, query_name);
   
   ///set the attributes of the span. This example is for a http request
   span->SetAttribute(SemanticConventions::kServerAddress, “server_name”);
   span->SetAttribute(SemanticConventions::kServerPort, “server_port”);
   span->SetAttribute(SemanticConventions::kHttpRequestMethod, request.method},
   span->SetAttribute(SemanticConventions::kUrlScheme, "http"},
   span->SetAttribute(SemanticConventions::kHttpRequestBodySize,
                                     static_cast<uint64_t>(request.content.length())},
   span->SetAttribute(SemanticConventions::kClientAddress, request.client}},
                                   options);
   
   //Every log must be changed to the format mentioned below to get it monitored in Opentelemetry
   //supported log level are kError, KDebug, KInfo, KWarn, Kfatal,KTrace
   
   logger->EmitLogRecord(opentelemetry::logs::Severity::kError, "Insertion of employee data into DB is unsuccessful", ctx.trace_id(), ctx.span_id(), ctx.trace_flags(),opentelemetry::common::SystemTimestamp(std::chrono::system_clock::now()));
                   
     //add an event on success or failure of the API
     span->AddEvent("dB insertion of employee data successful");
   
     //end the span
     span->End();

IP Table Configuration

The profile key configures the agents to send data to SnappyFlow server. When you want the traffic directed towards SnappyFlow, modify the IP table as described below.

Use the following code to enable redirect.

sudo iptables -t nat -A OUTPUT -p https --dport 9200 -d <destination_ip> -j DNAT --to-destination <proxy_ip>:<proxy_host_port>

sudo iptables-save

Where,

destination_ip: IP/DNS of the Ground SnappyFlow.

proxy_ip: SfProxy IP

proxy_host_port: SfProxy host Port

Disable default firewall of VM

If firewall prevents sending data to destination port 9200, disable firewall.

Use the following command to disable the firewall.

sudo ufw disable 

View Metrics and Logs

Follow the below steps to view the logs and traces collected from your application.

1. Go to the Application tab in SnappyFlow and navigate to your Project > Application > Dashboard.
2. To view logs, navigate to the Log Management section and select Primary Storage option.
3. To view trace data, navigate to the Tracing section and click the View Transactions button. You can view the traces in the Aggregate and the Real Time tabs.