RiverBed AppInternal Configuration

CHAPTER 23 Network Configuration Screen

Overview

This screen displays the current network configuration for the analysis server virtual machine. You can also use this
screen to change some network settings.
Note that the administrative command line interface (CLI) provides additional commands to configure network
settings. (See the “Command Line Interface Reference“ for more detail on the CLI.) Changes made using the CLI are
reflected on this screen.
The settings on this screen are typically specified through the network “wizard” (the CLI command “networkcfg“)
after the analysis server virtual machine first starts. See “Configuring Networking on the Virtual Machine“ in the
“Analysis Server Installation“ documentation.

Description of Fields

General Network Settings

Hostname

The host name for the analysis server virtual machine. The host name is typically specified after installation through
the network “wizard” (the CLI command “networkcfg“). You can also change the host name using the CLI command
“hostname“.
Be careful about changing the host name. If you change the host name after it is initially set, you will likely also have
to change it elsewhere:
 On every system that has the AppInternals agent installed, as described in “Changing the Agent’s Analysis

Server“.
 In the “Collection Server Address“ setting in the “Collector Configuration Screen“
 If you replaced the default security certificate in the “TLS Certificate Configuration Screen“ with a certificate

that specifies a host name, you will need a new certificate with the new host name.

Riverbed SteelCentral AppInternals Version 10.0 113

Network Configuration Screen

Gateway

The IP address for the gateway to external networks. The gateway is typically specified after installation through the
network “wizard” (the CLI command “networkcfg“). You can also change the gateway name using the CLI command
“ip default-gateway <address>“.

Primary DNS Server

The IP address for the primary DNS (Domain Name System) server to translate a domain name to an IP address.
The DNS server can also be configured through the network “wizard” (the CLI command “networkcfg“) or by using
the CLI command “dns <address>“.

Secondary DNS Server

The IP address for the secondary DNS (Domain Name System) server.

DNS Domain List

A list of domain suffixes in your environment. The analysis server will not require a fully-qualified domain name to
resolve addresses in these domains.

Interface Settings

There are separate sections for each network interface that has been configured for the virtual machine.
In almost all cases, there will be just a single area, for the default interface eth0. However, other interfaces may have
been configured in the hypervisor for the virtual machine. In this case, there will sections for those interfaces.

DHCP

Use DHCP (Dynamic Host Configuration Protocol) to dynamically assign an IP address for the network interface. This
is typically specified after installation through the network “wizard” (the CLI command “networkcfg“) as described
in “Configuring Networking on the Virtual Machine“ in the “Analysis Server Installation“ documentation.
The display shows the following values:
 The IP address and subnet mask obtained from DHCP. The CLI command “show interface name <name>“ also

shows these values.
 The gateway and DNS server values provided by DHCP or explicitly configured in any of the following ways:

– In the “General Network Settings“ area of this screen
– Through the network “wizard” (the CLI command “networkcfg“)
– Using the CLI commands “ip default-gateway <address>“ and “dns <address>“

Static

Specify a static IP address and subnet mask for the network interface. This is typically specified after installation
through the network “wizard” (the CLI command “networkcfg“) as described in“Configuring Networking on the
Virtual Machine“ in the “Analysis Server Installation“ documentation.

114 Riverbed SteelCentral AppInternals Version 10.0

Network Configuration Screen

These settings can also be specified through the “ip address <ip-addr> netmask <subnetmask>“argument to the
“interface <interfacename>“ CLI command.

Riverbed SteelCentral AppInternals Version 10.0 115

Network Configuration Screen
116 Riverbed SteelCentral AppInternals Version 10.0

CHAPTER 24 TLS Certificate Configuration Screen

This screen contains the TLS (Transport Layer Security) certificate that the analysis server presents to HTTPS clients.
(Secure Socket Layer, or SSL is the old name for TLS.) Use this screen to replace the default certificate with a secure
TLS or SSL certificate.

Overview

The analysis server presents this certificate on this page to clients that connect to it. These clients include:
 AppInternals users that log in to the AppInternals web user interface
 AppInternals agents that connect to the analysis server
 Browsers accessing HTTPS web pages being monitored through JavaScript instrumentation (see “Enable

JavaScript Instrumentation (End-User Experience)“ for more details)

Riverbed SteelCentral AppInternals Version 10.0 119

TLS Certificate Configuration Screen

The default certificate supplied with AppInternals is a self-signed certificate issued by an untrusted authority. This
default certificate allows AppInternals to work without initial configuration. However, self-signed certificates are not
secure and cause issues, including the following:
 Web browsers generate a warning when any site presents a self-signed certificate. This means that users

connecting to AppInternals web interface see a warning similar to the following:

 JavaScript instrumentation may not work for applications that serve pages using HTTPS (secure HTTP). Because
the certificate is untrusted, browsers (depending on how they are configured) may generate an error and not load
the injected JavaScript snippet. This prevents generation of end-user data.

To avoid these issues, replace the default certificate with a new private encryption key and associated certificate (also
a called key/certificate pair), where the public certificate is signed by a trusted certificate authority (CA). The
key/certificate pair must be PEM encoded.
Use the “Set New Certificate“ settings to replace the default certificate.

Set New Certificate

The key/certificate pair must be PEM encoded. Typically, you obtain the key/certificate pair from the security
administrator in your organization.
The key/certificate pair may be in a single file or in multiple files. The CA may also provide a password for the private
key, and a signing-certificate chain
Follow these steps to replace the default certificate:

1) Copy the PEM-encoded public certificate in the Public key certificate text area. The public certificate starts with
the line -----BEGIN CERTIFICATE----- and ends with -----END CERTIFICATE-----. Include
both the beginning and ending lines.

120 Riverbed SteelCentral AppInternals Version 10.0

TLS Certificate Configuration Screen

2) Copy the PEM-encoded private key in the Private key text area. The private key certificate starts with the line
-----BEGIN PRIVATE KEY----- and ends with -----END PRIVATE KEY-----. Include both the
beginning and ending lines.

3) The CA may provide a password for the private key. If provided, this password is required to decrypt the private
key. Paste it in the Private key password text area.

4) The CA may supply a signing-certificate chain that verifies an intermediate CA back to a “root” certificate
authority. If provided, the certificate chain is required. paste it in the Certificate chain text area.

5) Click the Save button to save the new key/certificate pair and restart the analysis server interface. This may take
a few minutes.

Import the CA Certificate into Web Browsers

To automatically accept a signed certificate, Web browsers must store a certificate (the CA root certificate) that
identifies the certificate authority that signed the public certificate presented by the analysis server.
Web browsers by default include CA certificates from many third-party certificate authorities. If the trusted source that
signed the public certificate is internal to your organization, the CA certificate may not be in users’ browsers. When
this happens, users will see a warning when first connecting to the AppInternals interface. This warning is slightly
different than the warning when the analysis server presents the self-signed certificate created by the installation.
In this example from Firefox, the message notes that “the issuer certificate not trusted”. Compare this with the
self-signed certificate warning shown in the “Overview“:

In this case, users must add a CA root certificate to the browser manually. They must obtain a signing-certificate chain
or the CA root certificate from the certificate authority (CA) that issued the signed key/certificate supplied in the “Set
New Certificate“ settings.

Riverbed SteelCentral AppInternals Version 10.0 121

TLS Certificate Configuration Screen
Browsers can import a variety of file formats for the certificate. The following example shows adding a CA certificate
file named CA1_cacert.der to Firefox.
From the Tools menu, users click Options > Advanced > Certificates > View Certificates > Authorities > Import
and then supply the file path to the certificate:

122 Riverbed SteelCentral AppInternals Version 10.0

CHAPTER 25 Software Upgrade Screen

Upgrading

By default, the analysis server connects to a Riverbed server to check for and download upgrades. (You can change
this behavior as described in “Disabling Automatic Checking for Upgrades“.) When the analysis server detects an
upgrade, it displays a message in the web interface:

Click the Learn More link to open the Software Upgrade screen. The screen shows the current version of
AppInternals. When there is an upgrade available, the screen shows the Download button.
 Click Download to download the latest available upgrade package. A download message appear with a progress

indicator.
 When the download finishes, the Install button appears where the Download button had been. Click Install to

install the upgrade package. The upgrade should finish without intervention.
Depending on which analysis server components are part of the upgrade, you may be logged out of the analysis server
interface and see the following message:

Riverbed SteelCentral AppInternals Version 10.0 123

Software Upgrade Screen
In other cases, the analysis server interface will not be available and the Server Offline message appears:

When the upgrade completes, the message is removed and you can use the web interface again without logging in.
Note: Use the CLI to Upgrade if the Web Interface is Unavailable
If the analysis server web interface is not available, you can log in to the analysis server virtual machine and upgrade
using the CLI as described in the “Upgrading the Analysis Server“.

Manually Upload an ISO Image File

If you disable use of the Riverbed server to check for and download upgrades (as described in “Disabling Automatic
Checking for Upgrades“ configuration topic), you must manually obtain the upgrade files.
You can obtain upgrades from the AppInternals download page on the Riverbed support site. Download the ISO image
file for a particular upgrade to your local system.
Click the Upload ISO File button and navigate to the ISO image file you downloaded. A status message opens with
a progress indicator. When the upload finishes, the Install button appears in the upper part of the screen. Click Install
to complete the upgrade.

124 Riverbed SteelCentral AppInternals Version 10.0

CHAPTER 26 ACCOUNT (Link to Authentication
Service)

The ACCOUNT menu option opens the SteelCentral Authentication Service in a separate browser window.
The Authentication Service is a required component for AppInternals. It provides authentication (including single
sign-on) and authorization (account access privileges) to Riverbed products and components. The virtual machine that
hosts the AppInternals analysis server includes an installation of SteelCentral Authentication Service.
Use Authentication Service to manage users and roles, create additional user accounts, and set up LDAP
authentication. For details, see the online help for the Authentication Service.
As described in “Changing the Server for the Authentication Service“, you can configure the analysis server to use an
authentication service installation on another system.

Riverbed SteelCentral AppInternals Version 10.0 125

ACCOUNT (Link to Authentication Service)
126 Riverbed SteelCentral AppInternals Version 10.0

Configuration on the Agent System

This material describes AppInternals agent configuration tasks you perform on agent systems.
See “Configuration in the Web Interface“ for details on tasks you perform using the Web interface.

Riverbed SteelCentral AppInternals Version 10.0 1

2 Riverbed SteelCentral AppInternals Version 10.0

CHAPTER 1 Agent Time Synchronization

AppInternals requires that times agent systems and the analysis server be synchronized through a network time service
(such as Network Time Protocol (NTP) or Windows Time Service).

This section describes time synchronization considerations for the agent. For the analysis server, you specify similar
settings in the “Date/Time Configuration Screen“.

Problems Resulting from Unsynchronized Times

Basic operation of AppInternals does not rely on synchronization. The analysis server collects data from agents and
processes it regardless of any time difference. However, unless times are synchronized, the analysis server will have
the problems described here.

 Misalignment of Metrics in Graphs: Any graphs that show metric values over time require close
synchronization of the system times on each agent. If times are not synchronized, metric values will not be
plotted correctly along the time axis of the graph.

 Failure to Detect Related Cross-Tier Transactions: A large time difference—greater than three
hours—between any agents will also affect how AppInternals detects related cross-tier transactions. The analysis
server limits its detection of related transactions to within three hours of that transaction. If an agent has a time
skew greater than three hours, its transactions will not be included in the processing to detect related transactions.

Confirm That the System Time Is Synchronized 3

This section describes how to:
 Check whether the system time is synchronized and, if so, with which time servers
 If the time is not synchronized, how to specify a time server and start synchronization

UNIX

Check that System Time is Synchronized

Use a command such as ntpstat or ntpq to check if NTP is running and which time servers it is using.

Riverbed SteelCentral AppInternals Version 10.0

Agent Time Synchronization

In this example, ntpstat shows that the system is using the time server on system 70.35.113.44:

# ntpstat
synchronised to NTP server (70.35.113.44) at stratum 3

time correct to within 189 ms
polling server every 1024 s

If NTP is not running, ntpstat displays results similar to this:

# ntpstat
unsynchronised
time server re-starting
polling server every 64 s

The ntpq -p command shows more detail:

# ntpq -p

remote refid st t when poll reach delay offset jitter

==============================================================================

+ns.tx.primate.n 204.123.2.72 2 u 954 1024 377 66.734 -5.904 3.935

+70.35.113.44 129.6.15.28 2 u 460 1024 377 91.817 0.249 2.077

+206.51.211.152 206.51.211.153 4 u 960 1024 377 50.319 -0.554 5.437

*fairy.mattnordh 200.98.196.212 2 u 963 1024 377 52.607 -5.722 9.449

(See the following link for details on ntpq output:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/s1-Checkin
g_the_Status_of_NTP.html)

Start Time Synchronization

If NTP is not running, edit the /etc/ntp.conf file. Add a server command if necessary. (This may not be necessary. The
default file contains server commands that specify valid NTP servers.)

Start the NTP daemon by running ntpd and verify that it is running:

# which ntpd 0.0.0.0:*
/usr/sbin/ntpd 0.0.0.0:*
# ntpd 0.0.0.0:*
# netstat -an | grep 123 :::*
udp 0 0 192.168.175.23:123 :::*
udp 0 0 127.0.0.1:123
udp 0 0 0.0.0.0:123
udp 0 0 ::1:123
udp 0 0 :::123

Use the chkconfig command to see if the NTP daemon, ntpd, is configured to run when the system boots. In this
example, it is not:

# ./sbin/chkconfig --list ntpd

ntpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off

If necessary, use chkconfig to enable ntpd to run when the system boots. The example below accepts the default,
which enables ntpd for run levels 2 through 5:

# ./sbin/chkconfig --list ntpd

ntpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
2:on 3:on 4:on 5:on 6:off
# ./sbin/chkconfig ntpd on

# ./sbin/chkconfig --list ntpd

ntpd 0:off 1:off

4 Riverbed SteelCentral AppInternals Version 10.0

Agent Time Synchronization

Windows

Check that System Time is Synchronized

Use the w32tm command-line tool to see if the system time is currently synchronized. In this example, the value of
the ntpserver entry indicates the time is synchronized with system myntpserver:

C:>w32tm /dumpreg /subkey:parameters

Value Name Value Type Value Data

--------------------------------------------------------------------

ServiceMain REG_SZ SvchostEntry_W32Time
ServiceDll REG_EXPAND_SZ C:\WINDOWS\system32\w32time.dll
Type REG_SZ NTP
ntpserver REG_SZ myntpserver,0x1

(The preceding command returns the contents of the
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\w32time\Parameters key in the Windows registry.)

Start Time Synchronization

If the system time is not synchronized, use another w32tm command to specify a network time source:

C:\>w32tm /config /syncfromflags:manual /manualpeerlist:myntpserver,0x1 /update
The command completed successfully.

Riverbed SteelCentral AppInternals Version 10.0 5

Agent Time Synchronization
6 Riverbed SteelCentral AppInternals Version 10.0

CHAPTER 2 Changing the Agent’s Analysis
Server

This section describes how to change the analysis server that an agent connects to when it starts. The agent installation
requires the analysis server host name or IP address. Typically, you do not have change this value after installation.
However, if the analysis server host name or address changes, you must change this value to match. (The analysis
server installation topic “Configuring Networking on the Virtual Machine“ describes changing the host name or
address on the analysis server.)
Follow these steps:

1) Stop the agent.
– On Windows systems, stop the Riverbed SteelCentral AppInternals Agent service.
– On UNIX systems, use the dsactl stop command. See “Using dsactl to Control Agent Software on UNIX“ for
details.

2) In the file <installdir>\Panorama\hedzup\mn\data\dsa.xml, change the value of the AnalysisServerHost
attribute. For example:

<Attribute name="AnalysisServerHost" value="myAnalysisServer.example.com"/>

3) Restart the agent.

Riverbed SteelCentral AppInternals Version 10.0 7

Changing the Agent’s Analysis Server
8 Riverbed SteelCentral AppInternals Version 10.0

CHAPTER 4 Using dsactl to Control Agent
Software on UNIX

The dsactl script starts and stops the agent (the dsa process) and sub-agent (the os_agent process) on UNIX.

You must start the agent and sub-agent after installation using the dsactl script. Otherwise, you typically only restart
the agent during troubleshooting or if it stops responding.

The following table shows the dsactl commands to start the DSA and OS sub-agent.

Component dsactl Command Process(es)
agent dsactl start dsa dsa
OS sub-agent dsactl start os os_agent

The dsactl script resides in the <installdir>/Panorama/hedzup/mn/bin directory. Run the script from that directory or
add the directory to the search path environment variable.

Running dsactl

The dsactl script requires a command argument (start, stop, restart, or status) and, optionally, dsa or the name of a
sub-agent on which the command operates. If you do not specify dsa or a sub-agent name, the DSA and sub-agents
are targeted by the preceding argument.
For all but the status command, you must be logged in to the account specified during the installation to use dsactl. If
not, dsactl generates an error:

# ./dsactl stop
dsactl: FATAL: You must be running as 'admin' to execute this script.
dsactl: Unable to continue the dsactl process due to errors.

Syntax

dsactl {start|stop|restart|status} [dsa] [os]

Riverbed SteelCentral AppInternals Version 10.0 11

Using dsactl to Control Agent Software on UNIX

Arguments

start

Starts the DSA, the specified sub-agents, or all sub-agents and the DSA, if no additional arguments are specified:

$ ./dsactl start dsa solaris_lite
Starting selected Riverbed SteelCentral AppInternals Version 10.0 components...
starting the DSA...

[waiting 10 seconds for the DSA to fully start up]
[started] UID:panorama PID:3676 dsa -d
[started] UID:panorama PID:3679 solaris_lite_agent

stop

Stops the DSA, the specified sub-agents, or enabled sub-agents and the DSA, if no additional arguments are specified:

$ ./dsactl stop
Stopping all Riverbed SteelCentral AppInternals Version 10.0 components...
[stopped] solaris_lite_agent
[stopped] apache_agent
[stopped] db2_agent
[stopped] oracleDB_agent

stopping the DSA... [ok]
[stopped] dsa

restart

Stops and starts the DSA, the specified sub-agents, or all sub-agents and the DSA, if no additional arguments are
specified:

$ ./dsactl restart solaris
Stopping selected Riverbed SteelCentral AppInternals Version 10.0 components...
[stopped] solaris_lite_agent

[DSA is already started]
Starting selected Riverbed SteelCentral AppInternals Version 10.0 components...
[started] UID:admin PID:11893 dsa -d
[started] UID:admin PID:12432 solaris_lite_agent

status

Displays the process status of the specified sub-agents, or all sub-agents and the DSA, if no sub-agents are specified:

$ ./dsactl status
[started] UID:admin PID:10273 dsa -d
[started] UID:admin PID:10277 solaris_lite_agent
[started] UID:admin PID:10278 apache_agent -s localhost -p 80

dsa os …

Specifies the components to target with the preceding command. Specify dsa or os or both. If you do not specify either
dsa or os, both are targeted by the preceding command.

12 Riverbed SteelCentral AppInternals Version 10.0

CHAPTER 5 Enabling Instrumentation of Java
Processes

Overview

AppInternals monitors Java applications by loading a profiler library into processes when they start:
 The profiler library determines if a user configured a process for instrumentation in the “Agent Details Screen“.

Only processes with the ““Instrument?” Option“ selected will be instrumented.
 For those processes, monitoring begins once the JVM startup completes. AppInternals generates transaction trace

data according to the settings specified in the corresponding “Configuration Settings Screen“ for the process.
This section describes required configuration on the agent system for Java processes to load the AppInternals profiler
library. Until you enable instrumentation as described here, processes will not load the profiler library and will not be
monitored.
In most cases, this configuration is manual and requires taking one of the approaches described here. The only case
where you do not have to manually enable instrumentation is if you installed the AppInternals agent on UNIX as root
and chose to enable system-wide instrumentation (see “Whether to enable automatic system-wide instrumentation“ in
the installation documentation). This installation choice performs the same steps as the “Enable Instrumentation
Automatically“ approach described here.

Symptoms of Not Enabling Java Instrumentation

The AppInternals analysis server cannot detect if Java instrumentation is enabled for specific processes. As a result,
unless Java instrumentation is enabled, the user interface in some cases shows a misleading status for Java processes
in the “Agent Details Screen“.
When users choose to instrument Java processes (via the ““Instrument?” Option“ in the “Agent Details Screen“) for
which instrumentation has not been enabled, they will not be instrumented. The status will incorrectly show as
Awaiting Restart, but restarting the affected processes will have no effect until you enable instrumentation as
described here. For example, for a Tomcat application:

Riverbed SteelCentral AppInternals Version 10.0 17

Enabling Instrumentation of Java Processes

You correct this confusing behavior when you enable instrumentation as described here and restart the application:
 If process had already been selected for instrumentation, AppInternals will instrument it and it will appear in the

Agent Details screen as Instrumented. For example, for Tomcat:

 If the process had NOT been selected for instrumentation, the status will appear as Running. Selecting the
Instrument? option will change the status to Awaiting Restart. However, because instrumentation had been
enabled, the status is now correct. Restarting the application really will cause AppInternals to instrument it.

A reliable way to determine if a particular Java process is instrumented is to check on the agent system for the presence
of a JIDA sub-agent log file that corresponds to the process. The log files are in the directory
<installdir>\Panorama\hedzup\mn\log. Their names include the process name and time the process started. For the
Tomcat process in the previous example, the log file name would be similar to the following:

DA-JIDA_Tomcat_TOMCAT7_20150610130750_5908.txt

Summary of Approaches

The remainder of this section describes a number of approaches for loading the profiler library and enabling
instrumentation. The following table summarizes the different approaches. Click a link to see more details for a
specific approach.

Approach Summary
“JidaRegister.exe“
(Windows) Program that sets the system-wide JAVA_TOOL_OPTIONS environment variable to
add the profiler library:
“LD_PRELOAD“
(UNIX) • JAVA_TOOL_OPTIONS attempts to load the profiler library in all Java processes
that start on the system.
“JAVA_TOOL_OPTIONS“
(UNIX) • If JAVA_TOOL_OPTIONS specifies the 64-bit library, applications that use a
32-bit JVM will not start (and vice-versa).

Environment variable set at the process level by modifying the application startup
script or at the user level by modifying a user profile (such as ~/.profile):

• Loads the profiler library in all processes within the scope of the variable.

• If agent software is removed without removing the option that loads the library
from LD_PRELOAD, Java processes will start but generate errors.

Environment variable set at the process level by modifying the application startup
script or at the user level by modifying a user profile (such as ~/.profile):

• Loads the profiler library in all Java processes within the scope of the variable.

• Causes an additional message to be written to standard error (stderr), which may
cause issues in some applications.

• If agent software is removed without removing the option that loads the library
from JAVA_TOOL_OPTIONS, Java processes will no longer start.

18 Riverbed SteelCentral AppInternals Version 10.0

Enabling Instrumentation of Java Processes

Approach Summary

“Enable Instrumentation Copies shared objects to the /lib and /lib64 system directories and adds an entry to the
Automatically“ /etc/ld.so.preload system file:
(UNIX) • Requires root access.
• Loads the profiler library in every process that starts on the system.
“Add -agentpath to Java
Command Line (Windows and Manually add the -agentpath Java command line option to the application startup:
UNIX)“ • Loads the profiler library only in processes that include the -agentpath option.
• Requires knowledge of where to supply the -agentpath option (such as in a startup

script, user interface, or configuration file).

Windows

On Windows, to enable Java instrumentation, use either of the following approaches:

 Set the JAVA_TOOL_OPTIONS environment variable by running
<installdir>\Panorama\hedzup\mn\bin>JidaRegister.exe. See “JidaRegister.exe“ for details.

 Manually add the -agentpath Java command line option to the application startup. See “Add -agentpath to Java
Command Line (Windows and UNIX)“.

JidaRegister.exe

The JAVA_TOOL_OPTIONS environment variable specifies the -agentpath Java option to load the AppInternals
profiler library. Run the <installdir>\Panorama\hedzup\mn\bin>JidaRegister.exe program to set
JAVA_TOOL_OPTIONS as a system-wide environment variable.

Run JidaRegister.exe and respond to prompts:

 Whether to instrument (load the profiler library) in Java processes. Specify y to proceed.

 Whether to load the 64-bit (specify y) or 32-bit version (specify n) of the library.

The following example specifies the 64-bit library:

C:\Panorama\hedzup\mn\bin>JidaRegister.exe
Running as user "NHX1-W2K8R2-12\Administrator"
Do you want to instrument your Java applications? [y|n]:y
Do your applications use a 64-bit JRE? [y|n]:y
Setting system environment variable:
JAVA_TOOL_OPTIONS -agentpath:C:\Panorama\hedzup\mn\bin\rpilj64.dll
done.
A log file has been written to C:\Panorama\hedzup\mn\log\JidaRegister.log
C:\Panorama\hedzup\mn\bin>

Riverbed SteelCentral AppInternals Version 10.0 19

Enabling Instrumentation of Java Processes

Once you run JidaRegister.exe, select the Java processes you want to instrument (via the ““Instrument?” Option“
option in the “Agent Details Screen“) and restart the corresponding Windows services. For example, for Tomcat:

After restarting the services, the applications will load the profiler library and they will appear in the Agent Details
screen as Instrumented. For example, for Tomcat:

Keep in mind the following points when using JidaRegister.exe:
 If JAVA_TOOL_OPTIONS had already been set with options unrelated to AppInternals, JidaRegister.exe

preserves those options and adds the option to load the profiler library at the beginning of the variable.
 Use the JidaRegister.exe with the uninstall argument to remove AppInternals options from

JAVA_TOOL_OPTIONS:

C:\Panorama\hedzup\mn\bin>JidaRegister.exe uninstall
Running as user "NHX1-W2K8R2-12\Administrator"
Unsetting environment variables...A log file has been written to C:\Panorama\hedzup\mn\log\Ji
daRegister.log

When you remove agent software on Windows (see “Removing Agent Software“), the uninstall program runs
JidaRegister.exe with the uninstall argument.
 Once you choose the 64-bit or 32-bit library, any JVM that does not match that choice will fail to start. For
example, after choosing the 64-bit library, running a 32-bit JVM fails:

C:\Users\Administrator>java -version
Picked up JAVA_TOOL_OPTIONS: -agentpath:C:\Panorama\hedzup\mn\bin\rpilj64.dll
Error occurred during initialization of VM
Could not find agent library C:\Panorama\hedzup\mn\bin\rpilj64.dll in absolute path, with err
or: Can't load AMD 64-bit .dll on a IA 32-bit platform

20 Riverbed SteelCentral AppInternals Version 10.0

Enabling Instrumentation of Java Processes

 Although Windows services started after running JidaRegister.exe pick up changes to JAVA_TOOL_OPTIONS,
other processes may not. For example, new Windows Command Prompt processes will not, so any applications
restarted from the command line will not load the profiler library. You must reboot the system to reliably
propagate changes from running JidaRegister.exe to all processes. You can propagate the changes to some (but
not all) new processes by clicking OK in the Windows Environment Variables window:

UNIX

On UNIX, to enable Java instrumentation, use any of the following approaches:
 Set the LD_PRELOAD environment variable at the process level by modifying the application startup script or at

a user level by modifying a user profile (such as ~/.profile). See “LD_PRELOAD“ for details.
 Set the JAVA_TOOL_OPTIONS environment variable at the process level by modifying the application startup

script or at a user level by modifying a user profile (such as ~/.profile). See “JAVA_TOOL_OPTIONS“ for
details.
 Automatically enable instrumentation by adding an entry to /etc/ld.so.preload. See “Enable Instrumentation
Automatically“.
 Manually add the -agentpath Java command line option to the application startup. See “Add -agentpath to Java
Command Line (Windows and UNIX)“.

LD_PRELOAD

The LD_PRELOAD environment variable specifies an absolute path to shared objects that processes will load before
any other library. Unlike “JAVA_TOOL_OPTIONS“, LD_PRELOAD affects all processes that start in the scope of
the environment variable, not only Java processes.

Riverbed SteelCentral AppInternals Version 10.0 21

Enabling Instrumentation of Java Processes

Typically, you set LD_PRELOAD in the startup script for applications. You could also add it to a user profile (such as
~/.profile) so that it would automatically affect all processes started by that user.

Set LD_PRELOAD as follows:

export LD_PRELOAD="<installdir>/Panorama/hedzup/mn/\$LIB/librpil.so $LD_PRELOAD"

Note the following:

 Replace <installdir> with the directory where the AppInternals agent software is installed.

 Do not change the \$LIB reference in the path. It is a Dynamic String Token (DST) that resolves to either the
<installdir>/Panorama/hedzup/mn/lib/ or <installdir>/Panorama/hedzup/mn/lib64/ directory. This means you
can use the same LD_PRELOAD value for both 32-bit and 64-bit JVMs.

The following example shows a script that adds LD_PRELOAD and starts multiple Tomcat servers:

[root@jidatest opt]# more begin_apps.sh
export LD_PRELOAD="/opt/Panorama/hedzup/mn/\$LIB/librpil.so $LD_PRELOAD"
/opt/apache-tomcat-t1Client/bin/startup.sh jidatest1
/opt/apache-tomcat-t2Flight/bin/startup.sh jidatest2
/opt/apache-tomcat-t3Schedl/bin/startup.sh jidatest3
[root@jidatest opt]#

After you set LD_PRELOAD, select the Java processes you want to instrument (via the ““Instrument?” Option“ option
in the “Agent Details Screen“) and restart the applications. After restarting, the applications will load the profiler
library and they will appear in the Agent Details screen as Instrumented.

JAVA_TOOL_OPTIONS

Use the JAVA_TOOL_OPTIONS environment variable to specify the -agentpath Java option to load the AppInternals
profiler library. Java processes that are in the scope of the environment variable add the option when they start.

JAVA_TOOL_OPTIONS is similar in its usage and effects to “LD_PRELOAD“. Unlike LD_PRELOAD, it affects
only Java processes within its scope. However, it has the following negative side effects that LD_PRELOAD does not,
which generally makes LD_PRELOAD preferable:

 It generates an additional message to standard error (stderr) about loading the AppInternals library. This may
cause issues in applications that, for example, check for the presence of output in standard error as a problem. The
following example uses java -version and redirects its standard error output to a file (2> show.stderr), then
shows the additional message in the file in bold:

bash-4.1$echo$JAVA_TOOL_OPTIONS
-agentpath:/opt/Panorama/hedzup/mn/$LIB/librpilj.so
bash-4.1$java-version2>show.stderr
bash-4.1$moreshow.stderr
PickedupJAVA_TOOL_OPTIONS:-agentpath:/opt/Panorama/hedzup/mn/$LIB/librpilj.so
javaversion"1.6.0_24"
OpenJDKRuntimeEnvironment(IcedTea61.11.1)(rhel-1.45.1.11.1.el6-x86_64)
OpenJDK64-BitServerVM(build20.0-b12,mixedmode)

 If the AppInternals agent software is removed without removing the applicable -agentpath option from
JAVA_TOOL_OPTIONS, Java processes within its scope will no longer start.

Typically, you set JAVA_TOOL_OPTIONS in the startup script for applications. You could also add it to a user profile
(such as ~/.profile) so that it would automatically affect all processes started by that user.

Set JAVA_TOOL_OPTIONS as follows:

export JAVA_TOOL_OPTIONS="-agentpath:<installdir>/Panorama/hedzup/mn/\$LIB/librpilj.so
$JAVA_TOOL_OPTIONS"

Note the following:

 Replace <installdir> with the directory where the AppInternals agent software is installed.

22 Riverbed SteelCentral AppInternals Version 10.0

Enabling Instrumentation of Java Processes

 Do not change the \$LIB reference in the path. It is a Dynamic String Token (DST) that resolves to either the
<installdir>/Panorama/hedzup/mn/lib/ or <installdir>/Panorama/hedzup/mn/lib64/ directory. This means you
can use the same JAVA_TOOL_OPTIONS value for both 32-bit and 64-bit JVMs.

The following example shows a script that adds JAVA_TOOL_OPTIONS and starts multiple Tomcat servers:

[root@jidatest opt]# more begin_apps.sh
export JAVA_TOOL_OPTIONS="-agentpath:/opt/Panorama/hedzup/mn/\$LIB/librpilj.so $JAVA_TOOL_OPTIONS"
/opt/apache-tomcat-t1Client/bin/startup.sh jidatest1
/opt/apache-tomcat-t2Flight/bin/startup.sh jidatest2
/opt/apache-tomcat-t3Schedl/bin/startup.sh jidatest3
[root@jidatest opt]#

After you set JAVA_TOOL_OPTIONS, select the Java processes you want to instrument (via the ““Instrument?”
Option“ option in the “Agent Details Screen“) and restart the applications. After restarting, the applications will load
the profiler library and they will appear in the Agent Details screen as Instrumented.

Enable Instrumentation Automatically

Use the script <installdir>/Panorama/hedzup/mn/bin/instrumentationoptions.sh to enable instrumentation
automatically for all processes on the system. This script generates messages that summarize the approaches for
enabling instrumentation on UNIX. (The agent installation also runs this script when run as root and the user chooses
to do so. See “Whether to display a summary of options for enabling instrumentation“ in the installation
documentation).

When instrumentationoptions.sh runs as root, after displaying the messages, the script prompts to enable
instrumentation. Respond yes to do so:

[root@nhv1-rh6-1 bin]# ./instrumentationoptions.sh
.
.
.
*** Option 4: ***
Enable instrumentation automatically system-wide. This is
easiest but adds an entry to the /etc/ld.so.preload system file.
Enable instrumentation automatically system-wide? [no]: yes
Process injection already disabled.
Successfully uninstalled process injection library.
Successfully installed process injection library.
Successfully enabled process injection.
[root@nhv1-rh6-1 bin]#

After you enable instrumentation system-wide, select the Java processes you want to instrument (via the
““Instrument?” Option“ option in the “Agent Details Screen“) and restart the applications. After restarting, the
applications will load the profiler library and they will appear in the Agent Details screen as Instrumented.

Enabling system-wide instrumentation does the following:

 Copies or creates symbolic links of the librpil*.so shared objects in <installdir>/Panorama/hedzup/mn/lib64 and
<installdir>/Panorama/hedzup/mn/lib64 in the /lib and /lib64 system directories, respectively.

 Adds the following entry to the system file /etc/ld.so.preload:

/$LIB/librpil.so

To disable system-wide instrumentation, run the script <installdir>/Panorama/hedzup/mn/bin/rpictrl.sh with the
disable argument:

[root@nhv1-rh6-1 bin]# ./rpictrl.sh disable
Successfully disabled process injection.

The disable operation removes the entry from /etc/ld.so.preload.

Riverbed SteelCentral AppInternals Version 10.0 23

Enabling Instrumentation of Java Processes

Any time system-wide instrumentation is enabled or disabled, AppInternals writes a message to the file
/var/log/messages. For example:

Jun 25 15:51:30 nhv1-rh6-1 root: Successfully disabled process injection.
Jun 25 15:52:29 nhv1-rh6-1 root: Successfully enabled process injection.

Add -agentpath to Java Command Line (Windows and UNIX)

You can specify the -agentpath option directly in the Java command that starts the application in which you want to
load the profiler library. You cannot use this approach for native applications with “embedded” JVMs that do not use
the Java launcher (java.exe) command line.
Specify the 64-bit or 32-bit version of the library and the AppInternals installation directory for the system:

Environment -agentpath Value and Example
Windows 32-bit JVM
Windows 64-bit JVM <installdir>\Panorama\hedzup\mn\bin\rpilj.dll
Linux 32-bit JVM
Linux 64-bit JVM -agentpath:C:\Panorama\hedzup\mn\bin\rpilj.dll

<installdir>\Panorama\hedzup\mn\bin\rpilj64.dll

-agentpath:C:\Panorama\hedzup\mn\bin\rpilj64.dll

<installdir>/Panorama/hedzup/mn/lib/librpilj.so

-agentpath:/opt/Panorama/hedzup/mn/lib/librpilj.so

<installdir>/Panorama/hedzup/mn/lib/librpilj64.so

-agentpath:/opt/Panorama/hedzup/mn/lib/librpilj64.so

Because -agentpath is configured for each process, it avoids a potential problem with the Windows system-wide
JAVA_TOOL_OPTIONS variable set by running “JidaRegister.exe“. With that approach, if there is a mismatch
between the 32-bit or 64-bit library specified by JAVA_TOOL_OPTIONS and the JVM that loads it, the JVM will not
start.

However, you must know where to specify the -agentpath options for the application. This varies by application and
may be in a startup script, user interface, or a configuration file.

24 Riverbed SteelCentral AppInternals Version 10.0

Enabling Instrumentation of Java Processes
For example, for Tomcat on Windows, you specify the option in the Tomcat properties dialog, in the Java tab:

After you specify the -agentpath option, select the Java processes you want to instrument (via the ““Instrument?”
Option“ option in the “Agent Details Screen“) and restart the applications. After restarting, the applications will load
the profiler library and they will appear in the Agent Details screen as Instrumented.

Riverbed SteelCentral AppInternals Version 10.0 25

Enabling Instrumentation of Java Processes
26 Riverbed SteelCentral AppInternals Version 10.0

CHAPTER 6 Instrumentation Techniques and
Troubleshooting

This section describes techniques for configuring and troubleshooting instrumentation using the JIDA and dotNet
sub-agents.

Hotspot Detection

AppInternals automatically tunes which methods it monitors to keep performance impact to a minimum. This
“hotspot” detection insures that AppInternals does not monitor short methods that are called with excessive frequency.

JIDA and the dotNet sub-agent dynamically monitor execution time of all methods they monitor. If a method is called
many times but has a very short execution time, AppInternals declares it a “hotspot” and automatically disables
monitoring.

As a result, over time, AppInternals' monitoring of these expensive methods automatically tapers to nothing. Although
there are typically only a few hotspot methods in applications (even large applications), testing shows that disabling
monitoring of these few methods heavily reduces AppInternals overhead.

AppInternals keeps a record of all the hotspot methods in the
<installdir>\Panorama\hedzup\mn\userdata\cache\jida_hotspots and dotnet_hotspots directories. The directory
contains zero-length files with the name of each method for which the sub-agents disabled monitoring. For example:

[nhapp1:panorama]>ls jida_hotspots/
com.ibm.ejs.container.EJSHome.activateBean
com.ibm.ejs.container.EJSHome.createBeanO
com.ibm.ejs.container.EJSHome.createWrapper
com.ibm.ejs.container.EJSHome.getFinderEntityBeanO
com.ibm.ejs.container.EJSHome.getWrapper
com.ibm.ejs.container.EJSHome.internalCreateWrapper
com.ibm.ejs.container.EJSHome.preFind
com.ibm.ejs.container.EJSHome.releaseFinderEntityBeanO
com.ibm.websphere.samples.trade.ejb.AccountBean.getDataBean
com.ibm.websphere.samples.trade.ejb.ConcreteAccountEJB_b5483e03.ejbActivate
.
.
.

JIDA and the dotNet sub-agent read the list of hotspot methods in the jida_hotspots directory whenever an application
starts, which offers the following benefits:

 The application gets an immediate performance boost since the sub-agents avoid the overhead of initial hotspot
detection.

Riverbed SteelCentral AppInternals Version 10.0 23

Instrumentation Techniques and Troubleshooting

 You can copy the <sub-agent>_hotspots directory from a testing environment to production and avoid the
overhead of initial detection.

 You can delete the <sub-agent>_hotspots directory for new versions of your application when you want to force
the sub-agents to go through hotspot detection again.

Troubleshooting Missing Classes

By default, AppInternals collects data for most application classes and methods (public only) that it determines are
“interesting”. The default automatic discovery and monitoring of interesting application classes and methods typically
works without problem and requires no manual configuration.

However, in some cases, the automatic monitoring may exclude classes and methods that are useful performance
indicators.

You can determine which classes and methods that the JIDA and dotNet sub-agents exclude from monitoring by using
the “Include methods not instrumented (Restart Required)“ option in the “Configuration Settings Screen“.

Note: This option makes log files grow very fast.

This option causes the sub-agents to log the name of each discovered method that is not instrumented and the reason
why it was not instrumented. Here is an example of messages that the JIDA sub-agent writes to its log file when this
setting is enabled:

05/21/2015 04:25:36 PM, JIDA , 2200 , INFO , Not instrumenting
, INFO , Not instrumenting
java.io.PrintWriter::append - size is only 6 bytes

05/21/2015 04:25:36 PM, JIDA , 2200

java.io.PrintWriter::write - method code not interesting

Agent and sub-agent log files reside in the <installdir>/Panorama/hedzup/mn/log directory on the agent system. Log
file names start with the prefix DA-dotNet or DA-JIDA and include the process name and time the process started. For
example:

DA-JIDA_Websphere_server1_20150613122039_22786.txt

Scan the log files for the process for which you want to see additional data. Look for Not instrumenting messages
and classes and methods that you want to monitor. You can then create matching custom filters to make sure they are
monitored, as described in “Manually choose packages, classes, and methods to instrument“.

To get access to the log files:

 Use a web browser. The agent serves log files at this URL: http://<agent-system>:2111/log/

 Create and download a “diagnostic bundle” that includes the log files. On the “Agent Details Screen“, click the
“Download the agent logs and diagnostic files“ link.

 Edit the log files directly in the <installdir>/Panorama/hedzup/mn/log directory.

Finding Classes and Methods That Are Instrumented

You can determine which classes and methods that the JIDA and dotNet sub-agents monitor by default by using the
“Verbose logging“ option in the “Configuration Settings Screen“.

24 Riverbed SteelCentral AppInternals Version 10.0

Instrumentation Techniques and Troubleshooting

This option causes the sub-agents to log the name of each method it discovers and will instrument, and when it actually
instruments the method once it runs. For example, for the dotNet sub-agent:

05/21/2015 03:55:48 PM, dotNet , 7060 , INFO , Will instrument

System.Web.UI.LiteralControl.Render:hasthis System.Void(System.Web.UI.HtmlTextWriter) with

com.altaworks.da.http.LiteralControl.__AW_Render

05/21/2015 03:56:07 PM, dotNet , 6216 , INFO , Instrumenting

System.Web.UI.LiteralControl.Render with com.altaworks.da.http.LiteralControl.__AW_Render

For the JIDA sub-agent:

05/21/2015 04:25:36 PM, JIDA , 2200 , INFO , Will instrument class:

java.io.PrintWriter - it is an instance of special class

05/21/2015 04:25:36 PM, JIDA , 2200 , INFO , Instrumented (metrics

trace) servlet print writer class java.io.PrintWriter with superclass java.io.Writer

Agent and sub-agent log files reside in the <installdir>/Panorama/hedzup/mn/log directory on the agent system. Log
file names start with the prefix DA-dotNet or DA-JIDA and include the process name and time the process started. For
example:

DA-JIDA_Websphere_server1_20150613122039_22786.txt

Scan the log files for the process. Look for Will instrument, Instrumenting (dotNet sub-agent) and Instrumented
(JIDA sub-agent) messages.

To get access to the log files:

 Use a web browser. The agent serves log files at this URL: http://<agent-system>:2111/log/

 Create and download a “diagnostic bundle” that includes the log files. On the “Agent Details Screen“, click the
“Download the agent logs and diagnostic files“ link.

 Edit the log files directly in the <installdir>/Panorama/hedzup/mn/log directory.

Troubleshooting Application Problems Caused by
Instrumentation

If some cases, AppInternals instrumentation of Java or .NET applications can cause problems, ranging from slow
performance to failure to start. A general approach to troubleshoot such problems is to:

1) Configure minimal instrumentation by disabling options in the “Configuration Settings Screen“.

2) Incrementally re-enable instrumentation, restarting the application each time. When the application encounters the
original problem, you have narrowed the cause to that option.

To configure minimal instrumentation:
 In the “Configurations Screen“, create a new configuration named Minimal Instrumentation or similar.
 Select the Manually choose packages, classes, and methods to instrument option.

Riverbed SteelCentral AppInternals Version 10.0 25

Instrumentation Techniques and Troubleshooting
 Edit the default filter by changing the “Interesting Only, Always, Never“ setting from Interesting Only to Never:

 In the “Advanced Settings“ area, clear all the “Optional Data to Report“ and “Optional Data to Report (Java)“
options:

 Click Save at the bottom of the Configuration Settings screen.

 In the “Bulk Configuration Screen“, select the application (listed in the Processes to Instrument column),
choose Minimal Instrumentation in the list of configurations, and click “Reassign“:

 Restart the application.

dotNet Sub-agent: Monitoring Fails if “IIS Management Scripts
and Tools” Not Installed

On Windows 2008 and 2012 systems, the dotNet sub-agent fails to instrument .NET applications unless the "IIS
Management Scripts and Tools" option is installed. In this case, the sub-agent generates an error similar to the
following in the DotNetAgent-Service.txt log file:

05/21/2014 04:41:34 PM, , , ERROR, Error, failed to retrieve
instance name. , , ERROR, WmiManager: No web site
05/21/2014 04:41:34 PM,

26 Riverbed SteelCentral AppInternals Version 10.0

Instrumentation Techniques and Troubleshooting

with Id='312401' found.

05/21/2014 04:41:34 PM, , , ERROR, at

DotNetAgentSvc.WebManagers.WmiManager.GetSiteName(String path, String siteIdentifier, Boolean

useIISappName)

Follow these steps to install this option:

1) From the Start menu, click Server Manager (or choose Administrative Tools and then Server Manager).

2) In the left navigation pane of the Server Manager, expand Roles, and then right-click Web Server (IIS) and click
Add Role Services.

`

3) The Select Role Services wizard opens. Scroll down to the IIS Management Scripts and Tools option. If the
option does not show (Installed) next to it, select it and click Next.

4) In Confirm Installations Selections panel, click Install.

JIDA Sub-agent: Specifying a Security Policy File for
Applications with Java Security Manager Enabled

By default, JIDA may not work with applications that are configured to start with JIDA and that enable the Java
security manager. Applications can enable the Java security manager using the -Djava.security.manager Java system
property or (typical for application servers) in a user interface.

Riverbed SteelCentral AppInternals Version 10.0 27

Instrumentation Techniques and Troubleshooting

The Java security manager specifies application actions that are unsafe or sensitive. Any actions not allowed by a
security policy cause the application to throw an exception. See the following link for more details on the Java security
manager:
http://docs.oracle.com/javase/tutorial/essential/environment/security.html
When the security manager is enabled, JIDA operations such as discovering JMX metrics or writing to disk may
generate the error java.security.AccessControlException: access denied in the JIDA log file.
For example:

ERROR, Caught: java.security.AccessControlException
09/10/2012 04:24:27 PM, JIDA , 4376 , ERROR, exception during AwJMXMetrics.init:
09/10/2012 04:24:27 PM, JIDA , 4376 , ERROR, java.security.AccessControlException: access denied
(javax.management.MBeanServerPermission findMBeanServer)
.
.
.

The managed node installation provides a Java security policy file in
<installdir>/Panorama/hedzup/mn/bin/JidaSecurity.policy. Applications that enable the security manager must add
this file to their list of policy files for JIDA to work.

Note: Identify the JRE for the Application Before Making Changes
Before changing the java.security file as described here, make sure it is the file for the Java Runtime Environment
(JRE) for the application you want to monitor with JIDA. Typically, systems have multiple JREs (the AppInternals
DSA has its own, for example) and it is easy to change the file for the wrong JRE.

Follow these steps:

1) Edit the lib/security/java.security file in the JRE for the application. Add a policy.url.x property that specifies the
JIDA policy file. The following example shows the added policy.url.3 property in bold:

# The default is to have a single system-wide policy
# file, and a policy file in the user's home directory.
policy.url.1=file:${java.home}/lib/security/java.policy
policy.url.2=file:${user.home}/.java.policy
policy.url.3=file:/${panorama.home}/bin/JidaSecurity.policy

On Windows, you must precede the property value with an additional slash character:

# The default is to have a single system-wide policy file,
# and a policy file in the user's home directory.
policy.url.1=file:${java.home}/lib/security/java.policy
policy.url.2=file:${user.home}/.java.policy
policy.url.3=file:/${panorama.home}/bin/JidaSecurity.policy

2) Confirm that the java.security file has the policy.expandProperties property set to true:

policy.expandProperties=true

3) In the startup options for the application, add the -Dpanorama.home system property. For example, for a default
installation on Windows systems:

-Dpanorama.home=c:\Panorama\hedzup\mn

4) Restart the application.
If JIDA still does not work after restarting the application, enable detailed application logging to diagnose why:

1) In the startup options for the application, add the -Djava.security.debug=policy system property.

28 Riverbed SteelCentral AppInternals Version 10.0

Instrumentation Techniques and Troubleshooting

2) Confirm that you changed lib/security/java.security as described above and restart the application.

3) Search the application log for policy: reading file messages. You should see entries that indicate the application
read several policy files, including JidaSecurity.policy. Note that the following error is expected when the
application reads JidaSecurity.policy and does not represent a problem:

java.lang.IllegalArgumentException: null KeyStore name

4) The error should be followed by an Adding policy entry message and several entries for the JidaSecurity.policy
file. If you see policy: reading file messages for other policy files, but not for JidaSecurity.policy, make sure you
did not change the java.security file for the wrong JRE.

5) If the log indicates that the application is correctly reading the JidaSecurity.policy file, but also contains
java.security.AccessControlException errors, it means that the JidaSecurity.policy file may
require changes. Contact Riverbed technical support with the full text of the log file.

JIDA Sub-agent: Overriding HTTP Proxy for localhost IP
Address

If your application server environment specifies a proxy server, you may need to explicitly override use of the proxy
server for the localhost IP address.
JIDA will not start if the IP address for the localhost host name (typically 127.0.0.1) is directed to a proxy server. If
the application server uses an HTTP proxy for that IP address, JIDA cannot connect to the DSA.
The following symptoms are characteristic of this problem:
 Messages similar to the following in the JIDA log files in <installdir>/Panorama/hedzup/mn/log/:

Using port 2111 to get a port assignment from the DSA.
Cannot connect to DSA.
Failure details: java.io.IOException: Service Unavailable

 The DSA log file (in the same directory) shows no messages about JIDA attempting to connect at that time.
 Other sub-agents on the same system have no problem connecting to the DSA.
To work around this problem, add the following -D system property after other command line modifications in the
application server startup configuration. Substitute the correct IP address if necessary:

-Dhttp.nonProxyHosts=127.0.0.1

Riverbed SteelCentral AppInternals Version 10.0 29

Instrumentation Techniques and Troubleshooting
30 Riverbed SteelCentral AppInternals Version 10.0

Configuration on the Analysis Server System

This material describes AppInternals agent configuration tasks you perform on the analysis server virtual machine.

Riverbed SteelCentral AppInternals Version 10.0 1

2 Riverbed SteelCentral AppInternals Version 10.0

CHAPTER 1 Configuration Tasks Using the CLI

This section describes configuration tasks that you perform on the AppInternals analysis server virtual machine using
the command line interface (CLI). See “Command Line Interface Reference“ for more detail on the CLI.
See “Configuration in the Web Interface“ for details on tasks you perform using the web interface.

Logging In to the Virtual Machine

The AppInternals analysis server runs on an SELinux virtual machine (VM). To use the CLI, you must you first log in
to the virtual machine. Use a Secure Shell (SSH) client to log in as the admin user to use the CLI. The default password
is admin. Use the “passwd <newpassword>“ command to change it.

login as: admin
[email protected]'s password:
Last login: Tue Nov 11 17:27:50 2014 from ksearlewin7.opnet.com
Successfully logged into AppInternals admin interface
Press the '?' key for a list of available commands
appinternals >

Adding Data Storage

The analysis server virtual machine uses the /mnt/data mount point to store AppInternals data. (This data includes raw
trace files, indexed trace files, and metric data.) The virtual machine by default provides 50 GB of storage. You can
add more storage to /mnt/data as described in this section.
Follow these steps:

1) In the hypervisor for the virtual machine, shut down the appinternals-vm virtual machine.

Riverbed SteelCentral AppInternals Version 10.0 3

Configuration Tasks Using the CLI
2) In the hypervisor for the virtual machine, add a new disk to appinternals-vm. The following example shows
adding a 50 GB disk named AddDiskExample in VirtualBox:

3) Start the VM and log in as the admin user.
4) Use the service supervisord stop command to stop related services that need to restart to recognize the additional

disk space.
5) Use the show drive unpartitioned command to obtain the drive name for the new disk (sdc in the following

example):

appinternals > show drive unpartitioned
sdc 50G

6) Enter configuration-mode from the default user-mode:

appinternals > enable
appinternals # configure terminal
appinternals (config) #

7) Use the partition disk command to make the new storage available in the logical volume which is mounted on
mnt/data:

4 Riverbed SteelCentral AppInternals Version 10.0

Configuration Tasks Using the CLI

appinternals (config) # partition disk sdc
Partitioning disk
Checking that no-one is using this disk right now ...
OK

Disk /dev/sdc: 6527 cylinders, 255 heads, 63 sectors/track
/dev/sdc: unrecognized partition table type

Old situation:
No partitions found
New situation:
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

Device Boot Start End #cyls #blocks Id System
6526 6527- 52428127 83 Linux
/dev/sdc1 0+ 0
- 0 0 0 Empty
/dev/sdc2 0 - 0 0 0 Empty
- 0 0 Empty
/dev/sdc3 0

/dev/sdc4 0

.

.

.

8) Use the show drive all command to confirm that the new disk is available.

appinternals (config) # show drive all

sda 16G [SWAP]
- dm-1 1.6G
- sda2 15.5G /
- dm-0 13.9G /boot
- sda1 500M

sdb 50G /mnt/data
- sdb1 50G
- dm-2 100G

sdc 50G /mnt/data
- sdc1 50G
- dm-2 100G

9) Use the service supervisord start command to start the services you stopped.

Note the following:

 The partition disk command automatically adds the new storage to the logical volume which is mounted on
mnt/data. The added storage is available for AppInternals data without further configuration.

 The amount of storage can only be increased. Once you add a drive, do not remove it. If you remove a drive (or
there is a failure) all the data in it is lost.

 Do not increase the size of an existing disk in the hypervisor. Adding storage in this manner is not supported

Changing Network Settings

“Configuring Networking on the Virtual Machine“ in the “Analysis Server Installation“ documentation describes
important configuration after starting the analysis server virtual machine for the first time.

You can change some settings in the “Network Configuration Screen“ in the AppInternals interface. However, if the
network settings are incorrectly configured, the web user interface may be inaccessible. In such cases, you must log in
to the virtual machine console and run the network “wizard” (the CLI command “networkcfg“) to change network
settings.

Riverbed SteelCentral AppInternals Version 10.0 5

Configuration Tasks Using the CLI

Changing the Server for the Authentication Service

SteelCentral Authentication Service is a required component for AppInternals. It provides authentication (including
single sign-on) and authorization (account access privileges) to Riverbed products and components.

The virtual machine that hosts the AppInternals analysis server also includes an installation of SteelCentral
Authentication Service. By default, the analysis server uses that installation.

This section describes how to use an authentication service installation on another system. This is useful to take
advantage of single sign-on, where users can log in to one component and not be prompted to log in again when they
use another component that uses the same authentication server. If your environment already has a SteelCentral
Authentication Service installation, follow these steps to use it (and enable single sign-on with other components that
use it):

1) Log in to the web user interface for the authentication service on the system you want to use. If the role named
AppInternalsAdmin does not exist, create it and assign it to the to user admin.

2) Log in to the AppInternalsVM as the admin user.

3) Enter configuration-mode from the default user-mode:

appinternals > enable
appinternals # configure terminal
appinternals (config) #

4) Change the server that is running SteelCentral Authentication Service. Use the auth hostname command. You
must use a fully-qualified domain name (FQDN) or an IP address (here, the new host name is
nhsrvc1.opnet.com):

appinternals (config) # auth hostname nhsrvc1.opnet.com
Updated Authentication service hostname to: nhsrvc1.opnet.com

5) If necessary, change the HTTPS port from the default of 8222 with the auth port command. Typically this is not
necessary:

appinternals (config) # auth port 8223
Updated Authentication service port to: 8223

6) Restart the analysis server web interface with the service webui stop and service webui start commands:

appinternals (config) # service webui stop

Using CATALINA_BASE: /opt/appinternals-webui

Using CATALINA_HOME: /opt/appinternals-webui

Using CATALINA_TMPDIR: /opt/appinternals-webui/temp

Using JRE_HOME: /usr/lib/jvm/jre-1.8.0/

Using CLASSPATH: /opt/appinternals-webui/bin/bootstrap.jar:/opt/appinternals-webui/bin/

tomcat-juli.jar

Using CATALINA_PID: /var/run/appinternals-webui.pid

Tomcat stopped.

Tomcat Stopped Successfully

appinternals (config) # service webui start

Using CATALINA_BASE: /opt/appinternals-webui

Using CATALINA_HOME: /opt/appinternals-webui

Using CATALINA_TMPDIR: /opt/appinternals-webui/temp

Using JRE_HOME: /usr/lib/jvm/jre-1.8.0/

Using CLASSPATH: /opt/appinternals-webui/bin/bootstrap.jar:/opt/appinternals-webui/bin/

tomcat-juli.jar

Using CATALINA_PID: /var/run/appinternals-webui.pid

Tomcat started.

Tomcat start successful

6 Riverbed SteelCentral AppInternals Version 10.0

Configuration Tasks Using the CLI

Upgrading the Analysis Server

Note that it is easier to use the “Software Upgrade Screen“ in the web user interface to upgrade the analysis server.
However, if the web user interface is inaccessible, you can upgrade using the CLI as described here.
You must be in “Configuration Mode“ to upgrade.

Downloading Upgrades Using the Riverbed Server

By default, the analysis server connects to a Riverbed server to check for and download upgrades. (You can change
this behavior as described in “Disabling Automatic Checking for Upgrades“.) Follow these steps to download the
upgrade from the Riverbed server and install it.

1) Enter configuration-mode from the default user-mode:

appinternals > enable
appinternals # configure terminal
appinternals (config) #

2) Check for an upgrade with the “show upgrade packages“ command. In the following example, there is a later
version (0.4.34) available:

appinternals (config) # show upgrade packages

current version: appinternals 0.4.33

upgrade info source: web

available versions:

- 0.4.34 453MB Upgradeable

appinternals #

3) Download the upgrade with the “upgrade download version <version>“ command:

appinternals (config) # upgrade download version 0.4.34
Download started for version 0.4.34
Download is 4% complete
Download is 6% complete
Download is 9% complete
Download is 13% complete
Download is 18% complete
Download is 30% complete
Download is 53% complete
Download is 89% complete
Download complete
appinternals (config) #

Now, version 0.4.34 shows as downloaded:

appinternals (config) # show upgrade packages

current version: appinternals 0.4.33

upgrade info source: web

available versions:

- 0.4.34 (downloaded) Upgradeable

4) Install the upgrade with the “upgrade install version <version>“ command. Depending on which internal
components are packaged in the upgrade, the installation may display many messages and restart various
components. This can take several minutes. For example:

appinternals (config) # upgrade install version 0.4.34
Loaded plugins: security
Cleaning repos: appinternals-local
Cleaning up Everything
Loaded plugins: security
Setting up Update Process
Resolving Dependencies
.
.
.

Riverbed SteelCentral AppInternals Version 10.0 7

Configuration Tasks Using the CLI

================================================================================

Package Arch Version Repository Size

================================================================================

Updating:

appinternals x86_64 0.4.34-1.el6 appinternals-local 2.6 k

appinternals-agentconfig x86_64 0.0.1-46 appinternals-local 4.9 k

appinternals-negotiator noarch 0.0.1-46 appinternals-local 37 k

appinternals-operators x86_64 0.0.1-76 appinternals-local 12 M

appinternals-silo x86_64 0.0.4-350 appinternals-local 68 M

appinternals-silo-dbg x86_64 0.1.0-350 appinternals-local 7.3 k

appinternals-webui x86_64 0.8.0-1300 appinternals-local 94 M

.

.

.

Updating : appinternals-webui-0.8.0-1300.x86_64 5/18
Tomcat stopped.
Tomcat Stopped Successfully
Tomcat started.
Tomcat start successful
.
.
.

Upgrade started
Mounting file: /var/lib/appinternals/upgrade_packages/riverbed_appinternals_v0.4.34.iso
Upgrade complete
appinternals (config) #

Manually Uploading ISO Image Files

If you disable use of the Riverbed server to check for and download upgrades (as described in “Disabling Automatic
Checking for Upgrades“), you must manually obtain the upgrade files.

You can obtain upgrades from the AppInternals download page on the Riverbed support site. Download the ISO image
file for a particular upgrade to your local system. Use the Upload ISO File button in the “Software Upgrade Screen“
to copy the file to the analysis server. See “Manually Upload an ISO Image File“ for details.

Disabling Automatic Checking for Upgrades

The AppInternals web interface automatically checks for available upgrades by connecting to a Riverbed server. When
it detects that an upgrade is available, a notification windows appears and users can go to the “Software Upgrade
Screen“ to download and install the upgrade. (The upgrades are never installed unless a user approves.)
If automatic connections to an external server are not allowed in your environment, you can disable this behavior.
Follow these steps:

1) Log in to the VM as the admin user.

2) Enter configuration-mode from the default user-mode:

appinternals > enable
appinternals # configure terminal
appinternals (config) #

3) Use the “upgrade repository url <url>“ command to override the default server. Specify a server value that will
always fail, such as localhost:

upgrade repository url http://localhost

8 Riverbed SteelCentral AppInternals Version 10.0

CHAPTER 2 Command Line Interface Reference

Overview

The AppInternals analysis server runs on an SELinux virtual machine (VM). Administrators can log on to the VM to
issue a limited set of commands using the command line interface (CLI) described here.
See “Configuration Tasks Using the CLI“ for details on specific tasks.

Logging In to the Virtual Machine

Use a Secure Shell (SSH) client to log in as the admin user to use the CLI. The default password is admin. Use the
“passwd <newpassword>“ command to change it.

login as: admin
[email protected]'s password:
Last login: Tue Nov 11 17:27:50 2014 from ksearlewin7.opnet.com

Successfully logged into AppInternals admin interface

Press the '?' key for a list of available commands

appinternals >

Keyboard Shortcuts

The CLI provides the following shortcuts:
 In any of the “CLI Modes“, type ? or the TAB key to see available commands. After typing a command, type ? or

the TAB key to see command options.
 Press the TAB key to complete a partial command automatically.
 The CLI accepts abbreviations for some commands. For example, conf t is an acceptable abbreviation for the

configure terminal command.

CLI Modes

The CLI has different modes that each allow access to additional commands. See the following topics for more details:
 “User-Mode Commands“

Riverbed SteelCentral AppInternals Version 10.0 9

Command Line Interface Reference

 “Enable-Mode Commands“
 “Configuration-Mode Commands“

User-Mode Commands

This section describes commands you can issue in user-mode. When you first log in to a CLI session, you are in
user-mode.
The command-line prompt for user-mode is hostname >.

login as: admin
[email protected]'s password:
Last login: Thu Oct 23 13:29:18 2014 from ksearlewin7.opnet.com

Successfully logged into AppInternals admin interface

Press the '?' key for a list of available commands

nhv1-ams-5 >

To log out of the CLI session (and the virtual machine), use the exit command in user-mode.

enable

Enter enable mode. The enable mode prompt is #.

appinternals > enable
appinternals #

exit

In user-mode and enable-mode, log out of the virtual machine session.

set silo_setting <setting> <value>

Changes settings that control aspects of the “silo” database services (see “service <name> { start | stop }“ for a
summary of the services). You must reboot the analysis server system (see the “reboot“ command) after changing any
of these values for the changes to take effect.

The following table describes the settings.

disk_quota_pct The maximum disk space that silo can use, as a percentage of the total space on the disk. 10 percent
disk_quota_min is the default.
disk_quota_max
log_diagnostic Disk space reserved for silo usage, in gibibytes (GiB). Valid values are 1 to 10. 4 GiB is the default
and lowest recommended value.

The maximum disk space that silo can use, in GiB. Valid values are 1 to 1000000. 10000 is the
default.

Enable logging of diagnostic messages in silo log files. (To see log files, create and download a
“diagnostic bundle” as described in the “Diagnostic Bundle Screen“ topic.) Valid values are true
and false. The default is false (suppress diagnostic messages) since they are typically useful only for
troubleshooting.

10 Riverbed SteelCentral AppInternals Version 10.0