TIBCO Spotfire Community

Welcome to TIBCO Spotfire Community Sign in | Join | Help

Monitor Information Link Usage

Introduction

This article shows one way of extracting Information Link usage to enable monitoring and analysis.

By default TIBCO Spotfire Analytics Server logs all SQL commands executed and the time it took to execute a specific Information Link to file. This data can be extracted, and published to a database and which in turn can be analyzed in TIBCO Spotfire.

Extract the data using external tools

The following steps are needed on the TIBCO Spotfire Analytics Server

1. Install cygwin, to get the grep and sed piping commands, and Oracle client tools and drivers.

2. Create an ilusage.sh text file with the following content:

rm /cygdrive /d/tsas101/server/logs/informationlinks.log

echo "Time;Thread;User;Something;Information Link;Guid;Result;Execution Time" > / cygdrive /d /tsas101/server/logs/informationlinks.log

grep -h ": Information Link" /cygdrive/d/tsas101/server/logs/sql.log* | sed -e "s/\([0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\} [0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\},[0-9]\{3\}\) \[\*Thread-\([0-9]\+\), \([a-zA-Z_0-9]\+\)[^]]*\]\: Information Link[ ]*\('\(.*\)'\)*[ ]*\((\([0-9a-f]\{8\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{12\}\))\)* \([a-z ]\+\), time: \([0-9E.]*\) seconds/\1;\2;\3;0;\5;\7;\8;\9/g" >> / cygdrive /d/tsas101/server/logs/informationlinks.log

3. Create a database table (below the following has Oracle specific syntax):

CREATE TABLE IL_STATISTICS (
TIME    TIMESTAMP,
THREAD INTEGER,
USERNAME    VARCHAR2(100),
SOMETHING    INTEGER,
INFORMATIONLINK    VARCHAR2(500),
GUID    CHAR(36),
RESULT    VARCHAR2(100),
EXECUTIONTIME    NUMBER(20,3)
)

4. Create a ctl file (ilusage.ctl) for Oracle sqlldr:

OPTIONS(SKIP=1)
Load DATA
INFILE 'd:/tsas101/server/logs/informationlinks.log'
INTO TABLE spotuser_iim.IL_STATISTICS TRUNCATE
FIELDS TERMINATED BY ';'
TRAILING NULLCOLS
(
    Time TIMESTAMP "YYYY-MM-DD HH24:MI:SS,FF3",
    THREAD Integer EXTERNAL,
    USERNAME CHAR,
    SOMETHING Integer EXTERNAL,
    INFORMATIONLINK CHAR,
    GUID CHAR,
    RESULT CHAR,
    EXECUTIONTIME FLOAT EXTERNAL
)

5. Create a bat file (update.bat) that should be run every day using Scheduled Tasks:

c:\cygwin\bin\bash ilusage.sh
sqlldr userid=user/password control=ilusage.ctl log=ilstatistics.log

6. Map the database table in Information Services and create an Information Link

Extract the data using a .net program

static void Main(string[] args)

{

    string directory = @".";

    string fileSearchPattern = "sql.log*";

    string matchExpression = ": Information Link";

    string replaceExpression = @"(?<Timestamp>[\d]{4}-[\d]{2}-[\d]{2} [\d]{2}:[\d]{2}:[\d]{2},[\d]{3}) \[\*Thread-(?<Thread>[\d]+),\s(?<User>[\w\d_]+)[^]]*\*\][^:]*: Information Link\s*(?:'(?<Informationlink>.*)')*\s*(\((?<Guid>[\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12})\))*\s*(?<Result>[a-z ]+), time: (?<Executiontime>[\dE.]*) seconds";

    char fieldSeperator = ';';

    Regex extractRegex = new Regex(replaceExpression);

    Regex matchRegex = new Regex(matchExpression);

   

    // Write Header

    string[] fieldNames = extractRegex.GetGroupNames();

    StringBuilder replacementStringBuilder = new StringBuilder();

   

    // Make sure group 0 and 1 is not included.

    for (int i = 2; i < fieldNames.Length; ++i)

    {

        if (i > 2)

        {

            Console.Write(fieldSeperator);

            replacementStringBuilder.Append(fieldSeperator);

        }

        Console.Write(fieldNames[i]);

        replacementStringBuilder.Append('$');

        replacementStringBuilder.Append('{');

        replacementStringBuilder.Append(fieldNames[i]);

        replacementStringBuilder.Append('}');

    }

    Console.WriteLine();

    // Read files and extract information.

    string line = string.Empty;

    string replacementString = replacementStringBuilder.ToString();

    foreach (FileInfo fileInfo in new DirectoryInfo(directory).GetFiles(fileSearchPattern))

    {

        using (TextReader reader = new StreamReader(fileInfo.FullName))

        {

            while (reader.Peek() > 0)

            {

               line = reader.ReadLine();

               if (matchRegex.IsMatch(line))

               {

                   string output = extractRegex.Replace(line, replacementString);

                   Console.WriteLine(output);

               }

           }

        }

    }

}

Import the Data

You’ll probably note that there are different results from the execution of an Information Link. On our production server I got three different results {canceled, executed successfully, job closed}. To simplify later analysis it may be easier to separate the successful cases from the other by importing these cases as two distinct tables.
  • If Information Links are used, one way of doing that is to create two filter elements in Information Designer: First use the conditions FC1 = 'executed successfully' and FC1 != 'executed successfully'. Then make a copy of the original Information Link. Finally add one of the filter elements to each Information Link.
  • Another way of doing this is to first add a prompt on the result column element in the Information Link. Then add two tables using the prompted Information Link: make sure that different prompt values are selected for the two cases. If the analysis is later saved as linked without prompting the selected filter, the values will be applied the next time the analysis is opened or the data tables are refreshed.

     

Sample analysis

Information link execution time over time, also added a horizontal line per Color, the plot is filtered down to only display two Information Links.

X: <Year([Timestamp]) NEST Week([Timestamp]) >
Y: Median([Executiontime]) as [Execution Time]
Color: <[Informationlink]>

As noted in the sample plot above, the execution time varies greatly over time. By combining features it is easy to mark weeks where there has been a 30% change in the median execution time per month.
  1. If not only successful execution exists, make sure to create a new calculated column from the timestamp which only have values for successful executions. if(([Result])=("executed successfully"),[Timestamp],null) call it Timestamp (Cleaned)
  2. Create a hierarchy using InformationLink, Year(Timestamp (Cleaned)), Month(Timestamp (Cleaned)) call it InformationLinkTime
  3. Create a new calculated column using the following expression
    if(([Result])=("executed successfully"),(Median([Executiontime]) OVER ([Hierarchy.InformationLinkTime]) - Median([Executiontime]) OVER (Intersect(Previous([Hierarchy.InformationLinkTime]),[Informationlink]))) / Median([Executiontime]) OVER ([Hierarchy.InformationLinkTime]),null) call it Change Previous Month
  4. Create a new calculated column
    if(([Change Previous Month])>(0.3),"Large Increase", null) call it Investigate
  5. Now, by using the new Find functionality in TIBCO Spotfire 2.2 it is easy to mark those weeks that have a large increase:
    Press Ctrl+F, start typing Large Increase, when there is only one item in the result list, press enter.

Note that changing the hierarchy InformationLinkTime, for example using Year,Quarter instead of Year,Month will trigger a recalculation of the dependent calculated columns.

It is useful to define the actual Limit (0.3) in a property outside the actual column expression, so that it is possible to change the limit without editing the actual expression. Currently there are three ways to use properties  in an expression using one of the following functions: {DocumentProperty, DataTableProperty, ColumnProperty}.

Column properties and data table properties can be defined in the Information Model using the Information Designer, or they can be set directly on the imported data table or column, document properties are defined in the analysis.

For example first add a real valued document property to the analysis, then change the expression for the Investigate column to something like
if(([Change Previous Month])>(DocumentProperty("ExecutionTime.ChangeLimit")),"Large Increase",null)
Now if the property value is changed, the calculated columns are re-evaluated.
Comments

About Daniel Vulcan

Daniel Vulcan is a developer with the TIBCO Spotfire and the TIBCO Spotfire Web Player teams.