Rainbird
  • 🏃‍♂️Getting started
    • What is Rainbird?
    • How does Rainbird work?
    • Example use cases
      • 📈Investment suitability assessment
      • 🤖Digital tax consultant
      • 🏥Covid risk assessment
      • 🚗Insurance claims liability
      • 💰Fraud identification
    • Hosting options
    • Quickstart guide
    • Onboarding with Rainbird
  • 🧠Knowledge Modelling
    • What is knowledge modelling?
    • What is a query?
    • Modelling
      • Concepts
        • Concept instances
      • Relationships
        • Question Configuration
        • Rules
          • Match, Infer, Ask process
          • Expressions List
      • Facts
      • Datasources
      • Other features
        • Markdown
        • Import/Export Knowledge Maps
      • Troubleshooting Tips
    • Testing
      • Manual tests
      • Automated tests
      • External User Acceptance Testing (UAT)
    • Versioning
    • Publishing
      • API Management
      • Setting a live version
      • Publishing an agent
    • Stats and Reporting
    • Managing your data
    • The library
      • How to: update a template
  • 🔍Evidence
    • What is evidence?
    • The Evidence Tree
      • The Salience Chart
  • 👩‍💻Developer guides
    • Overview
    • API Guide
      • API request flow
      • Run example queries
      • Environments
      • Error codes
      • Skipping an answer to a question
      • Retrieving a full Evidence Tree for use with a custom UI or application
      • Building an Evidence Tree URL
    • API interactive documentation
    • SDKs
  • Rainbird Labs
    • Overview
    • Consult
    • Generate from documentation
    • Co-author
    • /interact
    • /explain
Powered by GitBook
On this page
  • Stats & Reporting filters
  • Query usage
  • Interpreting this data
  • Interaction History
  • Viewing interaction history list
  • Viewing the interaction history details
  1. Knowledge Modelling

Stats and Reporting

Last updated 4 months ago

Stats and reporting tools provide insight into the usage of your knowledge maps. This includes:

  1. Query usage: Totals and daily aggregates of queries started and query results.

  2. Interaction history: If enabled, this provides details on the queries made, question and answers provided and query results. This also includes access to the Evidence Tree for all query results.

To access Stats & Reporting navigate to a Knowledge Map, then click Publish followed by Stats & Reporting.

Stats & Reporting filters

By default the Query usage will display with a date range of the past 3 months. If there is no live version, then the draft will be selected by default. However if a live version exists, this will become the default.

The date range and versions can be changed from the filters above the graph. These filters apply to both query usage and interaction history.

Query usage

The graph shows the utilisation of the knowledge map for the selected version and date range. The following information is displayed as a daily aggregate and total.

  1. Started queries: The number of query requests made.

  2. Query results: The number of all results received, including those where no answer could be inferred.

  3. Query results with answers: The number of results that contained an answer to the query (note: a query result can contain more than one answer, but this data is not included here)

This data can also be downloaded by clicking Export to CSV beneath the totals

Interpreting this data

Technical errors or user experience issues

A large delta between started queries and query results can indicate either high error rates or that queries are being abandoned by the user. This can be common to see with the draft version, so be sure to check this isn't selected. If interaction history is enabled, this can provide useful insight into where users are abandoning their interaction with Rainbird or where errors are occurring.

Knowledge gaps

A large delta between query results and query results with answers can indicate the knowledge map has a gap in the data and/or logic. If data is being provided that cannot be reasoned over, there may be an opportunity to understand usage and determine if the knowledge map should be expanded to handle these additional scenarios. If interaction history is enabled, this can provide useful insight into what data is being provided and enable new logic to be created to handle it.

Understanding usage of rate-limited queries

Only production queries are rate limited (all queries as part of building and testing are not counted) and only when a result is returned, regardless of whether it contains an answer or not.

Therefore to determine query utilisation against a contractually agreed limit, for each knowledge map filter it to the version(s) used in production across the selected date range. The figure for Query results will show the rate-limited queries.

Interaction History

Interaction history keeps a record of the queries made and their results, including access to the evidence tree of every result. Additionally where data has been required to reach a result, it will show the data injected, questions asked/answers given and/or datasources invoked, depending on what features are used in that knowledge map.

This can support with the auditability and traceability of decisions provided by Rainbird, as you get visibility of the data provided to the session, as well as the inferences made on this data to derive the result.

Enabling interaction history

Interaction history must be enabled per version. To enable this feature, navigate to Publish and Live Version and tick the Records interactions box for each applicable version.

Viewing interaction history list

Navigate to Publish, Stats & Reporting, then Interaction History.

A table will be displayed showing all queries for the selected version and date range.

This list view displays the following:

  • Start time: Date and time of the query request

  • Version: The version the query was made to

  • Session ID: Unique ID for the session in which the query was made

  • Query: Made up of the Subject, Relationship and Object, combined these fields will show the query (not the query result)

  • Flag: Manually mark queries with a coloured flag for review.

All columns can be sorted and filtered.

Viewing the interaction history details

Selecting any query from the list will open the details view showing all interaction events within that session.

Clicking --- x questions asked --- will minimise the questions and answers to get straight to the result and access the evidence tree. Clicking again will expand to display the questions and answers.

If multiple queries are made within the same session, each query will be shown separately in the list view to support filtering and sorting. However opening any query from the same session will open the same details view, as all queries within that session are shown.

Interaction history events

The following events may be displayed in the interaction history details view:

  • Start: Session started

  • Inject: What facts were injected into the session. This event will show a count. Selecting it will expand to show all injected facts

  • Query: The query, or queries, that were made

  • Questions: The questions that were asked by the reasoning engine. This event can be expanded to view detailed question properties such as available items the user could have selected or known answers

  • Answers: The answers and their certainty that were given by the end-user or system

  • Datasources: The datasources that were used by the knowledge map, including the facts that were generated from these datasources or datasource errors.

  • Result: The result, or results, that were given. An evidence tree link is available for each result.

  • Errors: Where an error occurred an error event will be displayed

The interaction history details can be downloaded as a CSV by clicking the download icon in the top-right.

🧠
Version and date range filters