Stats and Reporting
Last updated
Last updated
Stats and reporting tools provide insight into the usage of your knowledge maps. This includes:
Query usage: Totals and daily aggregates of queries started and query results.
Interaction history: If enabled, this provides details on the queries made, question and answers provided and query results. This also includes access to the Evidence Tree for all query results.
To access Stats & Reporting navigate to a Knowledge Map, then click Publish followed by Stats & Reporting.
By default the Query usage will display with a date range of the past 3 months. If there is no live version, then the draft will be selected by default. However if a live version exists, this will become the default.
The date range and versions can be changed from the filters above the graph. These filters apply to both query usage and interaction history.
The graph shows the utilisation of the knowledge map for the selected version and date range. The following information is displayed as a daily aggregate and total.
Started queries: The number of query requests made.
Query results: The number of all results received, including those where no answer could be inferred.
Query results with answers: The number of results that contained an answer to the query (note: a query result can contain more than one answer, but this data is not included here)
This data can also be downloaded by clicking Export to CSV beneath the totals
A large delta between started queries and query results can indicate either a high error rates or that queries are being abandoned by the user. This can be common to see with the draft version, so be sure check this isn't selected. If interaction history is enabled, this can provide useful insight into where users are abandoning their interaction with Rainbird or where errors are occurring.
A large delta between query results and query results with answers can indicate the knowledge map has a gap in the data and/or logic. It data is being provided that cannot be reasoned over, there may be an opportunity to understand usage and determine if the knowledge map should be expanded to handle these additional scenarios. If interaction history is enabled, this can provide useful insight into what data is being provided and enable new logic to be created to handle it.
Only production queries are rate limited (all queries as part of building and testing are not counted) and only when a result is returned, regardless of whether it contains an answer or not.
Therefore to determine query utilisation against a contractually agreed limit, for each knowledge map filter it to the version(s) used in production across the selected date range. The figure for Query results will show the rate-limited queries.
Interaction history keeps a record of the queries made and their results, including access to the evidence tree of every result. Additionally where data has been required to reach a result, it will show the data injected, questions asked/answers given and/or datasources invoked, depending on what features are used in that knowledge map.
This can support with the auditability and traceability of decisions provided by Rainbird, as you get visibility of the data provided to the session, as well as the inferences made on this data to derive the result.
Interaction history must be enabled per version. To enable this feature, navigate to Publish and Live Version and tick the Records interactions box for each applicable version.
Navigate to Publish, Stats & Reporting, then Interaction History.
A table will be displayed showing all queries for the selected version and date range.
This list view displays the following:
Start time: Date and time of the query request
Version: The version the query was made to
Session ID: Unique ID for the session in which the query was made
Query: Made up of the Subject, Relationship and Object, combined these fields will show the query (not the query result)
Flag: Manually mark queries with a coloured flag for review.
All columns can be sorted and filtered.
Selecting any query from the list will open the details view showing all interaction events within that session.
Clicking --- x questions asked --- will minimise the questions and answers to get straight to the result and access the evidence tree. Clicking again will expand to display the questions and answers.
If multiple queries are made within the same session, each query will be shown separately in the list view to support filtering and sorting. However opening any query from the same session will open the same details view, as all queries within that session are shown.
The following events may be displayed in the interaction history details view:
Start: Session started
Inject: What facts were injected into the session. This event will show a count. Selecting it will expand to show all injected facts
Query: The query, or queries, that were made
Questions: The questions that were asked by the reasoning engine. This event can be expanded to view detailed question properties such as available items the user could have selected or known answers
Answers: The answers and their certainty that were given by the end-user or system
Datasources: The datasources that were used by the knowledge map, including the facts that were generated from these datasources or datasource errors.
Result: The result, or results, that were given. An evidence tree link is available for each result.
Errors: Where an error occurred an error event will be displayed
The interaction history details can be downloaded as a CSV by clicking the download icon in the top-right.
