Human Annotation is a manual evaluation method. It is used to collaboratively annotate traces, sessions and observations with scores. In ABV you can use Annotation Queues to streamline working through reviewing larger batches of single interactions (trace-level), multiple interactions (session-level) or even single observations, below the trace level.

​

Why use Human Annotation?

• Collaboration: Enable team collaboration by inviting other internal members to annotate a subset of traces and observations. This manual evaluation can enhance the overall accuracy and reliability of your results by incorporating diverse perspectives and expertise.
• Annotation data consistency: Create score configurations for annotation workflows to ensure that all team members are using standardized scoring criteria. Hereby configure categorical, numerical or binary score types to capture different aspects of your data.
• Evaluation of new product features: This feature can be useful for new use cases where no other scores have been allocated yet.
• Benchmarking of other scores: Establish a human baseline score that can be used as a benchmark to compare and evaluate other scores. This can provide a clear standard of reference and enhance the objectivity of your performance evaluations.

​

Annotation of Single Traces, Sessions and Observations

Manual Annotation of single traces, sessions and observations is available in the trace, session and observation detail view.

​

1) Prerequisite: Create a Score Config

To use Human Annotation, you need to have at least one score configuration (Score Config) set up. See Scores Data Model for details.

​

2) Trigger Annotation on a Trace, Session or Observation

On a Trace, Session or Observation detail view click on Annotate to open the annotation form.

​

3) Select Score Configs to use

​

4) Set Score values

​

5) See newly added Scores

To see your newly added scores on traces or observations, click on the Scores tab on the trace or observation detail view. To see your newly added scores on sessions, click on the Scores tab on the session detail view. All scores are also available in the traces, sessions and observations table views respectively.

​

Annotation Queues

Annotation queues allow you to manage and prioritize your annotation tasks in a structured way. This feature is particularly useful for large-scale projects that benefit of human-in-the-loop evaluation at some scale. Queues streamline this process by allowing for specifying which traces, sessions or observations you’d like to annotate on which dimensions.

​

Create Annotation Queues

​

1) Prerequisite: Create a Score Config

To use Human Annotation, you need to have at least one score configuration (Score Config) set up. See Scores Data Model for details on how to create and manage Score Configs.

​

2) Go to Annotation Queues View

• Navigate to Your Project > Human Annotation to see all your annotation queues.
• Click on New queue to create a new queue.

​

3) Fill out Create Queue Form

• Select the Score Configs you want to use for this queue.
• Set the Queue name and Description (optional).

• Click on Create queue to create the queue.

​

Assign Users to Annotation Queues

​

1) Go to Annotation Queues View

• Navigate to Your Project > Human Annotation to see all your annotation queues.
• Click on New queue to assign users to a new queue or click the three dots on the right of an existing queue to assign users to it.

​

2) Assign Users to the Queue

• Expand the User Assignment section under the Advanced settings section.
• Search for the users you want to assign to the queue.
• Click on the user to assign them to the queue. You can also assign multiple users at once by selecting them in the search results.
• Review the users you have selected to be assigned to the queue.
• Click on Save to save the queue.

​

3) Manage Assignments

• Click on the three dots on the right of the queue or the Edit queue button in the queue detail view to manage the assignments.
• View the users assigned to the queue by expanding the User Assignment section.
• Assign new users by clicking the Assign users button and searching for the users you want to assign.
• Unassign current users by clicking the X icon on the right of the user.

​

Run Annotation Queues

​

1) Populate Annotation Queues

Once you have created annotation queues, you can assign traces, sessions or observations to them. Bulk Selection To add multiple traces, sessions or observations to a queue:

1. Navigate to the respective table view and optionally adjust the filters
2. Select Traces, Sessions or Observations via the checkboxes.
3. Click on the “Actions” dropdown menu
4. Click on Add to queue to add the selected traces, sessions or observations to the queue.
5. Select the queue you want to add the traces, sessions or observations to.

Single Item To add single traces, sessions or observations to a queue:

1. Navigate to the trace, session or observation detail view
2. Click on the Annotate dropdown
3. Select the queue you want to add the trace, session or observation to

​

2) Navigate to Annotation Queue

1. Navigate to Your Project > Human Annotation
2. Option 1: Click on the queue name to view the associated annotation tasks
3. Option 2: Click on “Process queue” to start processing the queue

​

3) Process Annotation Tasks

You will see an annotation task for each item in the queue.

1. On the Annotate Card add scores on the defined dimensions
2. Click on Complete + next to move to the next annotation task or finish the queue

​

Manage Annotation Queues via API

You can enqueue, manage and dequeue annotation tasks via the API. This allows for scaling and automating your annotation workflows.