Human Annotation

human annotation is a manual evaluation method it is used to collaboratively annotate traces, sessions and observations with scores in abv you can use human annotation /#annotation queues to streamline working through reviewing larger batches of of human annotation docid\ sus61b5z5hbx6njqctecn , human annotation docid\ sus61b5z5hbx6njqctecn or even human annotation docid\ sus61b5z5hbx6njqctecn , below the trace leve l why use human annotation? collaboration enable team collaboration by inviting other internal members to annotate a subset of traces and observations this manual evaluation can enhance the overall accuracy and reliability of your results by incorporating diverse perspectives and expertise annotation data consistency create score configurations for annotation workflows to ensure that all team members are using standardized scoring criteria hereby configure categorical, numerical or binary score types to capture different aspects of your data evaluation of new product features this feature can be useful for new use cases where no other scores have been allocated yet benchmarking of other scores establish a human baseline score that can be used as a benchmark to compare and evaluate other scores this can provide a clear standard of reference and enhance the objectivity of your performance evaluations annotation of single traces, sessions and observations manual annotation of single traces, sessions and observations is available in the trace, session and observation detail view 1\) prerequisite create a score config to use human annotation, you need to have at least one score configuration (score config) set up see how to create and manage score configs for details 2\) trigger annotation on a trace, session or observation on a trace, session or observation detail view click on annotate to open the annotation form 3\) select score configs to use 4\) set score values 5\) see newly added scores to see your newly added scores on traces or observations, click on the scores tab on the trace or observation detail view to see your newly added scores on sessions, click on the scores tab on the session detail view all scores are also available in the traces, sessions and observations table views respectively annotation queues annotation queues allow you to manage and prioritize your annotation tasks in a structured way this feature is particularly useful for large scale projects that benefit of human in the loop evaluation at some scale queues streamline this process by allowing for specifying which traces, sessions or observations you'd like to annotate on which dimensions create annotation queues 1\) prerequisite create a score config to use human annotation, you need to have at least one score configuration (score config) set up see how to create and manage score how to create and manage score configs for details 2\) go to annotation queues view navigate to your project > human annotation to see all your annotation queues click on new queue to create a new queue 3\) fill out create queue form select the score configs you want to use for this queue set the queue name and description (optional) click on create queue to create the queue assign users to annotation queues 1\) go to annotation queues view navigate to your project > human annotation to see all your annotation queues click on new queue to assign users to a new queue or click the three dots on the right of an existing queue to assign users to it 2\) assign users to the queue expand the user assignment section under the advanced settings section search for the users you want to assign to the queue click on the user to assign them to the queue you can also assign multiple users at once by selecting them in the search results review the users you have selected to be assigned to the queue click on save to save the queue 3\) manage assignments click on the three dots on the right of the queue or the edit queue button in the queue detail view to manage the assignments view the users assigned to the queue by expanding the user assignment section assign new users by clicking the assign users button and searching for the users you want to assign unassign current users by clicking the x icon on the right of the user run annotation queues 1\) populate annotation queues once you have created annotation queues, you can assign traces, sessions or observations to them bulk selection to add multiple traces, sessions or observations to a queue navigate to the respective table view and optionally adjust the filters select traces, sessions or observations via the checkboxes click on the "actions" dropdown menu click on add to queue to add the selected traces, sessions or observations to the queue select the queue you want to add the traces, sessions or observations to single item to add single traces, sessions or observations to a queue navigate to the trace, session or observation detail view click on the annotate dropdown select the queue you want to add the trace, session or observation to 2\) navigate to annotation queue navigate to your project > human annotation option 1 click on the queue name to view the associated annotation tasks option 2 click on "process queue" to start processing the queue 3\) process annotation tasks you will see an annotation task for each item in the queue on the annotate card add scores on the defined dimensions click on complete + next to move to the next annotation task or finish the queue manage annotation queues via api you can enqueue, manage and dequeue annotation tasks via the public api docid\ ezykvdguou0bjbdr0cynx this allows for scaling and automating your annotation workflows