{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n# Working with the LabelRowV2\n\nThe :class:`encord.objects.LabelRowV2` class is a wrapper around the Encord label row data format. It\nprovides a convenient way to read, create, and manipulate labels.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Imports and authentication\nFirst, import dependencies and authenticate a project manager.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from pathlib import Path\nfrom typing import List\n\nfrom encord import EncordUserClient, Project\nfrom encord.objects import (\n AnswerForFrames,\n Classification,\n LabelRowV2,\n Object,\n ObjectInstance,\n OntologyStructure,\n RadioAttribute,\n)\nfrom encord.objects.common import Option\nfrom encord.objects.coordinates import BoundingBoxCoordinates\nfrom encord.objects.frames import Range\nfrom encord.orm.project import Project as OrmProject" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

Note

To interact with Encord, you need to authenticate a client. You can find more details\n `here `.

\n\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Authentication: adapt the following line to your private key path\nprivate_key_path = Path.home() / \".ssh\" / \"id_ed25519\"\n\nwith private_key_path.open() as f:\n private_key = f.read()\n\nuser_client = EncordUserClient.create_with_ssh_private_key(private_key)\n\n# Find project to work with based on title.\nproject_orm: OrmProject = next(\n (\n p[\"project\"]\n for p in user_client.get_projects(title_eq=\"Your project name\")\n )\n)\nproject: Project = user_client.get_project(project_orm.project_hash)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Get metadata around labels\n\nSometimes you might want to inspect some metadata around the label rows, such as the label hash,\nwhen the label was created, the corresponding data hash, or the creation date of the label.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "label_rows: List[LabelRowV2] = project.list_label_rows_v2()\n\n\nfor label_row in label_rows:\n print(f\"Label hash: {label_row.label_hash}\")\n print(f\"Label created at: {label_row.created_at}\")\n print(f\"Annotation task status: {label_row.annotation_task_status}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Inspect the filters in :meth:`~encord.project.Project.list_label_rows_v2` to only get a subset of the label rows.\n\nYou can find more examples around all the available read properties by inspecting the properties of the\n:class:`~encord.objects.LabelRowV2` class.\n\n## Starting to read or write labels\n\nTo start reading or writing labels, you need to call the :meth:`~encord.objects.LabelRowV2.initialise_labels`\nmethod which will download the state of the label from the Encord server.\n\nOnce this method has been called, you can create your first label.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "first_label_row: LabelRowV2 = label_rows[0]\n\nfirst_label_row.initialise_labels()\n# ^ Check the reference for possible arguments\n\n# Code to add/manipulate some labels\n...\n\n# Once you have added new labels, you will need to call .save() to upload all labels to the server.\nfirst_label_row.save()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Creating/reading object instances\nThe :class:`encord.objects.LabelRowV2` class works with its corresponding ontology. If you add object instances\nor classification instances, these will be created from the ontology. You can read more about object instances\nhere: https://docs.encord.com/ontologies/use/#objects\n\nYou can think of an object instance as a visual label in the label editor. One bounding box would be one object\ninstance.\n\n### Finding the ontology object\n\nThe LabelRowV2 is designed to work with its corresponding ontology via the :class:`~encord.objects.ontology_labels_impl.OntologyStructure`.\nYou will need to use the title or feature node hash to find the right objects, classifications, attributes, or\nattribute options. See the example below to find the ontology object for the demonstrative \"Box of a human\" object.\n\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "ontology_structure: OntologyStructure = first_label_row.ontology_structure\nbox_ontology_object: Object = ontology_structure.get_child_by_title(\n title=\"Box of a human\", type_=Object\n)\n# ^ optionally specify the `type_` to narrow the return type and also have a runtime check." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Creating and saving an object instance\n\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Instantiate an object instance from the box ontology node.\nbox_object_instance: ObjectInstance = box_ontology_object.create_instance()\n\nbox_object_instance.set_for_frames(\n coordinates=BoundingBoxCoordinates(\n height=0.5,\n width=0.5,\n top_left_x=0.2,\n top_left_y=0.2,\n ),\n # Add the bounding box to the first frame\n frames=0,\n # There are multiple additional fields that can be set optionally:\n manual_annotation=True,\n)\n\n# Link the object instance to the label row.\nfirst_label_row.add_object_instance(box_object_instance)\n\n\nfirst_label_row.save() # Upload the label to the server" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Inspecting an object instance\n\nYou can now get all the object instances that are part of the label row.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Check the get_object_instances optional filters for when you have many different object/classification instances.\nall_object_instances: List[\n ObjectInstance\n] = first_label_row.get_object_instances()\n\nassert all_object_instances[0] == box_object_instance\nassert all_object_instances[0].get_annotation(frame=0).manual_annotation is True" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Adding object instances to multiple frames.\n\nSometimes, you might want to work with a video where a single object instance is present in multiple frames.\nFor example, you are tracking a car across multiple frames. In this case you would create one\nobject instance and place it on all the frames where it is present.\nIf objects are never present in multiple frames, you would always create a new object instance for a new frame.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Assume you have the coordinates of a single object for the first 3 frames of a video.\n# These are indexed by frame number.\ncoordinates_per_frame = {\n 3: BoundingBoxCoordinates(\n height=0.5,\n width=0.5,\n top_left_x=0.2,\n top_left_y=0.2,\n ),\n 4: BoundingBoxCoordinates(\n height=0.5,\n width=0.5,\n top_left_x=0.3,\n top_left_y=0.3,\n ),\n 5: BoundingBoxCoordinates(\n height=0.5,\n width=0.5,\n top_left_x=0.4,\n top_left_y=0.4,\n ),\n}\n\n\n# OPTION 1 - think in terms of \"the frames per object instance\"\nbox_object_instance_2: ObjectInstance = box_ontology_object.create_instance()\n\nfor frame_number, coordinates in coordinates_per_frame.items():\n box_object_instance_2.set_for_frames(\n coordinates=coordinates, frames=frame_number\n )\n\n# OPTION 2 - think in terms of the \"object instances per frame\"\nbox_object_instance_3: ObjectInstance = box_ontology_object.create_instance()\n\nfor frame_view in first_label_row.get_frame_views():\n frame_number = frame_view.frame\n if frame_number in coordinates_per_frame:\n frame_view.add_object_instance(\n object_instance=box_object_instance_3,\n coordinates=coordinates_per_frame[frame_number],\n )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Read access across multiple frames\n\nAs shown above with OPTION 1 and OPTION 2, you can think of the individual object instances and on which\nframes they are present or you can think of the individual frames and which objects they have.\nFor a read access thinking of the individual frames can be particularly convenient.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "for label_row_frame_view in first_label_row.get_frame_views():\n frame_number = label_row_frame_view.frame\n print(f\"Frame number: {frame_number}\")\n object_instances_in_frame: List[\n ObjectInstance\n ] = label_row_frame_view.get_object_instances()\n for object_instance in object_instances_in_frame:\n print(f\"Object instance: {object_instance}\")\n annotation = object_instance.get_annotation(frame=frame_number)\n print(f\"Coordinates: {annotation.coordinates}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Working with a classification instance\n\nCreating a classification instance is similar to creating an object instance. The only differences are that you\ncannot create have more than one classification instance of the same type on the same frame and that there is\nno coordinates to be set for classification instances.\n\nYou can read more about classification instances here: https://docs.encord.com/ontologies/use/#classifications\n\n### Get the ontology classification\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Assume that the following text classification exists in the ontology.\ntext_ontology_classification: Classification = (\n ontology_structure.get_child_by_title(\n title=\"Free text about the frame\", type_=Classification\n )\n)\ntext_classification_instance = text_ontology_classification.create_instance()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Add the classification instance to the label row\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# First set the value of the classification instance\ntext_classification_instance.set_answer(answer=\"This is a text classification.\")\n\n# Second, select the frames where the classification instance is present\ntext_classification_instance.set_for_frames(frames=0)\n\n# Then add it to the label row\nfirst_label_row.add_classification_instance(text_classification_instance)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Read classification instances\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Check the convenient filters of get_classification_instances() for your use cases\nall_classification_instances = first_label_row.get_classification_instances()\nassert all_classification_instances[0] == text_classification_instance" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Working with object/classification instance attributes\n\nBoth object instances and classification instances can have attributes. You can read more about examples\nusing these links: https://docs.encord.com/annotate/editor/images#frame-classification and\nhttps://docs.encord.com/annotate/editor/images#frame-classification\n\nIn the ontology you might have already configured text, radio, or checklist attributes for your object/classification.\nWith the LabelRowV2, you can set or get the values of these attributes. Here, we refer to as \"setting or getting an\nanswer to an attribute\".\n\n### Answering classification instance attributes\nThe case for answering classification instance attributes is simpler, so let's start with those.\n\nYou will again need to deal with the original ontology object to interact with answers to attributes. We have\nexposed convenient accessors to find the right attributes to get the attributes or the respective options by\ntheir title.\n\n

Note

When working with attributes, you will see that the first thing to do is often to grab the ontology object.\n Usually, when calling the `get_child_by_title` the `type_` is recommended, but still optional. However, for\n classifications this is often required.\n\n The reason is that the classification title is always equal to the title of the top level attribute of this\n classification. Therefore, it is important to distinguish what exactly you're trying to search for.

\n\n#### Text attributes\n\nAnswering text attributes is the simplest case and has already been shown in the section on classification instances\nabove.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Assume that the following text classification exists in the ontology.\ntext_ontology_classification: Classification = ontology_structure.get_child_by_title(\n title=\"Free text about the frame\",\n # Do not forget to specify the type here\n type_=Classification,\n)\ntext_classification_instance = text_ontology_classification.create_instance()\n\n# First set the value of the classification instance\ntext_classification_instance.set_answer(answer=\"This is a text classification.\")\n\nassert (\n text_classification_instance.get_answer()\n == \"This is a text classification.\"\n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We encourage you to read the `set_answer` and `get_answer` docstrings to understand the different behaviours and\npossible options which you can set.\n\n#### Checklist attributes\n\nAssume we have a checklist with \"all colours in the picture\" which defines a bunch of colours that we can\nsee in the image. You will need to get all the options from the checklist ontology that you would like to\nselect as answers.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "checklist_ontology_classification: Classification = ontology_structure.get_child_by_title(\n title=\"All colours in the picture\",\n # Do not forget to specify the type here\n type_=Classification,\n)\nchecklist_classification_instance = (\n checklist_ontology_classification.create_instance()\n)\n\n# Prefer using the `checklist_ontology_classification` over the `ontology_structure` to get the options.\n# The more specific the ontology item that you're searching from is, the more likely you will avoid title clashes.\ngreen_option: Option = checklist_ontology_classification.get_child_by_title(\n \"Green\", type_=Option\n)\nblue_option: Option = checklist_ontology_classification.get_child_by_title(\n \"Blue\", type_=Option\n)\n\nchecklist_classification_instance.set_answer([green_option, blue_option])\n\nassert sorted(checklist_classification_instance.get_answer()) == sorted(\n [green_option, blue_option]\n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Radio attributes\n\nLet's assume we have a radio classification called \"Scenery\" with the options \"Mountains\", \"Ocean\", and \"Desert\".\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "scenery_ontology_classification: Classification = ontology_structure.get_child_by_title(\n title=\"Scenery\",\n # Do not forget to specify the type here\n type_=Classification,\n)\n\nmountains_option = scenery_ontology_classification.get_child_by_title(\n title=\"Mountains\", type_=Option\n)\n\nscenery_classification_instance = (\n scenery_ontology_classification.create_instance()\n)\n\nscenery_classification_instance.set_answer(mountains_option)\n\nassert scenery_classification_instance.get_answer() == mountains_option" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Radio attributes can also be nested. You can read more about nested options here:\nhttps://docs.encord.com/ontologies/use/#nested-classifications\n\nLet's say that if you have the Mountains scenery, there is an additional radio classification called \"Mountains count\"\nwith the answers \"One\", \"Two\", and \"Many\". Continuing the example above, you can set the nested answer like this:\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "mountains_count_attribute = mountains_option.get_child_by_title(\n \"Mountains count\", type_=RadioAttribute\n)\ntwo_mountains_option = mountains_count_attribute.get_child_by_title(\n \"Two\", type_=Option\n)\n\nscenery_classification_instance.set_answer(two_mountains_option)\n\n# Note, that if for `set_answer` or `get_answer` the attribute of the classification cannot be inferred, we need\n# to manually specify it.\nassert (\n scenery_classification_instance.get_answer(\n attribute=mountains_count_attribute\n )\n == two_mountains_option\n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Answering object instance attributes\n\nSetting answers on object instances is almost identical to setting answers on classification instances.\nYou will need to possibly get the attribute, but also the answer options from the ontology.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "car_ontology_object: Object = ontology_structure.get_child_by_title(\n \"Car\", type_=Object\n)\ncar_brand_attribute = car_ontology_object.get_child_by_title(\n title=\"Car brand\", type_=RadioAttribute\n)\n# Again, doing ontology_structure.get_child_by_title(\"Mercedes\") is also possible, but might be more ambiguous.\nmercedes_option = car_brand_attribute.get_child_by_title(\n title=\"Mercedes\", type_=Option\n)\n\ncar_object_instance = car_ontology_object.create_instance()\n\ncar_object_instance.set_answer(mercedes_option)\n\n# The attribute cannot be inferred, so we need to specify it.\nassert (\n car_object_instance.get_answer(attribute=car_brand_attribute)\n == mercedes_option\n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setting answers for dynamic attributes\n\nDynamic attributes are attributes for object instances where the answer can change in each frame.\nYou can read more about them here: https://docs.encord.com/annotate/editor/videos/#dynamic-classification\n\nThese behave very similarly to static attributes, however, they expect that a frame is passed to the `set_answer`\nwhich will set the answer for the specific frame.\n\nThe read access, however, behaves slightly different to show which answers have been set for which frames.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "person_ontology_object: Object = ontology_structure.get_child_by_title(\n \"Person\", type_=Object\n)\n\nposition_attribute = person_ontology_object.get_child_by_title(\n title=\"Position\", # The options here are \"Standing\" or \"Walking\"\n type_=RadioAttribute,\n)\n\nperson_object_instance = person_ontology_object.create_instance()\n\n# Assume you would add the right coordinates of this person for frames 0-10 here.\n# Now assume the person is standing in frames 0-5 and walking in frames 6-10.\n\nperson_object_instance.set_answer(\n answer=position_attribute.get_child_by_title(\"Standing\", type_=Option),\n frames=Range(start=0, end=5),\n # Wherever you can set frames, you can either set a single int, a Range, or a list of Range.\n)\n\nperson_object_instance.set_answer(\n answer=position_attribute.get_child_by_title(\"Walking\", type_=Option),\n frames=Range(start=6, end=10),\n)\n\nassert person_object_instance.get_answer(attribute=position_attribute) == [\n AnswerForFrames(\n answer=position_attribute.get_child_by_title(\"Standing\", type_=Option),\n ranges=[Range(start=0, end=5)],\n ),\n AnswerForFrames(\n answer=position_attribute.get_child_by_title(\"Walking\", type_=Option),\n ranges=[Range(start=6, end=10)],\n ),\n]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Utils: Dealing with numeric frames\n\nYou will see that in many places you can use :class:`encord.objects.frames.Range` which allows you to\nspecify frames in a more flexible way. Use\n[one of the many helpers](https://python.docs.encord.com/api.html#module-encord.objects.frames)\naround frames to conveniently tranform between formats of a single frame, frame ranges, or a list of frames.\n\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.16" } }, "nbformat": 4, "nbformat_minor": 0 }