Working with labels

The encord.objects.LabelRowV2 class is a wrapper around the Encord label row data format. It provides a convenient way to read, create, and manipulate labels.

ℹ️

Note

The entire source code is provided for your reference.

ℹ️

Note

To interact with Encord, you need to authenticate a client. You can find more details here.

Specify a project

#Import dependencies
from pathlib import Path
from typing import List

from encord import EncordUserClient, Project
from encord.objects import (
    AnswerForFrames,
    Classification,
    LabelRowV2,
    Object,
    ObjectInstance,
    OntologyStructure,
    RadioAttribute,
)
from encord.objects.common import Option
from encord.objects.coordinates import BoundingBoxCoordinates
from encord.objects.frames import Range
from encord.orm.project import Project as OrmProject

# Instantiate Encord client by substituting the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)


# Find project to work with based on title.
project_orm: OrmProject = next(
    (
        p["project"]
        for p in user_client.get_projects(title_eq="Your project name")
    )
)
project = user_client.get_project(project_orm.project_hash)

View label metadata

To inspect metadata for label rows (data units), such as the label hash, when the label was created, the corresponding data hash, or the creation date of the label.

label_rows: List[LabelRowV2] = project.list_label_rows_v2()


for label_row in label_rows:
    print(f"Label hash: {label_row.label_hash}")
    print(f"Label created at: {label_row.created_at}")
    print(f"Annotation task status: {label_row.annotation_task_status}")

Inspect the filters in list_label_rows_v2() to only get a subset of the label rows.

You can find more examples around all the available read properties by inspecting the properties of the LabelRowV2 class.

Exporting labels

To export or download labels, or perform any other function that includes reading or writing labels, call the initialise_labels() method. This method downloads the current state of labels from the Encord server and create a label hash if none exists.

ℹ️

Note

See our Export labels documentation for a comprehensive code snippet for exporting label data.

Once this method has been called, you can create your first label.

first_label_row: LabelRowV2 = label_rows[0]

first_label_row.initialise_labels()
# ^ Check the reference for possible arguments

Saving labels

Once initialise_labels() has been called, you can create your first label.

first_label_row: LabelRowV2 = label_rows[0]

first_label_row.initialise_labels()
# ^ Check the reference for possible arguments

# Code to add/manipulate some labels goes here
...

# Once you have added new labels, you will need to call .save() to upload all labels to the server.
first_label_row.save()

Editing labels

Editing labels after they have been created involves the following steps:

  1. Download existing labels.
  2. Find the attribute you are looking for, and change the attribute answer.
  3. Save the changes.

The script below shows how to update the location of an object's bounding box, and attribute values. Make sure to substitute the following values:

  • <private_key_path> with the full path to your private key.
  • <project_hash> with the name of the project you want to update labels in.
  • <file_name> with the name of the file / data unit you want to edit labels for.
  • <object_name> with the name of the object.
  • <attribute_name> with the name of the attribute.
  • <option_name> with the name of the radio button option of the attribute.
from encord import EncordUserClient

# Instantiate Encord client by replacing <private_key_path> with the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Specify the project. Replace <project_hash> with the hash of your Project
project = user_client.get_project("<project_hash>")

# Specify the label row you want to edit labels for. This example takes the first label row
# Replace the <file_name.mp4> with the name of the file you want to change labels for
first_label_row = project.list_label_rows_v2(
    data_title_eq="<file_name>"
)[0]

# Download the existing labels 
first_label_row.initialise_labels()

# Get the first object instance in the label row
ontology_structure = first_label_row.ontology_structure

# Gets the object instance we want to edit the label for. This example uses the first
first_obj_instance = first_label_row.get_object_instances()[0]

# Specifies which Ontology object to change labels for
# Replace <object_name> with the name of your object. For example 'Flower'
my_object = ontology_structure.get_child_by_title(title="<object_name>")

# Sets coordinates for the specified object instance
# This example changes the location of a bounding box
first_obj_instance.set_for_frames(
    coordinates = BoundingBoxCoordinates(
        height=0.1474,
        width =0.1154,
        top_left_x=0.1097,
        top_left_y=0.3209
    ),
    # Overwrites if annotations are already present.
    manual_annotation=False, overwrite=True
)

# Selects a Radio button attribute
# Replace <attribute_name> with the name of the attribute you want to edit. For example 'Flower species'
my_attribute = my_object.get_child_by_title(title="<attribute_name>", type_=RadioAttribute)

# Selects the radio button option of the attribute
# Replace <option_name> with the name of the option. For example 'Sunflower' 
attribute_option = my_attribute.get_child_by_title(title="<option_name>", type_= Option)

# Sets the attribute to the attribute_option defined above
first_object_instance.set_answer(attribute_option, overwrite=True)

# Once you have updated labels and attributes, save your changes
first_label_row.save()

Creating/reading object instances

The encord.objects.LabelRowV2 class works with its corresponding Ontology. If you add object instances or classification instances, these will be created from the Ontology. You can read more about object instances here.

You can think of an object instance as a visual label in the Label Editor. One bounding box would be one object instance.

Finding the Ontology object

The LabelRowV2 is designed to work with its corresponding Ontology using the OntologyStructure. You must use the title or feature node hash to find the right objects, Classifications, attributes, or attribute options. See the example below to find the Ontology object for the demonstrative “Box of a human” object.

ontology_structure: OntologyStructure = first_label_row.ontology_structure
box_ontology_object: Object = ontology_structure.get_child_by_title(
    title="Box of a human", type_=Object
)
# ^ optionally specify the `type_` to narrow the return type and also have a runtime check.

Creating and saving an object instance

Creating and saving object instances is essential when importing annotations into Encord. Specify the label shape, the coordinates, and which frames the shape should occur on.

Bounding box 
# Find a bounding box annotation object in the project ontology
ontology_structure: OntologyStructure = label_row.ontology_structure
box_ontology_object: Object = ontology_structure.get_child_by_title(
        title="person", type_=Object
    )

# Instantiate an object instance from the box ontology node
box_object_instance: ObjectInstance = box_ontology_object.create_instance()

box_object_instance.set_for_frames(
    coordinates=BoundingBoxCoordinates(
        height=0.5,
        width=0.5,
        top_left_x=0.2,
        top_left_y=0.2,
    ),
    # Add the bounding box to the first frame
    frames=0,
    # There are multiple additional fields that can be set optionally:
    manual_annotation=True,
)

# Link the object instance to the label row.
first_label_row.add_object_instance(box_object_instance)

# Upload the label to the server
first_label_row.save()
Rotatable bounding box 
# Find a rotatable bounding box annotation object in the project ontology
ontology_structure: OntologyStructure = label_row.ontology_structure
rbb_ontology_object: Object = ontology_structure.get_child_by_title(
        title="person", type_=Object
    )

# Instantiate an object instance from the rotatable bounding box ontology node
rbb_object_instance: ObjectInstance = rbb_ontology_object.create_instance()

rbb_object_instance.set_for_frames(
    coordinates=RotatableBoundingBoxCoordinates(
        height=0.5,
        width=0.5,
        top_left_x=0.2,
        top_left_y=0.2,
        theta=305.7762
    ),
    # Add the bounding box to the first frame
    frames=0,
    # There are multiple additional fields that can be set optionally:
    manual_annotation=True,
)

# Link the object instance to the label row.
first_label_row.add_object_instance(rbb_object_instance)

# Upload the label to the server
first_label_row.save()
Polygon 
# Find a polygon annotation object in the project ontology
ontology_structure: OntologyStructure = label_row.ontology_structure
polygon_ontology_object: Object = ontology_structure.get_child_by_title(
        title="person", type_=Object
    )

# Instantiate an object instance from the polygon ontology node
polygon_object_instance: ObjectInstance = polygon_ontology_object.create_instance()

# The x,y coordinates of each polygon vertex are specified as follows
polygon_object_instance.set_for_frames(
    coordinates=PolygonCoordinates(
        [PointCoordinate(32,45), PointCoordinate(59,45), PointCoordinate(137,95), PointCoordinate(137,240)]
    ),
    # Add the polygon to the first frame
    frames=0,
    # There are multiple additional fields that can be set optionally:
    manual_annotation=True,
)

# Link the object instance to the label row.
first_label_row.add_object_instance(polygon_object_instance)

# Upload the label to the server
first_label_row.save()
Polyline 
# Find a polyline annotation object in the project ontology
ontology_structure: OntologyStructure = label_row.ontology_structure
polyline_ontology_object: Object = ontology_structure.get_child_by_title(
        title="person", type_=Object
    )

# Instantiate an object instance from the polyline ontology node
polyline_object_instance: ObjectInstance = polyline_ontology_object.create_instance()

# The x,y coordinates of each polyline vertex are specified as follows
polyline_object_instance.set_for_frames(
    coordinates=PolylineCoordinates(
        [PointCoordinate(32,45), PointCoordinate(59,45), PointCoordinate(137,95), PointCoordinate(137,240)]
    ),
    # Add the polyline to the first frame
    frames=0,
    # There are multiple additional fields that can be set optionally:
    manual_annotation=True,
)

# Link the object instance to the label row.
first_label_row.add_object_instance(polyline_object_instance)

# Upload the label to the server
first_label_row.save()
Keypoint 
# Find a keypoint annotation object in the project ontology
ontology_structure: OntologyStructure = label_row.ontology_structure
keypoint_ontology_object: Object = ontology_structure.get_child_by_title(
        title="person", type_=Object
    )

# Instantiate an object instance from the keypoint ontology node
keypoint_object_instance: ObjectInstance = keypoint_ontology_object.create_instance()

# The x,y coordinates of the keypoint are specified as follows
keypoint_object_instance.set_for_frames(
    coordinates=Pointcoordinate(
        x: 78
        y: 239
    ),
    # Add the keypoint to the first frame
    frames=0,
    # There are multiple additional fields that can be set optionally:
    manual_annotation=True,
)

# Link the object instance to the label row.
first_label_row.add_object_instance(keypoint_object_instance)

# Upload the label to the server
first_label_row.save()
Bitmask

Please see our dedicated documentation to learn how to import Bitmask annotations into Encord.

# Find a bitmask annotation object in the project ontology
ontology_structure: OntologyStructure = label_row.ontology_structure
bitmask_ontology_object: Object = ontology_structure.get_child_by_title(
    title="My bitmask feature", type_=Object
)

# Instantiate an object instance from the bitmask ontology node
bitmask_object_instance: ObjectInstance = bitmask_ontology_object.create_instance()

# The coordinates the bitmask are specified as follows

bitmask_ontology_object_instance.set_for_frames(
    # Create coordinates from provided numpy bitmask
    coordinates=BitmaskCoordinates(numpy_coordinates),
    # Add the bitmask to the first frame
    frames=0,
    # There are multiple additional fields that can be set optionally:
    manual_annotation=True,
)

# Link the object instance to the label row.
first_label_row.add_object_instance(bitmask_object_instance)

# Upload the label to the server
first_label_row.save()

Inspecting an object instance

You can now get all the object instances that are part of the label row.

# Check the get_object_instances optional filters for when you have many different object/classification instances.
all_object_instances: List[
    ObjectInstance
] = first_label_row.get_object_instances()

assert all_object_instances[0] == box_object_instance
assert all_object_instances[0].get_annotation(frame=0).manual_annotation is True

Adding object instances to multiple frames

Sometimes, you might want to work with a video where a single object instance is present in multiple frames. For example, you are tracking a car across multiple frames. In this case you would create one object instance and place it on all the frames where it is present. If objects are never present in multiple frames, you would always create a new object instance for a new frame.

# Assume you have the coordinates of a single object for the first 3 frames of a video.
# These are indexed by frame number.
coordinates_per_frame = {
    3: BoundingBoxCoordinates(
        height=0.5,
        width=0.5,
        top_left_x=0.2,
        top_left_y=0.2,
    ),
    4: BoundingBoxCoordinates(
        height=0.5,
        width=0.5,
        top_left_x=0.3,
        top_left_y=0.3,
    ),
    5: BoundingBoxCoordinates(
        height=0.5,
        width=0.5,
        top_left_x=0.4,
        top_left_y=0.4,
    ),
}


# OPTION 1 - think in terms of "the frames per object instance"
box_object_instance_2: ObjectInstance = box_ontology_object.create_instance()

for frame_number, coordinates in coordinates_per_frame.items():
    box_object_instance_2.set_for_frames(
        coordinates=coordinates, frames=frame_number
    )

# OPTION 2 - think in terms of the "object instances per frame"
box_object_instance_3: ObjectInstance = box_ontology_object.create_instance()

for frame_view in first_label_row.get_frame_views():
    frame_number = frame_view.frame
    if frame_number in coordinates_per_frame:
        frame_view.add_object_instance(
            object_instance=box_object_instance_3,
            coordinates=coordinates_per_frame[frame_number],
        )

Read access across multiple frames

As shown above with OPTION 1 and OPTION 2, you can think of the individual object instances and on which frames they are present or you can think of the individual frames and which objects they have. For a read access thinking of the individual frames can be particularly convenient.

for label_row_frame_view in first_label_row.get_frame_views():
    frame_number = label_row_frame_view.frame
    print(f"Frame number: {frame_number}")
    object_instances_in_frame: List[
        ObjectInstance
    ] = label_row_frame_view.get_object_instances()
    for object_instance in object_instances_in_frame:
        print(f"Object instance: {object_instance}")
        annotation = object_instance.get_annotation(frame=frame_number)
        print(f"Coordinates: {annotation.coordinates}")

Working with a classification instance

Creating a classification instance is similar to creating an object instance. The only differences are that you cannot create have more than one classification instance of the same type on the same frame and that there is no coordinates to be set for classification instances.

You can read more about classification instances here

Get the ontology classification

# Assume that the following text classification exists in the ontology.
text_ontology_classification: Classification = (
    ontology_structure.get_child_by_title(
        title="Free text about the frame", type_=Classification
    )
)
text_classification_instance = text_ontology_classification.create_instance()

Add the classification instance to the label row

# First set the value of the classification instance
text_classification_instance.set_answer(answer="This is a text classification.")

# Second, select the frames where the classification instance is present
text_classification_instance.set_for_frames(frames=0)

# Then add it to the label row
first_label_row.add_classification_instance(text_classification_instance)

Read classification instances

# Check the convenient filters of get_classification_instances() for your use cases
all_classification_instances = first_label_row.get_classification_instances()
assert all_classification_instances[0] == text_classification_instance

Working with object/classification instance attributes

Both object instances and classification instances can have attributes. You can read more about examples using these links: https://docs.encord.com/docs/annotate-label-editor#instances-and-frame-labels and https://docs.encord.com/docs/annotate-images#frame-classification

In the ontology you might have already configured text, radio, or checklist attributes for your object/classification. With the LabelRowV2, you can set or get the values of these attributes. Here, we refer to as “setting or getting an answer to an attribute”.

Answering classification instance attributes

The case for answering classification instance attributes is simpler, so let’s start with those.

You will again need to deal with the original ontology object to interact with answers to attributes. We have exposed convenient accessors to find the right attributes to get the attributes or the respective options by their title.

ℹ️

Note

When working with attributes, you will see that the first thing to do is often to grab the ontology object. Usually, when calling the getchild_by_title the type is recommended, but still optional. However, for classifications this is often required.
The reason is that the classification title is always equal to the title of the top level attribute of this classification. Therefore, it is important to distinguish what exactly you’re trying to search for.

Text attributes

Answering text attributes is the simplest case and has already been shown in the section on classification instances above.

# Assume that the following text classification exists in the ontology.
text_ontology_classification: Classification = ontology_structure.get_child_by_title(
    title="Free text about the frame",
    # Do not forget to specify the type here
    type_=Classification,
)
text_classification_instance = text_ontology_classification.create_instance()

# First set the value of the classification instance
text_classification_instance.set_answer(answer="This is a text classification.")

assert (
    text_classification_instance.get_answer()
    == "This is a text classification."
)

We encourage you to read the set_answer and get_answer docstrings to understand the different behaviours and possible options which you can set.

Checklist attributes

Assume we have a checklist with “all colours in the picture” which defines a bunch of colours that we can see in the image. You will need to get all the options from the checklist ontology that you would like to select as answers.

checklist_ontology_classification: Classification = ontology_structure.get_child_by_title(
    title="All colours in the picture",
    # Do not forget to specify the type here
    type_=Classification,
)
checklist_classification_instance = (
    checklist_ontology_classification.create_instance()
)

# Prefer using the `checklist_ontology_classification` over the `ontology_structure` to get the options.
# The more specific the ontology item that you're searching from is, the more likely you will avoid title clashes.
green_option: Option = checklist_ontology_classification.get_child_by_title(
    "Green", type_=Option
)
blue_option: Option = checklist_ontology_classification.get_child_by_title(
    "Blue", type_=Option
)

checklist_classification_instance.set_answer([green_option, blue_option])

assert sorted(checklist_classification_instance.get_answer()) == sorted(
    [green_option, blue_option]
)

Radio attributes

Let’s assume we have a radio classification called “Scenery” with the options “Mountains”, “Ocean”, and “Desert”.

scenery_ontology_classification: Classification = ontology_structure.get_child_by_title(
    title="Scenery",
    # Do not forget to specify the type here
    type_=Classification,
)

mountains_option = scenery_ontology_classification.get_child_by_title(
    title="Mountains", type_=Option
)

scenery_classification_instance = (
    scenery_ontology_classification.create_instance()
)

scenery_classification_instance.set_answer(mountains_option)

assert scenery_classification_instance.get_answer() == mountains_option

Radio attributes can also be nested. You can read more about nested options here: https://docs.encord.com/docs/annotate-working-with-ontologies#nested-classifications

Let’s say that if you have the Mountains scenery, there is an additional radio classification called “Mountains count” with the answers “One”, “Two”, and “Many”. Continuing the example above, you can set the nested answer like this:

mountains_count_attribute = mountains_option.get_child_by_title(
    "Mountains count", type_=RadioAttribute
)
two_mountains_option = mountains_count_attribute.get_child_by_title(
    "Two", type_=Option
)

scenery_classification_instance.set_answer(two_mountains_option)

# Note, that if for `set_answer` or `get_answer` the attribute of the classification cannot be inferred, we need
# to manually specify it.
assert (
    scenery_classification_instance.get_answer(
        attribute=mountains_count_attribute
    )
    == two_mountains_option
)

Answering object instance attributes

Setting answers on object instances is almost identical to setting answers on classification instances. You will need to possibly get the attribute, but also the answer options from the ontology.

car_ontology_object: Object = ontology_structure.get_child_by_title(
    "Car", type_=Object
)
car_brand_attribute = car_ontology_object.get_child_by_title(
    title="Car brand", type_=RadioAttribute
)
# Again, doing ontology_structure.get_child_by_title("Mercedes") is also possible, but might be more ambiguous.
mercedes_option = car_brand_attribute.get_child_by_title(
    title="Mercedes", type_=Option
)

car_object_instance = car_ontology_object.create_instance()

car_object_instance.set_answer(mercedes_option)

# The attribute cannot be inferred, so we need to specify it.
assert (
    car_object_instance.get_answer(attribute=car_brand_attribute)
    == mercedes_option
)

Setting answers for dynamic attributes

Dynamic attributes are attributes for object instances where the answer can change in each frame. You can read more about them here: https://docs.encord.com/docs/annotate-videos#dynamic-classification

These behave very similarly to static attributes, however, they expect that a frame is passed to the set_answer which will set the answer for the specific frame.

The read access, however, behaves slightly different to show which answers have been set for which frames.

person_ontology_object: Object = ontology_structure.get_child_by_title(
    "Person", type_=Object
)

position_attribute = person_ontology_object.get_child_by_title(
    title="Position",  # The options here are "Standing" or "Walking"
    type_=RadioAttribute,
)

person_object_instance = person_ontology_object.create_instance()

# Assume you would add the right coordinates of this person for frames 0-10 here.
# Now assume the person is standing in frames 0-5 and walking in frames 6-10.

person_object_instance.set_answer(
    answer=position_attribute.get_child_by_title("Standing", type_=Option),
    frames=Range(start=0, end=5),
    # Wherever you can set frames, you can either set a single int, a Range, or a list of Range.
)

person_object_instance.set_answer(
    answer=position_attribute.get_child_by_title("Walking", type_=Option),
    frames=Range(start=6, end=10),
)

assert person_object_instance.get_answer(attribute=position_attribute) == [
    AnswerForFrames(
        answer=position_attribute.get_child_by_title("Standing", type_=Option),
        ranges=[Range(start=0, end=5)],
    ),
    AnswerForFrames(
        answer=position_attribute.get_child_by_title("Walking", type_=Option),
        ranges=[Range(start=6, end=10)],
    ),
]

Utils: Dealing with numeric frames

You will see that in many places you can use encord.objects.frames.Range which allows you to specify frames in a more flexible way. Use one of the many helpers around frames to conveniently transform between formats of a single frame, frame ranges, or a list of frames.