If an artificial intelligence model was an engine, then its training data would be its fuel. And like an engine of an automobile, the quality of its functionality is largely dependent on the quality of what is fed into it. To paraphrase a well-known computer science phrase, if you put garbage in, you get garbage out.The key to ensuring that your models are providing accurate outputs lies in the training data, and data annotation is vital in providing the structure and context to datasets that allow AI models to learn effectively. Without well-structured context, those datasets are just noise.But annotating huge datasets, whether images or videos, can be a never-ending (and tedious) task, and the size of these tasks are set to grow significantly as AI models demand larger datasets. Thankfully, there are a range of tools designed to make image and video data annotation an efficient process. In this article, we will examine what is image annotation, who is using it, how to use it, and how CVAT might be a good choice for your organization’s computer vision or machine learning projects.What is Image Annotation?Image annotation (or image data labeling) is the process of adding labels and tags to image datasets for training computer vision models. Doing so provides context for the machine learning model to understand, and make predictions.With image data annotation software, the annotations generally come in the form of a shape such as a bounding box, polygon, or segmentation mask, along with a textual tag, or label. The geometric shape helps to visually and spatially define the object of interest in the image, while the textual tags help the AI model to identify and classify the object(s) in the image. Example of an image with bounding box annotations.‍Because in computer vision, identification and classification are key processes that help machines interpret and understand visual data, and image annotation is required for achieving this goal.Image annotation can be a huge undertaking in terms of the amount of time and resources needed. With datasets ranging in size from a few thousand images, to several million images, it’s important to determine the best strategy for both acquiring datasets, and for annotating them. Such strategies can involve usage of public versus proprietary datasets, and the choice to use in-house annotation versus professional annotation services. We will examine these strategies in more depth later.‍Where and What Image Annotation Is Used For?Autonomous vehicles, medical imaging, facial recognition, and satellite imagery, all use computer vision and artificial intelligence to perform tasks such as detection, classification and segmentation. All of these industrial applications make use of labeled datasets, and image annotation plays a significant role in the transformation from a dataset (for example, a collection of drone imagery), to a labelled dataset, that provides context for the AI model to use.Data annotation is mostly used for training machine learning models. And by training, we mean that we are teaching a computer vision model to identify objects in various kind of images. This is analogous to how a child learns by pointing at things, and calling them out by name. In short, image annotation is providing ground truth labels for the computer vision model. Image annotation is also used for supervised learning, in which the model learns from annotated examples in the form of input-output pairs. In this case, the annotated data assists the model in understanding how an object should be identified.Finally, image annotation can also be used with performance evaluation to assess the accuracy of a model’s learning. The accuracy can be tested by comparing the model’s output to the annotated data (ground truth).As mentioned in the introduction, quality training data for these systems is largely dependent on quality image annotation. A proper image annotation process can help to improve model accuracy/data consistency, reduce time, costs, and biases, and lead to generally more efficient model training.‍How Are Companies Doing Image Annotation?Any company or organization looking to develop their own computer vision AI models will require high-quality image datasets. These datasets tend to be quite huge, so consideration must be made as to how the datasets are obtained in the first place.This begs the question, should an entity opt for proprietary or open datasets? Let’s examine both cases in more detail.Creating Your Own Proprietary Image DatasetsThe benefits of creating a proprietary dataset include having complete control over the subject matter in the images, as well as the quality of the image. Many open datasets lack the selectivity of a particular subject matter, meaning that they might not be well suited for specialized models.Imagine that you wanted to train an AI to detect a particular type of defect on a manufacturing process developed in-house. By definition of the process’ proprietary nature, it would be close to impossible to find an open dataset with the images needed for training. The datasets, in this case, would necessarily need to be produced in house.A Google Street view car, capturing real world images as it drives around. Source: Wikimedia Commons‍The downside of creating such a dataset, is that the task requires vast resources in terms of image data collection (taking photos), data cleaning and preprocessing, annotation and labelling, quality control, storage and management. It is a time-consuming, and costly exercise.Some examples of companies making use of their own proprietary datasets include Tesla and Google. Tesla collects footage from their own vehicle sensors and makes use of this image data to train its self-driving AI feature (also known as FSD). Similarly, Google uses images gleaned from their own image assets for training Google Lens and Street View AI.Using Open DatasetsOne alternative (often used by smaller companies) is to use open image datasets, which reduce costs and speed up development of models. Many such datasets tend to be created by universities and government institutes, and are often freely available for research use and non-commercial use. Some are available for commercial use, but they are dependent on license conditions, and may require some fee to be paid.The downside of using open datasets is that they may lack specialization for specific tasks, as per the manufacturing example in the section on proprietary datasets.Panoptic segmentation datasets from COCO. Source: https://cocodataset.org.Think of proprietary and open datasets as bespoke tailored clothing versus clothing bought “ready to wear” from a shopping mall. With the tailored garment, you get to choose the material, the fit, the exact color, and any other features that you desire, but the customization comes at a premium price. With ready to wear clothing, you are restricted to whatever is on the shelf in terms of size, style and color, but you save a lot of money compared to the bespoke option. The table below shows several open datasets that can all be imported into CVAT image annotation tool for data labeling.{{image-annotation-table-1="/blog-banners"}}To summarize, your organization’s choice of proprietary or open datasets will depend largely on the resources available and the level of specialization needed for your training data.Proprietary datasets allow a high level of specialization, at the cost of time and financial resources, while open datasets allow faster development at a more cost-effective price point, but may take a penalty when it comes to specialization.Some companies might benefit from a hybrid approach, where open data is used for initial training, and then switch to real data to fine tune their own models.Each method has its own merits and trade-offs, and should be well considered before embarking on the task of developing an AI model.‍What Are Different Types of Image Annotation Tasks?So, you’ve decided exactly where your dataset is coming from, and are ready to begin the process of adding context to the data for training your model. This is where the image annotation phase (and image annotation software like CVAT) comes into play.At its core, the process of image annotation involves highlighting the item of interest in an image or video data, and adding context via text-based notes to the item in question. The type of annotation would depend on the intended use of the data.For example, if you wanted to train a model to recognize the presence of a cat in an image (image classification), you would upload image data consisting of cats in various scenes. You could then instruct your in-house or third-party image annotation services team to sort through the images, and add a text description indicating if a cat is present in the image, or not.More advanced tasks (such as “detection”) would require a bounding box to be drawn around the cat in the image, with various other descriptions (such as color or breed) added as tags.CVAT provides a variety of different image annotation tools for all of these tasks. Such tools include cuboid annotation (for objects with depth or volume), attribute annotation, tag annotation, and a plethora of different shape annotation tools for 2D objects.Image Annotation TasksThe image annotation process requires applying various labels to images (or video) in order to add structure and context to datasets.Image annotation tasks can generally be divided into three different categories, which are classification, detection, and segmentation. CVAT has a number of drawing tools which are aimed at each category. Before we delve into the drawing tools, let’s take a look at the three categories in more detail. Image ClassificationImage classification is the most basic of image annotation categories. It involves applying a label (or labels) to a singular image and simply helps the AI model to identify if such an object is present. With the image classification method, the object location is not specified, only its presence in the image. The label will then aid the computer vision model in identifying similar objects throughout the whole dataset. As an example, your team might be training an AI to recognize images of cats. With the image classification method, each image could be labelled as “cat” (if present) or “no cat” (if not present). Additional tags could be added to classify each image by breed, or by color. But the classification model would not be able to identify where exactly the cat is located within the image. With image classification, it is not necessary to use the shape drawing notation tools, as the labels/tags are applied to the entire image.To indicate where the object is located in the image, you need to use a detection model.‍Object DetectionDetection expands upon classification by adding a localization element. Detection not only identifies the presence of an object in an image, but adds spatial information, which helps to identify the object’s location in the image.Such tasks require the use of drawing tools (such as a rectangle/bounding box, polygon, or ellipse) to be added during notation to highlight where the object of interest is positioned. These drawing tools help the AI model understand both the object’s presence and its position in the image.Additionally, if there are multiple objects in the image, the detection model can specify how many there are, and more advanced models can even assign a confidence score, which indicates the likelihood that the identification is correct. Finally, more advanced models can also detect interactions and relationships between multiple models.So going back to our cat example, a detection model could identify that there are two cats, it could classify them according to breed (with a confidence score), and then infer one cat’s position in relation to another cat.Image segmentationSegmentation annotation is the most advanced of the three categories, and divides an image into discrete areas, providing pixel-level accuracy. There are four main subcategories of segmentation in computer vision, which are Semantic, Instance and Panoptic Segmentation types.Semantic SegmentationWith semantic segmentation, each pixel of the object of interest is assigned a class label. Whereas the detection model will use a bounding box to assign a general class and location within the image, defining the object at a pixel level allows the model to detect the shape with more precision.For example, an image might have a cat drinking out of a bowl while another cat sleeps nearby. During the annotation process, the annotator could use a brush tool to paint the pixels of both of the cats, or use a polygon tool. All the pixels within the masks would be classed as “cat”. Similarly, the bowl could also be annotated, with all the bowl pixels labeled as “bowl”.With semantic segmentation, the model does not distinguish between multiple objects of the same class. Both cats, despite being in their own discrete regions, would simply be classified as “cat”. To distinguish multiple instances of the same class, you need to use instance segmentation. Instance SegmentationAn instance segmentation model also uses masks to assign pixel-level classification to objects, but unlike semantic segmentation, it can identify different instances of the same class. For example, it could distinguish between two cats in an image, each with a different label. During the annotation process, the annotator would create a mask around each cat, showing the exact shape and boundaries of each cat. Unlike detection, which only provides a bounding box, instance segmentation gives a detailed pixel-by-pixel representation of each cat. Panoptic SegmentationThe final category is panoptic segmentation, which combines the benefits of both instance segmentation and semantic segmentation to create a more complete understanding of an image. In this approach, annotators categorize both background elements (such as a wall, or carpet) as well as countable objects such as people, cars, or cats.In this case, if there was an image of three cats lounging on a patterned rug, using the panoptic segmentation method, we would treat the rug as a single background element, applying one uniform label to it. Each cat would be identified individually, with separate segmentation masks, distinguishing them even if they are curled up together or partially overlapping (occluded). This method gives AI a more complete understanding of a scene, allowing it to recognize both the setting and the objects within it.‍Annotation Category SummaryBefore we take a more in-depth look at the various drawing tools, let’s just summarize the annotation categories in terms of their function, along with some non-feline related applications.{{image-annotation-table-2="/blog-banners"}}‍Types of Image Annotation TechniquesAn effective image annotation software should be capable of annotating objects that are both static (shape mode) and objects in motion, across multiple frames (frame mode). And it should be able to use any kind of common image file, such as JPEG, PNG, BMP, GIF, PPM and TIFF.To make shape annotation tasks a cinch, CVAT allows users to annotate with rectangles, polygons, polylines, ellipses, cuboids, skeletons, and with a brush tool.While the various shape annotation tools can be used interchangeably in many situations, each tool works optimally for specific types of task.RectanglesAnnotating with rectangles is one of the easiest methods of image annotation. Also known as a “bounding box”, this shape is best suited for the detection of uncomplicated objects such as doors on a building, street furniture, packing boxes, animals, and faces. They can even be used for notation of people both static and in motion. This is particularly useful for surveillance or tracking projects, although if pose estimation is required, more detailed annotations such as skeleton or polygons could be a better option.‍When multiple objects obstruct each other, the "Occluded" label can be used.Overall, notation with rectangles is an easy and computationally efficient method well-suited for quick object detection of a broad range of subjects. If you want a quick way to identify the general presence and location of the object, then notation with rectangles is a great place to start.Image annotation with rectangles is incredibly straightforward. In CVAT, simply select the rectangle icon in the controls sidebar, choose a label, and select a Drawing Method (2-point or 4-point). Click Shape (or Frame, if annotating video) to enter drawing mode.2-Point Method: Click two opposite corners (top-left and bottom-right) to create a rectangle.4-Point Method: Click the top, bottom, left, and right-most points of the object for a more precise fit. The rectangle is completed automatically after the fourth click.Users can adjust the boundaries of the resulting rectangle using the mouse, and rotate it to best fit the object of interest. Polygons Offering a higher level of precision than rectangles, annotating with polygons is better suited for objects with irregular shapes requiring a more accurate boundary delineation.Drawing a polygon allows a much higher level of detail, as it can closely follow the curves and shape of an object, making it well suited for tasks that require pixel-level analysis. Polygons can also be used for creating masks for semantic segmentation, instance segmentation and panoptic segmentation.Polygon annotation can be used for the detection of objects such as geographical features on satellite images, tumors in medical imagery, types of plants in plant identification, and pretty much anything where an object’s shape is too complex to be captured by a rectangle.If a rectangular notation is best suited for broad object detection tasks, polygons are more optimally used for tasks such as image localization, segmentation, or detailed recognition. To put it another way, while a rectangle annotation is fine for detecting faces, polygons are better for detecting facial features such as mouths, eyes, and noses.Like the rectangle annotation function, drawing polygons is uncomplicated. To draw a polygon in CVAT, locate the polygon option in the controls sidebar, and choose a label. Click Shape to enter drawing mode, then the polygon can be drawn with either of two methods. With the first method, the user can simply use the mouse to draw dots around the outline of the object. With the second method, the user can hold the shift key down, and trace the object with the mouse as a continuous contour. Dots will appear around the object automatically. You can see an example of this in the graphic below.‍Manual drawing of a polygon annotation‍Once the polygon is completed (with either method), the user can adjust the polygon by clicking on the dots, and dragging them until they are happy with the result.‍EllipsesAnnotating with ellipses is the method most useful for the detection of round objects, either elliptical, circular or spherical. If you want to quickly annotate objects such as wheels, various fruits, or even the eyes on a face, then ellipses are the perfect shape for the task.Applications where you might wish to use elliptical annotations include cell detection in medical imaging, pupils in eye tracking, astronomical objects, circular craters in geospatial mapping, or egg monitoring in a hatchery.‍‍In CVAT, ellipses are created in much the same way as rectangles. Simply specify two opposite points, and the ellipse will be inscribed in an imaginary rectangle. And like the rectangular notations, ellipses can also be rotated about a point. You can see how easy it is to annotate with ellipses in the video above.‍PolylinesThe previously mentioned notation types have focused on objects with enclosed regions. Polylines also allow for the notation of elongated, thin enclosed shapes, but also permit the notation of non-enclosed linear, continuous objects.To that end, polyline notation is the most optimal choice for objects with long boundaries and contours that do not need to be fully enclosed, such as railways lines or roads.It is also extremely handy when it comes to tasks requiring path-based analysis, such as object tracking, and for connecting key points in pose estimation tasks. Specific examples of applications using polylines include footpaths and rail lines in aerial mapping, general linear infrastructure inspection, text lines and paragraphs in OCR, animal and human skeletons in pose estimation, and moving objects in video sequences.‍A polyline on a continuous road marking‍To sum it up, polylines are at their most useful when the goal is to track, detect, or measure linear features.Drawing polylines in CVAT is similar to drawing a polygon. Simply select the polyline tool from the control panel, select the shape (or track), and set the number of points needed for the polyline. The drawing will complete when the specified number of points has been reached. Also, like the polygon tool, there are two ways in which a polyline can be drawn - it can be drawn with dots, or it can be traced along manually by holding down the shift key.‍Brush ToolThe brush tool is a free-form tool that allows the manual painting of objects, and the creation of masks. Masking is particularly useful for annotating singular objects that may appear split in two, such as a vehicle with a human standing in front of it. You can see an example of this in the graphic below.Example of image annotation with a brush tool‍The brush tool in CVAT features various modes such as brush shape selection, erase pixels, and polygon-to-mask. Polygon-to-mask mode enables quick conversion of polygon selections into masks. Annotations can be saved and modified via the Brush Tool menu, enhancing efficiency in detailed image segmentation tasks.Annotating with the brush tool is ideal for applications that require a high level of precision, such as medical imaging, object detection, or autonomous driving.‍SkeletonsAnnotating with skeletons is the best option when dealing with tasks requiring the analysis of complex and consistent structures, such as human figures. It’s also a little more involved than the other annotation processes we have looked at in this article, which is why we have saved it until the end!‍Example of a skeleton annotation‍A Skeleton consists of multiple points (also referred to as elements), which may be connected by edges. Each point functions as an individual object, with its own unique attributes and properties such as color, occlusion, and visibility.Skeleton annotations can be used for both static and moving images, although they are used in different ways for each type. When using skeleton notation with static images, they are best used when analyzing a single pose, whereas in video, they can be used for more dynamic applications (such as tracking movement over time).Other specific applications of skeleton-based annotations include gait analysis, workplace ergonomics assessments, gesture recognition for sign language, crime scene analysis, and avatar posture recognition in AR/VR environments.Out of the various other methods we have looked at for notating static images in this article, notating with skeletons is generally the most complex. However, the whole process of annotating with skeletons is made much more user-friendly with CVAT.If you wish to annotate with skeletons with CVAT, then the process is summarized as follows:There are two main methods of annotating with skeletons. The first is to do it manually, and the second is to load a skeleton from a model. The Skeleton Configurator allows users to add, move, and connect points, upload/download skeletons in .SVG format, and configure labels, attributes, and colors for each point.To use the Skeleton Configurator, set up a Skeleton task in the configurator, and click Setup Skeleton to enable manual creation. To create the skeleton, simply add points and edges in the drawing area, configure the required attributes, upload the files, and submit the task. ‍AI-Assisted Image Data AnnotationAs seen in the previous section, annotating with various shapes is a straightforward experience. But these tasks can be made easier still, thanks to various automation features.AI-assisted image annotation makes use of pre-trained ML models for the detection, classification, and segmentation of objects within image datasets. CVAT can use pre-installed models, and can also integrate with Hugging Face and Roboflow for cloud-hosted instances. For organizations using a self-hosted setup, custom models can be used with Nuclio. AI models in CVAT, such as YOLOv3, YOLOv7, RetinaNet, and OpenVINO-based models, provide accurate object detection, facial recognition, and text detection. CVAT’s automated shape annotations and labeling features can significantly accelerate the complex image annotation process, potentially improving speed by up to 10 times. These features leverage various machine learning algorithms for tasks like object detection, semantic segmentation, and tracking. Automatic Labeling using pre-trained deep learning models (e.g., OpenVINO, TensorFlow, PyTorch).Semi-Automatic Annotations (e.g., interactive segmentation).Automatic Mask Generation: AI models can generate segmentation masks for complex objects.Smart Polygon Tool: Automatically refines polygon shapes around detected objects.Pre-Trained Object Detectors: Detects and labels objects using AI models like YOLO, Mask R-CNN, or Faster R-CNN.We will do a deep dive into the automation side of image annotation in another post - we just thought we would draw your attention to its existence, just in case you wanted to know how AI itself can be used to make the model training process even more efficient.‍Easy Annotation & Labeling of Images with CVAT Annotation SoftwareAs you have seen in this article, there are numerous techniques in image annotation specific for a range of different computer vision projects and use cases. The good news is that CVAT offers all the aforementioned tools in a handy and easily accessible solution.So whether your team is training a computer vision model, engaging in supervised learning, or conducting a performance evaluation, then the CVAT platform can help with all of your data annotation needs.CVAT takes away the headaches of creating annotated datasets with its innovative and user-friendly approach to annotation and task allocation. With its image annotation tool , your organization can upload datasets of visual assets, break the sets down into smaller chunks, and distribute them to team members anywhere on Earth. Once the team members receive their tasks, they are able to use the intuitive image annotation engine to quickly add context to both image and video datasets.CVAT also integrates seamlessly with HUMAN Protocol’s innovative task distribution and compensation system, creating a seamless, efficient workflow for crowdsourcing annotations. And, if you don't have enough resources to do annotation in-house, CVAT's professional annotation services team is available to provide high-quality, expertly labeled datasets, ensuring your machine learning models receive the precise training data they need.So, to summarize - CVAT's image annotation platform can be used for any visual object, whether it's flat or three-dimensional, static or dynamic. And the drawing tools are fundamentally the same for whichever scenario. Naturally, there are more advanced features for power users, and if you would like to know more about those, you can learn more at this link.And if you haven’t yet got to grips with the basics of image annotation and would like to get started with the features in this article, you can try out the free SaaS version of CVAT right here. For those wanting to try the on-premise community version, you can find that over on Github.

Whether you're developing precision agriculture systems to detect crop diseases, creating AI-powered tools for early lung cancer detection from CT scans, or building theft detection systems for convenience stores, the success of your AI project hinges on one crucial element: high-quality annotated data. Even the most sophisticated AI models are only as good as the data they're trained on.‍“OK, but how do I make sure the data we get from our in-house annotation team or data labeling agency is actually good?”, you ask. And we answer: data labeling specifications.‍What are data labeling specifications? And, why does your project need them?Data labeling specifications (or annotation specifications) are documentation that provides clear instructions and guidelines for annotators on how to annotate or label data. Depending on the project, these guidelines may include class definitions, detailed descriptions of labeling rules, examples of edge cases, and visual references such as annotated images or diagrams.Labeling specifications serve several critical purposes:‍Ensure all annotators follow the same standardsMaintain consistency across large datasetsEnable quality controlHelp achieve the required accuracy for model trainingServe as a reference document for both the client and annotation teamThe lack of well-thought-out specifications leads to all sorts of issues for all stakeholders involved—clients, labeling service providers, annotation teams, and ultimately, the end users of the data:#1 Inconsistent annotation resultsPoor specifications result in inconsistent annotation outcomes, as annotators are left to make assumptions and interpret tasks as they see fit. For example, if the guidelines don't specify how to handle occluded objects (e.g., a pedestrian behind a car), one annotator might use a bounding box while another uses a polygon. These inconsistencies can make the dataset unusable for model training, and often require its complete re-annotation. Source: https://cocodataset.org#2 Wasted time and moneyInconsistent annotation results inevitably trigger a costly cycle of revisions and rework, with each iteration requiring additional time from annotators, reviewers, and project managers. The result? Blown budgets and missed deadlines that could have been avoided with clear specifications from the start.‍#3 Frustrated annotation teamNothing kills team morale faster than having to redo work that's already been done. When annotators spend hours labeling data only to learn that the requirements weren't clear or complete, it's more than just frustrating—it's demoralizing. Productivity drops, attention to detail suffers, and the entire project enters a downward spiral. ‍#4 Project management overheadUnclear specifications turn project managers into full-time firefighters. Instead of focusing on strategic tasks, they're stuck in an endless cycle of retraining annotators, clarifying instructions, and double-checking work. Every vague requirement creates a ripple effect of questions, corrections, and additional reviews. This translates into more management hours, higher costs, and project managers who can't focus on what really matters—delivering quality results on time. ‍So, what makes a good specification?A well-crafted specification is like a detailed roadmap—it guides annotators to their destination without leaving room for wrong turns. Based on our experience working with hundreds of clients, here's what separates great specifications from the rest:‍Project Context. Don't just tell annotators what to do—help them understand why they're doing it. Whether your AI will be scanning crops for disease or monitoring store security, this context helps annotators make better decisions when they encounter tricky cases.Comprehensive Class Definitions. Think of this as your annotation dictionary. Every object class should be clearly defined, along with its key characteristics. For instance, what exactly counts as a "ripe tomato" in your agricultural dataset? What specific visual indicators should annotators look for?Clear Annotation Rules. Spell out exactly how you want things labeled. Should that partially visible car be marked with a bounding box or a polygon? How precise should segmentation masks be? Leave no room for guesswork.Edge Case Playbook. Every dataset has its tricky cases. Maybe it's a car hidden behind a tree or a disease symptom that's barely visible. Document these scenarios and provide clear instructions on how to handle them consistently.Red Flags and Common Pitfalls. Show annotators what not to do. By highlighting common mistakes upfront, you can prevent errors before they happen and save countless hours of revision time.Visual Examples (That Actually Help). A picture is worth a thousand words. This is true for labeling specs too. Include plenty of annotated examples showing both perfect and poor annotations. These real-world references are often more valuable than written descriptions alone.When you nail your specifications, the benefits cascade throughout your entire project:‍Every annotator follows the same playbook, delivering uniform results that your AI models can actually learn from. No more dealing with a mishmash of annotation styles that confuse your training process.Clear instructions mean fewer mistakes and less back-and-forth. Your team can work confidently and efficiently, keeping your project timeline on track.Every round of corrections burns through your budget. With crystal-clear specifications, you slash the need for revisions and keep costs under control. Plus, modern annotation platforms like CVAT come with built-in specification support, making it even easier for your team to stay on track.Now, let's put it to the test and see how good vs. bad labeling specs play out with a real-world dataset.‍“Good vs. Bad” Labeling Specifications: A Head-to-Head Test‍Source: https://cocodataset.orgThe setupAn image of a parking lot with different cars, road signs, people, trees, and fences. .Two annotators.Two different specs.The specs‍The first annotator was given very basic instructions: ‍Annotate the road, signs, people and vehicles using masks. Transportation must additionally be annotated with boxes.‍That's it. No quality guidelines, no examples, no nothing.The second annotator got a bit more lucky, and received a few more details:Annotate only the driveway and exclude the sidewalk from the annotation.Annotate signs together with their posts.Use only a mask, not a bounding box, for vehicles with less than 50% visibility.The results‍‍The results are quite descriptive. Without extra clarification, the first annotation is less accurate, missing some attributes such as signposts, and incorrectly labeling the sidewalk as part of the street. The second annotation is 100% accurate.‍A super-simple example, but when applied to a real use case, leaving out extra details can lead to thousands of inconsistent annotations, missed deadlines, unhappy annotators, and, worst of all, AI models that fail to perform reliably in production.‍Build better AI with better specifications Creating thorough labeling specs takes time and effort, but it's an investment that pays off many times over through consistent results, faster delivery, and significant cost savings. ‍To help you get started, we've created a comprehensive data labeling specification template based on our experience with hundreds of successful annotation projects. It covers all the essential elements we discussed and includes practical examples you can adapt for your specific needs. ‍Free Data Labeling Specs Template‍Download our free template and set your AI project up for success from day one. ‍{{labeling-specs-banner="/blog-banners"}}

Manual data labeling can be a real slog, especially when you’re working with massive datasets. That’s why automated annotation is such a lifesaver—it speeds up the process, ensures consistency, and frees you up to focus on building smarter machine learning models. CVAT OGs know that both our platforms (SaaS and on-premises) support a number of options for automated annotation using AI models, including:‍Nuclio platformRoboflow and Hugging Face integrationCLI-based annotation on your own hardwareThese methods are used and loved by thousands of users, but because data annotation projects come in all shapes and sizes, they may not work for everyone. Nuclio functions, for example, are currently managed by the CVAT administrator and are limited to CVAT On-prem installations. Roboflow and Hugging Face support a limited range of model architectures. CLI-based annotation requires users to set up and run models only on their own machines, which can be hardware-intensive and time-consuming for some teams.‍Today, we’re excited to share that CVAT is addressing all these limitations with the launch of AI agents. ‍What is a CVAT AI agent?An AI agent in CVAT is a process (or service) that runs on your hardware or infrastructure and acts as a bridge between the CVAT platform and your AI model. Its main role is to receive auto-annotation requests from CVAT, transfer data (e.g., images) to your model for processing, retrieve the resulting annotations (e.g., object coordinates, masks, polygons), and send these results back to CVAT for automatic inclusion in your task. ‍In other words, CVAT AI agents work as a bridge between your custom model and the CVAT platform, enabling seamless integration of your model into the auto-annotation process.‍How are CVAT AI agents different from other automation methods?‍Customization and accuracy: Unlike with Roboflow, or Hugging Face integrations, you can now use your own AI models, tailored specifically to your datasets and tasks, to produce precise annotations that meet your exact training requirements.Collaboration and accessibility: Unlike CLI-based annotations, AI agents allow you to centralize your model setup and share it across your organization. Team members can access and use the models without any additional setup.‍‍Flexibility across platforms: AI agents don’t require CVAT administrator control and are available on both CVAT Online and On-prem (Enterprise, version 2.25 or later), giving you the freedom to deploy and manage your models in any environment.These features make CVAT AI agents a powerful tool for scaling your annotation processes while maintaining accuracy, collaboration, and control.‍How to annotate data with CVAT AI agentNow, let’s see how to set up automated data annotation with a custom model using a CVAT AI agent. For that, you will need:An account on a CVAT instance. In the tutorial we’ll use CVAT Online, but you can use your own CVAT On-prem instance if you wish - just substitute your instance’s URL in the commands.A CVAT task with labels from the COCO dataset (or a subset of them) and some images.You will also need to install Python and CVAT CLI to your machine.‍Refresher: CLI-based annotationLet’s first briefly review how CLI-based annotation works, since the agent-based method has a lot in common with it.‍First, you need a Python module that implements the auto-annotation function interface from the CVAT SDK. These modules will serve as bridges between CVAT and whatever deep learning framework you might use. For brevity, we will refer to such modules as native functions. ‍The CVAT SDK includes some predefined native functions (using models from torchvision), but for this article, we’ll use a custom function that uses YOLO11 from Ultralytics. ‍Here it is:‍import PIL.Image from ultralytics import YOLO import cvat_sdk.auto_annotation as cvataa _model = YOLO("yolo11n.pt") spec = cvataa.DetectionFunctionSpec( labels=[cvataa.label_spec(name, id) for id, name in _model.names.items()], ) def _yolo_to_cvat(results): for result in results: for box, label in zip(result.boxes.xyxy, result.boxes.cls): yield cvataa.rectangle(int(label.item()), [p.item() for p in box]) def detect(context, image): conf_threshold = 0.5 if context.conf_threshold is None else context.conf_threshold return list(_yolo_to_cvat(_model.predict( source=image, verbose=False, conf=conf_threshold)))‍Save it to yolo11_func.py., and then run:‍cvat-cli --server-host https://app.cvat.ai --auth "<user>:<password>" task auto-annotate <task id> --function-file yolo11_func.py --allow-unmatched-labels‍This will make the CLI download the images from your task, run the model on them and upload the resulting annotations back to the task.‍Note: long-time readers might notice a few changes since the last time we talked about CLI-based annotation on this blog. In particular, we changed the command structure of CVAT so that you have to use task auto-annotate rather than just auto-annotate. In addition, native functions can now support custom confidence thresholds, so our YOLO11 example reflects that.‍Registering the function with CVATNow, let’s see how we can integrate the same model as an agent-based function.‍An important thing to know is that the agent-based functions feature also uses native functions. In other words, if you already have a native function you’ve used with the cvat-cli task auto-annotate command, you can use the same function as an agent-based function, and vice versa. So let’s reuse the yolo11_func.py file we just created.‍First, we must let CVAT know about our function. Use the following command:‍cvat-cli --server-host https://app.cvat.ai --auth “<user>:<password>” function create-native "YOLO11" --function-file yolo11_func.py‍The string “YOLO11” here is just a name that CVAT will use for display purposes; you can use any name of your choosing.‍Now, if you open CVAT and go to the Models tab, you will see our model there, looking something like this:‍‍You can click on it and check that it has all the expected properties, such as label names. However, if you actually try to use this model for automatic annotation, it will not work. The request will stay “queued”, and after a while, it will automatically be aborted. That’s because we need to do one final step.‍Note: At no point in the process does the function itself (like the Python code or weights) get uploaded to CVAT. The only information the registration process transfers to CVAT is metadata about the function, such as the name and list of labels.‍Powering the function with an agentWe must now run an agent that will process requests for the function. Use the following command:‍cvat-cli --server-host https://app.cvat.ai --auth “<user>:<password>” function run-agent 58 --function-file yolo11_func.py‍Instead of 58, substitute the model ID you see in the CVAT UI. You can also find the same ID in the output of the function create-native command. This command starts an agent for our function, which runs indefinitely. The job of the agent is to process all incoming auto-annotation requests involving the function.‍While the agent is running, open your task in CVAT and click Actions -> Automatic annotation. You’ll be able to select the YOLO11 model and set the auto-annotation parameters, like for any other type of model CVAT supports.‍Click "Annotate." After a short delay, you should see the agent start printing messages about processing the new request. Once it’s done, CVAT should notify you that the annotation is complete. You can then examine the jobs of your task to see the new bounding boxes. The agent will keep running, ready to process more requests.‍CleanupNow that we’re done testing the function, we can remove it from CVAT. First, interrupt the agent by pressing Ctrl+C in the terminal. Second, delete the function by running the following command:‍cvat-cli --server-host https://app.cvat.ai --auth “<user>:<password>” function delete 58‍Alternatively, you can do this through the UI: find the model in the Models tab, click the ellipsis and select Delete.‍Working in an organizationIn the preceding tutorial, you added the function to your personal workspace. In this case, only you can annotate with it. Now let’s discuss what’s needed to share a function with an organization.‍First, you’ll need to add an --org parameter to all of your CLI commands:‍cvat-cli --org <your organization slug> ...‍Second, you should be aware of the permission policy when you work in an organization. A function can be…‍… added by any organization supervisor.… removed by its owner or any organization maintainer.… used to auto-annotate a task by any user that has write access to that task.These rules are the same as for Roboflow and Hugging Face functions.‍In addition, to power a function, an agent must run as that function’s owner or any organization maintainer. However, an agent must be able to access data for tasks it’s requested to process. So if you want to make it possible to use the function on any task in the organization, you should run the agent as a user with the maintainer role.‍Technical detailsThe following diagram shows the major components involved in agent-based functions. In the general case, the agent can run in a completely separate infrastructure from the CVAT server. The only requirement is that it’s able to connect to the CVAT server via the usual HTTPS port. The agent does not need to accept any incoming connections. Of course, if you run your own CVAT instance, you can run the agent in the same infrastructure, even on the same machine if you’d like.‍While so far we’ve been talking about the agent, you’re not actually limited to running one agent per function. If you’d like to be able to annotate more than one task at a time, you can run multiple agents:All annotation requests coming from the users are placed in a queue and distributed to agents on a first-come-first-serve basis. If one agent crashes or hangs while processing a request, that request will eventually be reassigned to another agent.‍What AI agents can’t do (yet)AI agents are still pretty new, so there are a few things they can’t do just yet (but don’t worry, we’re on it and will roll out updates soon): ‍Annotate just one frame,Work with skeletons,Handle videos or 3D data tasks, orSupport shapes with attributes.‍Get started with CVAT AI agentsCVAT AI agents are here to level up how teams automate data annotation. Now, you can use models trained just for your unique datasets or tasks, no matter if you’re on CVAT Online or On-Prem. This means: ‍✅ more precise annotations that are better aligned with your requirements, ✅ less manual fixing, and ✅ datasets that are ready to go for AI training or deployment. ‍And, with a centralized setup, your whole team can easily access the model, speeding up workflows and improving collaboration.‍Ready to take your automated annotation to the next level? Sign up or log in to your CVAT Online account or contact us to get CVAT with AI agents support on your server.‍

IntroductionChoosing the right data annotation service is a key step in any AI or machine learning project. High-quality labeling services are essential for training algorithms and ensuring accurate predictions. CVAT (Computer Vision Annotation Tool) and Clarifai are two leading platforms offering various annotation services. These platforms cater to a wide range of users, from individual researchers to large companies.In this comparison, we’ll examine the strengths and weaknesses of both. We will focus on performance, scalability, and ease of use. We will also consider the target audience and suitability for specific industries. This will help you make the best choice for your project.‍Performance and capabilitiesCVAT is an open-source tool designed for teams that need more control and customization over their annotation workflows. It offers the following annotation types.Annotation types2D Image Annotations: Support for detailed annotations like bounding boxes, polylines, points, skeletons and polygons for more intricate data.Video Annotations: Capabilities for object tracking, recognition, and event detection in video-based tasks.3D Sensor Fusion: Provides support for annotations involving 3D sensor data, making it ideal for applications like autonomous driving, robotics, and LiDAR tasks.‍One of CVAT's key strengths is its ability to handle complex annotations, like instance and semantic segmentation with high precision. This makes it ideal for industries like healthcare, automotive, and surveillance, where detailed accuracy is very important.Clarifai is a comprehensive platform that focuses on automating data annotation processes to improve efficiency. Its main features include:‍2D Image Annotations: Efficient handling of large-scale image classification tasks using AI-driven automation, including bounding boxes and polygons.Text Classification: Support for natural language processing (NLP) initiatives, making it suitable for text-based projects.Video Annotations: Offers video object tracking to automate and simplify video analysis.Document Analysis: Named entity recognition (NER) for processing and analyzing large volumes of text efficiently.‍Clarifai is highly adaptable for different annotation tasks due to its AI tools. This makes it a good fit for industries like e-commerce, finance, and media. These industries handle a large amount of data, but the annotations are less complex.‍Ease of UseCVAT provides an easy-to-use platform that doesn't require technical expertise. Users can quickly sign up on the CVAT cloud platform and start labeling process right away. Data scientists and AI researchers value its powerful customization features. However, smaller teams or individuals without much technical knowledge can also use it effortlessly. The platform also supports complex project setups and allows for collaboration among multiple users, making it suitable for team-based projects.‍Clarifai is also designed for ease of use, requiring minimal setup. Its intuitive platform includes many automated features that help reduce manual effort. This makes it a great choice for project managers or companies looking to outsource data labeling without getting into the technical details. Teams can quickly start using the platform, even if they don’t have extensive technical knowledge in data annotation.‍‍Scalability and FlexibilityScalability is crucial for teams and organizations looking to expand their AI projects. CVAT excels in this area, primarily because it is open-source. This allows teams to enhance their annotation operations by improving infrastructure, adding custom plugins, or adjusting workflows to fit specific needs. Such flexibility is particularly beneficial for large organizations and AI research teams. These teams are involved in complex projects that require tailored workflows or intricate annotations. Examples include projects in the autonomous driving or aerospace sectors.‍‍On the other hand, Clarifai offers a simple approach to scalability. With its global workforce and AI automation, it excels in projects that require quick deployment. Companies in sectors like retail, healthcare, and marketing can easily scale their annotation needs. They can do this using Clarifai’s fully managed services. These services help reduce operational burdens. This is particularly advantageous for businesses looking for fast results without the need to establish a dedicated in-house annotation team.‍Industry-Specific SuitabilityClarifai and CVAT are versatile tools that can be applied across various industries, though they approach data annotation differently. Clarifai emphasizes automated data labeling, ideal for large datasets requiring speed and efficiency. Its AI-driven labeling is fast, yet it also supports manual annotation when needed for flexibility. On the other hand, CVAT focuses on manual labeling. This makes it better suited for tasks that demand high accuracy and human oversight. CVAT also offers automated and semi-automated annotation options. This allows CVAT to adapt to projects where repetitive or simpler tasks can be handled by AI. More complex tasks are left for human annotators.‍The decision between manual and automated annotation depends on the complexity of the data and specific project requirements. Automated annotation excels with large, straightforward datasets, whereas manual annotation is essential for more precise and intricate work. Both tools successfully cater to the unique data annotation requirements of various sectors, ensuring high-quality results across industries, including:‍HealthcareAnnotation helps analyze medical images like X-rays and MRIs. It is important for diagnosing tumors and other diseases.‍Surveillance and Security In this field, annotation is used for video tasks like event detection and facial recognition. It improves accuracy in important situations.‍Autonomous VehiclesAnnotation is key for object tracking and 3D sensor fusion. It trains models for lane detection, pedestrian tracking, and obstacle recognition.‍E-commerceAnnotation assists in classifying images and tagging products. This makes it easier to handle large data volumes and enhances user experience.‍Retail and MarketingIn these areas, annotation analyzes customer data. It helps businesses gain insights and make predictions.‍RoboticsAnnotation trains robots for tasks like object recognition and navigation. It creates reliable models for complex environments, such as automated warehouses and factories.‍Pricing ModelData Labeling ServicesA labeling service is a data annotation service used to train artificial intelligence models. Specialists manually mark objects in images or text so that the AI can learn to recognize and categorize them. This process is crucial for creating high-quality training datasets. These datasets allow AI to accurately perform tasks such as facial recognition, object detection, or text analysis. CVAT and Clarifai offer data labeling services. Below, we will review their data annotation offerings:‍CVAT· Discussion of Requirements: First, you contact the CVAT team or your contacts to discuss the details of your project. This helps them understand your specific needs and goals.· Proof of concept (POC) annotation: CVAT will request a data sample and an initial specification. This will allow CVAT to demonstrate its expertise. It will also help prepare an accurate project quote and estimate the time required to complete the project. This phase is completely free for a customer!· Team Formation: Depending on the scope and complexity of the project, CVAT may form a specialized team of annotators. This team will be responsible for carrying out the annotations according to your requirements.· Project and Task Creation: CVAT creates a project on their platform, including tasks for annotation. These tasks contain instructions and examples to guide the annotators on how to work with your data.· Data Preparation and Upload: You provide your data (images, videos, etc.), which are then uploaded into the system. CVAT supports various formats, making the upload process easier.· Annotation Process: The annotators begin working on annotating the data. CVAT offers powerful annotation tools, allowing the team to perform their tasks efficiently.· Quality Control: During and after the annotation, quality control is conducted. This may include reviewing the annotators' work and using automated tools to ensure accuracy.· Documentation: CVAT provides documentation for the project, including reports on completed work, quality metrics, and any important comments. This is useful for analysis and reporting.· Delivery of Annotated Data: Once the project is completed, you receive the annotated data in the agreed format, ready for use in your project.· Feedback and Support: The CVAT team remains in contact to gather your feedback on the process and provide support for any questions that may arise.‍Clarifai· Easy Execution: Users can effortlessly upload data in various formats to the Clarifai platform. The labeled data will be returned to the specified format for continued training, whether on Clarifai or another platform.· Expert and Flexible Workforce: The platform reduces the daily management burden of data labeling pipelines by allocating a specialized team based on expertise. A single team will manage the entire project to ensure consistency.· Quality Assurance Checkpoints: Clarifai conducts tests against data samples to ensure quality before finalizing the labeling of the complete training dataset. Users receive regular updates and transparency regarding quality metrics and turnaround times.· More Secure: The platform offers a secure environment for handling image, video, and document data. It adheres to strict security standards and data privacy principles. This allows users to select teams with background checks. The annotation takes place in secure facilities.· Flexible Pricing: Clarifai provides flat-rate pricing, making it easier to outsource data labeling needs and reduce operational overhead. Pricing scales with project growth.· Speed Time to Production: The team utilizes a state-of-the-art platform. This platform employs AI automation to expedite dataset annotation and project completion. It ensures high levels of accuracy.CVAT’s flexible pricing includes options like per-object, per-image, or hourly billing based on project demands. The only limitation for CVAT is that the project cost cannot be less than $5,000.‍Clarifai offers a more fixed project evaluation system, but there is also the option for a customized approach to the project.‍‍Suggestions for self-service on the platform.There are also plans available for independent work on the platform. Below is a comparison.CVAT‍Clarifai‍Additional Areas of ComparisonTo assist you in making an informed choice, here are five distinctions between CVAT and Clarifai:‍Integration with Existing Tools:CVAT's open-source architecture allows for seamless integration with third-party tools and custom pipelines. This makes it a suitable choice for teams with established AI ecosystems. This flexibility enables organizations to tailor their workflows to specific needs. While Clarifai also provides integration options, its emphasis on ready-to-use AI models may limit customization for teams with advanced technical skills.Project Management:CVAT offers robust project management features. These features allow team leaders to assign tasks, monitor progress, and collaborate in real time. This can be particularly beneficial for complex projects involving larger teams. Clarifai provides managed services for annotation and project management, which can streamline processes and support team coordination.Annotation Accuracy:CVAT is equipped with comprehensive annotation tools that are ideal for tasks demanding high precision, such as autonomous driving or medical imaging. Its capabilities allow for detailed data management. Clarifai utilizes AI-driven automation to enhance efficiency. This may be sufficient for many applications. However, it may face challenges with highly complex datasets.Turnaround Time:Clarifai's AI automation and distributed workforce are recognized for delivering faster turnaround times, making it suitable for projects that prioritize speed. Conversely, CVAT focuses on meticulous manual and semi-automated annotation. This ensures a high quality of results. This can be particularly important for complex datasets, even if it may take longer.Security and Data Privacy:CVAT's open-source nature allows for on-premise hosting. This grants organizations full control over data privacy. This is an essential feature for businesses handling sensitive information. Clarifai provides cloud-based solutions with strong security measures. This may appeal to companies that prioritize data security. However, it may not offer the same level of direct data control as CVAT.‍ConclusionCVAT and Clarifai are both powerful data annotation platforms, each serving different needs and applications. CVAT is well-suited for those requiring customizable, precise, and scalable solutions, particularly in sectors like robotics, autonomous driving, healthcare, and surveillance. Its open-source nature allows for easy installation and project management, especially for teams with the technical expertise to handle complex annotation tasks.‍On the other hand, Clarifai is designed for teams that value user-friendliness, automation, and rapid scalability. Its focus on AI features and managed services makes it a strong contender across various industries.‍Are you ready to make your choice? Explore both CVAT and Clarifai to determine which platform aligns best with your project's unique needs and objectives!‍

CVAT, your go-to computer vision annotation tool, now supports the YOLOv8 dataset format.‍‍Version 2.17.0 of CVAT is currently live. Among the many changes and bug fixes, CVAT also introduced support for YOLOv8 datasets for all open-source, SaaS, and Enterprise customers. Starting now, you export annotated data to be compatible with YOLOv8 models.What is the YOLOv8 Dataset Format?YOLOv8, developed by Ultralytics, is the latest version of the YOLO (You Only Look Once) object detection series of models. YOLOv8 is designed for:‍‍Classification: Classifying or organizing an entire image into a set of predefined classes;‍Classification‍‍Object Detection: Detecting, locating, and identifying the class of object in the image or visual data.‍Object Detection‍‍Pose Estimation: Identifying the location and orientation of a person or object within an image by recognizing specific keypoints (also referred to as interest points).‍Pose Estimation‍‍Oriented Bounding Boxes: A step further than object detection and introduce an extra angle to locate objects more accurately in an image.Oriented Bounding Boxes‍‍Instance Segmentation: Pixel-accurate segmentation of objects or people in an image or visual data.‍Instance Segmentation‍With the help of CVAT’s data labeling and annotation tools, YOLOv8 models can be trained to perform the functions as accurately as possible.‍What are the Benefits of Using the YOLOv8 Model for Computer Vision?Ultralytics has used the knowledge and experience garnered from previous iterations of their AI models to create the latest and most advanced YOLOv8. The benefits of using YOLOv8 include, but are certainly not limited to: ‍Highly accurate object detection;Versatility when it comes to detecting multiple objects, classifying and segmenting them, and detecting keypoints within images;Efficient, as the YOLOv8 has been optimized for efficient hardware usage and doesn’t require much computing power to run;Open-source means that the YOLOv8 is always evolving and being built by a passionate community of developers and all its features are easily accessible;And a lot more that would require a much longer list than this. ‍Which Industries Can Benefit from Training YOLOv8 Models?A trained YOLOv8 model can then be used for a variety of tasks. The functionality that YOLOv8 computer vision models can provide will benefit the following industries.‍Computer vision and AI models trained to detect various objects related to automotive are the way of the future in the automotive industry. Self-driving vehicles and traffic management are just a few of the ways that YOLOv8 models will benefit the automotive industry.Automotive use case‍The YOLOv8 object detection model can also offer significant functionality for security. Thanks to highly accurate object tracking and pose estimation, YOLOv8 models can detect intrusions and monitor for unregistered activities or prohibited objects within a given area.‍Security us case‍Using computer vision in retail and logistics will improve the efficiency at which stores maintain their supply and stock. They can also use YOLOv8's powerful object detection function to detect which shelves need to be restocked to improve customer experience.‍Naturally, the robotics industry greatly benefits from AI models with accurate computer vision, as it helps significantly when it comes to problem-solving. With each advancement in computer vision, problem-solving robots get more and more sophisticated as a result.‍Robotics use case‍In construction and architecture, computer vision can identify weak support, foundational problems, and other structural errors. This can help construction crews to detect potentially disastrous errors before any serious problems occur. On top of that, visual surveillance can be paired with AI to help construction managers detect safety hazards before they take place.‍Safety hazards use case‍There are ton of functions for many other industries when it comes to Ultalytics' YOLOv8 model. For now, these are among the most popular use cases for this tech.‍Understanding the Technical Details of the YOLOv8 Dataset FormatThe YOLOv8 dataset format uses a text file for each image, where each line corresponds to one object in the image.‍Each line includes five values for detection tasks: class_id, center_x, center_y, width, and height. These coordinates are normalized to the image size, ensuring consistency across varying image dimensions.‍For tasks like pose estimation, the YOLOv8 format also includes additional keypoint coordinates. Segmentation tasks require the use of polygons or masks, represented by a series of points that define the object boundary. Additionally, oriented bounding boxes can be rotated, which helps in annotating objects not aligned with the image axes.‍Dataset StructureThe YOLOv8 dataset typically includes the following components:‍<dataset directory>/ ├── data.yaml # configuration file ├── train.txt # list of train subset image paths │ ├── images/ │ ├── train/ # directory with images for train subset │ │ ├── image1.jpg │ │ ├── image2.jpg │ │ ├── image3.jpg │ │ └── ... ├── labels/ │ ├── train/ # directory with annotations for train subset │ │ ├── image1.txt │ │ ├── image2.txt │ │ ├── image3.txt │ │ └── ...‍‍Images Folder: This folder contains the images you are training the model on. These images are referenced by the corresponding annotation files.‍Annotations: Each image has a corresponding .txt file with the same name located in the annotations folder. The file structure for detection tasks looks like this:‍# <image_name>.txt: # content depends on format # YOLOv8 Detection: # label_id - id from names field of data.yaml # cx, cy - relative coordinates of the bbox center # rw, rh - relative size of the bbox # label_id cx cy rw rh 1 0.3 0.8 0.1 0.3 2 0.7 0.2 0.3 0.1 # YOLOv8 Oriented Bounding Boxes: # xn, yn - relative coordinates of the n-th point # label_id x1 y1 x2 y2 x3 y3 x4 y4 1 0.3 0.8 0.1 0.3 0.4 0.5 0.7 0.5 2 0.7 0.2 0.3 0.1 0.4 0.5 0.5 0.6 # YOLOv8 Segmentation: # xn, yn - relative coordinates of the n-th point # label_id x1 y1 x2 y2 x3 y3 ... 1 0.3 0.8 0.1 0.3 0.4 0.5 2 0.7 0.2 0.3 0.1 0.4 0.5 0.5 0.6 0.7 0.5 # YOLOv8 Pose: # cx, cy - relative coordinates of the bbox center # rw, rh - relative size of the bbox # xn, yn - relative coordinates of the n-th point # vn - visibility of n-th point. 2 - visible, 1 - partially visible, 0 - not visible # if second value in kpt_shape is 3: # label_id cx cy rw rh x1 y1 v1 x2 y2 v2 x3 y3 v3 ... 1 0.3 0.8 0.1 0.3 0.3 0.8 2 0.1 0.3 2 0.4 0.5 2 0.0 0.0 0 0.0 0.0 0 2 0.3 0.8 0.1 0.3 0.7 0.2 2 0.3 0.1 1 0.4 0.5 0 0.5 0.6 2 0.7 0.5 2 # if second value in kpt_shape is 2: # label_id cx cy rw rh x1 y1 x2 y2 x3 y3 ... 1 0.3 0.8 0.1 0.3 0.3 0.8 0.1 0.3 0.4 0.5 0.0 0.0 0.0 0.0 2 0.3 0.8 0.1 0.3 0.7 0.2 0.3 0.1 0.4 0.5 0.5 0.6 0.7 0.5 # Note, that if there are several skeletons with different number of points, # smaller skeletons are padded with points with coordinates 0.0 0.0 and visibility = 0‍‍data.yaml: This configuration file defines the dataset structure for training. It includes paths to the images and annotation files and lists all class names. An example of a data.yaml file looks like this:‍path: ./ # dataset root dir train: train.txt # train images (relative to 'path') # YOLOv8 Pose specific field # First number is the number of points in a skeleton. # If there are several skeletons with different number of points, it is the greatest number of points # Second number defines the format of point info in annotation txt files kpt_shape: [17, 3] # Classes names: 0: person 1: bicycle 2: car # ... ‍This lightweight and modular format allows for flexibility and scalability in your machine-learning pipeline. It also means that it can undertake a wide range of computer vision tasks, including object detection, pose estimation, segmentation, and oriented bounding boxes. For more technical details and in-depth usage, you can explore the full YOLOv8 format documentation.‍How to Use the YOLOv8 Dataset Format in CVATExporting YOLOv8 DatasetsAfter completing annotations in CVAT, exporting them in a YOLOv8 format is straightforward. Here’s how you can do it:‍‍Export Your Dataset: Once your annotations are ready, CVAT allows you to export them in YOLOv8 format, ensuring they are perfectly structured for use in YOLOv8 models. This includes annotations for detection, pose, oriented bounding boxes, and segmentation tasks. For detailed instructions on exporting your dataset, you can refer to the Exporting Annotations Guide.‍Train Your YOLOv8 Model: With your annotations exported, you can now directly integrate them into Ultralytics' YOLOv8 training pipeline. The dataset will be ready to train your model for detection, pose estimation, or segmentation tasks without the need for conversion. For further guidance on training your YOLOv8 models using Python, check out the Ultralytics YOLOv8 Python Usage Guide.‍Importing YOLOv8 DatasetsIn addition to exporting datasets, CVAT also supports importing datasets that are already in the YOLOv8 format. This feature allows you to bring external datasets and annotations into CVAT for further refinement or use in different projects. You can import both annotations and images for detection, oriented bounding boxes, segmentation, and pose estimations.‍To learn more about how to import YOLOv8 datasets and annotations, follow the detailed instructions in our Dataset Import Guide.‍F.A.Q.Which CVAT users have access to YOLOv8 support?All CVAT users, including open-source, SaaS, and Enterprise, have access to annotation tools for the YOLOv8 computer vision model.‍How good is YOLOv8 object detection?A YOLOv8 computer vision model trained with data annotated through CVAT can be very accurate in identifying various objects in visual data. YOLOv8 models can identify object borders down to the pixel, making them incredibly powerful when it comes to object detection.‍What functions do YOLOv8 models perform in computer vision?As listed above, YOLOv8's functions include classification, object detection, pose estimation, oriented bounding boxes, and instance segmentation.‍Start Using YOLOv8 in CVAT Today!The additional support for YOLOv8 dataset formats is a major milestone for CVAT. All open-source, SaaS customers and Enterprise clients are welcome to try out CVAT to help you train a YOLOv8 model for all manner of computer vision uses.‍For more information, visit our YOLOv8 format documentation. ‍Not a CVAT.ai user? Click through and sign up here.‍Do not want to miss updates and news? Have any questions? Join our community:‍Facebook‍DiscordLinkedInGitterGitHub

‍We are excited to announce a new feature for our enterprise clients: we've added Security Assertion Markup Language Single Sign-On (SAML SSO) support into the CVAT platform. This addition underscores our commitment to providing secure and flexible solutions tailored to the needs of large organizations.‍What is SAML SSO and Why Does It Matter?‍SAML is a well-established and trusted SSO standard widely adopted by many companies due to its robust security features. It allows users to authenticate across multiple applications using a single set of credentials, significantly simplifying the login process while enhancing security. ‍CVAT.ai SSO Proposal‍Better User Experience: SAML SSO simplifies the login process for users by allowing them to access multiple applications with a single set of credentials. This reduces the time spent on managing various logins and enhances overall productivity.‍‍Improved Security: SAML is known for its rigorous security standards, making it the preferred choice for many large organizations. ‍We understand that every enterprise has unique requirements. That is why CVAT.ai supports both SAML and OIDC (OpenID Connect), another popular SSO protocol. Enterprises can choose the protocol that best fits their infrastructure and security policies.‍Get Started Today‍With the new SAML SSO integration, your enterprise can enjoy a more secure, streamlined, and flexible authentication process. Whether you already use CVAT or consider it part of your enterprise's workflow, this new feature ensures you have the best tools to manage security and access effectively.‍‍Not a CVAT.ai user? Click through and sign up here‍Do not want to miss updates and news? Have any questions? Join our community:‍Facebook‍DiscordLinkedInGitterGitHub‍

In a significant update for computer vision enthusiasts and professionals, the powerful Segment Anything 2 model has been integrated into the Computer Vision Annotation Tool (CVAT.ai). This cutting-edge technology, developed by Meta, improves the image segmentation speed and accuracy and streamlines the annotation process. ‍So, what's new in the SAM 2?‍SAM 2 dramatically improves over earlier methods in image annotation without prior training on 17 different datasets. It also reduces the need for human involvement by about three times, making the process much more efficient.SAM 2 performs better than its predecessor, SAM, on a suite of 23 datasets without prior training and operates six times faster.Using SAM 2 feels like real-time processing, as it can handle about 44 frames per second.Using SAM 2 for video segmentation annotation in the loop is 8.4 times quicker than manual per-frame annotation with the original SAM.‍CVAT.ai Cloud: Segment Anything Model v2 Now Available for Image Segmentation‍CVAT has integrated "Segment Anything 2" into its SaaS version, improving the platform's capabilities for image segmentation.Integrating Meta AI's advanced machine learning models transforms CVAT into a more powerful tool for various users, ranging from academic researchers to industry professionals. This integration highlights a mutual commitment to advancing the field of computer vision. For now, in CVAT.ai, SAM 2 works only for images today, but video support will be added soon!We've Added Bounding Box Input‍The public version of CVAT.ai now supports optional bounding box input for Segment Anything 2. This feature allows users to define areas to annotate more quickly and accurately, enhancing the efficiency of model training processes for various applications.‍‍CVAT.ai Enterprise Edition: Added Segment Anything Model v2 CVAT has stepped up its game for Enterprise users by integrating Segment Anything 2 interactor support. This edition is tailored to meet the high demands of corporate environments where precision and scalability are critical. Enterprises can leverage this feature to handle complex segmentation tasks more effectively, ensuring higher accuracy and productivity in machine learning projects.‍‍‍Not a CVAT.ai user? Click through and sign up here‍Do not want to miss updates and news? Have any questions? Join our community:‍Facebook‍DiscordLinkedInGitterGitHub‍

In the dynamic world of computer vision, staying current with technology advancements is not just beneficial—it's critical. This is particularly true for organizations that use self-hosted solutions for the Computer Vision Annotation Tool (CVAT.ai). ‍Regular updates to such a tool are essential for several reasons: security, improved functionality, ensuring compatibility, and maintaining operational efficiency. This article explores why regularly updating your self-hosted CVAT.ai solution is crucial for maintaining a competitive edge and operational reliability.‍This article is divided into two parts: the first addresses 'why' regular updates are necessary, and the second explains 'how' to implement these updates effectively.‍Why is it Necessary to Update CVAT.ai Regularly?‍Improved Security: One of the most compelling reasons to regularly update your self-hosted CVAT is to enhance security. Although the latest version of CVAT.ai is secure, the threat landscape constantly evolves. New vulnerabilities are discovered daily, and the CVAT.ai Team releases patches to mitigate these risks. By staying updated, you safeguard your system against vulnerabilities that malicious actors could otherwise exploit. Regular updates are crucial for maintaining the integrity of your data and ensuring the privacy of the information processed by CVAT.‍Access to Latest Features: CVAT is continuously improved by a community of developers who add new functionalities and enhancements. These updates can include everything from improved annotation algorithms, support for new formats, and enhanced user interfaces to integration capabilities with other tools and platforms. ‍Compatibility and Integration: As your IT environment evolves, new versions of dependent software and hardware are introduced. Regularly updating CVAT ensures compatibility with other software tools and infrastructure changes. For example, updates may be needed for CVAT.ai to operate smoothly with newer versions of browsers, operating systems, or integrations with third-party APIs and services. Maintaining an updated system prevents disruptions caused by compatibility issues, which can be costly and time-consuming to resolve after the fact.‍Operational Reliability: Regular updates introduce new features and improvements, including optimizations that enhance CVAT's performance and stability. These optimizations can lead to faster load times, improved response times, and more efficient data processing, enhancing the system's overall reliability. For businesses relying heavily on computer vision technologies, operational reliability is non-negotiable.‍How to Update CVAT?‍Before we delve into the procedure, it’s important to note that the steps described here apply only to standard CVAT.ai standard public images.‍If you have created a custom image that we need to be aware of, we assume you are technically proficient and can handle the necessary updates tailored to your image.‍Step 1: Back Up Your Data‍Before making any changes to your CVAT installation, it's essential to back up your data. This ensures you can restore your system to its previous state if something goes wrong during the update.For more information, see CVAT.ai Backup Guide.‍Step 2: Stop the Old Version‍You need to stop the currently running version of the application to avoid potential conflicts.Use the Docker compose command to stop the running CVAT.ai container.‍Step 3: Pull Updates from Repository‍Once the system is halted, you can safely update the software by pulling the latest changes from the CVAT GitHub repository. You must download the entire source code, not just the Docker Compose configuration file.To see if the new version was released and to check the latest changes, use CVAT.ai Changelog.You must also check and update the additional components at this stage.‍Step 4: Handle Personal Customizations‍If you have custom configurations, such as a database managed outside Docker, you must ensure these are compatible with the new version. Review your configurations and make necessary adjustments to ensure they work with the new version of CVAT. In some cases, you need to build images locally; see this Guide for details.‍Step 5: Run the New Version‍After updating the software and adjusting your customizations, you can start the new version of CVAT.Start CVAT container: Use Docker commands to run the new CVAT containers; see the Upgrade Guide for details.‍Step 6: Manual Updates Where Needed‍Sometimes, you may need to update custom external components or manually handle migration scripts.‍And that's it!You now have new updates CVAT.ai with all necessary security improvements and features!‍Looks Too Complicated?‍Updating and managing CVAT can sometimes feel complex, mainly when you're focused on annotating and training models for your work or research. If you'd prefer to leave the sysadmin and DevOps tasks to someone else, CVAT offers installation support and help managing Enterprise self-hosted solutions. Explore our enterprise proposals and plans to find the right level of support for your needs. Alternatively, consider using our online version—it's always up-to-date and secure, so you can focus solely on annotating without hassle.‍‍Not a CVAT.ai user? Click through and sign up here‍Do not want to miss updates and news? Have any questions? Join our community:‍Facebook‍DiscordLinkedInGitterGitHub‍

Computer Vision Annotation Tool (CVAT) was started by Intel in 2017 and launched publicly on GitHub in the middle of 2018. In 2022, the platform became a core IP of an independent CVAT.ai Corporation, which we consider our founding year. With over seven years of experience, CVAT.ai has embarked on a mission to transform the field of data annotation and image labeling. We are proud of our remarkable journey and the milestones we have achieved.‍Our platform has become a cornerstone for data scientists, machine learning engineers, researchers, and students striving for excellence in artificial intelligence. ‍An anniversary is more than just a date; it symbolizes our growth, achievements, and the vibrant community we have fostered.‍The following post will outline our achievements from last year and revisit the company's history!Best Moments of The Year‍There were some ups and downs, but we are here to celebrate the results of our efforts. No hard feelings—lessons were learned and will not be forgotten. Today, we focus on the good parts and celebrate the fruits of our hard work:‍September 2023: We've reached 10,000 stars on GitHub, and we're still going strong—today, we have nearly 12,000 stars! We want to thank every one of you for your support. We also welcome stars as a gift, so if you'd like to cheer us up and help make our data labeling tool even more popular, please visit our GitHub and give us a star.November 2023: CVAT.ai plays a crucial role in the crowdsourcing annotation of Computer Vision datasets; therefore, in collaboration with Human Protocol, we have successfully launched the crowdsourcing data annotation project in several iterations:some textFebruary 2023: We’ve aired the first experiment in Crowdsourcing Annotation with CVAT and Human Protocol.November 2023: We have continued to push forward, and through a combined effort with our Human Protocol partners, we have made the integration more user-friendly for annotators and clients whose data needs to be annotated.November 2023: With Human Protocol, we warmly welcomed speakers at the Newconomics 2023 Conference.This initiative makes data annotation more affordable for AI companies needing annotated data. We are continuing to collaborate with Human Protocol to unlock and democratize AI.‍February 2024: CVAT.ai joined Google Summer of Code 2023, and we are still actively working on the project, which we consider a success!April 2024: We've introduced Annual Plans, helping our loyal and devoted users save up to 30% on data annotation tools. We have maintained transparent pricing, which significantly aids in budget planning!May 2024: The CVAT.ai Labeling Service is stellar and thriving. We have several hundred annotators who work across various fields, consistently meeting deadlines and maintaining high-quality standards. Our client base includes large enterprises in retail and other sectors, featuring customers from the top 100 enterprises worldwide. Their satisfaction with our services brings us great joy.May 2024: CVAT.ai was recognized as a top-choice data annotation tool at the Embedded Vision Summit 2024 (EVS 2024).‍Looking Forward‍As we celebrate this milestone, we are more committed than ever to pushing the boundaries of what CVAT.ai can achieve. ‍We extend our heartfelt thanks to our users, contributors, and partners who have been part of this incredible journey. Your support and collaboration have been instrumental in our success.‍Here's to more years of innovation, growth, and success with CVAT.ai!‍Stay connected with us, be curious, keep annotating!‍‍Not a CVAT.ai user? Click through and sign up here‍Do not want to miss updates and news? Have any questions? Join our community:‍Facebook‍DiscordLinkedInGitterGitHub‍

In the first two parts of this article series, we discovered the cost of annotating images and videos yourself or with an in-house team. This part investigates the finances and resources you need to outsource the data annotation to the labeling service.‍‍However, let's first revisit our practical scenario: Imagine a leading robotics scientist developing a smart home assistant to distinguish between dirt and valuable objects in a home environment. Life's chaos often includes scattered toys, misplaced glasses, pet fur, and god knows what else. The proposed robot aims to clean efficiently and assist in locating misplaced items. Such functionality could benefit the elderly by helping them keep track of their possessions, for example. So, there is a niche for such products.‍As the project's lead, you are instrumental in guiding a compact research team that has gathered a dataset of 100,000 images, each depicting different room settings with items scattered across the floor. According to publicly available data, this dataset size is typical for robotics projects, ranging from thousands to millions of images. ‍With an average of 23 objects per image, the task involves annotating approximately 2.3 million objects. This series of articles explores various strategies for managing this large-scale annotation challenge, including do-it-yourself approaches, forming an in-house team, outsourcing, and utilizing crowdsourcing techniques.‍‍Welcome to the third part of our series, which explores the costs of outsourcing data annotation to cover the scientist's labeling needs.‍Case 1: You handle the task yourself or with minimal colleague help.Case 2: You hire annotators and annotate with your team.Case 3: You outsource the task to professionals.Case 4: Crowdsourcing.‍Case 3: You Outsource the Task to Professionals‍Let's start with a brief introduction and a statement that all data labeling companies operate similarly, with some variations that can significantly impact the quality of their labeling services. The devil is in the details. And CVAT.ai is not an exception. If the named scientist comes to us before jumping into the work, we will request some information from him and his team.‍Time and Stages‍To be precise, this is how the whole workflow will look, separated by stage and with time estimations. It might differ for different companies, so we are talking from our experience.‍We are not shy to state that our experience is vast and one of the best in the market, as we not only provide data labeling services but also own our data annotation platform. For clients, this means that we are flexible and can continuously adjust CVAT to make the annotation and validation process more efficient. Our clients can use the same platform internally and easily extend annotations. They also just log in to see how the data annotation process is going for them. ‍Without paying for anything., just try to annotate something, like millions of data scientists worldwide do. ‍But enough about us, let's see what annotation stages are there. ‍Stage 1: Annotation Proof of Concept (PoC)‍We will sign a Non-Disclosure agreement with the client to protect the data if necessary. We will request actual data samples (50-100 images or 1-2 videos) to start investigating it and see how it should be annotated.We will need the client's approved annotation specifications. At this stage, we will work together closely and ask questions to clarify corner cases and quality requirements. Following the efforts above, we will create a PoC and offer the precise project costs and durations.We will then send the client our proposal.‍We commit to initiating a PoC within one day of data reception and will provide detailed estimates and calculations within 3-5 days, depending on the project scope. Our initial project budget assessment is conducted with a high degree of accuracy. According to our experience, the final project cost typically deviates from the initial estimate by at most 10%.‍Stage 2: Documentation & Preparation‍Based on the conducted Proof of Concept (PoC), we will propose the most effective method for data annotation, refine and supplement the initial specification, and agree on the quality requirements and project annotation timelines.We will develop all the necessary documentation and sample agreements, including comprehensive information about our collaboration's terms and payment conditions. The client should only review the documentation and suggest any necessary revisions.Training the data annotation team is also entirely our responsibility. We will assign a dedicated manager who will be the direct and constant point of contact for resolving all operational issues and gathering all the necessary information about the project to build the training process for the annotation team.‍Document processing on our end will be completed within a week, barring any delays from the client. We immediately begin training and data annotation for expedited projects, bypassing bureaucratic delays.‍Stage 3: Annotation‍At this stage, we perform data annotation strictly following the instructions. However, we understand that requirements may change during the process, so we are always ready to be flexible and accommodate minor changes to the initial documentation.Since we understand that developing an AI model is a multi-step process, for large projects, we advocate delivering annotated data in batches without waiting for the entire dataset to be annotated. This approach allows our clients to conduct relevant experiments and adjust the process. The dedicated manager, responsible for the interim progress, will oversee the project from start to finish.We welcome regular feedback from the client and are ready to make additional revisions to the documentation as the project progresses to ensure the expected result.Typically, the most critical stage is annotating the first batch of data, during which all processes are fine-tuned, and the client's final requirements are understood. After successfully delivering the first batch of data, our team operates like a well-oiled machine, delivering high-quality results within the expected timelines.‍Most projects reach completion within one month.‍Stage 4: Validation‍We guarantee high-quality results to our clients because, before committing to specific obligations, we conduct experiments that help us understand the results we can deliver and how to improve them.‍We take full responsibility for a quality check; we can offer the following services for better results:‍Conduct manual and Сross Quality Assurance (QA), automate QA for Ground Truth (GT) annotation covering 3-10% of the dataset.Execute any final amendments at no additional cost and deliver a conclusive quality report.Compute and report quality metrics like Accuracy, Precision, Recall, Dice coefficient, and others, and provide a confusion matrix.Final validation and the conclusive report from our end will be completed within one week.‍Stage 5: Acceptance‍This is the final and best stage, where the client gets the final results.All that is left is to process payments and provide feedback regarding our labeling service.‍Following our previous article, in case there are no client delays and unexpected events, the whole process for the described project will take approximately 50 work days, 10 weeks, or 2.3 months. Of course, it depends on each case's requirements and circumstances.‍By entrusting us with your project, you commission a high-quality service with a pre-defined and documented guaranteed outcome. The client's role is limited to observing the process, accepting recommended changes from our side, reviewing the delivered data, and providing feedback on the results of the validated work. We take on all internal processes and guarantee the project's quality and timely delivery.‍Data Labeling Price‍Well, that’s a tricky question because the price heavily depends on the amount of data and the specific needs: the quality, the type of annotation, deadlines, and many more.‍Let’s use data publicly available online to estimate the cost of annotating 2,300,000 objects or 100,000 images. However, here's the issue—labeling service providers often lack transparency, and there aren't many published prices. Thus, we can only rely on fragments of information from sources like KILI Technology or Mindkosh to make our estimates. The number will usually be above $300,000 because semantic segmentation, used for this task, is one of the most expensive annotation types for now.‍But how much will it cost if the client comes to CVAT.ai? We used a flexible approach when we needed this amount of data to be annotated. Our pricing is built on the following assumptions:‍Estimation and Payment Models‍Per Object: This primary model charges for each data unit annotated—whether a frame, object, or attribute within an image or video. It suits projects with clearly defined unit sizes and quantities.Per Image/Video: Charges apply per image or video file processed, ideal for projects with consistent complexity or time demands per file.Per Hour: Costs are calculated based on the time annotators spend on the project, offering flexibility for projects with varying complexities or scope changes.‍Expected Project Budget Ranges‍$5K - $9.9K for Annotation Only, Manual, and Cross-Validation: This range is typical for projects focused on manual annotation, including thorough cross-validation for accuracy.Above $10K for Comprehensive Services: For budgets exceeding $10K, services extend beyond basic annotation to include AI engineer involvement, automated quality assurance, and potential custom AI solution development.The final cost of annotating 2,300,000 objects in CVAT.ai depends on the chosen approach. Using the "Per Object" method, the initial pricing begins at a set rate per unit. Due to the large volume, discounts ranging from 5% to 30% will be applied, reflecting our commitment to building long-term partnerships. By utilizing the highest discount tier, the total cost for annotating all objects will be approximately $225,400. This is an approximation, and the final price may vary based on the client's specific needs. Regardless of the exact cost, the results will be of the highest quality and delivered promptly.‍In general, you should expect the outsourcing price to be more than 1.5 times the cost of a potential in-house data annotation team. Hiring your data annotation team is one of the ways to achieve a better price while maintaining high quality. Read You hire annotators and annotate with your team for tips.‍Conclusion‍In summary, outsourcing your data annotation tasks to a professional service offers significant benefits in terms of time efficiency, quality assurance, and overall project management. While costs can vary based on project specifics, CVAT.ai provides a flexible pricing model that caters to different needs, ensuring high-quality results within a reasonable budget. With discounts available for larger volumes, we can offer competitive pricing without compromising quality.‍Next steps?‍Ready to label data with CVAT.ai? Email us: labeling@cvat.ai!Ensure you have all the necessary information—download our detailed takeaway now!‍‍‍Not a CVAT.ai user? Click through and sign up here‍Do not want to miss updates and news? Have any questions? Join our community:‍Facebook‍DiscordLinkedInGitterGitHub‍

In the first part of this series of articles, we emphasized the need for precise annotation of images and videos, essential for developing AI products capable of performing accurate analyses, making predictions, and delivering reliable outcomes. We focused on how time-consuming and money-consuming solo annotation might be.‍In this article, we will explore the costs and resources required to maintain an in-house team of data annotators.‍But before we jump into the topic, here's a reminder of our use case:‍A lead robotics scientist is creating a smart home assistant robot to differentiate between dirt and valuable items in a household setting. Life's chaos often includes scattered toys, misplaced glasses, pet fur, and god knows what else. The proposed robot would clean efficiently and help locate lost items, adding a layer of functionality beyond standard home cleaning devices. This can help elderly people keep track of their belongings, for example.‍As the lead scientist, you play a crucial role in this project. Along with your small research team, you've compiled a dataset of 100,000 images showing various room settings with items scattered on the floor. According to publicly available data, this dataset size is typical for robotics projects, which can range from thousands to millions of images.‍Each image features an average of 23 objects, so the task involves annotating approximately 2.3 million objects. This series presents various strategies to tackle this significant annotation task, including DIY methods, building an in-house team, outsourcing, and crowdsourcing.‍‍Welcome to part two of our series on the costs of data annotation. This article describes the cost of hiring annotators and building the annotation team yourself.‍Case 1: You handle the task yourself or with minimal help from colleagues.Case 2: You hire annotators and annotate with your team.Case 3: You outsource the task to professionals.Case 4: Crowdsourcing.‍Case 2: You hire annotators to annotate with your team‍Now, as with anything else on this planet, there are pros and cons to having an annotation team. Let's start with the advantages and address questions about the time required for annotation and the cost-effective impact of this approach.‍Here, we will calculate only the monthly expenses and costs. The minimum team to annotate 2.3 million objects consists of 35 annotators, supported by management personnel involved in onboarding, offboarding, and upskilling annotators.‍For these 35 annotators, one manager and 3-4 senior annotators are necessary to guide the team.‍Contracts and Team SizeData annotation teams vary in size from small (up to five members) to large groups, with larger teams requiring more coordination and management.‍Recruiting is straightforward for small teams, but complex for larger groups. Annotators may be full-time employees with fixed salaries or contractors. Contractors, however, pose challenges in retention and engagement due to their involvement in multiple projects and expectation for workload-aligned compensation.‍When working with contractors, as we do, extra effort is necessary to ensure availability. For instance, if you need 35 annotators, consider hiring between 60 to 70 to account for potential unavailability.Time and Costs‍From our experience the hiring process will take as much time as:‍Time to find a data annotation manager: 1 month or moreTime to find one annotator: Up to 1 monthTime to onboard one annotator: Up to 1 month‍You can conduct job interviews and onboarding concurrently. If you're fortunate, you might be able to hire between 5 to 10 annotators per month. But to hire and train a big data annotation team you need to have at least 3-4 months.‍Expenses wise it will be:Manager salary (per month): Up to $6000 (data from Indeed, June 2024)Annotators Salary (per hour): It depends on whether you can afford to hire abroad. If yes, starting from $1/h and up to $40 if you hire in the US or high level of qualification is required. Where to look for them? On Upwork, Indeed, LinkedIn—you name it. Again, the job posting price ranges from $0 to $500, in rare cases, with the help of the recruitment agency.Yes, if your service is as popular as platforms like CVAT.ai in the data annotation area, you can significantly reduce time and costs. Annotators will eagerly respond to your posted vacancies as soon as they are advertised.Set Up Time‍Next step is to prepare data: the dataset is the foundation of any robotics project.‍For this project, the scientist must sift through a vast collection of video footage to select relevant frames and then craft a comprehensive data annotation specification. In our case, this specification is planned to cover 40 different classes, each to be annotated with polygons individually. ‍On average, the complete guideline is 30-50 pages. It will include detailed instructions for annotating each class, examples of correct and incorrect annotations, and edge cases. Drafting this detailed specification is time-consuming; it might take several weeks. The data annotation specification will be updated during the project because it isn’t possible to describe all corner cases from the beginning.‍The time it takes to annotate each object with polygons will later be calculated, considering factors such as the object's complexity and size, the image's clarity, and the annotator's skill level.‍Simple Object (e.g., a rectangular object): 5-10 secondsModerately Complex Object (e.g., a car): 30-60 secondsHighly Complex Object (e.g., a human with detailed limb annotations): 1-3 minutes or more‍Operational Costs‍In addition to onboarding and training costs, the expenses for data annotation projects also include licenses and instance costs per annotator. Each annotator may require a license for the annotation software used, which can vary significantly in price depending on the complexity and capabilities of the software. ‍In the case of CVAT it will cost you $33 per seat or you can use the free open-source tools. ‍Remember that even free tools require time and resources to set up and support; time is money. So, while we say "free," it means that you can download and install the open-source tool, but the rest depends on your time, expertise, and effort (and how much of your paid time will be spent on this).‍Operational costs includes costs for accounting, contracts management and cannot be approximated, as they are company specific.‍Final calculations‍To calculate the total time required for 35 professional annotators to annotate 2,300,000 objects, where each object takes approximately 40 seconds in average to annotate, you can follow these steps:‍Calculate the Total Time for All Objects:‍Total time = 2,300,000 objects × 40 seconds per object = 92,000,000 seconds or 25,555.56 hours‍Divide by the Number of Annotators to Find Time per Annotator:‍Time per annotator = 25,555.56 hours / 35 annotators = 730.16 hours‍So, if all annotators work simultaneously and efficiently, each annotator will need about 18.25 work weeks, which is approximately 4.2 months, to complete the annotation of all 2,300,000 objects.‍To calculate the costs for the scenario described, let's break it down into its components and sum them up for the 4.2 months required for the project. We'll assume each annotator earns $550 per month and that there is a varying cost for licenses, from free to $33 per month. Additionally, management and validation cost is $6000 + 20% per month from the total cost of annotators.‍Total Salary Costs for Annotators (4.2 months): ‍Total annotator costs = $2,310 per annotator × 35 annotators = $80,850Management and validation Fees (for 4.2 Months):‍Total cost for a data annotation manager = $25,200Management and validation Fees = $80,850 * 20% = $16,170‍Conclusion: To annotate 100 000 images, that is 2,300,000 objects it will take 4.2 months and $122,220.‍To this number you need to add costs of the software licenses.‍Hidden and One Time Costs‍When calculating how much an annotation team costs it might be a good thing to take in account a one-time cost like hiring time and efforts. ‍As we’ve mentioned before, assembling a data annotation team starts with recruiting a crucial step that sets the tone for the team's development and effectiveness. Organizations typically choose between outsourcing recruitment or handling it internally.‍Time and Cost Estimates:‍Outsourcing RecruitmentTime: Recruitment agencies can expedite the process, typically taking 2 to 6 weeks to secure a position.Cost: Agencies charge a fee based on the position's annual salary, usually 15% to 30%.Internal RecruitmentTime: This method can take 4 to 8 weeks, depending on the efficiency of HR processes and candidate availability.Cost: Costs include job posting fees ($0 to $500) and internal HR labor (approximately $55,000 annually or $26 per hour). ‍The numbers provided are approximate and based on data from Indeed and LinkedIn; actual costs may vary and should be aligned with the company's internal processes. For example, at CVAT.ai, we have automated our hiring process, enabling us to recruit the best annotators on the market at competitive prices. We use Remote.com for onboarding candidates and are quite satisfied with this HR platform. Our annotators come from various countries, including Kenya, India, Nigeria, Ghana, Nepal, and Indonesia.‍Considerations for Hiring Relatives‍Small teams might consider hiring relatives for data annotation tasks. While this can add value in terms of trust and loyalty, it often leads to challenges such as the absence of professionalism and cost. Performance might not meet professional standards if the hiring criteria are not aligned with the job's technical demands.‍Management Overhead‍Post-recruitment, managing a data annotation team involves handling administrative tasks essential for maintaining AI development standards:‍Paperwork and Compliance: Managing contracts and compliance with labor laws.Financial Management: Overseeing accounts and payment systems.Work Environment Management: Providing training, managing workloads, and fostering a supportive work atmosphere.‍Additional Considerations‍Technology and Tools: Investments in data management and annotation tools can enhance efficiency.Team Dynamics: The interaction between team members and management style significantly impacts productivity.Market Conditions: Economic factors and labor availability can influence recruitment and operational costs.‍These elements are often seen as "hidden costs" and vary significantly by organization, affecting the overall expenses. They should be included in the final budget considerations due to their potential to impact the costs significantly.‍Conclusion‍What will be the total duration and costs of the entire project? Here, we are discussing the baseline minimal price, excluding hiring and hidden costs:‍Total Duration: 18.25 work weeks, which is approximately 4.2 months. Cost Range: The costs vary. They start from around $122,000 and may go up indefinetly, depending on team capacity and other factors, like where are you placed, do you hire locally or worldwide and so on. ‍What else should you take into account when reading this article?‍The calculation for the hiring process assumes linear and consistent recruitment and onboarding, which might not reflect real-world variations. Realistic scenarios may need buffer times for unexpected delays and additional costs for unplanned issues.The provided time and costs assume maximum efficiency. They may not account for variables such as sick leave, training efficacy, and turnover rates, which could significantly impact both time and cost estimates.Empirical data from similar past projects could further refine estimating the time taken to annotate objects and onboarding costs.‍Overall, the presented figures are reasonable but should be treated as approximations with potential for variation based on real-world execution.And that’s all for today. See you in the next article, where we will discuss how much it costs to outsource the data annotation to professionals.‍Not a CVAT.ai user? Click through and sign up here‍Do not want to miss updates and news? Have any questions? Join our community:‍Facebook‍DiscordLinkedInGitterGitHub‍

As time moves forward, CVAT.ai continues to grow, introducing new features, improving processes, and frequently updating its interface to make it more user-friendly. Our latest updates focus on making it easier for teams to work together on annotation projects.‍In our new video tutorial, we show how to quickly invite team members to collaborate on projects. The tutorial highlights recent updates to the user interface that simplify the process of adding new users and assigning roles. Keep an eye out for our next video, where we'll explain how to create projects, load data, and assign tasks to annotators. Join us to take advantage of the new, streamlined features at CVAT.ai.‍‍‍Not a CVAT.ai user? Click through and sign up here‍Do not want to miss updates and news? Have any questions? Join our community:‍Facebook‍DiscordLinkedInGitterGitHub‍