Try for Free
COMPANY
BlogPressAboutCareers
PRICING
CVAT.ai cloudOn-prem deployment
RESOURCES
InstallQuickstartDocumentation
COMMUNITY
DiscordGitterGitHubContribute
CONTACT US
contact@cvat.ai
How Much Does it Cost to Outsource Annotation to a Data Labeling Service?

In the first two parts of this article series, we discovered the cost of annotating images and videos yourself or with an in-house team.  This part investigates the finances and resources you need to outsource the data annotation to the labeling service.

However, let's first revisit our practical scenario: Imagine a leading robotics scientist developing a smart home assistant to distinguish between dirt and valuable objects in a home environment. Life's chaos often includes scattered toys, misplaced glasses, pet fur, and god knows what else. The proposed robot aims to clean efficiently and assist in locating misplaced items. Such functionality could benefit the elderly by helping them keep track of their possessions, for example.  So, there is a niche for such products.

As the project's lead, you are instrumental in guiding a compact research team that has gathered a dataset of 100,000 images, each depicting different room settings with items scattered across the floor. According to publicly available data, this dataset size is typical for robotics projects, ranging from thousands to millions of images.

With an average of 23 objects per image, the task involves annotating approximately 2.3 million objects. This series of articles explores various strategies for managing this large-scale annotation challenge, including do-it-yourself approaches, forming an in-house team, outsourcing, and utilizing crowdsourcing techniques.

Welcome to the third part of our series, which explores the costs of outsourcing data annotation to cover the scientist's labeling needs.

Case 3: You Outsource the Task to Professionals

Let's start with a brief introduction and a statement that all data labeling companies operate similarly, with some variations that can significantly impact the quality of their labeling services. The devil is in the details. And CVAT.ai is not an exception. If the named scientist comes to us before jumping into the work, we will request some information from him and his team.

Time and Stages

To be precise, this is how the whole workflow will look, separated by stage and with time estimations. It might differ for different companies, so we are talking from our experience.


We are not shy to state that our experience is vast and one of the best in the market, as we not only provide data labeling services but also own our data annotation platform. For clients, this means that we are flexible and can continuously adjust CVAT to make the annotation and validation process more efficient. Our clients can use the same platform internally and easily extend annotations. They also just log in to see how the data annotation process is going for them.


Without paying for anything., just try to annotate something, like millions of data scientists worldwide do. 


But enough about us, let's see what annotation stages are there.  

Stage 1: Annotation Proof of Concept (PoC)

  • We will sign a Non-Disclosure agreement with the client to protect the data if necessary.
  • We will request actual data samples (50-100 images or 1-2 videos) to start investigating it and see how it should be annotated.
  • We will need the client's approved annotation specifications. At this stage, we will work together closely and ask questions to clarify corner cases and quality requirements.
  • Following the efforts above, we will create a PoC and offer the precise project costs and durations.
  • We will then send the client our proposal.

We commit to initiating a PoC within one day of data reception and will provide detailed estimates and calculations within 3-5 days, depending on the project scope. Our initial project budget assessment is conducted with a high degree of accuracy. According to our experience, the final project cost typically deviates from the initial estimate by at most 10%.

Stage 2: Documentation & Preparation

  • Based on the conducted Proof of Concept (PoC), we will propose the most effective method for data annotation, refine and supplement the initial specification, and agree on the quality requirements and project annotation timelines.
  • We will develop all the necessary documentation and sample agreements, including comprehensive information about our collaboration's terms and payment conditions. The client should only review the documentation and suggest any necessary revisions.
  • Training the data annotation team is also entirely our responsibility. We will assign a dedicated manager who will be the direct and constant point of contact for resolving all operational issues and gathering all the necessary information about the project to build the training process for the annotation team.

Document processing on our end will be completed within a week, barring any delays from the client. We immediately begin training and data annotation for expedited projects, bypassing bureaucratic delays.

Stage 3: Annotation

  • At this stage, we perform data annotation strictly following the instructions. However, we understand that requirements may change during the process, so we are always ready to be flexible and accommodate minor changes to the initial documentation.
  • Since we understand that developing an AI model is a multi-step process, for large projects, we advocate delivering annotated data in batches without waiting for the entire dataset to be annotated. This approach allows our clients to conduct relevant experiments and adjust the process. The dedicated manager, responsible for the interim progress, will oversee the project from start to finish.
  • We welcome regular feedback from the client and are ready to make additional revisions to the documentation as the project progresses to ensure the expected result.
  • Typically, the most critical stage is annotating the first batch of data, during which all processes are fine-tuned, and the client's final requirements are understood. After successfully delivering the first batch of data, our team operates like a well-oiled machine, delivering high-quality results within the expected timelines.

Most projects reach completion within one month.

Stage 4: Validation

We guarantee high-quality results to our clients because, before committing to specific obligations, we conduct experiments that help us understand the results we can deliver and how to improve them.

We take full responsibility for a quality check; we can offer the following services for better results:

  • Conduct manual and Сross Quality Assurance (QA), automate QA for Ground Truth (GT) annotation covering 3-10% of the dataset.
  • Execute any final amendments at no additional cost and deliver a conclusive quality report.
  • Compute and report quality metrics like Accuracy, Precision, Recall, Dice coefficient, and others, and provide a confusion matrix.

Final validation and the conclusive report from our end will be completed within one week.

Stage 5: Acceptance

  • This is the final and best stage, where the client gets the final results.
  • All that is left is to process payments and provide feedback regarding our labeling service.

Following our previous article, in case there are no client delays and unexpected events, the whole process for the described project will take approximately 50 work days, 10 weeks, or 2.3 months. Of course, it depends on each case's requirements and circumstances.

By entrusting us with your project, you commission a high-quality service with a pre-defined and documented guaranteed outcome. The client's role is limited to observing the process, accepting recommended changes from our side, reviewing the delivered data, and providing feedback on the results of the validated work. We take on all internal processes and guarantee the project's quality and timely delivery.

Data Labeling Price

Well, that’s a tricky question because the price heavily depends on the amount of data and the specific needs: the quality, the type of annotation, deadlines, and many more.

Let’s use data publicly available online to estimate the cost of annotating 2,300,000 objects or 100,000 images. However, here's the issue—labeling service providers often lack transparency, and there aren't many published prices. Thus, we can only rely on fragments of information from sources like KILI Technology or Mindkosh to make our estimates. The number will usually be above $300,000 because semantic segmentation, used for this task, is one of the most expensive annotation types for now.

But how much will it cost if the client comes to CVAT.ai? We used a flexible approach when we needed this amount of data to be annotated. Our pricing is built on the following assumptions:

Estimation and Payment Models

  • Per Object: This primary model charges for each data unit annotated—whether a frame, object, or attribute within an image or video. It suits projects with clearly defined unit sizes and quantities.
  • Per Image/Video: Charges apply per image or video file processed, ideal for projects with consistent complexity or time demands per file.
  • Per Hour: Costs are calculated based on the time annotators spend on the project, offering flexibility for projects with varying complexities or scope changes.

Expected Project Budget Ranges

  • $5K - $9.9K for Annotation Only, Manual, and Cross-Validation: This range is typical for projects focused on manual annotation, including thorough cross-validation for accuracy.
  • Above $10K for Comprehensive Services: For budgets exceeding $10K, services extend beyond basic annotation to include AI engineer involvement, automated quality assurance, and potential custom AI solution development

.

The final cost of annotating 2,300,000 objects in CVAT.ai depends on the chosen approach. Using the "Per Object" method, the initial pricing begins at a set rate per unit. Due to the large volume, discounts ranging from 5% to 30% will be applied, reflecting our commitment to building long-term partnerships. By utilizing the highest discount tier, the total cost for annotating all objects will be approximately $225,400.  This is an approximation, and the final price may vary based on the client's specific needs. Regardless of the exact cost, the results will be of the highest quality and delivered promptly.

In general, you should expect the outsourcing price to be more than 1.5 times the cost of a potential in-house data annotation team. Hiring your data annotation team is one of the ways to achieve a better price while maintaining high quality. Read You hire annotators and annotate with your team for tips.

Conclusion

In summary, outsourcing your data annotation tasks to a professional service offers significant benefits in terms of time efficiency, quality assurance, and overall project management.   While costs can vary based on project specifics, CVAT.ai provides a flexible pricing model that caters to different needs, ensuring high-quality results within a reasonable budget. With discounts available for larger volumes, we can offer competitive pricing without compromising quality.

Next steps?

Ready to label data with CVAT.ai?  Email us: labeling@cvat.ai!

Ensure you have all the necessary information—download our detailed takeaway now!


Not a CVAT.ai user?  Click through and sign up here

Do not want to miss updates and news? Have any questions? Join our community:

Facebook

Discord

LinkedIn

Gitter

GitHub

July 25, 2024
CVAT Team
Go Back