Try for Free
COMPANY
BlogPressAboutCareers
PRICING
CVAT.ai cloudOn-prem deployment
RESOURCES
InstallQuickstartDocumentation
COMMUNITY
DiscordGitterGitHubContribute
CONTACT US
contact@cvat.ai
Mastering Image Annotation Crowdsourcing for Computer Vision with CVAT.ai and HUMAN Protocol

Introduction

Implementing annotation crowdsourcing right can be a challenging task, especially for Computer Vision problems. While crowdsourcing offers huge benefits, such as unlimited scaling and low overhead, it also has fundamental challenges, such as low resulting annotation quality, difficult validation, and overwhelming assignment management. Now, we’d like to introduce our solution for annotation at scale, based on CVAT.ai and HUMAN Protocol integration. Read on to learn how you can use it for your datasets and for annotation.

Overview

The solution is an online platform based on HUMAN Protocol services, which uses CVAT for data annotation and Web3 technologies for payments in crypto currencies.

On the platform, we bring together 2 key user roles: 

  • Requesters - individuals who need their data annotated. AI model delevelopers, researchers, and AI challenge organizers all fall into this category.
  • Annotators, workers - people who want to earn money by annotating data. This category includes people looking for a side hustle, freelancers, and professional data annotators.

For Requesters, the platform provides automatic data annotation assignment preparation and management, automatic annotation validation and merging, and payments based on the annotation quality. Together these features create a place where you can just ask for an annotated dataset - describe the task, provide the data, quality requirements, and reward details - and get the data annotated with the specified quality after some time. No more manual control and management is needed. Keep in mind that currently the platform supports only several Computer Vision task types, but it can be extended in future.

For Annotators, the platform is a place where they can start earning money in just several clicks. All they need to do is register, select a task, and ask for an assignment. Once they get an assignment, they can learn annotation requirements by reading the attached annotation guide, and start annotating - using one of the most popular open-source annotation tools in the Computer Vision community. Each assignment is small enough to be completed in a matter of minutes, allowing flexible participation for the workers. 

Bounty payments are based on cryptocurrency; so annotators are required to have a crypto wallet to get the money. For requesters, however, there is also an option to pay with a bank card. Funds for payments are reserved at the time of task creation and are disbursed automatically after the entire dataset has been annotated and validated.

Quick start

If you want to get your dataset annotated

Before initiating a new annotation task, please be aware of the following platform requirements:

  • A crypto wallet and a configured browser extension for wallet access
  • An Amazon S3 data bucket configured for  public access
  • A dataset with images in common formats (.jpg, .png, …), with at least 2 images
  • Support for only two task types: bounding box and single point annotations

Examples:

  • Only the MS COCO Object Detection / Keypoint Detection dataset format is supported, which is one of the most widely used formats in the industry. Note that points are encoded as 1-keypoint skeletons in the COCO Keypoint Detection format
  • Annotations for ground truth (validation) images should be  in the COCO .json file format

Please keep in mind that simpler tasks tend to be  easier and quicker to annotate. If your annotations require complex work, it can be a good idea to split the whole task into simpler subtasks, and annotate them separately for better quality and efficiency.

Now, let’s go over  the steps required to start an annotation task.

      1) Create a crypto wallet using one of the corresponding applications (e.g. MetaMask)

Example for MetaMask in Chrome:

            1. Install the browser extension.
            2. Open the extension page in the browser, create a password.
           3. Select the Polygon Mainnet network or add it from the list.
            4. Use an existing ETH-based wallet or click Add an account in MetaMask.
            5. Rename the account to “Job Requester” or any other name you like.
           6. Now, you'll have a wallet (ETH-based) in the network. It should show 0 MATIC on the balance.
           7. Transfer some MATIC from another ETH-based wallet or buy (e.g. $5). Make sure to buy in the Polygon Mainnet network.
            8. Click Import Tokens. Add the HMT token:
                        Id: 0xc748B2A084F8eFc47E086ccdDD9b7e67aEb571BF
                        Name: HMT

           9. Now, it should also show 0 HMT on the balance.
           10. Convert some MATIC into HMT

Now that you can perform payments in cryptocurrencies, it’s time to prepare your data and create an annotation task. 

     2) An AWS S3 bucket with public access is required. Сreate an AWS S3 bucket and upload your images as separate files into the bucket. Make sure to note the link to the directory containing the files within the bucket.
          You can obtain this URL by clicking Copy URL (the directory URL is needed).

Example:

    3) Select a small validation subset from the original images (e.g., 3%, 30 images from 1000), annotate it, and upload annotations in the COCO format to the bucket.
        You will only need a .json file. You can prepare such a dataset using CVAT  or other tools.
       Remember to note the URL to this file as well.

Example:

Using 0.1-5% of the images is recommended, depending on the dataset size. It is recommended to select just random images from the whole dataset. These images will appear randomly during the annotation process to check the annotation quality. The annotations produced by workers for these images are not included in the resulting annotated dataset; instead, the ground truth annotations are used.

Please note that only 1-keypoint skeletons are supported for the COCO Keypoints format.

   4) Proceed to the platform and register a new account, if you don’t have one.

   5) Click Create a Job.


   6) Select the payment method you prefer. You can pay via a crypto wallet or with a bank card.

   7) Once the payment method is selected, you’ll see the job configuration page. Please select the CVAT job type and the Polygon Mainnet network.

Let’s consider the fields one by one:

  • Type of job: Bounding Boxes / Points
  • Description: brief task description in 1-2 sentences
  • Labels: the list of label names (classes, categories) to be used during annotation
  • Data URL: a link to the bucket directory with dataset images
  • Ground truth URL: a link to the bucket file in .json format with GT annotations in COCO format
  • User guide URL: a link to the full task description document with public access. Such documents describe annotation rules for the workers. You can share this document via the same S3 bucket, Google Docs or other similar services
  • Accuracy target: the required accuracy target for annotations. The typical value range is 50-95% (keep in mind that normal human image classification accuracy is ~94%, and values from the 75-93 range are the most applicable. The metric used is IoU).

    

      8) At the next step, you’ll need to enter the bounty details for workers. The resulting value is the price for the whole dataset.
         The bounties for the annotators will be calculated automatically based on the number of images annotated.

Here you can choose to pay in crypto currency or with a bank card. The money will be reserved until the whole dataset is annotated.

     9) Once you’re ready, click Pay now.

  
    10) If everything is set up correctly, after the next several minutes, the job will appear in the job list. 

In the list, you can check the current annotation status and details of the jobs you have created. Now, the annotators will look for assignments and begin their work. The process is fully autonomous for you, so you will only need to wait for the process to finish. It can take some time, depending on the current market prices, available worker pool, and other conditions. The workers will be able to join the task and get an assignment via this platform link.

If, at some point, you decide to cancel a job, here you can find a button for this. The money you reserved for the work will get back to you.

   11) Once the data is annotated, the final annotations will be validated and merged automatically, and the result will be available by the URL you receive in the job details section.

If you want to earn money with image annotation and video annotation

1. Create a crypto wallet (check the explanation in the requester part of this guide).
2. Go to the annotator platform and register. You’ll need to pass a mandatory KYC procedure to finish the registration process, this is required by the applicable laws.
3. Currently, we require users to explicitly request participation in CVAT labeling. Please email us at app@humanprotocol.org to express your interest.
4. On the platform, open the list of available CVAT annotation tasks:

You will see the list of open annotation tasks, and  find assignment details such as bounty per assignment, brief task description, the size (i.e. the number of images) to be annotated, and the annotation type (bounding boxes. points, etc.):

5. To join a task, press the button on the right side of the task entry:

Once it is done, switch to the “My Jobs” tab, and you should see your assignment for this task:

6. To start annotation, press the “Go to job” button in the “Job URL” column

You will be navigated to the CVAT job, where you can draw annotations.

7. Make sure to check the job requirements by clicking on the Guide button:

8. Draw annotations as required in the task, and move between the job frames:

9. When all the job frames are finished, click Save.

10. Switch the job state to Completed.

Implementation details

In this section, we’ll discuss several implementation details to give you a clearer understanding of how everything works together. At the heart of the project lie three key components:

  • A tool that allows annotators to create high-quality annotations
  • A platform that manages assignments, tasks, and data
  • A contract that specifies the job, acceptance criteria, and bounty

All these components contribute to solving the crowdsourcing platform challenges, improving the overall efficiency of the system, and providing an effective solution, when combined. Now, let’s look closer at the specific problems and solutions we implemented in the platform.

Task creation and assignment management. The platform handles the datasets and worker assignments in an automatic way, which is measurable, consistent, and reliable. The platform splits the dataset given into small chunks (the assignments), allowing to annotate data efficiently in parallel. Each annotator gets a fixed amount of work in an assignment, and each assignment can be validated with no human interaction. As there is no human factor in the assignment creation or management, the process of getting an assignment is quick and is protected from occasional errors. 

Assignment validation. Once an assignment is annotated, it goes to validation. There are several validation strategies used in the industry, including consensus scoring, honey pots/ground truth, model validation, etc. In the current implementation, we decided to use ground truth-based scoring. With this approach, the task requester is required to provide a small number of ground truth annotations during task creation. Each assignment includes several images from the validation set so we can always check the quality of the annotations in the assignment. Then, we extrapolate the resulting quality for the whole assignment. If the assignment quality is below the required level, the annotations are discarded, and the assignment is sent for reannotation.

Payments. When the task is created, the requester configures the bounty and the money is reserved from their wallet. The platform relies on Smart Contracts in blockchain to guarantee fair payments and to allow cancellation of tasks. Each contract gets a clear definition of the work required to receive the bounty, which can be checked automatically. The workers are rewarded for each assignment completed, after the whole dataset quality is accepted during validation.

Not a CVAT.ai user?  Click through and sign up here

Do not want to miss updates and news? Have any questions? Join our community:

Facebook

Discord

LinkedIn

Gitter

GitHub

November 13, 2023
CVAT Team
Go Back