- Company
- Community
- Resources
- Pricing
- ⚡Labeling service
Implementing annotation crowdsourcing right can be a challenging task, especially for Computer Vision problems. While crowdsourcing offers huge benefits, such as unlimited scaling and low overhead, it also has fundamental challenges, such as low resulting annotation quality, difficult validation, and overwhelming assignment management. Now, we’d like to introduce our solution for annotation at scale, based on CVAT.ai and HUMAN Protocol integration. Read on to learn how you can use it for your datasets and for annotation.
The solution is an online platform based on HUMAN Protocol services, which uses CVAT for data annotation and Web3 technologies for payments in crypto currencies.
On the platform, we bring together 2 key user roles:
For Requesters, the platform provides automatic data annotation assignment preparation and management, automatic annotation validation and merging, and payments based on the annotation quality. Together these features create a place where you can just ask for an annotated dataset - describe the task, provide the data, quality requirements, and reward details - and get the data annotated with the specified quality after some time. No more manual control and management is needed. Keep in mind that currently the platform supports only several Computer Vision task types, but it can be extended in future.
For Annotators, the platform is a place where they can start earning money in just several clicks. All they need to do is register, select a task, and ask for an assignment. Once they get an assignment, they can learn annotation requirements by reading the attached annotation guide, and start annotating - using one of the most popular open-source annotation tools in the Computer Vision community. Each assignment is small enough to be completed in a matter of minutes, allowing flexible participation for the workers.
Bounty payments are based on cryptocurrency; so annotators are required to have a crypto wallet to get the money. For requesters, however, there is also an option to pay with a bank card. Funds for payments are reserved at the time of task creation and are disbursed automatically after the entire dataset has been annotated and validated.
Before initiating a new annotation task, please be aware of the following platform requirements:
Examples:
Please keep in mind that simpler tasks tend to be easier and quicker to annotate. If your annotations require complex work, it can be a good idea to split the whole task into simpler subtasks, and annotate them separately for better quality and efficiency.
Now, let’s go over the steps required to start an annotation task.
1) Create a crypto wallet using one of the corresponding applications (e.g. MetaMask)
Example for MetaMask in Chrome:
1. Install the browser extension.
2. Open the extension page in the browser, create a password.
3. Select the Polygon Mainnet network or add it from the list.
4. Use an existing ETH-based wallet or click Add an account in MetaMask.
5. Rename the account to “Job Requester” or any other name you like.
6. Now, you'll have a wallet (ETH-based) in the network. It should show 0 MATIC on the balance.
7. Transfer some MATIC from another ETH-based wallet or buy (e.g. $5). Make sure to buy in the Polygon Mainnet network.
8. Click Import Tokens. Add the HMT token:
Id: 0xc748B2A084F8eFc47E086ccdDD9b7e67aEb571BF
Name: HMT
9. Now, it should also show 0 HMT on the balance.
10. Convert some MATIC into HMT
Now that you can perform payments in cryptocurrencies, it’s time to prepare your data and create an annotation task.
2) An AWS S3 bucket with public access is required. Сreate an AWS S3 bucket and upload your images as separate files into the bucket. Make sure to note the link to the directory containing the files within the bucket.
You can obtain this URL by clicking Copy URL (the directory URL is needed).
Example:
3) Select a small validation subset from the original images (e.g., 3%, 30 images from 1000), annotate it, and upload annotations in the COCO format to the bucket.
You will only need a .json file. You can prepare such a dataset using CVAT or other tools.
Remember to note the URL to this file as well.
Example:
Using 0.1-5% of the images is recommended, depending on the dataset size. It is recommended to select just random images from the whole dataset. These images will appear randomly during the annotation process to check the annotation quality. The annotations produced by workers for these images are not included in the resulting annotated dataset; instead, the ground truth annotations are used.
Please note that only 1-keypoint skeletons are supported for the COCO Keypoints format.
4) Proceed to the platform and register a new account, if you don’t have one.
5) Click Create a Job.
6) Select the payment method you prefer. You can pay via a crypto wallet or with a bank card.
7) Once the payment method is selected, you’ll see the job configuration page. Please select the CVAT job type and the Polygon Mainnet network.
Let’s consider the fields one by one:
8) At the next step, you’ll need to enter the bounty details for workers. The resulting value is the price for the whole dataset.
The bounties for the annotators will be calculated automatically based on the number of images annotated.
Here you can choose to pay in crypto currency or with a bank card. The money will be reserved until the whole dataset is annotated.
9) Once you’re ready, click Pay now.
10) If everything is set up correctly, after the next several minutes, the job will appear in the job list.
In the list, you can check the current annotation status and details of the jobs you have created. Now, the annotators will look for assignments and begin their work. The process is fully autonomous for you, so you will only need to wait for the process to finish. It can take some time, depending on the current market prices, available worker pool, and other conditions. The workers will be able to join the task and get an assignment via this platform link.
If, at some point, you decide to cancel a job, here you can find a button for this. The money you reserved for the work will get back to you.
11) Once the data is annotated, the final annotations will be validated and merged automatically, and the result will be available by the URL you receive in the job details section.
1. Create a crypto wallet (check the explanation in the requester part of this guide).
2. Go to the annotator platform and register. You’ll need to pass a mandatory KYC procedure to finish the registration process, this is required by the applicable laws.
3. Currently, we require users to explicitly request participation in CVAT labeling. Please email us at app@humanprotocol.org to express your interest.
4. On the platform, open the list of available CVAT annotation tasks:
You will see the list of open annotation tasks, and find assignment details such as bounty per assignment, brief task description, the size (i.e. the number of images) to be annotated, and the annotation type (bounding boxes. points, etc.):
5. To join a task, press the button on the right side of the task entry:
Once it is done, switch to the “My Jobs” tab, and you should see your assignment for this task:
6. To start annotation, press the “Go to job” button in the “Job URL” column
You will be navigated to the CVAT job, where you can draw annotations.
7. Make sure to check the job requirements by clicking on the Guide button:
8. Draw annotations as required in the task, and move between the job frames:
9. When all the job frames are finished, click Save.
10. Switch the job state to Completed.
In this section, we’ll discuss several implementation details to give you a clearer understanding of how everything works together. At the heart of the project lie three key components:
All these components contribute to solving the crowdsourcing platform challenges, improving the overall efficiency of the system, and providing an effective solution, when combined. Now, let’s look closer at the specific problems and solutions we implemented in the platform.
Task creation and assignment management. The platform handles the datasets and worker assignments in an automatic way, which is measurable, consistent, and reliable. The platform splits the dataset given into small chunks (the assignments), allowing to annotate data efficiently in parallel. Each annotator gets a fixed amount of work in an assignment, and each assignment can be validated with no human interaction. As there is no human factor in the assignment creation or management, the process of getting an assignment is quick and is protected from occasional errors.
Assignment validation. Once an assignment is annotated, it goes to validation. There are several validation strategies used in the industry, including consensus scoring, honey pots/ground truth, model validation, etc. In the current implementation, we decided to use ground truth-based scoring. With this approach, the task requester is required to provide a small number of ground truth annotations during task creation. Each assignment includes several images from the validation set so we can always check the quality of the annotations in the assignment. Then, we extrapolate the resulting quality for the whole assignment. If the assignment quality is below the required level, the annotations are discarded, and the assignment is sent for reannotation.
Payments. When the task is created, the requester configures the bounty and the money is reserved from their wallet. The platform relies on Smart Contracts in blockchain to guarantee fair payments and to allow cancellation of tasks. Each contract gets a clear definition of the work required to receive the bounty, which can be checked automatically. The workers are rewarded for each assignment completed, after the whole dataset quality is accepted during validation.
Not a CVAT.ai user? Click through and sign up here
Do not want to miss updates and news? Have any questions? Join our community: