JSON Upload

JSON Format
Uploading images with a JSON file is the recommended solution if you need to add a large number of images.
In order to use the JSON format, the images that you want to add to the platform need to be reachable via a public URL. If that is not the case, you will find ways to upload images to a remote server below in the page.
Now that all the images are publicly reachable, we need to tell Deepomatic Studio where to find the images, as well as adding any useful information associated with images. In what follows, we will consider the following image coming from the Open Images dataset.
Simple JSON
If what you want to do is simply to add the image in the platform, you just need to list all the images location in the JSON, as seen in the example below.
Simple upload json
{
  "images": [
    {
      "location": "https://storage.googleapis.com/your-bucket/openimages/detection/images/train/000018acd19b4ad3.jpg",
      "data": 
        {
          "file_name": "my image",
          "source": "openimage dataset"
        }
    }
  ]
}
Each entry in the images list corresponds to an image and contains the following fields:
location: image URL
data: (optional) a dictionary in which you can add as many fields as you want. It will typically contain useful meta-data, such as: date, camera ID, etc.
Of course you can add multiple images at the same time: images is a list, so you may input as many as you want.
Add metadata to your images
You can add metadata to your images using the JSON Upload. The metadata are displayed in the information popup on Studio:
In order to add metadata, add a field data in the element of the images list. The following JSON was used to upload the image above:
{
  "images": [
    {
      "location": "https://www.wikichat.fr/wp-content/uploads/sites/2/comment-soigner-une-plaie-dun-chat.jpg",
      "data": 
        {
          "filename": "my image",
          "source": "openimage dataset",
          "animal type": "cat",
          "age": "2 months",
          "weight": "4 kg"
        }
    }
  ]
}
The fields File name, Created, Creator and Last annotator are set automatically.
Tagging/classification JSON
If you want your images to be annotated when you add them to the platform, you can provide these annotations using the JSON format. In addition to the location and optional data fields, you will need to add the following fields to the images entries of your JSON file:
stage: (optional) stage of the image, either train (training set), val (validation set).   If not provided, it is set train by default.
annotated_regions:  (optional) the list of annotations, each containing the following fields:
tags: the list of tags present on this region
region_type: always put Whole in this field, because the annotation concerns the whole image.
The difference between classification and tagging is that the classification can have only one tag per image while the tagging can have several.
If you want to add images with their annotations, you will have to add an additional tags field to the JSON. This is to check errors in the tags. Simply put, tags is a list of all possible tags for the images.
{
  "tags": ["Statue", "Bike", "Plane", "Crystal"],
  "images": [
    {
      "location": "https://storage.googleapis.com/dp-missions/openimages/detection/images/train/000018acd19b4ad3.jpg",
      "stage": "train",
      "annotated_regions": [
        {
          "tags": ["Statue", "Crystal"],
          "region_type": "Whole"
        }
      ]
    }
  ]
}
If a tag is present in the JSON tags field but not on Deepomatic Studio, it will be automatically created. This can be helpful when you have to manually create a lot of tags.
Detection JSON
Finally, you can also add annotations specific to a bounding box, meaning a certain region of the image (represented in the image below). 
If you upload the JSON below to the Deepomatic Studio platform on a detection dataset then you will see that both boxes are annotated.
Detection upload json
{
  "tags": ["Statue", "Bike", "Plane", "Crystal"],
  "images": [
    {
      "location": "https://storage.googleapis.com/dp-missions/openimages/detection/images/train/000018acd19b4ad3.jpg",
      "stage": "train",
      "annotated_regions": [
        {
          "tags": ["Statue"],
          "region_type": "Box",
          "region": {
            "xmin": 0.485351562,
            "xmax": 0.805664062,
            "ymin": 0.121522694,
            "ymax": 0.609077599
          }
        },
        {
          "tags": ["Crystal"],
          "region_type": "Box",
          "region": {
            "xmin": 0.231445312,
            "xmax": 0.706054688,
            "ymin": 0.273792094,
            "ymax": 0.775988287
          }
        }
      ]
    }
  ]
}
This time, note that we've added a new field region composed of xmin, xmax, ymin and ymaxcorresponding to the normalized bounding box  coordinates.
If the field annotated_regions is [ ], the image will be imported in the category Without concept whereas if the field is not mentioned, the image will be in the category Not annotated.
Upload to Remote Server
There are several options to upload images on a remote server, we will review the 3 main possibilities:
Uploading to a private server through ssh
​Google Cloud Platform​
​Amazon Web Services S3​
Private server
You might have access to a private server, the first step should be to contact your SysAdmin to enquire about your internal policies. You can also have a look at this tutorial on How To Configure SSH Key-Based Authentication on a Linux Server.
Once you've setup the server you can connect easily and upload your images using the scp command in bash. If your images are located in the /home/myname/images/ in your local computer and you want to upload them to /home/media/all_images/ on the remote server then you can use the command below.
scp -P 333 -r /home/myname/images/ [email protected]:/home/media/all_images/
The -r option means that you want to transfer the complete directory, -P 333 is the port number to use and yourusername/yourservername should be provided to you by your SysAdmin.
You will find a comprehensive tutorial on scp at Linux Academy.
Google Cloud Platform
If you have a GCP account then the first step will be to install gsutil, Google Services Command Line Interface. Simply refer to the Google official tutorial.
Install Google Services CLI
$ curl https://sdk.cloud.google.com | bash
$ exec -l $SHELL
$ gcloud init
You are now ready to transfer images to Google Cloud. The transfer is composed of two steps:
Upload the images to the platform.
Make them public so that Deepomatic Studio can download them.
Upload images to GCP and make them public
$ gsutil -m cp -r /home/myname/images/ gs://yourbucketname/all_images/
$ gsutil -m acl set -R -a public-read gs://yourbucketname/all_images/
The -m option in the gsutil commands ensure that the operations are parallelized, accelerating processing time depending on your machine, but it may consume a significant amount of network bandwidth.
Amazon Web Services S3
If you use AWS S3 instead of GCP you might want to have a look at the Amazon Installion Guide and also the Amazon Configuration Guide.
Install Amazon Web Services CLI
$ pip install awscli --upgrade --user
$ aws configure
Upload images to AWS s3 and make them public
$ aws s3 sync /home/myname/images/ yourbucketname/all_images/ --acl public-read
Note that both the uploading and making file public is done in one command.
Drag & Drop
Deepomatic CLI - Studio
Last modified 2yr ago