Use dataset_client

Client name: dataset_client

Rename tags

Command name: rename_tags

Command that rename a given tag in one or multiple input files

Return: A new import file

Options

Name
Type
Default
Required
Multiple
Description

-t / --tags

string

None

True

True

Tags to rename

-i / --input_files

path

None

True

True

Path to the input file where to rename tags

-n / --name

string

None

True

False

New tag name

-s / --suffix

string

None

False

False

Add suffix. If a suffix is specify we suffix view's name and real_name and all the concepts

Examples

Case 1: Rename one tag from a single input_file

dataset_client rename_tags -i client/input_file.json -t previous_tag -n NEW_TAG

Case 2: Rename one tag from multiple input_files

dataset_client rename_tags -i client/input_file.json -i input_file_2.json -t previous_tag -n NEW_TAG

Case 3: Rename multiples tags from a single input_files

dataset_client rename_tags -i client/input_file.json -t previous_tag -t prev_tag -n NEW_TAG

Case 4: Rename tag and add suffix

dataset_client rename_tags -i client/input_file.json -t previous_tag -n NEW_TAG -s _suffix

Import local images

Command name: import_local_images

Command that import local images and annotate them

Return: A new import file

Options

Name
Type
Default
Required
Multiple
Description

-apk / --api-key

string

None

True

False

API Key (you can find your API Key in the admin django platform)

-deepo-env

Choice

None

True

False

Deepo environment: stag,staging,dev,dev2,dev3,dev4,pre-prod,prod,prod-us1,us1>

-f / --folder

path

None

True

True

Folder to import

-o / --org-slug

string

None

True

False

Destination organization slug

-p / --project-slug

string

None

True

False

Destination project slug

Requirements

Metadata

In each folder you want to import, you have to add a metadata.txt file.

Metadata file should contain

view_uuid="abcd1234-efgh5678-ijkl9012"
tags=["tag 1", "tag 2", ...]

If you specify tags="", the images will be imported in the project as Not annotated If you specify tags="without concept", the images will be imported and annotated as Without concept If you specify tags=["dog", "wolf"], the images will be imported in the project as dog and wolf

Examples

Case 1: Import local folder

dataset_client import_local_images -apk your_api_key --depo-env env -f path/to/your/folder/ -o org_slug -p project_slug

Export view

Command name: export_view

Command that export all views and annotations that match the given tags

Options

Name
Type
Default
Required
Multiple
Description

-t / --tags

string

None

True

True

Tags to find

-i / --input_files

path

None

True

True

Path to the input file where to rename tags

-s / --suffix

string

None

False

False

Add suffix. If a suffix is specify we suffix view's name and real_name and all the concepts

-flat / --flatten

None

False

False

Flatten all the views (all views are view root)

-tree / --tree

None

False

False

Tree views (views kept their tree)

--tag_only / --tag_only

None

False

False

Indicate that we only want to export this specific tag and associated annotations

If -flat or -tree aren't specify, it will be flatten by default

Examples

Case 1: Export views from a single input_file

dataset_client export_view -i input_file.json -t tag

Case 2: Export views from multiple input_files

dataset_client export_view -i input_file.json -i input_file_bis.json -t tag

Case 3: Export views and make it flatten (generate only view root)

dataset_client export_view -i input_file.json -t tag -flat

Case 4: Export views and keep the view tree (root, children)

dataset_client export_view -i input_file.json -t tag -tree

Case 5: Export views and add suffix

dataset_client export_view -i input_file.json -t tag -flat -s _suffix

Case 6: Export view that only match the given tag

dataset_client export_view -i input_file.json -t tag --only_tag

Case 7: Export view that only match the given tags

dataset_client export_view -i input_file.json -t tag -t tag2 -t tag3 --only_tag

Download one vesta uri

dataset_client download --api-key <VESTA_API_KEY> --deepo-env <dev3|stag|...> -i <vesta_image uri>

Download images from an import file (with vesta_uri)

dataset_client download --api-key <VESTA_API_KEY> --deepo-env <dev3|stag|...> -f <import file>

You can also specify an output directory to save downloaded images:

dataset_client download --api-key <VESTA_API_KEY> --deepo-env <dev3|stag|...> -f <import file> -o <my output dir>

Ops

Buckets creation

export ENV=dev-us1
export BUCKET_NAME=dp-dataset-internal-tmp-${ENV}
export BUCKET_GCP_PROJECT=deepomatic-development-us1
export BUCKET_LOCATION=us-central1
export BUCKET_STORAGE_CLASS=Standard

gsutil mb -p ${BUCKET_GCP_PROJECT} -c ${BUCKET_STORAGE_CLASS} -l ${BUCKET_LOCATION} gs://${BUCKET_NAME}/

Permissions

  • Apply objectCreator for dataset_client users on targeted bucket

IAM_GROUPS=(
    dev.team@deepomatic.com
    sa.team@deepomatic.com
    se.team@deepomatic.com
    product.team@deepomatic.com
)

for IAM_GROUP in ${IAM_GROUPS[@]}; do
    gsutil iam ch group:${IAM_GROUP}:objectCreator gs://${BUCKET_NAME}
done

  • Apply read access permission to vesta and vesta-job GSAs (See this doc for GSA creation)

# Vesta GSA
export GSA_ENV=dev-us1
export GSA_GCP_PROJECT=deepomatic-development-us1
export GSA_NAMES=(
    vesta-${GSA_ENV}@${GSA_GCP_PROJECT}
    vesta-job-${GSA_ENV}@${GSA_GCP_PROJECT}
)

for GSA_NAME in ${GSA_NAMES[@]}; do
       gcloud --project=${BUCKET_GCP_PROJECT} storage buckets add-iam-policy-binding gs://${BUCKET_NAME} \
                --member "serviceAccount:${GSA_NAME}.iam.gserviceaccount.com" \
                --role "roles/storage.objectViewer"

       gcloud --project=${BUCKET_GCP_PROJECT} storage buckets add-iam-policy-binding gs://${BUCKET_NAME} \
                --member "serviceAccount:${GSA_NAME}.iam.gserviceaccount.com" \
                --role "roles/storage.legacyBucketReader"
done