Use dataset_client

Client name: dataset_client

Rename tags

Command name: rename_tags

Command that rename a given tag in one or multiple input files

Return: A new import file

Options

Name	Type	Default	Required	Multiple	Description
-t / --tags	string	None	True	True	Tags to rename
-i / --input_files	path	None	True	True	Path to the input file where to rename tags
-n / --name	string	None	True	False	New tag name
-s / --suffix	string	None	False	False	Add suffix. If a suffix is specify we suffix view's name and real_name and all the concepts

Examples

Case 1: Rename one tag from a single input_file

dataset_client rename_tags -i client/input_file.json -t previous_tag -n NEW_TAG

Case 2: Rename one tag from multiple input_files

dataset_client rename_tags -i client/input_file.json -i input_file_2.json -t previous_tag -n NEW_TAG

Case 3: Rename multiples tags from a single input_files

dataset_client rename_tags -i client/input_file.json -t previous_tag -t prev_tag -n NEW_TAG

Case 4: Rename tag and add suffix

dataset_client rename_tags -i client/input_file.json -t previous_tag -n NEW_TAG -s _suffix

Import local images

Command name: import_local_images

Command that import local images and annotate them

Return: A new import file

Options

Name	Type	Default	Required	Multiple	Description
-apk / --api-key	string	None	True	False	API Key (you can find your API Key in the admin django platform)
-deepo-env	Choice	None	True	False	Deepo environment: stag,staging,dev,dev2,dev3,dev4,pre-prod,prod,prod-us1,us1>
-f / --folder	path	None	True	True	Folder to import
-o / --org-slug	string	None	True	False	Destination organization slug
-p / --project-slug	string	None	True	False	Destination project slug

Requirements

Metadata

In each folder you want to import, you have to add a metadata.txt file.

Metadata file should contain

view_uuid="abcd1234-efgh5678-ijkl9012"
tags=["tag 1", "tag 2", ...]

If you specify tags="", the images will be imported in the project as Not annotated If you specify tags="without concept", the images will be imported and annotated as Without concept If you specify tags=["dog", "wolf"], the images will be imported in the project as dog and wolf

Examples

Case 1: Import local folder

dataset_client import_local_images -apk your_api_key --depo-env env -f path/to/your/folder/ -o org_slug -p project_slug

Export view

Command name: export_view

Command that export all views and annotations that match the given tags

Options

Name	Type	Default	Required	Multiple	Description
-t / --tags	string	None	True	True	Tags to find
-i / --input_files	path	None	True	True	Path to the input file where to rename tags
-s / --suffix	string	None	False	False	Add suffix. If a suffix is specify we suffix view's name and real_name and all the concepts
-flat / --flatten		None	False	False	Flatten all the views (all views are view root)
-tree / --tree		None	False	False	Tree views (views kept their tree)
--tag_only / --tag_only		None	False	False	Indicate that we only want to export this specific tag and associated annotations

If -flat or -tree aren't specify, it will be flatten by default

Examples

Case 1: Export views from a single input_file

dataset_client export_view -i input_file.json -t tag

Case 2: Export views from multiple input_files

dataset_client export_view -i input_file.json -i input_file_bis.json -t tag

Case 3: Export views and make it flatten (generate only view root)

dataset_client export_view -i input_file.json -t tag -flat

Case 4: Export views and keep the view tree (root, children)

dataset_client export_view -i input_file.json -t tag -tree

Case 5: Export views and add suffix

dataset_client export_view -i input_file.json -t tag -flat -s _suffix

Case 6: Export view that only match the given tag

dataset_client export_view -i input_file.json -t tag --only_tag

Case 7: Export view that only match the given tags

dataset_client export_view -i input_file.json -t tag -t tag2 -t tag3 --only_tag

Download one vesta uri

dataset_client download --api-key <VESTA_API_KEY> --deepo-env <dev3|stag|...> -i <vesta_image uri>

Download images from an import file (with vesta_uri)

dataset_client download --api-key <VESTA_API_KEY> --deepo-env <dev3|stag|...> -f <import file>

You can also specify an output directory to save downloaded images:

dataset_client download --api-key <VESTA_API_KEY> --deepo-env <dev3|stag|...> -f <import file> -o <my output dir>

Ops

Buckets creation

export ENV=dev-us1
export BUCKET_NAME=dp-dataset-internal-tmp-${ENV}
export BUCKET_GCP_PROJECT=deepomatic-development-us1
export BUCKET_LOCATION=us-central1
export BUCKET_STORAGE_CLASS=Standard

gsutil mb -p ${BUCKET_GCP_PROJECT} -c ${BUCKET_STORAGE_CLASS} -l ${BUCKET_LOCATION} gs://${BUCKET_NAME}/ 

Permissions

• Apply objectCreator for dataset_client users on targeted bucket

IAM_GROUPS=(
    [email protected]
    [email protected]
    [email protected]
    [email protected]
)

for IAM_GROUP in ${IAM_GROUPS[@]}; do
    gsutil iam ch group:${IAM_GROUP}:objectCreator gs://${BUCKET_NAME}
done

• Apply read access permission to vesta and vesta-job GSAs (See this doc for GSA creation)

# Vesta GSA
export GSA_ENV=dev-us1
export GSA_GCP_PROJECT=deepomatic-development-us1
export GSA_NAMES=(
    vesta-${GSA_ENV}@${GSA_GCP_PROJECT}
    vesta-job-${GSA_ENV}@${GSA_GCP_PROJECT}
)

for GSA_NAME in ${GSA_NAMES[@]}; do
       gcloud --project=${BUCKET_GCP_PROJECT} storage buckets add-iam-policy-binding gs://${BUCKET_NAME} \
                --member "serviceAccount:${GSA_NAME}.iam.gserviceaccount.com" \
                --role "roles/storage.objectViewer"

       gcloud --project=${BUCKET_GCP_PROJECT} storage buckets add-iam-policy-binding gs://${BUCKET_NAME} \
                --member "serviceAccount:${GSA_NAME}.iam.gserviceaccount.com" \
                --role "roles/storage.legacyBucketReader"
done