> For the complete documentation index, see [llms.txt](https://docs.deepomatic.com/platform-documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.deepomatic.com/platform-documentation/deepomatic-drive/adding-and-managing-data/adding-images/use-dataset_client.md). # Use dataset\_client Client name: `dataset_client` ## Rename tags Command name: `rename_tags` Command that rename a given tag in one or multiple input files Return: A new import file ### Options | Name | Type | Default | Required | Multiple | Description | | ---------------------- | -------- | ------- | -------- | -------- | -------------------------------------------------------------------------------------------- | | `-t` / `--tags` | `string` | None | `True` | `True` | Tags to rename | | `-i` / `--input_files` | `path` | None | `True` | `True` | Path to the input file where to rename tags | | `-n` / `--name` | `string` | None | `True` | `False` | New tag name | | `-s` / `--suffix` | `string` | None | `False` | `False` | Add suffix. If a suffix is specify we suffix view's name and real\_name and all the concepts | ### Examples #### Case 1: Rename one tag from a single input\_file `dataset_client rename_tags -i client/input_file.json -t previous_tag -n NEW_TAG` #### Case 2: Rename one tag from multiple input\_files `dataset_client rename_tags -i client/input_file.json -i input_file_2.json -t previous_tag -n NEW_TAG` #### Case 3: Rename multiples tags from a single input\_files `dataset_client rename_tags -i client/input_file.json -t previous_tag -t prev_tag -n NEW_TAG` #### Case 4: Rename tag and add suffix `dataset_client rename_tags -i client/input_file.json -t previous_tag -n NEW_TAG -s _suffix` ## Import local images Command name: `import_local_images` Command that import local images and annotate them Return: A new import file ### Options | Name | Type | Default | Required | Multiple | Description | | ----------------------- | -------- | ------- | -------- | -------- | -------------------------------------------------------------------------------------------------- | | `-apk` / `--api-key` | `string` | None | `True` | `False` | API Key (you can find your API Key in the admin django platform) | | `-deepo-env` | `Choice` | None | `True` | `False` | Deepo environment: `stag`,`staging`,`dev`,`dev2`,`dev3`,`dev4`,`pre-prod`,`prod`,`prod-us1`,`us1`> | | `-f` / `--folder` | `path` | None | `True` | `True` | Folder to import | | `-o` / `--org-slug` | `string` | None | `True` | `False` | Destination organization slug | | `-p` / `--project-slug` | `string` | None | `True` | `False` | Destination project slug | ### Requirements #### Metadata In each folder you want to import, you have to add a `metadata.txt` file. Metadata file should contain ```json view_uuid="abcd1234-efgh5678-ijkl9012" tags=["tag 1", "tag 2", ...] ``` If you specify `tags=""`, the images will be imported in the project as `Not annotated` If you specify `tags="without concept"`, the images will be imported and annotated as `Without concept` If you specify `tags=["dog", "wolf"]`, the images will be imported in the project as `dog` and `wolf` ### Examples #### Case 1: Import local folder `dataset_client import_local_images -apk your_api_key --depo-env env -f path/to/your/folder/ -o org_slug -p project_slug` ## Export view Command name: `export_view` Command that export all views and annotations that match the given tags ### Options | Name | Type | Default | Required | Multiple | Description | | --------------------------- | -------- | ------- | -------- | -------- | -------------------------------------------------------------------------------------------- | | `-t` / `--tags` | `string` | None | `True` | `True` | Tags to find | | `-i` / `--input_files` | `path` | None | `True` | `True` | Path to the input file where to rename tags | | `-s` / `--suffix` | `string` | None | `False` | `False` | Add suffix. If a suffix is specify we suffix view's name and real\_name and all the concepts | | `-flat` / `--flatten` | | None | `False` | `False` | Flatten all the views (all views are view root) | | `-tree` / `--tree` | | None | `False` | `False` | Tree views (views kept their tree) | | `--tag_only` / `--tag_only` | | None | `False` | `False` | Indicate that we only want to export this specific tag and associated annotations | If `-flat` or `-tree` aren't specify, it will be `flatten` by default ### Examples #### Case 1: Export views from a single input\_file `dataset_client export_view -i input_file.json -t tag` #### Case 2: Export views from multiple input\_files `dataset_client export_view -i input_file.json -i input_file_bis.json -t tag` #### Case 3: Export views and make it flatten (generate only view root) `dataset_client export_view -i input_file.json -t tag -flat` #### Case 4: Export views and keep the view tree (root, children) `dataset_client export_view -i input_file.json -t tag -tree` #### Case 5: Export views and add suffix `dataset_client export_view -i input_file.json -t tag -flat -s _suffix` #### Case 6: Export view that only match the given tag `dataset_client export_view -i input_file.json -t tag --only_tag` #### Case 7: Export view that only match the given tags `dataset_client export_view -i input_file.json -t tag -t tag2 -t tag3 --only_tag` ## Download one vesta uri `dataset_client download --api-key --deepo-env -i ` ## Download images from an import file (with vesta\_uri) `dataset_client download --api-key --deepo-env -f ` You can also specify an output directory to save downloaded images: `dataset_client download --api-key --deepo-env -f -o ` ## Ops ### Buckets creation ``` export ENV=dev-us1 export BUCKET_NAME=dp-dataset-internal-tmp-${ENV} export BUCKET_GCP_PROJECT=deepomatic-development-us1 export BUCKET_LOCATION=us-central1 export BUCKET_STORAGE_CLASS=Standard gsutil mb -p ${BUCKET_GCP_PROJECT} -c ${BUCKET_STORAGE_CLASS} -l ${BUCKET_LOCATION} gs://${BUCKET_NAME}/ ``` ### Permissions * Apply `objectCreator` for `dataset_client` users on targeted bucket ``` IAM_GROUPS=( dev.team@deepomatic.com sa.team@deepomatic.com se.team@deepomatic.com product.team@deepomatic.com ) for IAM_GROUP in ${IAM_GROUPS[@]}; do gsutil iam ch group:${IAM_GROUP}:objectCreator gs://${BUCKET_NAME} done ``` * Apply read access permission to `vesta` and `vesta-job` GSAs (See this [doc](https://github.com/Deepomatic/vesta/blob/master/web/deploy/jobnest/README.md#gsa-creation-for-gcp-resource-access) for GSA creation) ``` # Vesta GSA export GSA_ENV=dev-us1 export GSA_GCP_PROJECT=deepomatic-development-us1 export GSA_NAMES=( vesta-${GSA_ENV}@${GSA_GCP_PROJECT} vesta-job-${GSA_ENV}@${GSA_GCP_PROJECT} ) for GSA_NAME in ${GSA_NAMES[@]}; do gcloud --project=${BUCKET_GCP_PROJECT} storage buckets add-iam-policy-binding gs://${BUCKET_NAME} \ --member "serviceAccount:${GSA_NAME}.iam.gserviceaccount.com" \ --role "roles/storage.objectViewer" gcloud --project=${BUCKET_GCP_PROJECT} storage buckets add-iam-policy-binding gs://${BUCKET_NAME} \ --member "serviceAccount:${GSA_NAME}.iam.gserviceaccount.com" \ --role "roles/storage.legacyBucketReader" done ```