Data export

Introduction

The Data Export Service automatically exports data related to your operations into a standard format at a chosen interval and stores it in a configured external location or provides through one email notification a temporary link to access it.

The data export is designed for BI purposes, to export your data in operational applications or to trigger processes with a near real time latency, contact your implementation manager or customer support manager.

Global Architecture

Cloud Architecture integration diagram for the cloud bucket connector

Configuration

To enable the data export you should notify it to your customer success manager and with them choose the right configuration for your export. After finalizing your configuration, we will validate and deploy the solution for you.

Base configuration parameters

Parameter
site_id	List of uuids	A list of one or more site identifiers for which you want to export data.
export_type	work-orders-states or work-orders-timeline	Indicates the type of data to export. See Structured data format The timeline format is the default one.
file_format	json	The output format for the exported data files. Note: we only support Jsonline.
interval	once, daily, weekly, monthly, ..	The frequency at which the export job will run. See Export frequency
connector_type	email / cloud_storage	Determines how the structured data and images will be sent to you. See Connector types
connector configuration	Depending on the connector	See Connector types

Export frequency

You can have a one shot export or configure a regular data export with a defined frequency.

• Daily

• Weekly

• Monthly

The export will be generated and sent at the chosen interval.@

For a one time export or for the initialization of the export, restrictions may apply.

Connector types

The connector_type section determines how and where your exported data is delivered. Two primary connector types are supported:

1. Email with temporary link

2. Your cloud bucket

Below are the configuration details that differ based on your chosen connector type.

Email connector

This connector allows you to receive at your email address one temporary link to the structured data export file data export generated.

If you choose the email connector you will not have access to the images.

Email connector Config

Parameters	Values / type	Description
emails	List of strings (valid email addresses)	Recipients to be notified when data is available.

GCP Cloud bucket connector

This connector allows you to receive directly into your cloud bucket the data export you decided, both the raw data and the images.

Cloud Storage connector Config

Parameters	Values / type	Descripti
storage_uri	String (e.g., gs://customer-cloud-bucket-export/folder/customer-external/)	Prefixed URI of the storage location where the exported files will be saved on the customer cloud. Note: Must be a valid Google Cloud Storage (GCS) bucket path. Example: gs://my-bucket/deepomatic-data
export_images	True / False	Determines whether images associated with the exported data should also be included in the export.

storage_uri

We already have a separation between different sites, but we advise to have one extra layer to separate different environments.

• Separate the production / testing sites in different folders

  • gs://customer-cloud-bucket-export/deepomatic/testing/

  • gs://customer-cloud-bucket-export/deepomatic/production/

export_images

When enabled we will transfer images related to your export_type from our infrastructure to yours. (See image export section)

GCP setup:

Given that we will transfer the data from our infrastructure to yours, to enable the Deepomatic GCP Cloud Storage Connector:

1. We will provide you one Deepomatic Principal email (a Deepomatic GCP Service Account):

customer-<customer_name>[e-mail]

1. To this GSA you will need to assign these permissions on the bucket specified in the external_storage_uri:

   1. storage.objects.create

   2. storage.objects.list

   3. storage.objects.get

   4. storage.buckets.get

Structure of the data in the target bucket

The new data exports and images will be added to your bucket following this structure :

<external_storage_uri>/
  433a60f6-9756-473e-b1e9-9e89e24bb787/ # < your site Id
    data/
      work-orders-states/ # < export type of your choice 
        2025-03-16_00:00:00_2025-03-17_00:00:00.json # < date from - date to
        2025-03-17_00:00:00_2025-03-18_00:00:00.json
        ...
    inputs/ # < images
      13ead234c-8236-4sfa-b748-sdfb409cfe5.png # < refer to the inpud id in the export 
      338e3fda-4aab-4d20-8aa0-e9b234ae4779.png
      ... 

S3 cloud bucket connector

TBD - WIP

SFTP connector

TBD - WIP

Structured data format

Timeline data format

Each generated file contains structured data on all the work orders created or updated during the last interval time frame, including all the analysis performed on them.

This format can be used for:

• book keeping

• field ops analysis, AI and FTR analysis

• field operator experience analysis

The work-order-timeline file is in the jsonline format.

Schema

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "id": {"type": ["string", "null"], "format": "uuid"},
        "name": {"type": ["string", "null"]},
        "site_id": {"type": ["string", "null"], "format": "uuid"},
        "create_date": {"type": ["string", "null"], "format": "date-time"},
        "update_date": {"type": ["string", "null"], "format": "date-time"},
        "metadata": {"type": ["object", "null"]},
        "progress_score": {"type": ["number", "null"]},
        "work_items": {
            "type": "array",
            "items": {
                "type": "object",
                "required": ["id", "name", "input"],
                "properties": {
                    "id": {"type": ["string", "null"], "format": "uuid"},
                    "name": {"type": ["string", "null"]},
                    "input": {
                        "type": "array",
                        "items": {
                            "type": "object",
                            "required": [
                                "id",
                                "create_date",
                                "file_extension",
                                "metadata",
                                "data_conformity",
                                "job_conformity"
                            ],
                            "properties": {
                                "id": {"type": ["string", "null"], "format": "uuid"},
                                "create_date": {"type": ["string", "null"], "format": "date-time"},
                                "file_extension": {"type": ["string", "null"]},
                                "metadata": {"type": ["object", "null"]},
                                "data_conformity": {
                                    "type": ["array", "null"],
                                    "items": {
                                        "type": "object",
                                        "required": ["check", "code", "level"],
                                        "properties": {
                                            "check": {"type": ["string", "null"]},
                                            "code": {"type": ["number", "null"]},
                                            "level": {"type": ["string", "null"]}
                                        }
                                    }
                                },
                                "job_conformity": {
                                    "type": "array",
                                    "items": {
                                        "type": "object",
                                        "required": [
                                            "task_name",
                                            "analyzed_value",
                                            "corrected_value",
                                            "corrected_date",
                                            "message",
                                            "is_conformity_validated"
                                        ],
                                        "properties": {
                                            "task_name": {"type": ["string", "null"]},
                                            "analyzed_value": {
                                                "type": [
                                                    "string",
                                                    "boolean",
                                                    "number",
                                                    "null"
                                                ]
                                            },
                                            "corrected_value": {
                                                "type": [
                                                    "string",
                                                    "boolean",
                                                    "number",
                                                    "null"
                                                ]
                                            },
                                            "corrected_date": {
                                                "type": ["string", "null"],
                                                "format": "date-time"
                                            },
                                            "message": {"type": ["string", "null"]},
                                            "is_conformity_validated": {
                                                "type": ["boolean", "null"]
                                            }
                                        },
                                        "additionalProperties": false
                                    }
                                }
                            },
                            "additionalProperties": false
                        }
                    }
                },
                "additionalProperties": false
            }
        }
    },
    "required": [
        "id",
        "name",
        "site_id",
        "create_date",
        "update_date",
        "metadata",
        "work_items"
    ],
    "additionalProperties": false
}

Fields definitions

Work order level fields

Field	Type	Description
id	UUID (string)	Unique identifier of the work order
name	string	Descriptive name of the work order
site_id	UUID (string)	Identifier of the related site
create_date	ISO 8601 string	Creation timestamp with timezone
update_date	ISO 8601 string	Last updated timestamp with timezone
metadata	object	Additional information about the work order
progress_score	number	Optional progress indicator (e.g., between 0 and 1).
work_items	array	List of items to be processed. Each contains an id, name, and input. A work item relates to one taskgroup and has the same name

Work item

Field	Type	Description
id	UUID (string)	Identifier of the related work_item
name	string	Name of the work item
input	array	List of input, images, and their analysis results

input

Field	Type	Description
id	UUID (string)	Input identifier (image id)
metadata	object	May contain metadata of the image like geolocation or timestamp.
data_conformity	array of objects	List of conformity checks on raw input data. Can be empty.
job_conformity	array of objects	List of job-specific conformity checks. Can be empty.

data_conformity fields

Fields	Type	Description
check	string	The data conformity check.
code	string	One code per data conformity check.
level	string	Error, information or warning. Error is blocking, Warning and information are not blocking.

job_conformity fields

Fields	Type	Description
task_name	string	Name of the task being validated.
analyzed_value	string / number / bool	Output of the automatic analysis (optional).
corrected_value	string / number / bool	Manually corrected value (optional).
corrected_date	ISO 8601 string	When correction was made (nullable).
message	string	Explanation, error, or note about the result.
is_conformity_validated	boolean	if this task is considered validate.

Example

{
   "id": "a71805b6-93f2-4ea3-9e42-a46b397ad448",
   "name": "test-BB6D70A6",
   "site_id": "433a60f6-9756-473e-b1e9-9e89e24bb787",
   "create_date": "2025-03-19T14:21:40.105568+00:00",
   "update_date": "2025-03-19T14:21:40.125857+00:00",
   "metadata": {
      "created_with": "mobile_app"
   },
   "progress_score": null,
   "work_items": [
      {
         "id": "4e5b8dd8-9d50-4888-977b-4064e4b6a357",
         "name": "l4_civil_open",
         "input": [
            {
               "id": "9a739279-0904-4710-b740-53971660f2ae",
               "metadata": {
                  "timestamp_ms": 1742394153532
               },
               "data_conformity": [],
               "job_conformity": [
                  {
                     "task_name": "l4_civil_open_geolocation_maps",
                     "analyzed_value": "UNAVAILABLE",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "UNAVAILABLE",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_open_ctx",
                     "analyzed_value": true,
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "OK",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_open_geolocation",
                     "analyzed_value": "UNAVAILABLE",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "UNAVAILABLE",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_open_earthing",
                     "analyzed_value": false,
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "KO",
                     "is_conformity_validated": null
                  }
               ]
            }
         ]
      },
      {
         "id": "d43a33f6-a4b2-42fc-8865-0c689f2d1c36",
         "name": "l4_civil_close",
         "input": [
            {
               "id": "9fe06e48-e29e-4735-92df-82bcb8827b51",
               "metadata": {
                  "timestamp_ms": 1742394107294
               },
               "data_conformity": [],
               "job_conformity": [
                  {
                     "task_name": "l4_civil_close_type",
                     "analyzed_value": "UNAVAILABLE",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "UNAVAILABLE",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_close_size",
                     "analyzed_value": "existing",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "existing",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_close_geolocation",
                     "analyzed_value": "UNAVAILABLE",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "UNAVAILABLE",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_close_ctx",
                     "analyzed_value": true,
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "OK",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_close_pon_reference",
                     "analyzed_value": "CMIB-G-X-4NOZAJ",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "PON Reference: CMIB-G-X-4NOZAJ",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_close_geolocation_maps",
                     "analyzed_value": "UNAVAILABLE",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "UNAVAILABLE",
                     "is_conformity_validated": null
                  }
               ]
            }
         ]
      },
      {
         "id": "09f90b26-eb2b-44b7-a235-8e6973a6306f",
         "name": "l4_civil_open_floor_closeup",
         "input": [
            {
               "id": "ac58cbf8-d144-408b-9b23-ee78c72e479d",
               "metadata": {
                  "timestamp_ms": 1742394787589
               },
               "data_conformity": [],
               "job_conformity": [
                  {
                     "task_name": "l4_civil_open_floor_state",
                     "analyzed_value": "ANOMALY",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "ANOMALY",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_open_bottom_state_validation",
                     "analyzed_value": false,
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "KO",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_open_floor_closeup_ctx",
                     "analyzed_value": true,
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "OK",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_open_duct_count",
                     "analyzed_value": 0.0,
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "0.0",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_open_floor_closeup_geolocation",
                     "analyzed_value": "UNAVAILABLE",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "UNAVAILABLE",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_open_floor_closeup_geolocation_maps",
                     "analyzed_value": "UNAVAILABLE",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "UNAVAILABLE",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_open_duct_count_validation",
                     "analyzed_value": false,
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "KO",
                     "is_conformity_validated": null
                  }
               ]
            }
         ]
      }
   ]
}

Work order state data format

This format is the same as the timeline data format, except that to simplify the analysis, there is only one item in the input array. The input exported is always the last one created.

Image data format

The images analyzed during each interval timeframe will be uploaded into the customer target cloud storage.

File name

The name of each image is composed of the input uuid and the file extension. Both information can be found in the data file in the fields input and file_extension.

Path

In the data file, within each input you have the id and file_extension fields. The absolute path to the images is therefore:

<storage_uri>/<site_id>/inputs/<id><file_extension>

# ex:
gs://customer-cloud-bucket-export/deepomatic/prod/433a60f6-9756-473e-b1e9-9e89e24bb787/inputs/67dd3441-bdee-4deb-9a48-f7663637ed78.png