Deepomatic Platform
  • Overview
  • Release notes
    • May 2025
    • January 2025
    • November 21, 2024
    • October 17, 2024
    • September 19, 2024
    • July 18, 2024
    • June 27, 2024
    • May 23, 2024
    • April 18, 2024
    • March 21, 2024
    • February 22, 2024
    • January 18, 2024
    • December 13, 2023
    • October 26, 2023
    • July 20, 2023
    • June 29, 2023
    • May 29, 2023
    • April 27, 2023
    • March 30, 2023
    • February 17, 2023
    • January 19, 2023
    • December 22, 2022
    • November 18, 2022
    • October 19, 2022
    • September 19, 2022
    • July 27, 2022
    • June 26, 2022
    • May 17, 2022
    • April 13, 2022
    • March 17, 2022
    • February 10, 2022
    • December 21, 2021
    • October 26, 2021
  • Getting started
  • ADMIN & USER MANAGEMENT
    • Invite and manage users
      • Invite group of users at once
      • SSO
        • OIDC
        • SAML
  • Deepomatic Engage
    • Integrate applications
      • Deepomatic vocabulary
      • Deepomatic connectors
        • Set-up
        • Camera Connector
        • Work Order Connector
      • API integration
        • Authentication
        • Errors
        • API reference
          • Work order management
          • Analysis
            • Guide field workers
            • Perform an analysis
            • Correct an analysis
          • Data retrieval
          • Endpoints' list
      • Batch processing
        • Format
        • Naming conventions
        • Processing
        • Batch status & errors
      • Data export
    • Use the mobile application
      • Configure a mobile application
      • Create & visualize work orders
      • Complete work orders
      • Offline experience
    • Manage your business operations with customisable solutions
      • Roles
      • Alerting
      • Field services
        • Reviewing work orders
        • Exploring work orders
        • Grouping work orders
        • Monitoring assets performance
      • Insights
  • Security
    • Security
    • Data Protection
Powered by GitBook
On this page
  • Introduction
  • Global Architecture
  • Configuration
  • Export frequency
  • Connector types
  • Email connector
  • GCP Cloud bucket connector
  • S3 cloud bucket connector
  • SFTP connector
  • Structured data format
  • Timeline data format
  • Work order state data format
  • Image data format

Was this helpful?

  1. Deepomatic Engage
  2. Integrate applications

Data export

Last updated 2 days ago

Was this helpful?

Introduction

The Data Export Service automatically exports data related to your operations into a standard format at a chosen interval and stores it in a configured external location or provides through one email notification a temporary link to access it.

The data export is designed for BI purposes, to export your data in operational applications or to trigger processes with a near real time latency, contact your implementation manager or customer support manager.

Global Architecture

Configuration

To enable the data export you should notify it to your customer success manager and with them choose the right configuration for your export. After finalizing your configuration, we will validate and deploy the solution for you.

Base configuration parameters

Parameter

site_id

List of uuids

A list of one or more site identifiers for which you want to export data.

export_type

work-orders-states or work-orders-timeline

Indicates the type of data to export.

file_format

json

The output format for the exported data files.

Note: we only support Jsonline.

interval

once, daily, weekly, monthly, ..

The frequency at which the export job will run.

connector_type

email / cloud_storage

connector configuration

Depending on the connector

Export frequency

You can have a one shot export or configure a regular data export with a defined frequency.

  • Daily

  • Weekly

  • Monthly

The export will be generated and sent at the chosen interval.@

For a one time export or for the initialization of the export, restrictions may apply.

Connector types

The connector_type section determines how and where your exported data is delivered. Two primary connector types are supported:

  1. Email with temporary link

  2. Your cloud bucket

Below are the configuration details that differ based on your chosen connector type.

Email connector

This connector allows you to receive at your email address one temporary link to the structured data export file data export generated.

If you choose the email connector you will not have access to the images.

Email connector Config

Parameters
Values / type
Description

emails

List of strings (valid email addresses)

Recipients to be notified when data is available.

GCP Cloud bucket connector

This connector allows you to receive directly into your cloud bucket the data export you decided, both the raw data and the images.

Cloud Storage connector Config

Parameters
Values / type
Descripti

storage_uri

String (e.g., gs://customer-cloud-bucket-export/folder/customer-external/)

Prefixed URI of the storage location where the exported files will be saved on the customer cloud.

Note: Must be a valid Google Cloud Storage (GCS) bucket path. Example: gs://my-bucket/deepomatic-data

export_images

True / False

Determines whether images associated with the exported data should also be included in the export.

storage_uri

We already have a separation between different sites, but we advise to have one extra layer to separate different environments.

  • Separate the production / testing sites in different folders

    • gs://customer-cloud-bucket-export/deepomatic/testing/

    • gs://customer-cloud-bucket-export/deepomatic/production/

export_images

When enabled we will transfer images related to your export_type from our infrastructure to yours. (See image export section)

GCP setup:

Given that we will transfer the data from our infrastructure to yours, to enable the Deepomatic GCP Cloud Storage Connector:

  1. We will provide you one Deepomatic Principal email (a Deepomatic GCP Service Account):

customer-<customer_name>-data-export@deepomatic-160015.iam.gserviceaccount.com

  1. To this GSA you will need to assign these permissions on the bucket specified in the external_storage_uri:

    1. storage.objects.create

    2. storage.objects.list

    3. storage.objects.get

    4. storage.buckets.get

Structure of the data in the target bucket

The new data exports and images will be added to your bucket following this structure :

<external_storage_uri>/
  433a60f6-9756-473e-b1e9-9e89e24bb787/ # < your site Id
    data/
      work-orders-states/ # < export type of your choice 
        2025-03-16_00:00:00_2025-03-17_00:00:00.json # < date from - date to
        2025-03-17_00:00:00_2025-03-18_00:00:00.json
        ...
    inputs/ # < images
      13ead234c-8236-4sfa-b748-sdfb409cfe5.png # < refer to the inpud id in the export 
      338e3fda-4aab-4d20-8aa0-e9b234ae4779.png
      ... 

S3 cloud bucket connector

TBD - WIP

SFTP connector

This connector allows you to receive data exported from Deepomatic directly to your own SFTP server. The SFTP connector functions similarly to the GCP bucket connector but sends data to your SFTP server instead of a cloud storage bucket.

Configuration

Parameter
Values / Type
Description

sftp_host

String

Hostname of the SFTP server

sftp_port

Integer

Port number (default: 22)

sftp_private_key_path

String

Path to SSH private key

sftp_private_key_passphrase

String

Passphrase for SSH private key (optional)

sftp_remote_path

String

Base directory path on the remote server

export_images

Boolean

Whether to include images in the export (True/False)

Setup Process

Setting up the SFTP connector involves a clear exchange of information and configurations between your organization and Deepomatic:

1. Server Information

  • Your organization provides:

    • The SFTP server hostname/URI

    • Server port number (typically 22)

    • Base directory path where exports should be stored

2. Authentication Setup

  • Your organization prepares:

    • A dedicated SFTP user account for Deepomatic

  • Deepomatic provides:

    • An SSH public key for authentication

  • Your organization configures:

    • The Deepomatic SSH key in the authorized_keys file for the dedicated user

    • Appropriate file/directory permissions for the user

3. Network Configuration

  • Deepomatic provides:

    • A list of IP addresses/ranges that will connect to your SFTP server

  • Your organization configures:

    • Firewall rules to allow connections from Deepomatic's IP ranges

    • Any necessary network routing or proxy settings

4. Directory Structure

  • Your organization creates the directory structure with the site Id and export type of your choice, given by Deepomatic.

<external_storage_uri>/
  433a60f6-9756-473e-b1e9-9e89e24bb787/ # < your site Id
    data/
      work-orders-states/ # < export type of your choice 
        2025-03-16_00:00:00_2025-03-17_00:00:00.json # < date from - date to
        2025-03-17_00:00:00_2025-03-18_00:00:00.json
        ...
    inputs/ # < images
      13ead234c-8236-4sfa-b748-sdfb409cfe5.png # < refer to the inpud id in the export 
      338e3fda-4aab-4d20-8aa0-e9b234ae4779.png
      ... 

5. Validation and operation

  • Deepomatic will:

    • Perform connection tests to verify proper setup

    • Begin exporting data at the configured intervals

    • Organize files according to the specified directory structure

    • Transfer data and/or images (as configured)

Security Considerations

  • Restrict the SFTP user's access to only the directory needed for exports

  • Use host key verification to ensure you're connecting to the correct server

  • Enable connection encryption for data protection

  • Set appropriate file permissions on the SFTP server

Testing Your Connection

Before full deployment, Deepomatic will work with you to test the SFTP connection:

  1. We'll perform a test export to verify connectivity

  2. We'll validate file permissions and directory access

  3. We'll confirm the proper structure of exported data

Contact your Deepomatic representative if you experience any issues with the SFTP connector setup or data exports.

Structured data format

Timeline data format

Each generated file contains structured data on all the work orders created or updated during the last interval time frame, including all the analysis performed on them.

This format can be used for:

  • book keeping

  • field ops analysis, AI and FTR analysis

  • field operator experience analysis

The work-order-timeline file is in the jsonline format.

Schema

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "id": {"type": ["string", "null"], "format": "uuid"},
        "name": {"type": ["string", "null"]},
        "site_id": {"type": ["string", "null"], "format": "uuid"},
        "create_date": {"type": ["string", "null"], "format": "date-time"},
        "update_date": {"type": ["string", "null"], "format": "date-time"},
        "metadata": {"type": ["object", "null"]},
        "progress_score": {"type": ["number", "null"]},
        "work_items": {
            "type": "array",
            "items": {
                "type": "object",
                "required": ["id", "name", "input"],
                "properties": {
                    "id": {"type": ["string", "null"], "format": "uuid"},
                    "name": {"type": ["string", "null"]},
                    "input": {
                        "type": "array",
                        "items": {
                            "type": "object",
                            "required": [
                                "id",
                                "create_date",
                                "file_extension",
                                "metadata",
                                "data_conformity",
                                "job_conformity"
                            ],
                            "properties": {
                                "id": {"type": ["string", "null"], "format": "uuid"},
                                "create_date": {"type": ["string", "null"], "format": "date-time"},
                                "file_extension": {"type": ["string", "null"]},
                                "metadata": {"type": ["object", "null"]},
                                "data_conformity": {
                                    "type": ["array", "null"],
                                    "items": {
                                        "type": "object",
                                        "required": ["check", "code", "level"],
                                        "properties": {
                                            "check": {"type": ["string", "null"]},
                                            "code": {"type": ["number", "null"]},
                                            "level": {"type": ["string", "null"]}
                                        }
                                    }
                                },
                                "job_conformity": {
                                    "type": "array",
                                    "items": {
                                        "type": "object",
                                        "required": [
                                            "task_name",
                                            "analyzed_value",
                                            "corrected_value",
                                            "corrected_date",
                                            "message",
                                            "is_conformity_validated"
                                        ],
                                        "properties": {
                                            "task_name": {"type": ["string", "null"]},
                                            "analyzed_value": {
                                                "type": [
                                                    "string",
                                                    "boolean",
                                                    "number",
                                                    "null"
                                                ]
                                            },
                                            "corrected_value": {
                                                "type": [
                                                    "string",
                                                    "boolean",
                                                    "number",
                                                    "null"
                                                ]
                                            },
                                            "corrected_date": {
                                                "type": ["string", "null"],
                                                "format": "date-time"
                                            },
                                            "message": {"type": ["string", "null"]},
                                            "is_conformity_validated": {
                                                "type": ["boolean", "null"]
                                            }
                                        },
                                        "additionalProperties": false
                                    }
                                }
                            },
                            "additionalProperties": false
                        }
                    }
                },
                "additionalProperties": false
            }
        }
    },
    "required": [
        "id",
        "name",
        "site_id",
        "create_date",
        "update_date",
        "metadata",
        "work_items"
    ],
    "additionalProperties": false
}

Fields definitions

Work order level fields

Field
Type
Description

id

UUID (string)

Unique identifier of the work order

name

string

Descriptive name of the work order

site_id

UUID (string)

Identifier of the related site

create_date

ISO 8601 string

Creation timestamp with timezone

update_date

ISO 8601 string

Last updated timestamp with timezone

metadata

object

Additional information about the work order

progress_score

number

Optional progress indicator (e.g., between 0 and 1).

work_items

array

List of items to be processed. Each contains an id, name, and input. A work item relates to one taskgroup and has the same name

Work item

Field
Type
Description

id

UUID (string)

Identifier of the related work_item

name

string

Name of the work item

input

array

List of input, images, and their analysis results

input

Field
Type
Description

id

UUID (string)

Input identifier (image id)

metadata

object

May contain metadata of the image like geolocation or timestamp.

data_conformity

array of objects

List of conformity checks on raw input data. Can be empty.

job_conformity

array of objects

List of job-specific conformity checks. Can be empty.

data_conformity fields

Fields
Type
Description

check

string

The data conformity check.

code

string

One code per data conformity check.

level

string

Error, information or warning. Error is blocking, Warning and information are not blocking.

job_conformity fields

Fields
Type
Description

task_name

string

Name of the task being validated.

analyzed_value

string / number / bool

Output of the automatic analysis (optional).

corrected_value

string / number / bool

Manually corrected value (optional).

corrected_date

ISO 8601 string

When correction was made (nullable).

message

string

Explanation, error, or note about the result.

is_conformity_validated

boolean

if this task is considered validate.

Example

{
   "id": "a71805b6-93f2-4ea3-9e42-a46b397ad448",
   "name": "test-BB6D70A6",
   "site_id": "433a60f6-9756-473e-b1e9-9e89e24bb787",
   "create_date": "2025-03-19T14:21:40.105568+00:00",
   "update_date": "2025-03-19T14:21:40.125857+00:00",
   "metadata": {
      "created_with": "mobile_app"
   },
   "progress_score": null,
   "work_items": [
      {
         "id": "4e5b8dd8-9d50-4888-977b-4064e4b6a357",
         "name": "l4_civil_open",
         "input": [
            {
               "id": "9a739279-0904-4710-b740-53971660f2ae",
               "metadata": {
                  "timestamp_ms": 1742394153532
               },
               "data_conformity": [],
               "job_conformity": [
                  {
                     "task_name": "l4_civil_open_geolocation_maps",
                     "analyzed_value": "UNAVAILABLE",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "UNAVAILABLE",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_open_ctx",
                     "analyzed_value": true,
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "OK",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_open_geolocation",
                     "analyzed_value": "UNAVAILABLE",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "UNAVAILABLE",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_open_earthing",
                     "analyzed_value": false,
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "KO",
                     "is_conformity_validated": null
                  }
               ]
            }
         ]
      },
      {
         "id": "d43a33f6-a4b2-42fc-8865-0c689f2d1c36",
         "name": "l4_civil_close",
         "input": [
            {
               "id": "9fe06e48-e29e-4735-92df-82bcb8827b51",
               "metadata": {
                  "timestamp_ms": 1742394107294
               },
               "data_conformity": [],
               "job_conformity": [
                  {
                     "task_name": "l4_civil_close_type",
                     "analyzed_value": "UNAVAILABLE",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "UNAVAILABLE",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_close_size",
                     "analyzed_value": "existing",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "existing",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_close_geolocation",
                     "analyzed_value": "UNAVAILABLE",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "UNAVAILABLE",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_close_ctx",
                     "analyzed_value": true,
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "OK",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_close_pon_reference",
                     "analyzed_value": "CMIB-G-X-4NOZAJ",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "PON Reference: CMIB-G-X-4NOZAJ",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_close_geolocation_maps",
                     "analyzed_value": "UNAVAILABLE",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "UNAVAILABLE",
                     "is_conformity_validated": null
                  }
               ]
            }
         ]
      },
      {
         "id": "09f90b26-eb2b-44b7-a235-8e6973a6306f",
         "name": "l4_civil_open_floor_closeup",
         "input": [
            {
               "id": "ac58cbf8-d144-408b-9b23-ee78c72e479d",
               "metadata": {
                  "timestamp_ms": 1742394787589
               },
               "data_conformity": [],
               "job_conformity": [
                  {
                     "task_name": "l4_civil_open_floor_state",
                     "analyzed_value": "ANOMALY",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "ANOMALY",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_open_bottom_state_validation",
                     "analyzed_value": false,
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "KO",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_open_floor_closeup_ctx",
                     "analyzed_value": true,
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "OK",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_open_duct_count",
                     "analyzed_value": 0.0,
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "0.0",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_open_floor_closeup_geolocation",
                     "analyzed_value": "UNAVAILABLE",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "UNAVAILABLE",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_open_floor_closeup_geolocation_maps",
                     "analyzed_value": "UNAVAILABLE",
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "UNAVAILABLE",
                     "is_conformity_validated": null
                  },
                  {
                     "task_name": "l4_civil_open_duct_count_validation",
                     "analyzed_value": false,
                     "corrected_value": null,
                     "corrected_date": null,
                     "message": "KO",
                     "is_conformity_validated": null
                  }
               ]
            }
         ]
      }
   ]
}

Work order state data format

This format is the same as the timeline data format, except that to simplify the analysis, there is only one item in the input array. The input exported is always the last one created.

Image data format

The images analyzed during each interval timeframe will be uploaded into the customer target cloud storage.

File name

The name of each image is composed of the input uuid and the file extension. Both information can be found in the data file in the fields input and file_extension.

Path

In the data file, within each input you have the id and file_extension fields. The absolute path to the images is therefore:

<storage_uri>/<site_id>/inputs/<id><file_extension>

# ex:
gs://customer-cloud-bucket-export/deepomatic/prod/433a60f6-9756-473e-b1e9-9e89e24bb787/inputs/67dd3441-bdee-4deb-9a48-f7663637ed78.png

See The timeline format is the default one.

See

Determines how the structured data and images will be sent to you. See

See

Structured data format
Export frequency
Connector types
Connector types
Cloud Architecture integration diagram for the cloud bucket connector