Workflow performance evaluation

The perf command is used to compute and display performance scores per task. It calculates both the preview score for tasks associated with app version in production and the same tasks when running the appointed workflow in the local environment. Consider that for doing the analysis we select only the work_orders tagged with "ai_performance_review" and "review completed".

The command generates a JSON file (perf_results_data.json) in the missions/your_project/workflow_v2/perf_tests directory. The structure of the JSON file looks like this:

[
    {
        "work_order": "70199838-ead7-4c3a-a710-4931a2941295",
        "task_group": "cable",
        "task_name": "check_timestamp",
        "task_groundtruth": true,
        "task_value": {
            "result": false
        },
        "input_path": [
            "image_input:perf_tests/perf_test_2024-09-06_12-08-50/521483884/cable/cable_1.jpg"
        ]
    },
    {
        "work_order": "70199838-ead7-4c3a-a710-4931a2941295",
        "task_group": "form",
        "task_name": "form",
        "task_groundtruth": true,
        "task_value": {
            "result": true
        },
        "input_path": [
            "image_input:perf_tests/perf_test_2024-09-06_12-08-50/521483884/form/form_1.jpg"
        ]
    }
]

In the output example:

• "task_groundtruth" corresponds to an analyzed or corrected value if such a value exists in production.

• "task_value" is the result of the task executed locally.

Command Usage:

To execute the command and generate performance results, use the following:

wf_client perf --org vodafone-es --site_id '6bc48feb-35fc-4d91-94b9-13ee07b9ff18' --app_version_id 'f8a59d70-efd8-4eba-aa32-a4fc717ca902' --cwd ../missions/.../workflow_v2 --env ../.env --pdb

Parameters:

• --org: Organization identifier.

• --site_id: The unique ID of the site.

• --app_version_id: The version ID of the app in production where you want to run the analysis.

• --cwd: The working directory path for your local project, where the workflow to be analyzed is located.

• --env: Path to the environment file containing project-specific environment variables.

• --pdb: Enables Python Debugger (optional, if debugging is needed).