evaluation_jobs

Creates, updates, deletes, gets or lists a evaluation_jobs resource.

Overview

Name	evaluation_jobs
Type	Resource
Id	google.datalabeling.evaluation_jobs

Fields

Name	Datatype	Description
name	string	Output only. After you create a job, Data Labeling Service assigns a name to the job with the following format: "projects/{project_id}/evaluationJobs/ {evaluation_job_id}"
description	string	Required. Description of the job. The description can be up to 25,000 characters long.
annotationSpecSet	string	Required. Name of the AnnotationSpecSet describing all the labels that your machine learning model outputs. You must create this resource before you create an evaluation job and provide its name in the following format: "projects/{project_id}/annotationSpecSets/{annotation_spec_set_id}"
attempts	array	Output only. Every time the evaluation job runs and an error occurs, the failed attempt is appended to this array.
createTime	string	Output only. Timestamp of when this evaluation job was created.
evaluationJobConfig	object	Configures specific details of how a continuous evaluation job works. Provide this configuration when you create an EvaluationJob.
labelMissingGroundTruth	boolean	Required. Whether you want Data Labeling Service to provide ground truth labels for prediction input. If you want the service to assign human labelers to annotate your data, set this to true. If you want to provide your own ground truth labels in the evaluation job's BigQuery table, set this to false.
modelVersion	string	Required. The AI Platform Prediction model version to be evaluated. Prediction input and output is sampled from this model version. When creating an evaluation job, specify the model version in the following format: "projects/{project_id}/models/{model_name}/versions/{version_name}" There can only be one evaluation job per model version.
schedule	string	Required. Describes the interval at which the job runs. This interval must be at least 1 day, and it is rounded to the nearest day. For example, if you specify a 50-hour interval, the job runs every 2 days. You can provide the schedule in crontab format or in an English-like format. Regardless of what you specify, the job will run at 10:00 AM UTC. Only the interval from this schedule is used, not the specific time of day.
state	string	Output only. Describes the current state of the job.

Methods

Name	Accessible by	Required Params	Description
projects_evaluation_jobs_get	SELECT	evaluationJobsId, projectsId	Gets an evaluation job by resource name.
projects_evaluation_jobs_list	SELECT	projectsId	Lists all evaluation jobs within a project with possible filters. Pagination is supported.
projects_evaluation_jobs_create	INSERT	projectsId	Creates an evaluation job.
projects_evaluation_jobs_delete	DELETE	evaluationJobsId, projectsId	Stops and deletes an evaluation job.
projects_evaluation_jobs_patch	UPDATE	evaluationJobsId, projectsId	Updates an evaluation job. You can only update certain fields of the job's EvaluationJobConfig: humanAnnotationConfig.instruction, exampleCount, and exampleSamplePercentage. If you want to change any other aspect of the evaluation job, you must delete the job and create a new one.
projects_evaluation_jobs_pause	EXEC	evaluationJobsId, projectsId	Pauses an evaluation job. Pausing an evaluation job that is already in a PAUSED state is a no-op.
projects_evaluation_jobs_resume	EXEC	evaluationJobsId, projectsId	Resumes a paused evaluation job. A deleted evaluation job can't be resumed. Resuming a running or scheduled evaluation job is a no-op.

SELECT examples

Lists all evaluation jobs within a project with possible filters. Pagination is supported.

SELECT
name,
description,
annotationSpecSet,
attempts,
createTime,
evaluationJobConfig,
labelMissingGroundTruth,
modelVersion,
schedule,
state
FROM google.datalabeling.evaluation_jobs
WHERE projectsId = '{{ projectsId }}';

INSERT example

Use the following StackQL query and manifest file to create a new evaluation_jobs resource.

All Properties

/*+ create */
INSERT INTO google.datalabeling.evaluation_jobs (
projectsId,
job
)
SELECT 
'{{ projectsId }}',
'{{ job }}'
;

Manifest

- name: your_resource_model_name
  props:
    - name: job
      value:
        - name: name
          value: string
        - name: description
          value: string
        - name: state
          value: string
        - name: schedule
          value: string
        - name: modelVersion
          value: string
        - name: evaluationJobConfig
          value:
            - name: imageClassificationConfig
              value:
                - name: annotationSpecSet
                  value: string
                - name: allowMultiLabel
                  value: boolean
                - name: answerAggregationType
                  value: string
            - name: boundingPolyConfig
              value:
                - name: annotationSpecSet
                  value: string
                - name: instructionMessage
                  value: string
            - name: textClassificationConfig
              value:
                - name: allowMultiLabel
                  value: boolean
                - name: annotationSpecSet
                  value: string
                - name: sentimentConfig
                  value:
                    - name: enableLabelSentimentSelection
                      value: boolean
            - name: inputConfig
              value:
                - name: textMetadata
                  value:
                    - name: languageCode
                      value: string
                - name: gcsSource
                  value:
                    - name: inputUri
                      value: string
                    - name: mimeType
                      value: string
                - name: bigquerySource
                  value:
                    - name: inputUri
                      value: string
                - name: dataType
                  value: string
                - name: annotationType
                  value: string
                - name: classificationMetadata
                  value:
                    - name: isMultiLabel
                      value: boolean
            - name: evaluationConfig
              value:
                - name: boundingBoxEvaluationOptions
                  value:
                    - name: iouThreshold
                      value: number
            - name: humanAnnotationConfig
              value:
                - name: instruction
                  value: string
                - name: annotatedDatasetDisplayName
                  value: string
                - name: annotatedDatasetDescription
                  value: string
                - name: labelGroup
                  value: string
                - name: languageCode
                  value: string
                - name: replicaCount
                  value: integer
                - name: questionDuration
                  value: string
                - name: contributorEmails
                  value:
                    - string
                - name: userEmailAddress
                  value: string
            - name: bigqueryImportKeys
              value: object
            - name: exampleCount
              value: integer
            - name: exampleSamplePercentage
              value: number
            - name: evaluationJobAlertConfig
              value:
                - name: email
                  value: string
                - name: minAcceptableMeanAveragePrecision
                  value: number
        - name: annotationSpecSet
          value: string
        - name: labelMissingGroundTruth
          value: boolean
        - name: attempts
          value:
            - - name: attemptTime
                value: string
              - name: partialFailures
                value:
                  - - name: code
                      value: integer
                    - name: message
                      value: string
                    - name: details
                      value:
                        - object
        - name: createTime
          value: string

UPDATE example

Updates a evaluation_jobs resource.

/*+ update */
UPDATE google.datalabeling.evaluation_jobs
SET 
name = '{{ name }}',
description = '{{ description }}',
state = '{{ state }}',
schedule = '{{ schedule }}',
modelVersion = '{{ modelVersion }}',
evaluationJobConfig = '{{ evaluationJobConfig }}',
annotationSpecSet = '{{ annotationSpecSet }}',
labelMissingGroundTruth = true|false,
attempts = '{{ attempts }}'
WHERE 
evaluationJobsId = '{{ evaluationJobsId }}'
AND projectsId = '{{ projectsId }}';

DELETE example

Deletes the specified evaluation_jobs resource.

/*+ delete */
DELETE FROM google.datalabeling.evaluation_jobs
WHERE evaluationJobsId = '{{ evaluationJobsId }}'
AND projectsId = '{{ projectsId }}';