pipelines

Creates, updates, deletes, gets or lists a pipelines resource.

Overview

Name	pipelines
Type	Resource
Id	google.datapipelines.pipelines

Fields

Name	Datatype	Description
name	string	The pipeline name. For example: projects/PROJECT_ID/locations/LOCATION_ID/pipelines/PIPELINE_ID. PROJECT_ID can contain letters ([A-Za-z]), numbers ([0-9]), hyphens (-), colons (:), and periods (.). For more information, see Identifying projects. LOCATION_ID is the canonical ID for the pipeline's location. The list of available locations can be obtained by calling google.cloud.location.Locations.ListLocations. Note that the Data Pipelines service is not available in all regions. It depends on Cloud Scheduler, an App Engine application, so it's only available in App Engine regions. * PIPELINE_ID is the ID of the pipeline. Must be unique for the selected project and location.
createTime	string	Output only. Immutable. The timestamp when the pipeline was initially created. Set by the Data Pipelines service.
displayName	string	Required. The display name of the pipeline. It can contain only letters ([A-Za-z]), numbers ([0-9]), hyphens (-), and underscores (_).
jobCount	integer	Output only. Number of jobs.
lastUpdateTime	string	Output only. Immutable. The timestamp when the pipeline was last modified. Set by the Data Pipelines service.
pipelineSources	object	Immutable. The sources of the pipeline (for example, Dataplex). The keys and values are set by the corresponding sources during pipeline creation.
scheduleInfo	object	Details of the schedule the pipeline runs on.
schedulerServiceAccountEmail	string	Optional. A service account email to be used with the Cloud Scheduler job. If not specified, the default compute engine service account will be used.
state	string	Required. The state of the pipeline. When the pipeline is created, the state is set to 'PIPELINE_STATE_ACTIVE' by default. State changes can be requested by setting the state to stopping, paused, or resuming. State cannot be changed through UpdatePipeline requests.
type	string	Required. The type of the pipeline. This field affects the scheduling of the pipeline and the type of metrics to show for the pipeline.
workload	object	Workload details for creating the pipeline jobs.

Methods

Name	Accessible by	Required Params	Description
get	SELECT	locationsId, pipelinesId, projectsId	Looks up a single pipeline. Returns a "NOT_FOUND" error if no such pipeline exists. Returns a "FORBIDDEN" error if the caller doesn't have permission to access it.
list	SELECT	locationsId, projectsId	Lists pipelines. Returns a "FORBIDDEN" error if the caller doesn't have permission to access it.
create	INSERT	locationsId, projectsId	Creates a pipeline. For a batch pipeline, you can pass scheduler information. Data Pipelines uses the scheduler information to create an internal scheduler that runs jobs periodically. If the internal scheduler is not configured, you can use RunPipeline to run jobs.
delete	DELETE	locationsId, pipelinesId, projectsId	Deletes a pipeline. If a scheduler job is attached to the pipeline, it will be deleted.
patch	UPDATE	locationsId, pipelinesId, projectsId	Updates a pipeline. If successful, the updated Pipeline is returned. Returns NOT_FOUND if the pipeline doesn't exist. If UpdatePipeline does not return successfully, you can retry the UpdatePipeline request until you receive a successful response.
run	EXEC	locationsId, pipelinesId, projectsId	Creates a job for the specified pipeline directly. You can use this method when the internal scheduler is not configured and you want to trigger the job directly or through an external system. Returns a "NOT_FOUND" error if the pipeline doesn't exist. Returns a "FORBIDDEN" error if the user doesn't have permission to access the pipeline or run jobs for the pipeline.
stop	EXEC	locationsId, pipelinesId, projectsId	Freezes pipeline execution permanently. If there's a corresponding scheduler entry, it's deleted, and the pipeline state is changed to "ARCHIVED". However, pipeline metadata is retained.

SELECT examples

Lists pipelines. Returns a "FORBIDDEN" error if the caller doesn't have permission to access it.

SELECT
name,
createTime,
displayName,
jobCount,
lastUpdateTime,
pipelineSources,
scheduleInfo,
schedulerServiceAccountEmail,
state,
type,
workload
FROM google.datapipelines.pipelines
WHERE locationsId = '{{ locationsId }}'
AND projectsId = '{{ projectsId }}';

INSERT example

Use the following StackQL query and manifest file to create a new pipelines resource.

All Properties

/*+ create */
INSERT INTO google.datapipelines.pipelines (
locationsId,
projectsId,
displayName,
name,
type,
pipelineSources,
scheduleInfo,
workload,
state,
schedulerServiceAccountEmail
)
SELECT 
'{{ locationsId }}',
'{{ projectsId }}',
'{{ displayName }}',
'{{ name }}',
'{{ type }}',
'{{ pipelineSources }}',
'{{ scheduleInfo }}',
'{{ workload }}',
'{{ state }}',
'{{ schedulerServiceAccountEmail }}'
;

Manifest

- name: your_resource_model_name
  props:
    - name: displayName
      value: string
    - name: name
      value: string
    - name: jobCount
      value: integer
    - name: type
      value: string
    - name: pipelineSources
      value: object
    - name: scheduleInfo
      value:
        - name: timeZone
          value: string
        - name: schedule
          value: string
        - name: nextJobTime
          value: string
    - name: workload
      value:
        - name: dataflowLaunchTemplateRequest
          value:
            - name: gcsPath
              value: string
            - name: validateOnly
              value: boolean
            - name: projectId
              value: string
            - name: location
              value: string
            - name: launchParameters
              value:
                - name: update
                  value: boolean
                - name: environment
                  value:
                    - name: workerZone
                      value: string
                    - name: maxWorkers
                      value: integer
                    - name: enableStreamingEngine
                      value: boolean
                    - name: machineType
                      value: string
                    - name: additionalExperiments
                      value:
                        - string
                    - name: zone
                      value: string
                    - name: bypassTempDirValidation
                      value: boolean
                    - name: kmsKeyName
                      value: string
                    - name: additionalUserLabels
                      value: object
                    - name: ipConfiguration
                      value: string
                    - name: tempLocation
                      value: string
                    - name: numWorkers
                      value: integer
                    - name: serviceAccountEmail
                      value: string
                    - name: network
                      value: string
                    - name: subnetwork
                      value: string
                    - name: workerRegion
                      value: string
                - name: parameters
                  value: object
                - name: jobName
                  value: string
                - name: transformNameMapping
                  value: object
        - name: dataflowFlexTemplateRequest
          value:
            - name: validateOnly
              value: boolean
            - name: launchParameter
              value:
                - name: update
                  value: boolean
                - name: transformNameMappings
                  value: object
                - name: parameters
                  value: object
                - name: environment
                  value:
                    - name: enableStreamingEngine
                      value: boolean
                    - name: subnetwork
                      value: string
                    - name: serviceAccountEmail
                      value: string
                    - name: numWorkers
                      value: integer
                    - name: tempLocation
                      value: string
                    - name: zone
                      value: string
                    - name: additionalUserLabels
                      value: object
                    - name: network
                      value: string
                    - name: maxWorkers
                      value: integer
                    - name: kmsKeyName
                      value: string
                    - name: flexrsGoal
                      value: string
                    - name: machineType
                      value: string
                    - name: workerZone
                      value: string
                    - name: workerRegion
                      value: string
                    - name: additionalExperiments
                      value:
                        - string
                    - name: ipConfiguration
                      value: string
                - name: containerSpecGcsPath
                  value: string
                - name: launchOptions
                  value: object
                - name: jobName
                  value: string
            - name: projectId
              value: string
            - name: location
              value: string
    - name: state
      value: string
    - name: lastUpdateTime
      value: string
    - name: createTime
      value: string
    - name: schedulerServiceAccountEmail
      value: string

UPDATE example

Updates a pipelines resource.

/*+ update */
UPDATE google.datapipelines.pipelines
SET 
displayName = '{{ displayName }}',
name = '{{ name }}',
type = '{{ type }}',
pipelineSources = '{{ pipelineSources }}',
scheduleInfo = '{{ scheduleInfo }}',
workload = '{{ workload }}',
state = '{{ state }}',
schedulerServiceAccountEmail = '{{ schedulerServiceAccountEmail }}'
WHERE 
locationsId = '{{ locationsId }}'
AND pipelinesId = '{{ pipelinesId }}'
AND projectsId = '{{ projectsId }}';

DELETE example

Deletes the specified pipelines resource.

/*+ delete */
DELETE FROM google.datapipelines.pipelines
WHERE locationsId = '{{ locationsId }}'
AND pipelinesId = '{{ pipelinesId }}'
AND projectsId = '{{ projectsId }}';