Actions14
- Marketplace Dataset Actions
- Web Scraper Actions
- Web Unlocker Actions
Overview
The "Deliver Snapshot" operation of the Marketplace Dataset resource in this node enables users to deliver a snapshot of data from Bright Data's marketplace to various external storage or messaging services. This operation supports multiple delivery targets such as cloud storage providers (Amazon S3, Aliyun OSS, Google Cloud Storage, Microsoft Azure), messaging systems (Google Cloud PubSub), SFTP servers, Snowflake data warehouse, and webhooks.
This node is beneficial when you want to automate the export or distribution of dataset snapshots for further processing, archiving, or integration with other systems. For example:
- Delivering a snapshot file to an Amazon S3 bucket for backup or downstream analytics.
- Sending snapshot data to a Google Cloud PubSub topic for real-time processing.
- Uploading snapshot files to an SFTP server for secure transfer.
- Delivering data directly into a Snowflake stage for immediate querying.
Properties
| Name | Meaning |
|---|---|
| Snapshot ID | The unique identifier of the snapshot to be delivered. |
| Deliver Type | The target service/type where the snapshot will be delivered. Options: Aliyun Object Storage Service, Amazon S3, Google Cloud PubSub, Google Cloud Storage, Microsoft Azure, SFTP, Snowflake, Webhook. |
| Webhook Endpoint | URL of the webhook endpoint to which the snapshot will be delivered (only shown if Deliver Type is Webhook). |
| Filename Template | Template string defining the filename for the delivered snapshot, supporting placeholders. Required for all delivery types except possibly webhook. |
| File Extension | File extension/format for the delivered file. Options: JSON, JSONL, CSV. Required for all delivery types except possibly webhook. |
| Topic ID | Google PubSub topic ID where the snapshot message will be published (required for Google Cloud PubSub delivery). |
| Client Email | Client email credential for Google Cloud authentication (required for Google Cloud PubSub and Google Cloud Storage). |
| Private Key | Private key credential for Google Cloud authentication (required for Google Cloud PubSub and Google Cloud Storage). |
| Attributes | JSON object containing additional attributes to include in the PubSub message (optional, Google Cloud PubSub only). |
| Container | Name of the Azure storage container (required for Microsoft Azure delivery). |
| Bucket | Name of the storage bucket (required for Amazon S3, Aliyun OSS, Google Cloud Storage). |
| AWS Access Key | AWS Access Key ID credential (required for Amazon S3 delivery). |
| AWS Secret Key | AWS Secret Access Key credential (required for Amazon S3 delivery). |
| Access Key | Access key credential for Aliyun OSS (required for Aliyun OSS delivery). |
| Secret Key | Secret key credential for Aliyun OSS (required for Aliyun OSS delivery). |
| Account | Azure storage account name (required for Microsoft Azure delivery). |
| Key | Azure storage key credential (required for Microsoft Azure delivery). |
| SAS Token | Azure SAS token for access (required for Microsoft Azure delivery). |
| Role ARN | Optional AWS role ARN for Amazon S3 delivery. |
| External ID | Optional external ID for AWS role assumption in Amazon S3 delivery. |
| Directory | Target directory/path inside the storage or delivery location (applicable for most delivery types). |
| Region | AWS region for Amazon S3 or Aliyun OSS delivery. |
| Host | SFTP server hostname (required for SFTP delivery). |
| Port | SFTP server port number (default 22). |
| Path | Remote path on the SFTP server to store the file. |
| Username | Username credential for SFTP authentication (required for SFTP delivery). |
| Password | Password credential for SFTP authentication (required for SFTP delivery). |
| SSH Key | SSH private key credential for SFTP authentication (required for SFTP delivery). |
| Passphrase | Passphrase for the SSH key if applicable (optional for SFTP delivery). |
| Database | Snowflake database name (required for Snowflake delivery). |
| Schema | Snowflake schema name (required for Snowflake delivery). |
| Stage | Snowflake stage name (required for Snowflake delivery). |
| Role | Snowflake role (required for Snowflake delivery). |
| Warehouse | Snowflake warehouse (required for Snowflake delivery). |
| Snowflake Account | Snowflake account credential (required for Snowflake delivery). |
| Snowflake User | Snowflake user credential (required for Snowflake delivery). |
| Snowflake Password | Snowflake password credential (required for Snowflake delivery). |
| Compress | Boolean flag indicating whether to compress the delivered snapshot in gzip format. |
Output
The node outputs the response from the Bright Data API after requesting the snapshot delivery. The output is structured as JSON data representing the status or result of the delivery request.
If the delivery involves binary data (e.g., files), the node itself does not output the binary content but rather triggers the delivery to the specified external system.
Dependencies
- Requires an API key credential for authenticating with Bright Data's API.
- Depending on the chosen delivery type, additional credentials are required:
- Cloud storage keys and secrets for Amazon S3, Aliyun OSS, Google Cloud Storage, Microsoft Azure.
- Authentication credentials for SFTP (username/password or SSH key).
- Snowflake connection credentials.
- Google Cloud service account credentials (client email and private key) for PubSub and GCS.
- Network connectivity to the respective external services.
- Proper configuration of permissions and roles in the external services to allow writing/uploading data.
Troubleshooting
- Invalid Credentials: Errors related to authentication failures usually indicate incorrect or missing credentials for the selected delivery type. Verify that all required keys, tokens, and passwords are correctly provided.
- Permission Denied: If the node cannot write to the target bucket/container/stage, check that the credentials have sufficient permissions.
- Invalid Snapshot ID: Providing a wrong or non-existent snapshot ID will cause errors. Ensure the snapshot ID is correct and accessible.
- Network Issues: Connectivity problems to external services (SFTP, cloud storage) can cause timeouts or failures.
- Missing Required Parameters: Some delivery types require specific parameters (e.g., Topic ID for PubSub, Host for SFTP). Make sure all required fields are filled.
- File Naming Errors: Incorrect filename templates or unsupported extensions may cause delivery failures.
- Compression Issues: If compression is enabled but the receiving end does not support it, data might be unusable.