Multipart Upload

Upload large OAA payloads using chunked multipart requests

Overview

For OAA payloads that exceed 100 MB after compression, you can use multipart upload to split the payload into smaller chunks and upload them sequentially. The Veza platform assembles the chunks server-side before processing.

Multipart upload is useful when:

• The payload is too large for a single HTTP request

• Network reliability is a concern for large transfers

• You need to monitor upload progress for very large data sources

The standard :push endpoint supports payloads up to 100 MB (compressed or uncompressed). Use multipart upload when your payload approaches or exceeds this limit, or when you need more reliable transfer for large payloads.

How it works

1. Start the upload by sending a start request — the server creates an upload session and returns an upload_id.

2. Upload chunks sequentially, each base64-encoding a raw slice of the payload. Include the upload_id from step 1 and an incrementing sequence_number.

3. Finalize by sending a complete request with the upload_id and total sequence_count.

Each chunk must be base64-encoded from raw bytes before sending. The server decodes and reassembles chunks in sequence order when the complete operation is received.

API endpoint

POST /api/v1/providers/custom/{id}/datasources/{data_source_id}:parts

Start request body

Field	Type	Description
operation	string	Must be start

The response returns an upload_id (UUID string) to use in all subsequent requests for this upload session.

Chunk request body

Field	Type	Description
operation	string	Must be upload
upload_id	string	UUID returned by the start response
sequence_number	integer	Sequence number for this chunk, starting from 1. Maximum 99 chunks per upload.
data	string	Base64-encoded chunk of the raw payload

Finalization request body

Send a separate POST to the same endpoint after all chunks are uploaded:

Field	Type	Description
operation	string	Must be complete
upload_id	string	UUID from the start response
sequence_count	integer	Total number of chunks uploaded

Example

PAYLOAD_FILE="large_payload.json"
CHUNK_SIZE=10485760  # 10 MB per chunk

# 1. Start the upload session
RESPONSE=$(curl -s -X POST "https://$VEZA_URL/api/v1/providers/custom/$PROVIDER_ID/datasources/$DATASOURCE_ID:parts" \
  -H "authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"operation":"start"}')
UPLOAD_ID=$(echo "$RESPONSE" | python3 -c "import sys,json; print(json.load(sys.stdin)['upload_id'])")

# 2. Split into raw chunks and upload each
SEQUENCE=0
split -b $CHUNK_SIZE "$PAYLOAD_FILE" /tmp/oaa_chunk_
for CHUNK_FILE in $(ls /tmp/oaa_chunk_* | sort); do
  SEQUENCE=$((SEQUENCE + 1))
  ENCODED=$(base64 -w 0 < "$CHUNK_FILE")  # use -b 0 on macOS

  curl -s -X POST "https://$VEZA_URL/api/v1/providers/custom/$PROVIDER_ID/datasources/$DATASOURCE_ID:parts" \
    -H "authorization: Bearer $API_KEY" \
    -H "Content-Type: application/json" \
    -d "{
      \"operation\": \"upload\",
      \"upload_id\": \"$UPLOAD_ID\",
      \"sequence_number\": $SEQUENCE,
      \"data\": \"$ENCODED\"
    }"
done

# 3. Finalize the upload
curl -s -X POST "https://$VEZA_URL/api/v1/providers/custom/$PROVIDER_ID/datasources/$DATASOURCE_ID:parts" \
  -H "authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"operation\": \"complete\",
    \"upload_id\": \"$UPLOAD_ID\",
    \"sequence_count\": $SEQUENCE
  }"

rm /tmp/oaa_chunk_*

Using the Python SDK

The oaaclient Python SDK handles multipart upload automatically when enabled:

from oaaclient.client import OAAClient

client = OAAClient(url=veza_url, token=veza_api_key)
client.enable_multipart = True

# Push as usual - the SDK automatically chunks large payloads
client.push_application(provider_name, data_source_name, application_object=app)

When enable_multipart is set to True, the SDK automatically switches to multipart for payloads exceeding 50 MB (uncompressed) and:

• Splits the payload into chunks and base64-encodes each one individually

• Starts an upload session and tracks the upload_id

• Uploads each chunk sequentially with the correct sequence_number

• Sends the completion operation to trigger server-side assembly

Notes

• All chunks for a given upload_id must be uploaded before sending the completion operation.

• You can re-upload a chunk with the same sequence_number and upload_id if needed — the last upload for a given sequence number is used. All chunks must be present before the completion operation is sent.

• Chunk order is determined by sequence_number, not upload order.