# Large File Streaming

Sydekyks user and operator guide. Check feature availability and release-specific installation requirements.

Large file streaming lets a piece process a big file by passing it through as a stream,
transferring the bytes a chunk at a time, instead of loading the whole file into memory
first. It is useful when a file is too large to hold in RAM at either the source or the
destination. Because the file is never fully buffered, a flow can move files far larger
than the worker’s memory budget.

## ​Why it matters

A worker runs your piece code inside a sandbox with a bounded memory budget (about 1 GB,
minus overhead, see Limits (https://docs.sydekyks.com/install/reference/limits)). Reading a large file into a
Buffer holds the entire file in that budget at once, so a big enough file exhausts the
memory and the worker is OOM-killed.

Streaming avoids this: the bytes flow through as a Node Readable, roughly 5 MB at a
time, so no process ever holds the whole file. The transfer is transparent: there are no
extra “chunk” steps in your flow; a streaming action looks and behaves like any other.

Reading a stored file as a stream requires S3 file storage. Moving bytes from one
third-party service to another - the common case - never touches Sydekyks storage and
works on any backend. But when the file being read already lives in Sydekyks storage,
only S3 (AP_FILE_STORAGE_LOCATION=S3) serves it incrementally: with the default database
(DB) storage the bytes cannot be read out of a column a chunk at a time, so the server
loads the whole row into memory before sending it to your action, which defeats the memory
savings. Self-hosted installs default to DB; set S3 to get the benefit. See
Set up S3 (https://docs.sydekyks.com/install/configure-operate/setup-s3).

## ​When to use it

Which approach should you use?

- Buffer (a Buffer, the default): small files of known size where holding the whole
file in memory is cheap and simple.

- Stream (a Readable): large files, files of unknown size, or app-to-app transfers
(e.g. downloading a large object from one service and uploading it to another) where
buffering would risk exhausting memory.

Typical cases: large media files, multi-hundred-MB CSV or log exports, database dumps, and
storage-to-storage transfers.

## ​Pieces that support streaming

Streaming is enabled per action. “In” means the action reads its input file as a stream;
“out” means it writes the file it produces out as a stream.

 | Piece | Streams in | Streams out

 | Amazon S3 | Upload File | Read File

 | Azure Blob Storage | Create Blob | Read Blob

 | Dropbox | Upload file | Download File

 | Google Drive | Upload file | Read File Content, List files, Set public access, New File (trigger)

 | Microsoft OneDrive | Upload file | Get File

 | Microsoft SharePoint | Upload File | N/A

 | FTP/SFTP | Upload File | Read File Content

 | Subflows | Stream CSV to Subflows | N/A

More pieces are being enabled over time. Actions not listed here still work: they buffer
the file in memory, which is fine within the size limit (https://docs.sydekyks.com/install/reference/limits).

Stream CSV to Subflows streams a CSV straight from the URL you give it and never writes the
file into storage, so it is the one entry above that does not need S3 file storage.

### ​Per-service upload ceilings

Streaming removes the memory ceiling, not the destination API’s own limit. Where a service caps
what a single request can carry, the action switches to a chunked upload session above that cap:

 | Piece | One-request cap | Above it

 | Dropbox | 150 MB (https://developers.dropbox.com/dbx-performance-guide) (/2/files/upload answers 409 payload_too_large) | /2/files/upload_session/*, 8 MiB chunks

 | Microsoft SharePoint | 250 MB (https://learn.microsoft.com/en-us/graph/api/driveitem-put-content) (Graph’s simple PUT …/content) | Graph upload session, 10 MiB chunks

 | Microsoft OneDrive | 4 MiB | Graph upload session, 10 MiB chunks

These are pre-existing API limits rather than streaming limits, and each action now handles its own.
One difference matters if your source doesn’t report a size: Dropbox’s session is offset-based, so it
just streams the chunks as they arrive, while Graph wants the file’s total length in every fragment’s
Content-Range header, so the two Microsoft actions buffer once to learn the length before they can
chunk. Give those a source that reports Content-Length when you can.

## ​Building streaming actions

If you are building a piece (https://docs.sydekyks.com/build-pieces/building-pieces/overview), you can stream on both sides: read an
input file as a stream, and write an output file as a stream.

### ​Writing a file as a stream

ctx.files.write accepts a Buffer or a Readable. Pass a Readable, such as an S3
object body or a streaming HTTP response, and the engine drains it for you, counting the
bytes against AP_MAX_FILE_SIZE_MB as they arrive. Your action never builds a Buffer
itself, though the write is not streamed end-to-end: that cap sits far below the sandbox’s
memory budget, so the upload declares a real Content-Length - which is what makes it
retryable. It returns a file reference string you return from the action, exactly like the
buffered form (see Files (https://docs.sydekyks.com/build-pieces/piece-reference/files)).

```
async run(context) {
  const s3 = await resolveS3Client({ authProps, server: context.server });

  const { Body } = await s3.getObject({ Bucket: bucket, Key: key });

  // Body is a Readable: hand it straight to files.write, no Buffer in between
  return context.files.write({
    fileName: key,
    data: Body,
  });
}

```

### ​Reading a file as a stream

Add streaming: true to a Property.File. On platforms 0.87.0 and later the property
resolves to an ApStreamingFile; older engines still deliver a buffered ApFile, so the
value is typed as ApStreamingFile | ApFile:

```
type ApStreamingFile = {
  filename: string;
  extension?: string;
  size?: number;      // may be undefined when the source doesn't report a length
  body: Readable;
};

```

Never read .body or .size off the property directly - normalize it first with
streamUtils.toStreamingBody from @activepieces/pieces-common. It passes an
ApStreamingFile through untouched and wraps an ApFile’s buffer into a Readable with an
exact size:

```
import { streamUtils } from '@activepieces/pieces-common';

const { body, size } = streamUtils.toStreamingBody(context.propsValue.file);

```

Consume body directly. How you hand it to the destination depends on what that destination’s
client accepts, in order of preference:

1. A chunking uploader (best). Accepts a stream of unknown length and buffers each part
before sending it, so no content length is needed and parts are individually replayable. For S3
that is Upload from @aws-sdk/lib-storage; for Azure Blob Storage,
blockBlobClient.uploadStream.

```
props: {
  file: Property.File({
    displayName: 'File',
    required: true,
    streaming: true,
  }),
},
async run(context) {
  const { body } = streamUtils.toStreamingBody(context.propsValue.file);
  const s3 = await resolveS3Client({ authProps, server: context.server });

  await new Upload({
    client: s3,
    params: {
      Bucket: bucket,
      Key: finalFileName,
      Body: body,
    },
  }).done();
}

```

2. An SDK that takes a stream directly. Some clients accept a Readable as-is: Google
Drive’s media.body, SFTP’s client.put. Just pass body.

3. A single-request HTTP upload. If the destination is a plain PUT/POST that needs an
explicit Content-Length, you have to use size, and size is best-effort, so this
path needs a buffered fallback for when it is missing. Dropbox, SharePoint and OneDrive all
look like this:

```
import { buffer as readableToBuffer } from 'node:stream/consumers';

const { body: streamedBody, size } = streamUtils.toStreamingBody(context.propsValue.file);
const headers: Record<string, string> = { 'Content-Type': 'application/octet-stream' };
let body;
if (size != null) {
  headers['Content-Length'] = String(size);
  body = streamedBody;
} else {
  body = await readableToBuffer(streamedBody);
}

```

size is informational and best-effort: it is undefined when the source reports no
Content-Length, and it is also dropped when the response is compressed
(Content-Encoding: gzip/br/deflate), because the decompressed body no longer matches the
advertised length. Don’t require it: prefer pattern 1 or 2, which never need it.

Property.File() without streaming is unchanged: it still resolves to an ApFile with a
data buffer, so existing actions keep working.

## ​Limits & storage

- Writing into storage is capped, and buffered by the engine. A stream passed to
ctx.files.write is counted against AP_MAX_FILE_SIZE_MB (Cloud: 10 MB, self-hosted
default: 25 MB) while the bytes flow; crossing it aborts the transfer and fails the
step before anything is uploaded. See Limits (https://docs.sydekyks.com/install/reference/limits).

- Reading a streamed input is not capped. A streaming: true file input has no
AP_MAX_FILE_SIZE_MB ceiling; that is deliberate, since the point of the feature is to move
files larger than the cap out to an external service.

- Storage backend. Streaming a file out of Sydekyks storage requires S3; see the
callout at the top of this page. Writes are capped and engine-buffered on every backend.

- Individual parts retry, the whole transfer doesn’t. A multipart uploader buffers each
~5 MB part before sending it, so it can replay that part on a transient error. But the
source Readable can be read only once, so there is no retry of the transfer as a whole,
and an action that fails after its stream is drained cannot be re-run against the same
stream - it has to fetch the source again.

- httpClient does not retry a stream body at all. The retry loop in pieces-common
reuses the body it serialized before the first attempt, and a stream is one-shot, so retrying
would replay a drained stream and send a truncated body. So a request whose body is a
Readable (or a form-data payload, which is streamed too) runs its retries setting as
0. This applies to any piece sending a stream through httpClient, not just file
actions. If you need real retries, buffer the body instead.

- Multipart webhook signatures aren’t verified. Verifying an HMAC signature over an
uploaded file needs the raw bytes held in memory, which is exactly what streaming avoids.
Streamed multipart webhook uploads therefore skip signature verification. JSON, XML, form,
and text webhook bodies are unaffected.