Render Farm Upload Automation with Python: A paramiko and rsync Guide

The slowest part of a cloud render is often not the render. For a studio pushing a multi-camera VFX shot or a 400 GB Houdini cache to a farm every night, the bottleneck is moving the bytes — getting the project up reliably, and pulling the finished frames back down before anyone arrives in the morning. Doing that by hand, watching a progress bar at midnight, does not scale. Scripting it does.

This guide is about automating that transfer layer in Python. We run Super Renders Farm, a fully managed farm, and "fully managed" has a precise meaning for automation: you do not remote into machines, install software, or manage licenses, so the part of the pipeline you script is the part you own — the files going up and the renders coming down. The transfer surface is genuinely scriptable over SFTP, and Python is a first-class way to drive it. Our own transfer documentation names Python's paramiko library directly as a supported client, alongside rsync over SSH and the standard sftp command line.

We will be equally precise about what you cannot automate from Python today, because a pipeline built on a feature that does not exist is a pipeline that fails on its first unattended run. If you want the broader conceptual map — what "headless" versus "unattended" means, how to prepare scenes for command-line rendering — that is a separate companion guide. This one stays at the code level, focused on the transfer layer: paramiko, rsync, SSH keys, retries, and a nightly sync pattern you can drop into a studio's automation.

What you can automate in Python today

It helps to draw the boundary before writing any code. On a managed farm, some of the pipeline is fully scriptable, some of it is deliberately GUI-driven, and one part sits in between. Being honest about which is which keeps your automation from quietly breaking.

Matrix diagram showing which render-farm pipeline steps can be automated in Python on a managed cloud render farm: Upload project (fully scriptable via paramiko, rsync, sftp), Download output (fully scriptable), Detect job completion (partial — poll the SFTP output directory, no status API), Submit render job (GUI only — web form, Client App, or DCC plugin), Manage account and billing (GUI only), with green, amber, and grey status columns

• Upload a project — fully scriptable. SFTP from paramiko, rsync, or scripted sftp commands all work against our SFTP server. This is the core of what follows.
• Download finished frames — fully scriptable. Output lands in a per-job directory you can pull with the same tools.
• Detect that a job finished — partial. There is no public status API to poll. What you can do is poll the SFTP output directory and watch for the frames to appear and stabilize. It is a heuristic, not an official signal, and we treat it as such below.
• Submit a render job — GUI today. Submission runs through the web form, the SuperRenders Client App, or a per-DCC submission plugin. A public REST API for job submission, status polling, and output retrieval is on our roadmap, but at this time no public API endpoints are available for direct integration. If your pipeline is blocked specifically on a submission API, contact support and share the use case — the roadmap is shaped by real pipeline requirements.
• Manage account, credits, or billing — GUI. Out of scope for transfer automation.

So the automatable surface is transfer: push the project up, pull the renders down, and bridge the gap in between by watching the output directory. The submission step stays a deliberate handoff, and we will mark it clearly in the final pipeline rather than pretend it away. For a fuller treatment of what managed rendering does and does not expose, our guide on what a fully managed render farm is covers the model; new accounts can start from the getting-started walkthrough.

Prerequisites: SFTP access, SSH keys, and a Python environment

Three things need to be in place before the first scripted upload.

SFTP access, enabled per account. SFTP is enabled per-account on request. Sign in, look for "SFTP access" in your account settings, and generate credentials there; if it is not visible, ask support to enable it. Your credentials include a server hostname (it varies by region and storage allocation, so treat it as a value you read from configuration, never hard-code), a username tied to your account, a password or SSH key, and the standard SFTP port 22. Two paths matter: /uploads/<your-project-folder>/ is your write area, and /output/<job-id>/ is where finished renders appear.

An SSH key, not a password. For anything automated, SSH key authentication is the right choice — it keeps secrets out of your scripts and survives unattended runs without an interactive prompt. Generate a modern keypair and register the public half on your account:

ssh-keygen -t ed25519 -C "pipeline@yourstudio.example"
# add the contents of ~/.ssh/id_ed25519.pub in
# superrendersfarm.com -> Settings -> SFTP -> SSH Keys

A note on account security: two-factor authentication is not currently supported on accounts, so for SFTP the strongest hardening is a passphrase on the key file plus an SSH agent holding it for the session. The key, plus knowledge of its passphrase, plays a similar role to a second factor — possession plus secret.

A Python environment with paramiko. Everything below uses paramiko, the standard pure-Python SSH/SFTP implementation, and shells out to rsync for large incremental transfers.

python3 -m venv .venv && source .venv/bin/activate
pip install paramiko

Diagram of the SFTP account layout and key-based authentication for a managed render farm: a local pipeline machine holding an ed25519 private key connects over SSH port 22 to the farm SFTP server, which exposes two directories — /uploads/<project>/ as the write area for incoming projects and /output/<job-id>/ as the read area for finished frames; the matching public key is registered in account settings

Packaging a project so it survives an unattended upload

Most failed render jobs are not engine bugs — they are packaging gaps. A scene that renders on the artist's workstation fails on a fresh worker because a texture path points at a local-only drive, or a referenced sub-scene was never bundled. Automation amplifies this: an unattended upload of a broken package produces an unattended failure. Two rules keep packages clean.

First, make the project self-contained with relative paths. Run your DCC's collect-and-package command (Archive, Collect Files, Save Project with Assets) so every texture, proxy, and cache resolves relative to the project root. Second, mind the archive format if you compress before upload: we support tar, tar.gz, and 7z, but not .zip — repack as .tar.gz, or skip archiving entirely and let rsync transfer the folder tree, which is usually the better choice for ongoing projects anyway. As a practical ceiling, keep a single upload under ~300 GB; above that, lean on rsync with resume rather than one monolithic transfer.

Uploading a project with paramiko

The first building block is a recursive uploader. It connects with a key, walks the local project tree, recreates the directory structure under /uploads/, and puts each file. We pin host keys with RejectPolicy and read connection details from the environment so nothing sensitive lives in the script.

import os
import paramiko

def connect():
    host = os.environ["SRF_SFTP_HOST"]
    user = os.environ["SRF_SFTP_USER"]
    key_path = os.environ["SRF_SFTP_KEY"]      # path to private key

    client = paramiko.SSHClient()
    client.load_system_host_keys()             # trust ~/.ssh/known_hosts only
    client.set_missing_host_key_policy(paramiko.RejectPolicy())
    client.connect(hostname=host, port=22, username=user, key_filename=key_path)
    return client, client.open_sftp()

def _ensure_remote_dir(sftp, remote_dir):
    # mkdir -p over SFTP: build the path one segment at a time
    path = ""
    for segment in remote_dir.strip("/").split("/"):
        path += "/" + segment
        try:
            sftp.stat(path)
        except IOError:
            sftp.mkdir(path)

def upload_dir(sftp, local_dir, remote_dir):
    for root, _dirs, files in os.walk(local_dir):
        rel = os.path.relpath(root, local_dir)
        remote_root = remote_dir if rel == "." else f"{remote_dir}/{rel.replace(os.sep, '/')}"
        _ensure_remote_dir(sftp, remote_root)
        for name in files:
            local_path = os.path.join(root, name)
            remote_path = f"{remote_root}/{name}"
            sftp.put(local_path, remote_path)
            print(f"uploaded {remote_path}")

Driving it for one project:

client, sftp = connect()
try:
    upload_dir(sftp, "/local/projects/archviz-tower", "/uploads/archviz-tower-2026-06")
finally:
    sftp.close()
    client.close()

This is enough for small and mid-sized projects. sftp.put also accepts a callback= argument that receives bytes-transferred and total, which you can wire to a progress meter or a log line per file. For the large, repeated transfers that define studio work, though, rsync is the better tool.

Flow diagram of a paramiko upload routine for a render farm: a local project folder is walked file by file with os.walk, the remote directory tree is created under /uploads with an mkdir-p loop, then each file is sent with sftp.put over an SSH key-authenticated connection, with a progress callback logging each completed file

Incremental sync with rsync over SSH

A render project is rarely uploaded once. You tweak a shader, re-cache a sim, fix a light, and re-upload. Sending the whole folder each time wastes hours; rsync sends only what changed. For a studio uploading nightly, this is the single biggest time saver in the transfer layer, because it transfers the delta rather than the project.

The canonical invocation:

rsync -avz --partial --progress \
  /local/projects/archviz-tower/ \
  "$SRF_SFTP_USER@$SRF_SFTP_HOST:/uploads/archviz-tower-2026-06/"

-a preserves structure and timestamps, -z compresses in flight, --partial keeps partially transferred files so a dropped connection resumes instead of restarting, and --progress reports per-file. Re-running the same command after a change transfers only the modified files. Because the goal is automation, wrap it in Python so it lives in the same script as everything else, and so you can react to its exit code:

import subprocess

def rsync_up(local_dir, remote_subdir):
    host = os.environ["SRF_SFTP_HOST"]
    user = os.environ["SRF_SFTP_USER"]
    dest = f"{user}@{host}:/uploads/{remote_subdir}/"
    cmd = ["rsync", "-avz", "--partial", "--progress",
           f"{local_dir.rstrip('/')}/", dest]
    subprocess.run(cmd, check=True)   # raises CalledProcessError on failure

To run it unattended, schedule it. A studio mirroring a working directory to the farm every night at 1 a.m. needs one cron line:

0 1 * * *  cd /studio/pipeline && /usr/bin/python3 nightly_sync.py >> sync.log 2>&1

For rsync over SSH to authenticate without a prompt, point it at your key with -e "ssh -i ~/.ssh/id_ed25519", or let an SSH agent hold the unlocked key for the session.

Before-and-after diagram contrasting a full re-upload with an rsync incremental sync to a render farm: on the left, every file in a project is re-sent each night; on the right, rsync compares local and remote and transfers only the changed files (one modified shader and one new cache), with the unchanged bulk skipped, illustrating the delta-only transfer that makes nightly studio uploads fast

Downloading finished frames automatically

When a job completes, output frames are written to /output/<job-id>/ on the SFTP server. The download side mirrors the upload side — a recursive get with paramiko, or an rsync pull. The paramiko version walks the remote directory and recreates it locally:

import stat

def download_dir(sftp, remote_dir, local_dir):
    os.makedirs(local_dir, exist_ok=True)
    for entry in sftp.listdir_attr(remote_dir):
        remote_path = f"{remote_dir}/{entry.filename}"
        local_path = os.path.join(local_dir, entry.filename)
        if stat.S_ISDIR(entry.st_mode):
            download_dir(sftp, remote_path, local_path)
        else:
            sftp.get(remote_path, local_path)
            print(f"downloaded {local_path}")

For large output sets, the rsync pull is again the more efficient and resumable choice:

rsync -avz --progress \
  "$SRF_SFTP_USER@$SRF_SFTP_HOST:/output/<job-id>/" \
  /local/downloads/<job-id>/

One operational detail matters for unattended pipelines: rendered output is retained for 45 days after job completion, then automatically deleted. SFTP does not extend that window. The safe pattern is a nightly sync that mirrors output to local archive as soon as it appears, so retention is never the thing standing between you and your frames.

Detecting job completion without a status API

This is where the honest boundary becomes a concrete engineering choice. There is no public endpoint to ask "is job 12345 done?" — but the output directory itself is observable over SFTP. The pragmatic pattern is to poll /output/<job-id>/, count the files, and wait for the count to reach the expected frame total and hold steady across consecutive checks (so you do not start downloading mid-write).

import time

def wait_for_output(sftp, output_dir, expected_frames, poll=120, stable_checks=2):
    last_count, stable = -1, 0
    while True:
        try:
            files = sftp.listdir(output_dir)
        except IOError:
            files = []                      # folder not created yet -> not started
        count = len(files)
        if count >= expected_frames and count == last_count:
            stable += 1
            if stable >= stable_checks:
                return files                # count met and steady -> treat as done
        else:
            stable = 0
        last_count = count
        time.sleep(poll)

Be clear-eyed about what this is. Frames appear incrementally, so presence alone is not completion; checking the count against the expected total and confirming it is stable between polls is what makes it reliable enough for automation. It is a directory heuristic, not a contract. When the public API ships, this whole function collapses into a status call — until then, watching the output directory is the grounded way to bridge the gap, and it depends on nothing that does not already exist.

Sequence diagram of polling the SFTP output directory to detect render completion without a status API: a pipeline script repeatedly lists /output/<job-id>/, compares the frame count to the expected total, waits until the count both meets the target and stays stable across two consecutive checks, then proceeds to download — with an early empty-folder state shown as job-not-started

Putting it together: an unattended transfer pipeline

The pieces compose into a single nightly script. The shape is: sync the project up, hand off to submission, wait for output to appear and settle, pull it down, verify, and archive. The submission step is the deliberate GUI handoff — on a managed farm you submit through the web form, the Client App, or a DCC submission plugin, and the per-DCC plugins can be driven from the host application's own scripting environment (MAXScript, Python inside the DCC) when submission lives inside a tool you already script. We mark that step honestly rather than wrapping it in a function that pretends an API exists.

def nightly_pipeline(project_dir, remote_subdir, job_id, expected_frames):
    client, sftp = connect()
    try:
        # with_retries() (defined in the next section) wraps the fragile network calls
        with_retries(lambda: rsync_up(project_dir, remote_subdir))    # 1. push delta up
        # 2. SUBMIT: GUI / Client App / DCC plugin -- not a public API (yet)
        files = wait_for_output(                                      # 3. watch output dir
            sftp, f"/output/{job_id}", expected_frames)
        with_retries(lambda:                                          # 4. pull finished frames
            download_dir(sftp, f"/output/{job_id}", f"/local/downloads/{job_id}"))
        print(f"job {job_id}: {len(files)} frames retrieved")
    finally:
        sftp.close()
        client.close()

Steps 1, 3, and 4 are fully automated; step 2 is the handoff. When a public submission API arrives, steps 2 and 3 become API calls and the directory poll retires. The architecture does not change — only the submission and status legs move from GUI and heuristic to endpoint.

End-to-end flow diagram of an unattended render-farm transfer pipeline driven from Python: step 1 rsync uploads the project delta to /uploads, step 2 is a clearly marked GUI handoff where the job is submitted via web form, Client App, or DCC plugin (no public API), step 3 polls /output/<job-id> until frames are complete and stable, step 4 downloads and archives the finished frames locally — with automated steps shown in cyan and the manual submission step shown in grey

Error handling, retries, and resumable transfers

Unattended means no one is watching when a transfer hiccups, so the script has to recover on its own. Three habits cover most failures.

Retry transient failures with backoff. Network blips and brief disconnects are normal over long transfers. Wrap the fragile calls — as nightly_pipeline does above — so a single drop does not kill the run. Catch the specific transient errors rather than everything: paramiko's SSHException, the OSError family for sockets, and CalledProcessError for a failed rsync.

def with_retries(fn, attempts=3, backoff=5):
    transient = (paramiko.SSHException, OSError, subprocess.CalledProcessError)
    for i in range(1, attempts + 1):
        try:
            return fn()
        except transient:                  # retry SSH / network / rsync hiccups
            if i == attempts:
                raise
            time.sleep(backoff * i)        # back off: 5s, 10s, 15s

Lean on resumability. rsync --partial already resumes interrupted files, and re-running an rsync is idempotent — it only sends what is missing — so a retried sync is cheap, not a restart. For paramiko transfers, a retry plus a re-walk achieves the same effect because already-present files transfer near-instantly.

Handle host-key and connectivity errors explicitly. A "host key verification failed" error means the cached key in ~/.ssh/known_hosts no longer matches the server's — most often after a rare host-key rotation. Remove the stale line the error names and reconnect to accept the new key. Connection refused or timeout usually means a studio firewall is blocking outbound TCP 22; allow it or ask support about alternatives. And if throughput sits well below your link rate, SFTP's per-packet overhead is the cause on long-haul links — lftp with parallel segments, or several concurrent SFTP sessions, recovers most of the gap.

Summary: what to automate, and how

The transfer layer is the part of a managed-farm pipeline you own in code, and Python covers all of it.

Automate the upload, automate the download, bridge the middle by watching the output directory, and keep the submission handoff explicit. That gives you a nightly pipeline that is honest about its seams and reliable because of it. For large simulation-heavy projects where transfer reliability matters most — multi-terabyte Houdini caches and the like — the same patterns scale directly on Super Renders Farm; our Houdini cloud render farm page covers that workload.

FAQ

Q: Which Python library should I use to upload to the render farm? A: paramiko is the standard choice and is named directly in our SFTP documentation as a supported client. It is pure Python, handles SFTP cleanly, and works well for upload and download logic. For very large or frequently repeated transfers, shell out to rsync over SSH from Python with subprocess — it only sends changed files and resumes interrupted ones, which paramiko does not do natively.

Q: Is there a public API to submit render jobs from my Python pipeline? A: Not yet. A public REST API for submission, status polling, and output retrieval is on our roadmap, but no public endpoints are available today. Current programmatic submission paths are the SuperRenders Client App and the per-DCC submission plugin, which integrates with the host application's own scripting environment such as MAXScript or Python inside the DCC. If your pipeline is blocked specifically on a public submission API, contact support and share the use case — the roadmap is informed by real pipeline requirements.

Q: How do I detect that a render job finished if there is no status API? A: Poll the job's SFTP output directory, /output/<job-id>/, and watch the frame count. Treat the job as complete only when the count reaches the expected total and stays stable across consecutive checks, so you do not start downloading while frames are still being written. It is a directory heuristic rather than an official status signal, but it relies only on capabilities that exist today.

Q: Should I use SSH keys or a password for automated transfers? A: Use an SSH key. Hard-coding a password in a script is a security risk, and key authentication runs unattended without an interactive prompt. Generate an ed25519 key, register the public half in Settings → SFTP → SSH Keys, and protect the private key with a passphrase held by an SSH agent. Since two-factor authentication is not currently supported on accounts, the key plus its passphrase is the strongest practical hardening for SFTP access.

Q: Can I upload a .zip archive from my script? A: No — .zip archives are not supported. Repack as .tar.gz (or .tar / .7z), or skip archiving and let rsync transfer the folder tree directly, which is usually the better option for projects that change between uploads. Keep a single upload under roughly 300 GB and use rsync --partial for anything larger so a dropped connection resumes rather than restarts.

Q: How large a project can I move this way? A: Multi-terabyte transfers are supported over SFTP; the practical limit is your own upload bandwidth, not a farm-imposed cap. A 1 TB upload at 100 Mbps takes roughly a day, so plan around your link. For maximum throughput on fat or long-haul connections, use lftp with parallel segments or several concurrent SFTP sessions, since a single SFTP stream is limited by per-packet overhead.

Q: How long are my rendered frames available to download? A: Output is retained for 45 days after a job completes, then automatically deleted, and SFTP does not extend that window. For an unattended pipeline, mirror output to local archive as soon as it appears — a nightly rsync pull of /output/<job-id>/ keeps retention from ever becoming the reason a frame is lost.

Q: How is this different from your headless and unattended workflow guide? A: That guide is the conceptual map — what headless rendering means, how to prepare scenes for command-line rendering, and how the unattended loop fits together on a managed farm. This one is the code-level companion focused on the transfer layer: the actual paramiko and rsync you write to move projects up and frames down. Read the workflow guide for the shape; use this one for the implementation.