Firebase cloud function [ Error: memory limit exceeded. Function invocation was interrupted.] on youtube video upload

Question

I was trying to upload videos to youtube using the firebase cloud function.

What I need is when a user uploads a video to firebase cloud storage, functions.storage.objec

Answer 1

The only writeable part of the filesystem in Cloud Functions is the /tmp directory. As per the documentation here:

This is a local disk mount point known as a "tmpfs" volume in which data written to the volume is stored in memory. Note that it will consume memory resources provisioned for the function.

This is why you hit the memory limit with bigger files.

Your options are:

• Allocate more memory to your function (currently up to 2 GB)
• Execute the upload from an environment where you can write to filesystem. For example, your Cloud Function could call an App Engine Flexible service to execute the upload.

Answer 2

You can also use a resumable video upload following a series of steps:

1. Your GCS-triggered functions gets triggered when the video finishes the upload.
2. The function starts a resumable upload session, calculates what are reasonable chunks to upload, and inserts the chunk definitions into pubsub with the range for each chunk and the session id
3. You create a new pubsub-triggered function with that topic that receives that message, downloads the chunk from GCS using the range header (undocumented on the JSON API, but I already reported it), and uploads the chunk to Youtube

I have not tried, but this might even allow parallel uploads to Youtube from different functions uploading different chunks (which would greatly improve performance, although the docs suggest that the chunks need to be uploaded in order). You can download an arbitrary chunk from a GCS object, so the GCS side of things are not a problem for parallelization.

If parallel uploads are not allowed, you can just insert a new pubsub message when a functions finishes uploading it's chunk with the last byte uploaded, so the execution of functions is ordered (while it allows for parallel uploads of different videos).

This is a little more involved, but allows you to upload arbitrary-sized videos (up to the current 128 GB limit on Youtube) from small functions.

Take care to handle failures properly (maybe re-inserting the chunk into the pubsub topic).