Open in app

Sign in

Write

Sign in

How to automatically publish Zoom recordings to discourse.org

Oli Zimpasser
11 min readDec 22, 2021

While Zoom cloud recordings are easy to make, they are not easily accessible for other people.

We at id5 are using discourse.org within our Intranet to share information and so we want to have each Zoom cloud recording available as a discourse post, where anyone can easily access those recordings.

The solution described in this article is a Zoom cloud recording to Vimeo to discourse.org automated upload and post pipeline.

Supported features

Any meeting with a hashtag in its title and a Zoom cloud recording should be automatically uploaded to Vimeo and a discourse.org post should be created under the category of its hashtag. Those categories should be sub-categories of "Videos".

Typical usage in a Google Calendar integration with Zoom

If the meeting title also contains the hashtag #Exp it should be used to set an expiry date when the video and its discourse post will be deleted automatically.

Videos on Vimeo should be password protected.

AWS architecture

This diagram shows the architecture using AWS infrastructure:

AWS architecture and its integration

Before we dive into details, let's look at the general building blocks:

  • a Zoom cloud API webhook calls an AWS API Gateway for each finished Zoom cloud recording, which puts a message into AWS SNS
  • an AWS Lambda function is called for each SNS message, this function downloads the video file, uploads it into Vimeo and finally puts an entry into a AWS DynamoDB table
  • A second AWS Lambda function runs every 5 minutes and for each entry in the DynmoDB table it checks if Vimeo has finished the transcoding for this video. When the transcoding is completed, it creates a discourse.org post within a certain category. It might also create a new DynamoDB entry in a second table to set the date and time for an automated deletion of this video and post
  • To implement the deletion process, the second AWS Lambda function also checks against the DynamoDB table holding the expiration information. When a video is expired this lambda deletes the video on Vimeo and deletes the post in discourse.org

You might ask why the API Gateway isn't directly connected to a Lambda function. According to this AWS documentation page a lambda connected to an API Gateway cannot have more than 30 seconds of execution time, which might not be enough to download and upload the video. De-coupling the integration gives us a timeout of max 15 minutes.

Now let's look at the different components in detail.

AWS DynamoDB tables

Start with creating a table zoom-to-videoplatform-upload and another table zoom-to-videoplatform-expiry.

Both tables should be of type “On-demand” and have a partition key “videoUrl” of type String.

DynamoDB tables

AWS API Gateway and SNS

Next create a SNS topic called "zoom-to-vimeo-topic". Then create an API Gateway of type REST called "zoom-to-vimeo-gateway". Create a new resource with a path name of "ingest". Add the POST method with an integration of the SNS service to it. We will add the Auth lambda later.

API Gateway setup

Create a deployment for it and write down the endpoint URL. I have taken most of the information on how to set up the API Gateway to SNS integration from https://www.alexdebrie.com/posts/aws-api-gateway-service-proxy/, it is worth to read it as well.

AWS SNS to Lambda

Create a Lambda function using nodejs and attach it via an EventBridge to the SNS, which makes our lambda a subscription of the SNS topic.

The JavaScript code looks like this:

const AWS = require('aws-sdk');
const stream = require('stream');
const {promisify} = require('util');
const got = require('got');const pipeline = promisify(stream.pipeline);const generatePassword = require('./random-password');AWS.config.update({region: 'eu-central-1'});
const ddb = new AWS.DynamoDB({apiVersion: '2012-08-10'});const accessToken = '<<here goes the Vimeo API token>>';const prepareUpload = async (meetingTitle, accessToken, fileSize) => {
    const password = generatePassword();
    const postResponse = await got.post(
                            'https://api.vimeo.com/me/videos', 
    {
        json: {
            "upload": {
                "approach":"tus",
                "size": fileSize
            }, 
            "name": meetingTitle,
            "description": "Video uploaded on: " + new Date(),
            "password": password,
            "privacy": {
                "view": "password"
            }    
        },
        responseType: 'json',
        headers: {
            "Authorization": `Bearer ${accessToken}`,
            "Accept": "application/vnd.vimeo.*+json;version=3.4"
        }
    });
    const response = postResponse.body;
    return {
        uploadLink: response.upload.upload_link,
        videoUri: response.uri,
        videoFullLink: response.link,
        password: password
    };
}const sendHead = async (targetLocation) => {
    await got.head(targetLocation, {
        // see https://github.com/sindresorhus/got/issues/1489
        retry: 0, 
        headers: {
            "Accept": "application/vnd.vimeo.*+json;version=3.4",
            "Tus-Resumable": "1.0.0"
        }
    });
}const downloadAndUpload = async (url, targetLocation, accessToken) => { 
    await pipeline(
         got.stream(url),
         got.stream.patch(targetLocation, {
            headers: {
                "Content-Type": "application/offset+octet-stream",
                "Authorization": `Bearer ${accessToken}`,
                "Accept":   
                         "application/vnd.vimeo.*+json;version=3.4",
                "Tus-Resumable": "1.0.0",
                "Upload-Offset": "0"
            }
        })
 );
}const isNumeric = (n) => {
    return !isNaN(parseInt(n)) && isFinite(n);
}const prepareParams = (body) => {
    const meetingTitle = body.payload.object.topic;
    const {withHash} = meetingTitle.split(/\s+/).reduce((op,inp)=>{
        let hash = /^#.+/.test(inp)
        let key = hash ? 'withHash' : 'withOutHash'
        op[key] = op[key] || []
        op[key].push(inp)
        return op
        },{withHash: [],withOutHash: []})
    const expiry = withHash.filter(e => 
                                e.toLowerCase().startsWith('#exp'));
    const otherHashes = withHash.filter(e => 
                               !e.toLowerCase().startsWith('#exp'));
    if (otherHashes.length === 0) {
        // no hashtag at all means: do not upload this
        return { doNotUpload: true }
    }
    // we only look at the first hashtag
    const firstHashtag = otherHashes[0].substr(1);
    const expiryInDaysAsString = 
                             expiry[0] ? expiry[0].substr(4) : null;
    const expiryInDays = 
          isNumeric(expiryInDaysAsString) ? 
          parseInt(expiryInDaysAsString) : -1;    // find the download URL and its size for video
    const downloadToken = body.download_token;
    let url = null;
    let fileSize = -1;
    body.payload.object.recording_files.filter(e => 
          e.recording_type === 'shared_screen_with_speaker_view')
      .forEach(e => {
          url = e.download_url + "?access_token=" + downloadToken;
          fileSize = e.file_size;
    });
    return {
        meetingTitle,
        firstHashtag,
        url,
        fileSize,
        doNotUpload: false,
        expiryInDays
    }
}const writeDynamoDb = async (uploadLink, videoUri, videoFullLink, 
             password, firstHashtag, meetingTitle, expiryInDays) => {
    const dbEntry = {
        TableName: 'zoom-to-videoplatform-upload',
        Item: {
            uploadLink: {S: uploadLink},
            videoUri: {S: videoUri},
            videoFullLink: {S: videoFullLink},
            password: {S: password},
            firstHashtag: {S: firstHashtag},
            meetingTitle: {S: meetingTitle},
            expiryInDays: {N: `${expiryInDays}`}
        }
    };
    await ddb.putItem(dbEntry).promise();
}exports.handler = async (event) => {
    if (!event || 
          !event.Records || 
          !event.Records[0] || 
          !event.Records[0].Sns || 
          !event.Records[0].Sns.Message || 
          typeof event.Records[0].Sns.Message !== 'string' ) {
        return {
            statusCode: 400,
            body: JSON.stringify( 
                         'bad request (input event not valid)'),
        };
    }
    const body = JSON.parse(event.Records[0].Sns.Message);
    if (body.event !== 'recording.completed') {
        return {
            statusCode: 400,
            body: JSON.stringify(
                        'bad request (recording not completed)'),
        };
    }
    
    const { meetingTitle, firstHashtag, url, fileSize, doNotUpload, 
                             expiryInDays } = prepareParams(body);
    if (doNotUpload) {
        return {
            statusCode: 200,
            body: JSON.stringify('no hashtag found'),
        };
    }    const {uploadLink, videoUri, videoFullLink, password} = 
            await prepareUpload(meetingTitle, accessToken, fileSize);        await downloadAndUpload(url, uploadLink, accessToken);
    
    await sendHead(uploadLink);
    
    await writeDynamoDb(uploadLink, videoUri, videoFullLink, 
            password, firstHashtag, meetingTitle, expiryInDays);
    
    return {
        statusCode: 200,
        body: JSON.stringify('ok'),
    };
};

You need to make sure to upload the got npm module and a second JavaScript file called "random-password.js" which exports a function to generate a fixed or random password, like:

const generatePassword = () => {
    return "our-secret-vimeo-password";
}

module.exports = generatePassword

You have to change the timeout for this lambda to 15 minutes, also you might want to give it more memory.

As this lambda writes into the DynamoDB table "zoom-to-videoplatform-upload", it also needs more permissions. Add this to the existing role:

{
  "Effect": "Allow",
  "Action": [
      "dynamodb:PutItem"
  ],
  "Resource": [
      "<<arn of the DynamoDB table zoom-to-videoplatform-upload>>"
  ]
}

AWS Authorizer lambda

To protect your API gateway from anybody being able to add Videos to your intranet, you have to implement an Authorizer lambda function.

Create a new lambda, add this code and don't forget to change the token to your Zoom token later on.

const generatePolicy = (principalId, effect, resource) => {
    const authResponse = {};
    
    authResponse.principalId = principalId;
    if (effect && resource) {
        const policyDocument = {};
        policyDocument.Version = '2012-10-17'; 
        policyDocument.Statement = [];
        const statementOne = {};
        statementOne.Action = 'execute-api:Invoke'; 
        statementOne.Effect = effect;
        statementOne.Resource = resource;
        policyDocument.Statement[0] = statementOne;
        authResponse.policyDocument = policyDocument;
    }
    
    authResponse.context = {
    };
    return authResponse;
}exports.handler = async (event) => {
    if (!event.headers || !event.headers.Authorization) {
        throw new Error("Bad Gateway");
    }
    
    const token = event.headers.Authorization;
    if (token === '<<Your Zoom verification token goes here>>') {
        return generatePolicy('user', 'Allow', event.methodArn);
    }
    
    throw new Error("Unauthorized");
};

Add this lambda to the API Gateways Authorizers section. Then go to /ingest POST and under "Method Request" use it as an Authorization.

The 2nd AWS lambda

As shown in the diagram, this lambda has 2 jobs:

  • looking at the DynamoDB table "zoom-to-videoplatform-upload" and check for any finished transcoding on Vimeo, if so, find the right category on discourse — if this category doesn't exist yet, create it — then post the video on discourse, send a slack notification and delete the entry in "zoom-to-videoplatform-upload"
  • looking at the DynamoDB table "zoom-to-videoplatform-expiry" and for each expired entry, delete the respective Vimeo video and the discourse topic, then delete the entry in "zoom-to-videoplatform-expiry"

The full JavaScript code looks like this:

const AWS = require('aws-sdk');
const got = require('got');AWS.config.update({region: 'eu-central-1'});
const ddb = new AWS.DynamoDB({apiVersion: '2012-08-10'});const accessToken = '<<here goes the Vimeo API token>>';
const discourseToken = {
    user: 'system',
    key: '<<here goes the Discourse API token>>'
}
const discourseRoot = "https://test.trydiscourse.com"; // change!
const defaultDiscourseCategoryId = 14; // change!const notifySlack = async (meetingTitle, videoUri, password, 
                                                  topicId) => {
    const body = {
        "channel": "#zoom-recordings",
        "username": "webhookbot",
        "text": `New video ${meetingTitle} at ${videoUri} ` + 
                `(password: ${password}) - find it also ` + 
                `at ${discourseRoot}/t/${topicId}`,
        "icon_emoji": ":ghost:"
    }
    const postResponse = await got.post(
                'https://hooks.slack.com/services/xxx/xxx/xxxx', {
        form: {
            payload: JSON.stringify(body)
        }
    });
}const getOEmbed = async(videoFullLink) => {
    return got.get(`https://vimeo.com/api/oembed.json?url=${encodeURI(videoFullLink)}&width=1024`, { 
      responseType: 'json' 
    });
}const postToDiscourse = async (meetingTitle, password, embedHtml, 
                                        categoryId, expiryTime) => {
    let rawText = "Password: " + password;
    if (expiryTime) {
        rawText += "\r\n\r\nThis video will be deleted on " + expiryTime;
    }
    rawText += "\r\n\r\n" + embedHtml;
    const postReponse = await got.post(
                                    `${discourseRoot}/posts.json`, {
        json: {
            "title": `${meetingTitle} (${new Date()})`,
            "raw": rawText,
            "category": categoryId
        },
        responseType: 'json',
        headers: {
            "Api-Key": discourseToken.key,
            "Api-Username": discourseToken.user
        }
    })
    return postReponse.body.topic_id;
}const findCatoryOnDiscourse = async (firstHashtag) => {    
    const postReponse = await got.get(
                                     `${discourseRoot}/site.json`, {
        responseType: 'json',
        headers: {
            "Api-Key": discourseToken.key,
            "Api-Username": discourseToken.user
        }
    })
    let categoryId = -1;
    postReponse.body.categories.filter(e => 
            e.name.toLowerCase() === firstHashtag.toLowerCase() && 
            e.parent_category_id === defaultDiscourseCategoryId)
    .forEach(e => {
        categoryId = e.id
    });
    if (categoryId === -1) {
        const postCreateReponse = await got.post(
                              `${discourseRoot}/categories.json`, {
            json: {
                "name": firstHashtag, 
                "parent_category_id": defaultDiscourseCategoryId,
                "color": "0088CC", 
                "text_color": "FFFFFF"
            },
            responseType: 'json',
            headers: {
                "Api-Key": discourseToken.key,
                "Api-Username": discourseToken.user
            }
        })
        categoryId = postCreateReponse.body.category.id;
    }
    return categoryId;
}const deleteDynmoEntry = async (videoUri, tableName) => {
    const params = {
        TableName: tableName,
        Key: {
            'videoUri': {S: videoUri}
        }
    };
    await ddb.deleteItem(params).promise();
}const deleteVimeoVideo = async (videoUri) => {    
    await got.delete(
                     `https://api.vimeo.com${videoUri}`, {
        headers: {
            "Authorization": `Bearer ${accessToken}`
        }
    });
}const deleteDiscourseTopic = async (topicId) => {
    await got.delete(`${discourseRoot}/t/${topicId}.json`, {
        headers: {
            "Api-Key": discourseToken.key,
            "Api-Username": discourseToken.user
        }
    })
}const scanTable = async (tableName) => {
    const params = {
        TableName: tableName,
    };
    const scanResults = [];
    let items;
    do {
        items =  await ddb.scan(params).promise();
        items.Items.forEach((item) => scanResults.push(item));
        params.ExclusiveStartKey = items.LastEvaluatedKey;
    } while (typeof items.LastEvaluatedKey !== "undefined");
    
    return scanResults;
};const writeExpiraryRecord = async (videoUri, videoFullLink, topicId, 
                                      expiryInDays, expiryTime) => {
    const dbEntry = {
        TableName: 'zoom-to-videoplatform-expiry',
        Item: {
            videoUri: {S: videoUri},
            videoFullLink: {S: videoFullLink},
            topicId: {N: `${topicId}`},
            expiryInDays: {N: `${expiryInDays}`},
            expiryTime: {S: expiryTime}
        }
    };
    await ddb.putItem(dbEntry).promise();
}const processTranscodingUploads = async () => {
    const allItems = await scanTable(
                                   'zoom-to-videoplatform-upload');
    for (let i = 0 ; i < allItems.length ; i++) {
        const dbEntry = allItems[i];
        const videoUri = dbEntry.videoUri.S;        
        try {
            const statusResponse = await got.get(`https://api.vimeo.com${videoUri}?fields=uri,upload.status,transcode.status`, {
                responseType: 'json',
                headers: {
                    "Authorization": `Bearer ${accessToken}`
                }
            });
            if (
               statusResponse.body.transcode.status == 'complete') {                const videoFullLink = dbEntry.videoFullLink.S;
                const password = dbEntry.password.S;
                const firstHashtag = dbEntry.firstHashtag.S;
                const meetingTitle = dbEntry.meetingTitle.S;
                const expiryInDays = 
                                   parseInt(dbEntry.expiryInDays.N);                let expiryTime = null;
                if (expiryInDays > 0) {
                    const nowDate = new Date();
                    expiryTime = new 
                            Date(nowDate.setDate(nowDate.getDate() + 
                            expiryInDays)).toISOString();
                }
    
                const oEmbed = await getOEmbed(videoFullLink);               
                const embedHtml = oEmbed.body.html;

    
                const categoryId = await 
                                  findCatoryOnDiscourse(firstHashtag);
                
    
                const topicId = await postToDiscourse(meetingTitle, 
                       password, embedHtml, categoryId, expiryTime);
                
                if (expiryTime) {
                    await writeExpiraryRecord(videoUri, 
                             videoFullLink, topicId, expiryInDays, 
                             expiryTime);
                }
                
                await notifySlack(meetingTitle, 
                                  videoFullLink, password, topicId);
                
                await deleteDynmoEntry(videoUri, 
                                    'zoom-to-videoplatform-upload');
                
            }
        } catch (err) {
            console.log("Failed to process");
            console.log(err);
        }
    }    
}const processExpiredVideos = async () => {
    const allItems = await scanTable(
                                    'zoom-to-videoplatform-expiry');
    const nowDate = new Date();
    for (let i = 0 ; i < allItems.length ; i++) {
        const dbEntry = allItems[i];
        const expiryTimeStr = dbEntry.expiryTime.S;        
        const expiryTime = new Date(Date.parse(expiryTimeStr));
        try {
            if (expiryTime < nowDate) {
                const videoUri = dbEntry.videoUri.S;
                const topicId = dbEntry.topicId.N;                await deleteVimeoVideo(videoUri);

                await deleteDiscourseTopic(topicId);
                
                await deleteDynmoEntry(videoUri, 
                                    'zoom-to-videoplatform-expiry');
            }
        } catch (err) {
            console.log("Failed to process");
            console.log(err);
        }
    }
}exports.handler = async (event) => {    
    await processTranscodingUploads();
    await processExpiredVideos();    
    return {
        statusCode: 200,
        body: JSON.stringify('ok'),
    };
    
};

To give this Lambda function the needed permission on the DynamoDB tables, add this to the execution role of your Lambda:

{
  "Effect": "Allow",
  "Action": [
      "dynamodb:Scan",
      "dynamodb:DeleteItem"
  ],
  "Resource": [
      "<<arn of the DynamoDB table zoom-to-videoplatform-upload>>"
  ]
},
{
    "Effect": "Allow",
    "Action": [
        "dynamodb:Scan",
        "dynamodb:DeleteItem",
        "dynamodb:PutItem"
    ],
    "Resource": [
        "<<arn of the DynamoDB table zoom-to-videoplatform-expiry>>"
    ]
}

To configure it properly you need to change some values:

// this is the Vimeo Access token
const accessToken = '<<here goes the Vimeo API token>>';// this is the Disource.org Access token. Choose "User Level"
// as "All Users" and Scope to "Global"
const discourseToken = {
    user: 'system',
    key: '<<here goes the Discourse API token>>'
}// this needs to match your discourse.org domain
const discourseRoot = "https://test.trydiscourse.com";// Create a root category names "Video" and put the id here
const defaultDiscourseCategoryId = 14;

If you also want a Slack integration you have to add a Webhook integration on one of your channels and replace `https://hooks.slack.com/services/xxx/xxx/xxxx` with your endpoint URL.

Zoom integration

Under https://marketplace.zoom.us/develop/ you need to create a Webhook Only integration.

Enter all relevant data and make sure you have selected the event type "All Recordings have completed" as shown in these screenshots:

Event setup on Zoom
Zoom Webhook setup

Make sure to use the endpoint URL you got from the AWS Gateway deployment and put the Verification token into the Authorizer Lambda function.

Discourse.org configuration

To support embedded Vimeo videos on discourse.org you have to allow iframes from player.vimeo.com. Go to the administration area of discourse and add under Security a new "allowed iframes" entry:

discourse.org setting change to allow iframes vom Vimeo

AWS cost

The AWS cost to run this setup is negligible.We usually pay in the area of 0.01 to 0.1 USD per month for this. All resources are paid by usage which makes them very cheap — of course only as long as you made sure to use On-Demand DynamoDB tables.

Zoom
AWS
Discourse
Vimeo

--

--

Oli Zimpasser

Written by Oli Zimpasser

17 Followers

Did you ever want to manage a Kebab restaurant in JavaScript? https://codeyourrestaurant.com

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams