Home » FAQ » Encoding » I'm going to submit a lot of jobs, can Zencoder handle it? Should I let you know before I start?

permalink I'm going to submit a lot of jobs, can Zencoder handle it? Should I let you know before I start?

Zencoder can definitely handle it. Unless you're going to be running in to our API rate limit of 1,000 job creation requests per minute you shouldn't need to warn us before submitting your jobs.

In fact, we're more concerned that we might overwhelm your servers if you're sending through a very large volume of jobs. The only delay you should see from Zencoder will be the time it takes to bring the necessary encoding servers online to handle your jobs if the volume exceeds our current capacity. If this happens it should take no more than a few minutes from your initial request before additional capacity is available. After that, the biggest concerns will be how your servers can handle the traffic from Zencoder. If that initial delay will impact your transcoding, get in touch with us before you start submitting so we can bring additional servers online in anticipation.

File Transfer Concerns to Consider

If you're using S3 for input and output destinations then you should be good to go, as they can handle significant transfer volume, especially within the same region.

If you're using FTP servers you may want to throttle your jobs back some and run them in batches of 50 to 500, depending on the capacity of your servers. We've had experiences with large job volume completely overwhelming FTP servers and causing jobs to fail since we can't get to the files.

Be sure to account for the necessary bandwidth if the output files are going to the same server as the input files. Downloading may work fine on the first batch of jobs but start to suffer once uploading of outputs begins.

Notification Concerns to Consider

If you are using notifications the same concern applies. Will your server be able to handle (potentially) hundreds of POSTs from Zencoder in rapid succession?

This is a bit less of a concern, as we will retry failed notifications up to 25 times, and the files will already have been uploaded. However, if your videos are on the same server, or being uploaded to the same server, that is receiving notifications, this could cause successive jobs to fail if the server becomes overwhelmed.

The best way to avoid notification overload on your server is to respond successfully to Zencoder as quickly as possible when the notification comes in. We recommend doing any processing of the body of the notification outside of the scope of the response, using a job queue or other background processing. This will ensure that Zencoder doesn't timeout the notification and attempt again a few seconds later.