Transcoding Fails on Large Files

[jsimons256]

Environment (please complete the following information):

• OS: Ubuntu 20 on TrueNAS Scale VM
• CPU: 2 Intel Xeon 2630s, 6 Cores each, Hyperthreading Enabled
• Installation method: single server install
• Edited: All h264 encodings are turned off and all vp9 encodings are turned on This happens with all high res encodings

Describe the issue
Encodings fail on large files at high resolutions. I am running MediaCMS on a VM through TrueNas Scale with media_files and pg_dump mounted as NFS shares. I initially assigned 24 threads and 8 GB or RAM to the VM. It worked well with 1080p mp4s under 1 GB with all encodings succeeding. For 1080p encodings over 1 GB, the 720p and under encodings work while the 1080p encoding fails. I changed from allocating 8 to 12 GB or RAM and the 1080p encoding worked for ~1 GB files but failed for 2 GB files.

I also noticed that the chunks and hls folders are empty. Additionally, when I delete the mp4, the chunks in the originals folder labeled 0x_..._.mp4.mkv do not get deleted. I initially thought this was a permission error with the NFS shares, but I don't understand why allocating more RAM can fix the error. I'm also pretty sure that my setup is enough to handle these encodings. Any help would appreciated.

Logs

celery_long.log

[2023-01-06 04:06:01,801: INFO/ForkPoolWorker-7] encode_media[7c5829a6-b598-4312-bb97-f6677e41a653]: Saved 4.57
[2023-01-06 04:06:03,628: WARNING/MainProcess] Soft time limit (7200s) exceeded for encode_media[5f0daa50-f971-4c6c-b271-014ca4e6a5e1]
[2023-01-06 04:06:03,847: ERROR/ForkPoolWorker-4] Task encode_media[5f0daa50-f971-4c6c-b271-014ca4e6a5e1] raised unexpected: ValueError("The 'media_file' attribute has no file associated with it.")
Traceback (most recent call last):
File "/home/mediacms.io/mediacms/files/tasks.py", line 308, in encode_media
output = next(encoding_command)
File "/home/mediacms.io/mediacms/files/backends.py", line 50, in encode
out = process.stderr.read(10)
File "/home/mediacms.io/lib/python3.8/site-packages/billiard/pool.py", line 229, in soft_timeout_sighandler
raise SoftTimeLimitExceeded()
billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/mediacms.io/lib/python3.8/site-packages/celery/app/trace.py", line 412, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/mediacms.io/lib/python3.8/site-packages/celery/app/trace.py", line 704, in protected_call
return self.run(*args, **kwargs)
File "/home/mediacms.io/mediacms/files/tasks.py", line 335, in encode_media
encoding.save(update_fields=["status", "logs"])
File "/home/mediacms.io/mediacms/files/models.py", line 1115, in save
super(Encoding, self).save(*args, **kwargs)
File "/home/mediacms.io/lib/python3.8/site-packages/django/db/models/base.py", line 753, in save
self.save_base(using=using, force_insert=force_insert,
File "/home/mediacms.io/lib/python3.8/site-packages/django/db/models/base.py", line 801, in save_base
post_save.send(
File "/home/mediacms.io/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 177, in send
return [
File "/home/mediacms.io/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 178, in
(receiver, receiver(signal=self, sender=sender, **named))
File "/home/mediacms.io/mediacms/files/models.py", line 1554, in encoding_file_save
chunks_paths = [f.media_file.path for f in chunks]
File "/home/mediacms.io/mediacms/files/models.py", line 1554, in
chunks_paths = [f.media_file.path for f in chunks]
File "/home/mediacms.io/lib/python3.8/site-packages/django/db/models/fields/files.py", line 57, in path
self._require_file()
File "/home/mediacms.io/lib/python3.8/site-packages/django/db/models/fields/files.py", line 39, in _require_file
raise ValueError("The '%s' attribute has no file associated with it." % self.field.name)
ValueError: The 'media_file' attribute has no file associated with it.

[livvius]

I can confirm that issue

[Micrufun]

Temporary solution

A temporary workaround is disabling the Encode profiles for any profile other than 240 and 360:

https://github.com/mediacms-io/mediacms/blob/main/docs/admins_docs.md#12-video-transcoding

Add / remove resolutions and profiles by modifying the database table of Encode profiles through https://your_installation/admin/files/encodeprofile/

For example, the Active state of any profile can be toggled to enable or disable it.

Not the best solution, just a temporary workaround.

[mgogoulos]

The part on the above log

Soft time limit (7200s) exceeded for encode_media[

suggests that the specific job (or celery task more specifically) has a limit of 2hours and then fails the job, this is set as a check mechanism that a process wouldn't run for a long time. You may change this limit and restart celery_long, in this case it will take the time to complete the video encoding

files/tasks.py , under EncodingTask

@task(
    name="encode_media",
    base=EncodingTask,
    bind=True,
    queue="long_tasks",
    soft_time_limit=settings.CELERY_SOFT_TIME_LIMIT,
)
def encode_media(

set soft_time_limit to something bigger than settings.CELERY_SOFT_TIME_LIMIT , which is defined on cms/settings.py (CELERY_SOFT_TIME_LIMIT = 2 * 60 * 60)

[bouabid-ma]

I had the same problem, by increasing significantly this parameter, it works now
previously I had increased UPLOAD_MAX_SIZE

[jameskitt616]

What did you also change? I am using the default docker compose file, i set the UPLOAD_MAX_SIZE to 30GB and uploading still fails for <10GB files.