Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to ICU tokenizer #939

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Use ICU system package

d585a63
Select commit
Loading
Failed to load commit list.
Open

Switch to ICU tokenizer #939

Use ICU system package
d585a63
Select commit
Loading
Failed to load commit list.
firefoxci-taskcluster / clean-corpus-opus-ada83_v1-ru-en succeeded Nov 22, 2024 in 5m 41s

FirefoxCI (pull_request)

Clean opus ada83_v1 dataset ru-en use OpusCleaner true

Details

View task in Taskcluster | View logs in Taskcluster | View task group in Taskcluster

Task Status

Started: 2024-11-22T22:55:01.482Z
Resolved: 2024-11-22T22:56:12.325Z
Task Execution Time: 1 minute, 10 seconds, 843 milliseconds
Task Status: completed
Reason Resolved: completed
RunId: 0

Artifacts

- public/build/ada83_v1.en.zst
- public/build/ada83_v1.ru-en.filters.json
- public/build/ada83_v1.ru.zst
- public/logs/live_backing.log
- public/logs/live.log


[taskcluster 2024-11-22 22:55:01.573Z] Task ID: R12XdGPpRoCqwNhtpLocRw
[taskcluster 2024-11-22 22:55:01.573Z] Worker ID: 4891173128478963054
[taskcluster 2024-11-22 22:55:01.573Z] Worker Group: us-central1-b
[taskcluster 2024-11-22 22:55:01.573Z] Worker Node Type: projects/887720501152/machineTypes/n2-highmem-32
[taskcluster 2024-11-22 22:55:01.573Z] Worker Pool: translations-1/b-linux-large-gcp-300gb
[taskcluster 2024-11-22 22:55:01.573Z] Worker Version: 38.0.5
[taskcluster 2024-11-22 22:55:01.573Z] Public IP: 34.56.6.144
[taskcluster 2024-11-22 22:55:01.573Z] Hostname: translations-1-b-linux-large-gcp-300gb-nebgdu5etayv9h0ofgiocq
[taskcluster 2024-11-22 22:55:01.573Z] using cache "translations-level-1-checkouts-v3-7afeb851dd97df8f3607-KnyIE1GvSz67R9mjL97Now" -> /builds/worker/checkouts

[taskcluster 2024-11-22 22:55:04.867Z] Downloading artifact "public/image.tar.zst" from task ID: KnyIE1GvSz67R9mjL97Now.
[taskcluster 2024-11-22 22:55:09.870Z] Download Progress: 90.53%
[taskcluster 2024-11-22 22:55:10.416Z] Downloaded artifact successfully.
[taskcluster 2024-11-22 22:55:10.416Z] Downloaded 287.207 mb
[taskcluster 2024-11-22 22:55:10.417Z] Decompressing downloaded image
[taskcluster 2024-11-22 22:55:12.473Z] Loading docker image from downloaded archive.
[taskcluster 2024-11-22 22:55:26.870Z] Image 'public/image.tar.zst' from task 'KnyIE1GvSz67R9mjL97Now' loaded.  Using image ID sha256:d31e1900b8212f46ff27eab4217df610f5d7a124bb4975b4b8ea07a64443f3ba.
[taskcluster 2024-11-22 22:55:27.039Z] === Task Starting ===
[setup 2024-11-22T22:55:36.631Z] run-task started in /builds/worker
[setup 2024-11-22T22:55:36.631Z] Invoked by command: --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c pip install -r $VCS_PATH/pipeline/clean/requirements/clean.txt && if [ ${USE_OPUSCLEANER} == "true" ]; then dir="clean/opuscleaner"; else dir="clean"; fi && $VCS_PATH/pipeline/${dir}/clean-corpus.sh $MOZ_FETCHES_DIR/ada83_v1 $TASK_WORKDIR/artifacts/ada83_v1 auto opus_ada83/v1 ${OPUSCLEANER_MODE} 2>&1

...(2606 lines hidden)...

[task 2024-11-22T22:56:03.693Z] 120550K .......... .......... .......... .......... .......... 94%  230M 0s
[task 2024-11-22T22:56:03.693Z] 120600K .......... .......... .......... .......... .......... 94%  193M 0s
[task 2024-11-22T22:56:03.693Z] 120650K .......... .......... .......... .......... .......... 94%  223M 0s
[task 2024-11-22T22:56:03.693Z] 120700K .......... .......... .......... .......... .......... 94%  236M 0s
[task 2024-11-22T22:56:03.694Z] 120750K .......... .......... .......... .......... .......... 94%  242M 0s
[task 2024-11-22T22:56:03.694Z] 120800K .......... .......... .......... .......... .......... 94%  213M 0s
[task 2024-11-22T22:56:03.694Z] 120850K .......... .......... .......... .......... .......... 94%  239M 0s
[task 2024-11-22T22:56:03.694Z] 120900K .......... .......... .......... .......... .......... 94%  235M 0s
[task 2024-11-22T22:56:03.695Z] 120950K .......... .......... .......... .......... .......... 94%  222M 0s
[task 2024-11-22T22:56:03.695Z] 121000K .......... .......... .......... .......... .......... 94%  198M 0s
[task 2024-11-22T22:56:03.695Z] 121050K .......... .......... .......... .......... .......... 94%  232M 0s
[task 2024-11-22T22:56:03.695Z] 121100K .......... .......... .......... .......... .......... 94%  223M 0s
[task 2024-11-22T22:56:03.695Z] 121150K .......... .......... .......... .......... .......... 94%  232M 0s
[task 2024-11-22T22:56:03.696Z] 121200K .......... .......... .......... .......... .......... 94%  212M 0s
[task 2024-11-22T22:56:03.696Z] 121250K .......... .......... .......... .......... .......... 94%  240M 0s
[task 2024-11-22T22:56:03.696Z] 121300K .......... .......... .......... .......... .......... 94%  233M 0s
[task 2024-11-22T22:56:03.696Z] 121350K .......... .......... .......... .......... .......... 94%  231M 0s
[task 2024-11-22T22:56:03.696Z] 121400K .......... .......... .......... .......... .......... 94%  209M 0s
[task 2024-11-22T22:56:03.697Z] 121450K .......... .......... .......... .......... .......... 94%  234M 0s
[task 2024-11-22T22:56:03.697Z] 121500K .......... .......... .......... .......... .......... 94%  240M 0s
[task 2024-11-22T22:56:03.697Z] 121550K .......... .......... .......... .......... .......... 94%  245M 0s
[task 2024-11-22T22:56:03.697Z] 121600K .......... .......... .......... .......... .......... 94%  218M 0s
[task 2024-11-22T22:56:03.697Z] 121650K .......... .......... .......... .......... .......... 94%  249M 0s
[task 2024-11-22T22:56:03.698Z] 121700K .......... .......... .......... .......... .......... 94%  239M 0s
[task 2024-11-22T22:56:03.698Z] 121750K .......... .......... .......... .......... .......... 95%  245M 0s
[task 2024-11-22T22:56:03.698Z] 121800K .......... .......... .......... .......... .......... 95%  204M 0s
[task 2024-11-22T22:56:03.698Z] 121850K .......... .......... .......... .......... .......... 95%  230M 0s
[task 2024-11-22T22:56:03.699Z] 121900K .......... .......... .......... .......... .......... 95%  234M 0s
[task 2024-11-22T22:56:03.699Z] 121950K .......... .......... .......... .......... .......... 95%  242M 0s
[task 2024-11-22T22:56:03.699Z] 122000K .......... .......... .......... .......... .......... 95%  202M 0s
[task 2024-11-22T22:56:03.699Z] 122050K .......... .......... .......... .......... .......... 95%  238M 0s
[task 2024-11-22T22:56:03.699Z] 122100K .......... .......... .......... .......... .......... 95%  237M 0s
[task 2024-11-22T22:56:03.700Z] 122150K .......... .......... .......... .......... .......... 95%  233M 0s
[task 2024-11-22T22:56:03.700Z] 122200K .......... .......... .......... .......... .......... 95%  202M 0s
[task 2024-11-22T22:56:03.700Z] 122250K .......... .......... .......... .......... .......... 95%  239M 0s
[task 2024-11-22T22:56:03.700Z] 122300K .......... .......... .......... .......... .......... 95%  232M 0s
[task 2024-11-22T22:56:03.701Z] 122350K .......... .......... .......... .......... .......... 95%  239M 0s
[task 2024-11-22T22:56:03.701Z] 122400K .......... .......... .......... .......... .......... 95%  220M 0s
[task 2024-11-22T22:56:03.701Z] 122450K .......... .......... .......... .......... .......... 95%  256M 0s
[task 2024-11-22T22:56:03.701Z] 122500K .......... .......... .......... .......... .......... 95%  240M 0s
[task 2024-11-22T22:56:03.701Z] 122550K .......... .......... .......... .......... .......... 95%  241M 0s
[task 2024-11-22T22:56:03.702Z] 122600K .......... .......... .......... .......... .......... 95%  201M 0s
[task 2024-11-22T22:56:03.702Z] 122650K .......... .......... .......... .......... .......... 95%  251M 0s
[task 2024-11-22T22:56:03.702Z] 122700K .......... .......... .......... .......... .......... 95%  244M 0s
[task 2024-11-22T22:56:03.702Z] 122750K .......... .......... .......... .......... .......... 95%  225M 0s
[task 2024-11-22T22:56:03.702Z] 122800K .......... .......... .......... .......... .......... 95%  212M 0s
[task 2024-11-22T22:56:03.703Z] 122850K .......... .......... .......... .......... .......... 95%  241M 0s
[task 2024-11-22T22:56:03.703Z] 122900K .......... .......... .......... .......... .......... 95%  260M 0s
[task 2024-11-22T22:56:03.703Z] 122950K .......... .......... .......... .......... .......... 95%  239M 0s
[task 2024-11-22T22:56:03.703Z] 123000K .......... .......... .......... .......... .......... 95%  206M 0s
[task 2024-11-22T22:56:03.703Z] 123050K .......... .......... .......... .......... .......... 96%  254M 0s
[task 2024-11-22T22:56:03.704Z] 123100K .......... .......... .......... .......... .......... 96%  245M 0s
[task 2024-11-22T22:56:03.704Z] 123150K .......... .......... .......... .......... .......... 96%  252M 0s
[task 2024-11-22T22:56:03.704Z] 123200K .......... .......... .......... .......... .......... 96%  223M 0s
[task 2024-11-22T22:56:03.704Z] 123250K .......... .......... .......... .......... .......... 96%  231M 0s
[task 2024-11-22T22:56:03.704Z] 123300K .......... .......... .......... .......... .......... 96%  235M 0s
[task 2024-11-22T22:56:03.705Z] 123350K .......... .......... .......... .......... .......... 96%  246M 0s
[task 2024-11-22T22:56:03.705Z] 123400K .......... .......... .......... .......... .......... 96% 89.8M 0s
[task 2024-11-22T22:56:03.707Z] 123450K .......... .......... .......... .......... .......... 96% 29.5M 0s
[task 2024-11-22T22:56:03.707Z] 123500K .......... .......... .......... .......... .......... 96%  213M 0s
[task 2024-11-22T22:56:03.707Z] 123550K .......... .......... .......... .......... .......... 96%  233M 0s
[task 2024-11-22T22:56:03.707Z] 123600K .......... .......... .......... .......... .......... 96%  214M 0s
[task 2024-11-22T22:56:03.708Z] 123650K .......... .......... .......... .......... .......... 96% 57.0M 0s
[task 2024-11-22T22:56:03.709Z] 123700K .......... .......... .......... .......... .......... 96%  217M 0s
[task 2024-11-22T22:56:03.709Z] 123750K .......... .......... .......... .......... .......... 96%  228M 0s
[task 2024-11-22T22:56:03.710Z] 123800K .......... .......... .......... .......... .......... 96% 37.3M 0s
[task 2024-11-22T22:56:03.710Z] 123850K .......... .......... .......... .......... .......... 96%  149M 0s
[task 2024-11-22T22:56:03.711Z] 123900K .......... .......... .......... .......... .......... 96%  230M 0s
[task 2024-11-22T22:56:03.711Z] 123950K .......... .......... .......... .......... .......... 96%  228M 0s
[task 2024-11-22T22:56:03.711Z] 124000K .......... .......... .......... .......... .......... 96%  156M 0s
[task 2024-11-22T22:56:03.711Z] 124050K .......... .......... .......... .......... .......... 96%  226M 0s
[task 2024-11-22T22:56:03.712Z] 124100K .......... .......... .......... .......... .......... 96%  246M 0s
[task 2024-11-22T22:56:03.712Z] 124150K .......... .......... .......... .......... .......... 96%  249M 0s
[task 2024-11-22T22:56:03.712Z] 124200K .......... .......... .......... .......... .......... 96%  190M 0s
[task 2024-11-22T22:56:03.712Z] 124250K .......... .......... .......... .......... .......... 96%  222M 0s
[task 2024-11-22T22:56:03.712Z] 124300K .......... .......... .......... .......... .......... 97%  252M 0s
[task 2024-11-22T22:56:03.713Z] 124350K .......... .......... .......... .......... .......... 97%  227M 0s
[task 2024-11-22T22:56:03.713Z] 124400K .......... .......... .......... .......... .......... 97%  216M 0s
[task 2024-11-22T22:56:03.714Z] 124450K .......... .......... .......... .......... .......... 97% 73.8M 0s
[task 2024-11-22T22:56:03.714Z] 124500K .......... .......... .......... .......... .......... 97%  230M 0s
[task 2024-11-22T22:56:03.715Z] 124550K .......... .......... .......... .......... .......... 97%  243M 0s
[task 2024-11-22T22:56:03.715Z] 124600K .......... .......... .......... .......... .......... 97% 53.6M 0s
[task 2024-11-22T22:56:03.715Z] 124650K .......... .......... .......... .......... .......... 97% 79.1M 0s
[task 2024-11-22T22:56:03.716Z] 124700K .......... .......... .......... .......... .......... 97%  230M 0s
[task 2024-11-22T22:56:03.716Z] 124750K .......... .......... .......... .......... .......... 97%  106M 0s
[task 2024-11-22T22:56:03.716Z] 124800K .......... .......... .......... .......... .......... 97%  215M 0s
[task 2024-11-22T22:56:03.717Z] 124850K .......... .......... .......... .......... .......... 97%  121M 0s
[task 2024-11-22T22:56:03.717Z] 124900K .......... .......... .......... .......... .......... 97%  242M 0s
[task 2024-11-22T22:56:03.718Z] 124950K .......... .......... .......... .......... .......... 97% 33.9M 0s
[task 2024-11-22T22:56:03.719Z] 125000K .......... .......... .......... .......... .......... 97%  192M 0s
[task 2024-11-22T22:56:03.719Z] 125050K .......... .......... .......... .......... .......... 97%  233M 0s
[task 2024-11-22T22:56:03.719Z] 125100K .......... .......... .......... .......... .......... 97%  223M 0s
[task 2024-11-22T22:56:03.719Z] 125150K .......... .......... .......... .......... .......... 97%  219M 0s
[task 2024-11-22T22:56:03.720Z] 125200K .......... .......... .......... .......... .......... 97%  207M 0s
[task 2024-11-22T22:56:03.720Z] 125250K .......... .......... .......... .......... .......... 97%  240M 0s
[task 2024-11-22T22:56:03.720Z] 125300K .......... .......... .......... .......... .......... 97%  227M 0s
[task 2024-11-22T22:56:03.720Z] 125350K .......... .......... .......... .......... .......... 97%  224M 0s
[task 2024-11-22T22:56:03.720Z] 125400K .......... .......... .......... .......... .......... 97%  194M 0s
[task 2024-11-22T22:56:03.721Z] 125450K .......... .......... .......... .......... .......... 97%  231M 0s
[task 2024-11-22T22:56:03.721Z] 125500K .......... .......... .......... .......... .......... 97%  223M 0s
[task 2024-11-22T22:56:03.721Z] 125550K .......... .......... .......... .......... .......... 97%  240M 0s
[task 2024-11-22T22:56:03.721Z] 125600K .......... .......... .......... .......... .......... 98%  211M 0s
[task 2024-11-22T22:56:03.722Z] 125650K .......... .......... .......... .......... .......... 98%  243M 0s
[task 2024-11-22T22:56:03.722Z] 125700K .......... .......... .......... .......... .......... 98%  238M 0s
[task 2024-11-22T22:56:03.722Z] 125750K .......... .......... .......... .......... .......... 98%  224M 0s
[task 2024-11-22T22:56:03.722Z] 125800K .......... .......... .......... .......... .......... 98%  195M 0s
[task 2024-11-22T22:56:03.722Z] 125850K .......... .......... .......... .......... .......... 98%  238M 0s
[task 2024-11-22T22:56:03.723Z] 125900K .......... .......... .......... .......... .......... 98%  240M 0s
[task 2024-11-22T22:56:03.723Z] 125950K .......... .......... .......... .......... .......... 98%  246M 0s
[task 2024-11-22T22:56:03.723Z] 126000K .......... .......... .......... .......... .......... 98%  201M 0s
[task 2024-11-22T22:56:03.723Z] 126050K .......... .......... .......... .......... .......... 98%  223M 0s
[task 2024-11-22T22:56:03.723Z] 126100K .......... .......... .......... .......... .......... 98%  232M 0s
[task 2024-11-22T22:56:03.724Z] 126150K .......... .......... .......... .......... .......... 98%  258M 0s
[task 2024-11-22T22:56:03.724Z] 126200K .......... .......... .......... .......... .......... 98%  218M 0s
[task 2024-11-22T22:56:03.724Z] 126250K .......... .......... .......... .......... .......... 98%  229M 0s
[task 2024-11-22T22:56:03.724Z] 126300K .......... .......... .......... .......... .......... 98%  241M 0s
[task 2024-11-22T22:56:03.725Z] 126350K .......... .......... .......... .......... .......... 98%  252M 0s
[task 2024-11-22T22:56:03.725Z] 126400K .......... .......... .......... .......... .......... 98%  203M 0s
[task 2024-11-22T22:56:03.725Z] 126450K .......... .......... .......... .......... .......... 98%  253M 0s
[task 2024-11-22T22:56:03.725Z] 126500K .......... .......... .......... .......... .......... 98%  239M 0s
[task 2024-11-22T22:56:03.725Z] 126550K .......... .......... .......... .......... .......... 98%  251M 0s
[task 2024-11-22T22:56:03.726Z] 126600K .......... .......... .......... .......... .......... 98%  209M 0s
[task 2024-11-22T22:56:03.726Z] 126650K .......... .......... .......... .......... .......... 98%  125M 0s
[task 2024-11-22T22:56:03.726Z] 126700K .......... .......... .......... .......... .......... 98%  224M 0s
[task 2024-11-22T22:56:03.726Z] 126750K .......... .......... .......... .......... .......... 98%  234M 0s
[task 2024-11-22T22:56:03.727Z] 126800K .......... .......... .......... .......... .......... 98%  215M 0s
[task 2024-11-22T22:56:03.727Z] 126850K .......... .......... .......... .......... .......... 98%  236M 0s
[task 2024-11-22T22:56:03.727Z] 126900K .......... .......... .......... .......... .......... 99%  209M 0s
[task 2024-11-22T22:56:03.727Z] 126950K .......... .......... .......... .......... .......... 99%  222M 0s
[task 2024-11-22T22:56:03.728Z] 127000K .......... .......... .......... .......... .......... 99% 91.5M 0s
[task 2024-11-22T22:56:03.728Z] 127050K .......... .......... .......... .......... .......... 99%  234M 0s
[task 2024-11-22T22:56:03.728Z] 127100K .......... .......... .......... .......... .......... 99%  232M 0s
[task 2024-11-22T22:56:03.728Z] 127150K .......... .......... .......... .......... .......... 99%  245M 0s
[task 2024-11-22T22:56:03.729Z] 127200K .......... .......... .......... .......... .......... 99% 65.0M 0s
[task 2024-11-22T22:56:03.729Z] 127250K .......... .......... .......... .......... .......... 99%  231M 0s
[task 2024-11-22T22:56:03.730Z] 127300K .......... .......... .......... .......... .......... 99%  226M 0s
[task 2024-11-22T22:56:03.730Z] 127350K .......... .......... .......... .......... .......... 99%  243M 0s
[task 2024-11-22T22:56:03.730Z] 127400K .......... .......... .......... .......... .......... 99%  115M 0s
[task 2024-11-22T22:56:03.731Z] 127450K .......... .......... .......... .......... .......... 99%  152M 0s
[task 2024-11-22T22:56:03.731Z] 127500K .......... .......... .......... .......... .......... 99%  154M 0s
[task 2024-11-22T22:56:03.731Z] 127550K .......... .......... .......... .......... .......... 99%  234M 0s
[task 2024-11-22T22:56:03.732Z] 127600K .......... .......... .......... .......... .......... 99% 51.8M 0s
[task 2024-11-22T22:56:03.732Z] 127650K .......... .......... .......... .......... .......... 99%  225M 0s
[task 2024-11-22T22:56:03.732Z] 127700K .......... .......... .......... .......... .......... 99%  237M 0s
[task 2024-11-22T22:56:03.733Z] 127750K .......... .......... .......... .......... .......... 99%  226M 0s
[task 2024-11-22T22:56:03.733Z] 127800K .......... .......... .......... .......... .......... 99% 67.5M 0s
[task 2024-11-22T22:56:03.734Z] 127850K .......... .......... .......... .......... .......... 99%  184M 0s
[task 2024-11-22T22:56:03.734Z] 127900K .......... .......... .......... .......... .......... 99%  236M 0s
[task 2024-11-22T22:56:03.735Z] 127950K .......... .......... .......... .......... .......... 99% 50.7M 0s
[task 2024-11-22T22:56:03.735Z] 128000K .......... .......... .......... .......... .......... 99%  194M 0s
[task 2024-11-22T22:56:03.735Z] 128050K .......... .......... .......... .......... .......... 99%  220M 0s
[task 2024-11-22T22:56:03.736Z] 128100K .......... .......... .......... .......... .......... 99%  241M 0s
[task 2024-11-22T22:56:03.736Z] 128150K .......... .......... .......... .........            100%  224M=0.9s
[task 2024-11-22T22:56:03.736Z] 
[task 2024-11-22T22:56:03.736Z] 2024-11-22 22:56:03 (134 MB/s) - ‘/builds/worker/.local/lib/python3.10/site-packages/opuscleaner/filters/large.bin’ saved [131266198/131266198]
[task 2024-11-22T22:56:03.736Z] 
[task 2024-11-22T22:56:03.737Z] + echo '### Generating cleaning config: opus_ada83/v1.ru-en.filters.json'
[task 2024-11-22T22:56:03.737Z] ### Generating cleaning config: opus_ada83/v1.ru-en.filters.json
[task 2024-11-22T22:56:03.737Z] + filter_path=/builds/worker/artifacts/ada83_v1.ru-en.filters.json
[task 2024-11-22T22:56:03.737Z] + python3 generate_filters.py /builds/worker/fetches/ada83_v1 ru en opus_ada83/v1 /builds/worker/artifacts/ada83_v1.ru-en.filters.json custom
[task 2024-11-22T22:56:03.766Z] Using filter /builds/worker/checkouts/vcs/pipeline/clean/opuscleaner/configs/default.filters.json
[task 2024-11-22T22:56:03.770Z] + test -s /builds/worker/artifacts/ada83_v1.ru-en.filters.json
[task 2024-11-22T22:56:03.770Z] + echo '### Cleaning /builds/worker/fetches/ada83_v1 with filter /builds/worker/artifacts/ada83_v1.ru-en.filters.json'
[task 2024-11-22T22:56:03.770Z] ### Cleaning /builds/worker/fetches/ada83_v1 with filter /builds/worker/artifacts/ada83_v1.ru-en.filters.json
[task 2024-11-22T22:56:03.771Z] + opuscleaner-clean --parallel 32 --batch-size=50000 --input=- /builds/worker/artifacts/ada83_v1.ru-en.filters.json ru en
[task 2024-11-22T22:56:03.771Z] ++ zstdmt -dc /builds/worker/fetches/ada83_v1.ru.zst
[task 2024-11-22T22:56:03.771Z] + cut -f2
[task 2024-11-22T22:56:03.771Z] + paste /dev/fd/63 /dev/fd/62
[task 2024-11-22T22:56:03.771Z] + tee /dev/fd/63
[task 2024-11-22T22:56:03.771Z] ++ zstdmt -dc /builds/worker/fetches/ada83_v1.en.zst
[task 2024-11-22T22:56:03.771Z] + zstdmt
[task 2024-11-22T22:56:03.771Z] ++ cut -f1
[task 2024-11-22T22:56:03.772Z] ++ zstdmt
[task 2024-11-22T22:56:04.659Z] + echo '### Checking length of the files'
[task 2024-11-22T22:56:04.659Z] ### Checking length of the files
[task 2024-11-22T22:56:04.659Z] + test -s /builds/worker/artifacts/ada83_v1.ru.zst
[task 2024-11-22T22:56:04.659Z] + test -s /builds/worker/artifacts/ada83_v1.en.zst
[task 2024-11-22T22:56:04.659Z] ++ zstdmt -dc /builds/worker/artifacts/ada83_v1.ru.zst
[task 2024-11-22T22:56:04.660Z] ++ wc -l
[task 2024-11-22T22:56:04.663Z] + new_len_src=3988
[task 2024-11-22T22:56:04.664Z] ++ zstdmt -dc /builds/worker/artifacts/ada83_v1.en.zst
[task 2024-11-22T22:56:04.664Z] ++ wc -l
[task 2024-11-22T22:56:04.667Z] + new_len_trg=3988
[task 2024-11-22T22:56:04.667Z] ++ zstdmt -dc /builds/worker/fetches/ada83_v1.ru.zst
[task 2024-11-22T22:56:04.667Z] ++ wc -l
[task 2024-11-22T22:56:04.671Z] + orig_len_src=4122
[task 2024-11-22T22:56:04.671Z] + [[ 3988 -ge 1 ]]
[task 2024-11-22T22:56:04.671Z] + [[ 3988 -ge 1 ]]
[task 2024-11-22T22:56:04.671Z] + [[ 3988 = \3\9\8\8 ]]
[task 2024-11-22T22:56:04.671Z] + echo '### Filtered length: 3988 / 4122'
[task 2024-11-22T22:56:04.671Z] ### Filtered length: 3988 / 4122
[task 2024-11-22T22:56:04.671Z] + echo '### Clean /builds/worker/fetches/ada83_v1 is written to  /builds/worker/artifacts/ada83_v1'
[task 2024-11-22T22:56:04.671Z] ### Clean /builds/worker/fetches/ada83_v1 is written to  /builds/worker/artifacts/ada83_v1
[task 2024-11-22T22:56:04.671Z] + echo '###### Done: Cleaning corpus with OpusCleaner'
[task 2024-11-22T22:56:04.671Z] ###### Done: Cleaning corpus with OpusCleaner
[fetches 2024-11-22T22:56:04.671Z] removing /builds/worker/fetches
[fetches 2024-11-22T22:56:04.672Z] finished
[taskcluster 2024-11-22 22:56:10.972Z] === Task Finished ===
[taskcluster 2024-11-22 22:56:11.690Z] Successful task run with exit code: 0 completed in 70.118 seconds