AniDB Creators - Ban and performance

[Rand-Random]

Version of Shoko Server: 4.2.2.147 (f469fb2)

The new addition of AniDB creators enhancement took 33 minutes for the database update.

[2024-09-12 00:37:20:282] Info|Shoko.Server.Databases.BaseDatabase`1 > Starting Server: Datenbank - Schema Patches anwenden...121.11 - [ScheduleAnidbCreators]
[2024-09-12 00:37:20:282] Info|Shoko.Server.Databases.DatabaseFixes > Scheduling 12634 AniDB Creators for a refresh.
[2024-09-12 01:07:07:854] Info|Shoko.Server.Services.Connectivity.ConnectivityService > Checking LAN Connectivity…
[2024-09-12 01:07:07:887] Info|Shoko.Server.Services.Connectivity.ConnectivityService > Found a local gateway to use
[2024-09-12 01:07:07:887] Info|Shoko.Server.Services.Connectivity.ConnectivityService > Checking WAN Connectivity…
[2024-09-12 01:07:07:887] Info|Shoko.Server.Services.Connectivity.ConnectivityService > Successfully connected to 4/4 internet service endpoints
[2024-09-12 01:11:07:713] Info|Shoko.Server.Databases.DatabaseFixes > Scheduled 12634 AniDB Creators took 00:33:47.4254840

Since it was taking so long, maybe consider some kind of progress indicator.
It stood on the same message for over half an hour which may make people nervous and believe shokoserver is stuck.

After the success of the datbase update, 12,634 creators were in the queue to process.
A good night sleep later, and I have a anidb udp ban.

[2024-09-12 01:12:19:485] Info|GetAniDBCreatorJob_CreatorID:225 > Processing GetAniDBCreatorJob
[2024-09-12 01:12:19:579] Info|GetAniDBCreatorJob_CreatorID:225 > Found AniDB Creator: Sakurai Takahiro (ID=225,Type=Person)
[2024-09-12 01:12:27:902] Info|GetAniDBCreatorJob_CreatorID:233 > Processing GetAniDBCreatorJob
[2024-09-12 01:12:27:982] Info|GetAniDBCreatorJob_CreatorID:233 > Found AniDB Creator: Miki Shin'ichirou (ID=233,Type=Person)
[2024-09-12 01:12:36:319] Info|GetAniDBCreatorJob_CreatorID:451 > Processing GetAniDBCreatorJob
[2024-09-12 01:12:36:427] Info|GetAniDBCreatorJob_CreatorID:451 > Found AniDB Creator: Touchi Hiroki (ID=451,Type=Person)
[2024-09-12 01:12:44:748] Info|GetAniDBCreatorJob_CreatorID:308 > Processing GetAniDBCreatorJob
[2024-09-12 01:12:44:849] Info|GetAniDBCreatorJob_CreatorID:308 > Found AniDB Creator: Nazuka Kaori (ID=308,Type=Person)
[2024-09-12 01:12:53:167] Info|GetAniDBCreatorJob_CreatorID:356 > Processing GetAniDBCreatorJob
[2024-09-12 01:12:53:249] Info|GetAniDBCreatorJob_CreatorID:356 > Found AniDB Creator: Toyoguchi Megumi (ID=356,Type=Person)
[2024-09-12 01:13:01:602] Info|GetAniDBCreatorJob_CreatorID:351 > Processing GetAniDBCreatorJob
[2024-09-12 01:13:01:656] Info|GetAniDBCreatorJob_CreatorID:351 > Found AniDB Creator: Okiayu Ryoutarou (ID=351,Type=Person)
[2024-09-12 01:13:10:199] Info|GetAniDBCreatorJob_CreatorID:374 > Processing GetAniDBCreatorJob

5 hours 30 minutes later and still 9,631 Tasks to go.

[2024-09-12 06:45:38:932] Info|GetAniDBCreatorJob_CreatorID:54405 > Found AniDB Creator: Shutou Yukina (ID=54405,Type=Person)
[2024-09-12 06:45:43:814] Info|GetAniDBCreatorJob_CreatorID:46282 > Processing GetAniDBCreatorJob
[2024-09-12 06:45:43:893] Info|GetAniDBCreatorJob_CreatorID:46282 > Found AniDB Creator: Makino Amane (ID=46282,Type=Person)
[2024-09-12 06:45:48:815] Info|GetAniDBCreatorJob_CreatorID:58276 > Processing GetAniDBCreatorJob
[2024-09-12 06:45:48:886] Info|GetAniDBCreatorJob_CreatorID:58276 > Found AniDB Creator: Sekiguchi Risa (ID=58276,Type=Person)
[2024-09-12 06:45:53:812] Info|GetAniDBCreatorJob_CreatorID:59837 > Processing GetAniDBCreatorJob
[2024-09-12 06:45:53:856] Warn|Shoko.Server.Providers.AniDB.UDP.AniDBUDPConnectionHandler > AniDB UDP Banned!
[2024-09-12 06:45:53:856] Error|Quartz.Core.JobRunShell > Job AniDB.GetAniDBCreatorJob_CreatorID:59837 threw a JobExecutionException: : Parameters: refire = False, unscheduleFiringTrigger = False, unscheduleAllTriggers = False 
 Quartz.JobExecutionException: Exception of type 'Shoko.Server.Providers.AniDB.AniDBBannedException' was thrown.
 ---> Shoko.Server.Providers.AniDB.AniDBBannedException: Exception of type 'Shoko.Server.Providers.AniDB.AniDBBannedException' was thrown.
   at Shoko.Server.Providers.AniDB.UDP.Generic.UDPRequest`1.ParseResponse(String response, Boolean returnFullResponse) in D:\a\ShokoServer\ShokoServer\Shoko.Server\Providers\AniDB\UDP\Generic\UDPRequest.cs:line 110
   at Shoko.Server.Providers.AniDB.UDP.Generic.UDPRequest`1.Send() in D:\a\ShokoServer\ShokoServer\Shoko.Server\Providers\AniDB\UDP\Generic\UDPRequest.cs:line 44
   at Shoko.Server.Scheduling.Jobs.AniDB.GetAniDBCreatorJob.Process()
   at Shoko.Server.Scheduling.Jobs.BaseJob.Execute(IJobExecutionContext context) in D:\a\ShokoServer\ShokoServer\Shoko.Server\Scheduling\Jobs\BaseJob.cs:line 32
   --- End of inner exception stack trace ---
   at Shoko.Server.Scheduling.Jobs.BaseJob.Execute(IJobExecutionContext context) in D:\a\ShokoServer\ShokoServer\Shoko.Server\Scheduling\Jobs\BaseJob.cs:line 43
   at Quartz.Core.JobRunShell.Run(CancellationToken cancellationToken) [See nested exception: Shoko.Server.Providers.AniDB.AniDBBannedException: Exception of type 'Shoko.Server.Providers.AniDB.AniDBBannedException' was thrown.
   at Shoko.Server.Providers.AniDB.UDP.Generic.UDPRequest`1.ParseResponse(String response, Boolean returnFullResponse) in D:\a\ShokoServer\ShokoServer\Shoko.Server\Providers\AniDB\UDP\Generic\UDPRequest.cs:line 110
   at Shoko.Server.Providers.AniDB.UDP.Generic.UDPRequest`1.Send() in D:\a\ShokoServer\ShokoServer\Shoko.Server\Providers\AniDB\UDP\Generic\UDPRequest.cs:line 44
   at Shoko.Server.Scheduling.Jobs.AniDB.GetAniDBCreatorJob.Process()

[Rand-Random]

Update:
Current amount of tasks still pending: 3,573.
Hope its done on tuesday.

Ban occured each day after completing ~3,000 tasks.

[harshithmohan]

It shouldn't have been scheduled at startup in the first place.
I removed it from the startup so it will not affect the stable (when it releases) and also anyone who hasn't updated their daily to that version yet.

Edit: if you don't actually need the creator (not VA, it's for other staff) images immediately, you can clear the queue. If you update the anidb info of a series, the images will be downloaded them.

[Rand-Random]

Thanks for the info, at this point I am commited to it completing.

It shouldn't have been scheduled at startup in the first place.

Okay, than should I go ahead and close the issue?

[ElementalCrisis]

Yeah we can close this issue for now, if you notice any other weird activity, we can look into it further.

[revam]

@Rand-Random just letting you know that the new optional action to schedule the get creator jobs now have a progress indicator in the log.

the new action also won't schedule the any creators which have had the job previously ran, in case anyone want (or need) to run the action at some point, then they can rest assured that it won't schedule every creator once again, as it will only schedule the ones with missing metadata.