forked from DIRACGrid/DIRAC
-
Notifications
You must be signed in to change notification settings - Fork 0
/
release.notes
8436 lines (6141 loc) · 295 KB
/
release.notes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
[v7r2-pre6]
FIX: python 2-3 compatibility fixes
*Core
FIX: (#4283) Added protection against empty certificates directory in dirac-install
*WMS
NEW: (#4367) Added option "CompressJDLs" to JobDB, which compresses JDLs before they are
added to the JobJDLs table
*ResourceStatusSystem
CHANGE: (#4235) Removes locals() way of passing parameters
*tests
NEW: (#4179) Set up Gitlab CI pipeline using Docker containers
[v7r1-pre18]
NEW: Add environment.yml file for preparing an environment with conda
*Core
NEW: initial support for mysql8
NEW: (#4236) DISET - pass the client DIRAC version as part of the request description structure
NEW: (#4274) Allow installation from a local directory
FIX: (#4289) Use subprocess32 if possible for significantly better performance
FIX: (#4289) TypeLoader should not be a singleton as it has a cache
CHANGE: (#4362) RPC parameters are cast to list/tuples when needed
FIX: (#4410) M2Crypto transport cast socket.error to strings
CHANGE: (#4410) (JSON) rpcStub uses list instead of tuples
NEW: (#4410) add utilities for JSON Serialization (strToIntDict and ignoreEncodeWarning)
CHANGE: (#4354) ThreadPool - replaced by ThreadPoolExecutor(native python)
FIX: (#4439) MySQLDB _connect method
*Configuration
CHANGE: (#4157) add new helper methods for IdP/Proxy Providers, OAuthManager. Add methods that
read DN options in dirac user section
NEW: (#4241) Slave configuration servers are automatically updated as part of the new configuration
changes commit
NEW: (#4241) Configuration Server - added forceGlobalConfigurationUpdate() interface to force all
registered service to update their configuration
NEW: (#4257) Add search emails for group method in Registry
NEW: (#4397) ConfigurationClient - added getOptionsDictRecursively method, add docs
*Framework
CHANGE: (#4157) add possibility to work with Proxy Providers and "clean" proxy without
voms/dirac extensions
NEW: (#4257) Filter of duplicate mails in Notification
FIX: (#4257) Optimize test in NotificationHandler
FIX: (#4289) Avoid leaking std(in|out|err) file handles when running runsvdir
CHANGE: (#4362) BundleDeliveryFramework uses list instead of tuple
CHANGE: (#4410) (JSON) use of list instead of tuples for SecurityLog
*WMS
CHANGE: (#4362) SandboxStoreClient uses list instead of tuple
CHANGE: (#4396) Removed outdated PilotMonitorAgent
CHANGE: (#4410) (JSON) use of list instead of tuples for JobReport and Watchdog
CHANGE: (#4416) As explained in #4412, Completed jobs is used as major status for jobs during
completion while it is also used for jobs that have a pending request and require a further
action before being Done or Failed.
CHANGE: (#4416) Completing jobs are treated exactly like Running jobs, i.e. set Stalled if they
didn't give sign of life, and are not set Failed directly as before. Jobs to be checked
are those that had not sent a heartbeat within the stalledTime period, in order to reduce
the number of jobs to check.
NEW: (#4416) Created a file JobStatus.py for replacing the literal job statuses, and updated many
modules using these new variables. Note that there are still many others to change, but
could be done gradually...
FIX: (#4451) StalledJobAgent - use int factor to the seconds time delta
FIX: (#4451) ProxyDB - set UserName in setPersistencyFlag() if it is a first record in ProxyDB_Proxies
*DMS
CHANGE: (#4243) FileCatalog - standard components are loaded using ObjectLoader mechanism
NEW: (#4279) FileCatalog - added VOMSSecurityManager class
CHANGE: (#4279) FileCatalog - dropped PolicyBasedSecurityManager and VOMSPolicy classes
CHANGE: (#4410) (JSON) correct casts for serialization for FTS
CHANGE: (#4364) remove all the SE mangling in the DFC (SEPrefix, resolvePFN, SEDefinition)
CHANGE: (#4423) FileMetadata - split getFileMetadataFields into getFileMetadataFields and _getFileMetadataFields
CHANGE: (#4423) DirectoryMetadata - split getMetadataFields into getMetadataFields and _getMetadataFields
*RMS
CHANGE: (#4410) (JSON) correct casts for serialization
*Resources
NEW: (#4157) ProxyProvider resources with implementation for DIRAC CA and PUSP proxy providers
NEW: (#4276) IdProvider resources to represent external user identity providers
FIX: (#4455) discrepancy in CE._reset() methods return value in different CE implementations
*RSS
CHANGE: (#4362) SQL query orders using list instead of tuples
CHANGE: (#4410) (JSON) db ordering uses list instead of tuples
*StorageManagement
CHANGE: (#4410) (JSON) use lists instead of tuples
*tests
CHANGE: (#4279) use VOMSSecurityManager for integration instead of PolicyBasedSecurityManager/VOMSPolicy
FIX: (#4284) DMS client tests to use DIRAC user name, rather than the local system user name
NEW: (#4289) CI for unit/lint/integration testing with GitHub actions
NEW: (#4289) Start using shell linting of the test scripts
FIX: (#4289) Exit quickly if the installation fails to make it easier to find issues
CHANGE: (#4410) call pytest directly instead of as a module
FIX: (#4426) restored system test, moved Jenkins test to appropriate location
CHANGE: (#4439) integrate test_MySQLDB to all_integration_server_tests
*docs
NEW: (#4289) Document how to run integration tests in docker
[v7r0p15]
*Accounting
NEW: (#4464) StorageOccupancy accounting
*RSS
NEW: (#4464) FreeDiskSpaceCommand used to fill the StorageOccupancy accounting
CHANGE: (#4464) FreeDiskSpaceCommand can clean the from not-anymore-existing SEs
*WMS
CHANGE: (#4471) Optimizer: treat input sandboxes uploaded as LFNs ad Input Data
[v7r0p14]
*Framework
CHANGE: (#4458) Remove LHCb specific code used to install cx_Oracle
*Core
CHANGE: (#4460) agents exit with sys.exit(2) in case of errors
*Resources
CHANGE: (#4460) InProcess CE correctly reports number of processors available
*WMS
FIX: (#4454) PilotWrapper: try to untar also with system tar
[v7r0p13]
*Core
FIX: (#4448) SubProcess: rewritten function for getting child PIDs
*ResourceStatusSystem
NEW: (#4419) Backported GGUSTicketsPolicy from LHCb
CHANGE: (#4419) removed dangerous script dirac-rss-policy-manager
*tests
NEW: (#4429) Allow extensions to extend the continuous integration tests
*docs
NEW: (#4431) dirac-doc-get-release-notes can take a sinceLatestTag option, header and footer messages,
and can create github/gitlab release
[v7r0p12]
*WMS
FIX: (#4432) Better logging for WMS Optimizers
NEW: (#4435) PilotCS2JSONSynchronizer: added options pilotRepoBranch and pilotVORepoBranch
*DMS
FIX: (#4436) dirac-dms-clean-directory: also works on single empty directory, fixes #4420
*Framework
FIX: (#4436) dirac-install-component: fix bug prohibiting use of the -m/--module parameter
*tests
NEW: (#4429) Allow extensions to extend the continuous integration tests
NEW: (#4433) added README.rst for every subdirectory
*docs
FIX: (#4438) update notes for dirac-distribution (add extensions)
[v7r0p11]
*Core
FIX: (#4411) Make dirac-install tool python 3 compatible
NEW: (#4409) Allow --dirac-os-version argument to dirac-install.py to be a path or URL to a tarball
[v7r0p10]
*Core
FIX: (#4394) RequestHandler - fix log output for the case where the ActionTuple is actually a string
FIX: (#4394) AuthManager - manage the case where ExtraCredentials are in a form of a list
NEW: (#4379) dirac-install accepts userEnvVariables switch, adds user-requested env variables
to *rc* files
*WMS
FIX: (#4404) Fix the matching delay functionality, fixes #2983
NEW: (#4403) JobCleaningAgent: Add possibility to remove HeartBeatLoggingInfo before jobs are
completely removed.
FIX: (#4394) SiteDirector - do not add downloading files ti the pilot if Pilot3 flag is False
FIX: (#4390) Watchdog: get the CPU consumed by children processes too
FIX: (#4370) SiteDirector: Pilot3 option easier to interpret
CHANGE: (#4338) getStatus method in ARC CE now uses a JobSupervisor to get status of multiple pilots at once.
CHANGE: (#4338) SiteDirector.updatePilotStatus has been parallelized using threads.
NEW: (#4338) AvailableSlotsUpdateCycleFactor is a parameter part of the configuration allowing to control
the rate of the update of the available slots in the queues.
FIX: (#4338) Revert to queueCECache being an object attribute to fix the CEs instantiation.
*RMS
CHANGE: (#4400) ReqDB: Add pool_recycle=3600 parameter for sql engine setup, might prevent
occasional "Mysql Server has gone away" errors
*Resources
FIX: (#4380) ComputingElement classes: use GridEnv if present
CHANGE: (#4360) in HTCondor CEs, the pilot stamps are now used to retrieve outputs and logging info
*DMS
CHANGE: (#4400) FTS3DB: Add pool_recycle=3600 parameter for sql engine setup, might prevent
occasional "Mysql Server has gone away" errors
*Docs
CHANGE: (#4402) dirac-docs-get-release-notes.py does not require a GITLABTOKEN
[v7r0p9]
*Core
CHANGE: (#4302) Simplify and convert command wrapper scripts to bash. Also removed some obsolete MacOS code.
FIX: (#4361) restore the possibility to set a global timeout for a Client
*DMS
CHANGE: (#4353) FTS3 persistOperation method checks that the caller is allowed to do so
CHANGE: (#4353) FTS3Manager: do not expose updateJobStatus and updateFileStatus on the service
CHANGE: (#4353) FTS3 operation is canceled if the matching request is canceled
CHANGE: (#4353) when a source file does not exist, defunct the FTS3File associated
CHANGE: (#4353) use the JEncode serializer instead of the custom FTS3Serializer
CHANGE: (#4353) cancel an FTS3Job together with its associated FTS3Files if the job is not found on the server
*RSS
CHANGE: (#4336) remove a call to shifterProxy configuration which would not work for multi VO Dirac
*docs
CHANGE: (#4378) using docker images from docker hub
[v7r0p8]
*Configuration
CHANGE: Let update the config, even if the Pilot info is not in the CS
[v7r0p7]
*Core
CHANGE: {bash,tchs,diracos}rc set the PYTHONPATH to only dirac (ignore existing PYTHONPATH)
CHANGE: dirac-configure might work without the need of a proxy
CHANGE: the timeout of an RPC call is always updated
[v7r0p8]
*Configuration
CHANGE: Let update the config, even if the Pilot info is not in the CS
[v7r0p7]
*Core
CHANGE: {bash,tchs,diracos}rc set the PYTHONPATH to only dirac (ignore existing PYTHONPATH)
CHANGE: dirac-configure might work without the need of a proxy
CHANGE: the timeout of an RPC call is always updated
[v7r0p6]
*Core
NEW: Add environment.yml file for preparing an environment with conda
CHANGE: Use subprocess32 if possible for significantly better performance
FIX: TypeLoader should not be a singleton as it has a cache
*FrameworkSystem
FIX: Avoid leaking std(in|out|err) file handles when running runsvdir
*WorkloadManagementSystem
NEW: To activate the pilot logging mechanism the following option must be present in CS: Pilot/PilotLogging with a value set to "true", "yes" or "y".
*Tests
NEW: CI for unit/lint/integration testing with GitHub actions
NEW: Start using shell linting of the test scripts
FIX: Exit quickly if the installation fails to make it easier to find issues
*Docs
NEW: Document how to run integration tests in docker
CHANGE: The options in the command reference sections: 'ignore' and 'scripts', are replaced by 'exclude' and 'manual' respectively. The entries in 'exclude' will now reject scripts that are otherwise picked up by the 'patterns'. Entries in 'manual' will be added to the indexFile, but their rst file has to be provided by hand. Fixes #4345
[v7r0p5]
FIX: changes from v6r22p15 included
[v7r0p4]
*Core
FIX: (#4337) remove unexisting import from dirac-info
*WMS
CHANGE: (#4317) dealing with min and max number of processors from the job
*docs
CHANGE: (#4317) added WMS admin doc (job state machine)
[v7r0p3]
*Core
FIX: (#4298) install_site.sh : set UpdatePilotCStoJSONFile=False for initial installations
*WMS
FIX: (#4294) JobState - restored logic of having database clients at the class level
*TS
CHANGE: (#4308) Rename Operations option Productions/ProductionFilesMaxResetCounter to Transformations/FilesMaxResetCounter
*Resources, WMS, tests
CHANGE: (#4295) save the number of processors as jobLimits in pilot cfg
*tests
CHANGE: (#4291) removed pilot2 tests
*docs
NEW: (#4309) Added how to create a dedicated dirac file catalog
[v7r0p2]
*Accounting
FIX: (#4290) Allow longer user and site names.
*WMS
NEW: (#4287) dirac-wms-get-wn and dirac-wms-pilot-job-info scripts (ported from LHCbDIRAC)
*docs
CHANGE: (#4266) the AdministratorGuide has been restructured
[v7r0p1]
FIX: Removing some deprecated codes
[v7r0]
FIX: (#3962) use print function instead of print statement
FIX: (#3962) remove the usage of the long suffix to distinguish between long and int
FIX: (#3962) convert octal literals to new syntax
NEW: (#3962) Add pylint test to check python 3 compatibility
*Configuration
CHANGE: (#4249) The flag UpdatePilotCStoJSONFile has been moved to Operations/[]/Pilot section
NEW: (#4255) Resources - added getStorageElements() method
*Framework
CHANGE: (#4180) Removed local MySQL handling (DIRACOS won't have install it)
CHANGE: (#4180) Removed setup of old portal
*Accounting
CHANGE: (Multi)AccountingDB - Grouping Type and Object loader together with the MonitoringSystem ones.
*WorkloadManagementSystem
NEW: Add JobElasticDB.py with getJobParameters and setJobParameter methods to work with ElasticSearch (ES) backend.
NEW: Add gJobElasticDB variable and indicates the activation of ES backend.
CHANGE: Modify export_getJobParameter(s) to report values from ES if available.
CHANGE: (#3748) Reduced (removed, in fact) interactions of Optimizers with JobParameters, using only OptimizerParameters
NEW: (#3760) Add Client/JobStateUpdateClient.py
NEW: (#3760) Add Client/JobManagerClient.py
CHANGE: (#3760) Use the above Client classes instead of invoking RPCClient()
NEW: Added ES backend to WMSAdministratorHandler to get JobParameters.
NEW: Added separate MySQL table for JobsStatus in JobDB, and modified code accordingly.
CHANGE: ElasticJobDB.py: Modify setJobParameter method to register JobAttributes like Owner, Proxy, JobGroup etc.
CHANGE: ElasticJobDB.py: added getJobParametersAndAttributes method to retrieve both parameters and attributes for a given JobID.
CHANGE: Pilot Watchdog - using python UNIX services for some watchdog calls
CHANGE: (#3890) JobState always connects to DBs directly
CHANGE: (#3890) remove JobStateSync service
CHANGE: (#3873) Watchdog: use Profiler instead of ProcessMonitor
NEW: (#4163) SiteDirectors send by default Pilot3 (flag for pilot2 is still kept)
FIX: (#4163) removed unicode_literals (messes up things with DEncode)
NEW: (#4224) PilotWrapper can get the Pilot files from a list of locations
CHANGE: (#4224) PilotWrapper is compatible with several python versions
NEW: (#4260) New dirac-wms-match command to test a given job for matching computing resources
NEW: (#4260) QueueUtilities - utilities relevant to queues manipulation
FIX: (#4265) urllib.urlopen() call with and without the 'context' parameter in Utilities/PilotWrapper.py
according to the version of urllib instead of the python version
CHANGE: (#4244) Modified flag for using the JobParameters on ElasticSearch database
*Core
NEW: (#3744) Add update method to the ElasticSearchDB.py to update or if not available create the values
sent from the setJobParameter function. Uses update_by_query and index ES APIs.
CHANGE: (#3744) generateFullIndexName() method made static under the ElasticSearchDB class.
CHANGE: (#3744) removed unused/outdated stuff from Distribution module
FIX: check for empty /etc/grid-security/certificates dir
NEW: (#3842) Add createClient decorator to add wrapper for all export_ functions automatically
NEW: (#3678) dirac-install can install a non released code directly from the git repository
FIX: (#3931) AuthManager - modified to work with the case of unregistered DN in credDict
FIX: (#4182) Add CountryName OID and be more permissive in the String type for decoding VOMSExtensions
FIX: (#4211) change orders of @deprecated and @classmethod decorators in X509Chain
NEW: (#4229) dirac-install defines XRD_RUNFORKHANDLER environment variable
*TransformationSystem
NEW: (#4124) InputDataQuery and OutputDataQuery parameters can be set for Transformations before
they are added to the transformation system. Solves #4071
CHANGE: (#4238) TransformationDB.getTransformationMetaQuery returns ENOENT if no meta query exists
FIX: (#4238) InputDataAgent: silence warnings about transformations not having a input data query
*ProductionManagement
NEW: (#3703) ProductionManagement system is introduced
*Interfaces
CHANGE: (#4163) removed deprecated methods from Dirac.py
*Resources
FIX: (#4229) add proxy location as a virtual user for xroot urls
FIX: (#4234) future import in executeBatch script that prevented the pilot deployment on SSH CE, fixes #4233
FIX: (#4231) SE __executeMethod: only pass protocols parameter to getTransportURL
NEW: (#4255) New dirac-resource-info command to display computing and storage resources available
to a given VO
*ResourceStatusSystem
CHANGE: (#4177) use the ObjectLoader for Clients, in PEP
*Monitoring
FIX: MonitoringReporter - make processRecords() method thread safe, fixes #4193
*tests
NEW: Add ChangeESFlag.py script to modify useES flag in dirac.cfg. To be integrated with Jenkins code.
CHANGE: (#3760) Use the above Client classes instead of invoking RPCClient()
NEW: (#3744) Added performance tests for ES and MySQL for WMS DB backends
NEW: (#3744) Added miniInstallDIRAC function for Jenkins jobs
FIX: (#4177) restored test of JobsMonitor().JobParameters
NEW: (#4224) added a test for PilotWrapper
NEW: (#4244) integration test for JobParameters on ElasticSearch database
*Docs
FIX: Better dirac.cfg example configuration for web
NEW: (#3744) Add WMS documentation in DeveloperGuide/Systems
NEW: Added script (docs/Tools/UpdateDiracCFG.py) to collate the ConfigTemplate.cfg
files into the main dirac.cfg file
CHANGE: (#4110) updated basic tutorial for CC7 instead of SLC6.
NEW: (#4170) added Production system documentation
CHANGE: (#4224) Pilot 3 is the default
NEW: (#4244) Added a few notes on using the JobParameters on ElasticSearch database
[v6r22p23]
*DMS
CHANGE: (#4472) DM.getFile only checks metadata of files with replicas
[v6r22p22]
*Framework
CHANGE: (#4457) Web portal compilation is done during the DIRAC distribution, it is not
required to compile during the installation
*RSS
FIX: (#4452) PublisherHandler - fix UnboundLocalError
[v6r22p21]
*Core
FIX: (#4442) If no port is set in the CS for ES assume that the URL points to right location
*RSS
FIX: (#4441) DowntimeCommand - fix typo that was preventing to update expired or deleted
downtimes from RSS downtime cache.
*ConfigurationSystem
FIX: (#4447) remove logging from inside getConfigurationTree
*MonitoringSystem
FIX: (#4447) add check if MQ is setup for Monitoring
CHANGE: (#4443) Read the IndexPrefix for the CS and use this index prefix when creating ES indices,
if not given use the name of the setup
[v6r22p20]
*Core
CHANGE: (#4424) SocketInfo.py - DEFAULT_SSL_CIPHERS updated following the issue #4393
*WorkloadManagement
FIX: (#4440) ServerUtils.py - server name in ServerUtils.getPilotAgentsDB()
*Transformation
FIX: (#4434) TaskManager.py - fix bug in finding the settings for Clinics in the hospital
[v6r22p19]
*Framework
CHANGE: (#4415) ComponentInstaller - do not start runsvdir if not needed
*Resources
FIX: (#4414) Enable setting of WaitingToRunningRatio from Global CE Defaults, fixes #4368
*DMS
CHANGE: (#4413) FTS3Agent - change proxy lifetime of FTS3Agent from 2 to 12 hours
[v6r22p18]
*TS
CHANGE: (#4331) TaskManager - give possibility to run jobs for different problematic productions at
different Hospital sites
[v6r22p17]
CHANGE: Just update the Web version
[v6r22p16]
*Core
CHANGE: remove many calls to gLogger.debug() of low level DB modules (in particular MySQL.py) that may slowdown services
*FrameworkSystem
FIX: PRIVATE_LIMITED_DELEGATION role can download its own proxies
*ResourceStatusSystem
FIX: PublisherHandler: Convert set to list, because set can not be serialized using DEncode.
[v6r22p15]
*Resources
NEW: GFAL2_StorageBase: Disable GRIDFTP SESSION_REUSE by default. It can be enabled via
an environment variable export DIRAC_GFAL_GRIDFTP_SESSION_REUSE=True. This export
should be added in server bashrc files.
[v6r22p14]
*Framework
FIX: (#4318) NotificationHandler: fix name of initializeHandler function so the
handler is properly initialized and the periodicTasks are created
CHANGE: (#4325) InstalledComponentsDB.addInstalledComponents: Instead of creating a new
Host entry if only the CPU model changed, update the CPU field in the DBCHANGE:
InstalledComponentsDB.addInstalledComponents: Instead of creating a new Host entry
if only the CPU model changed, update the CPU field in the DB
*WMS
FIX: (#4329) Fix exception for PilotManager or PilotStatusAgent: itertems -> iteritems
*DMS
FIX: (#4327) avoid introducing inconsistencies in the DB due to the FTS inconsistencies
*Resources
CHANGE: (#4313) increase the SE logging level for plugin errors
*TS
CHANGE: (#4315) DataRecoveryAgent: Tweak information printout for email formatting;
clarify use of default values; reduce logging verbosity
[v6r22p13]
*Resources
FIX: (#4299) CREAMComputingElement - possibility to defined CEQueueName to be used in the pilot submission command
CHANGE: (#4297) SingularityCE: Delete workDir at end of job by default.
FIX: (#4297) SingularityCE: Always stop proxy renewal thread at end of job.
*TS
NEW: (#4301) DataRecoveryAgent to perform and apply consistency checks for transformations
NEW: (#4301) dirac-transformation-recover-data : script to manually run consistency checks on
individual transformations, for debugging and testing of the DataRecoveryAgent
[v6r22p12]
FIX: fixes from v6r21p16
[v6r22p11]
*Interfaces
FIX: (#4277) Dirac.py - fix error handling in getJobParameters()
[v6r22p10]
*Resources
FIX: (#4271) StorageElement - loadObject of occupancyPlugin is called with the path to OccupancyPlugins
[v6r22p9]
*Core
FIX: (#4269) Workflow - revert changes introduced in #4253
[v6r22p8]
*WMS
FIX: (#4256) handle empty results of getJobParameter in StalledJobAgent
*Resources
NEW: (#4223) Storage - occupancy plugin for WLCG standard accounting JSON
FIX: (#4259) SingularityComputingElement - ensure lcgBundle is installed in container if LCGBundleVersion is set.
[v6r22p7]
*Core
FIX: (#4253) Workflow - accept unicode file path to workflow XML
*WMS
NEW: (#4205) Statistics of Pilot submission is sent to Accounting System by SiteDirector.
FIX: (#4250) use host credentials to query the SandboxMetadataDB for async removal
*Resources
CHANGE: (#4242) DFC SEManagerDB: only acquire the lock if the cache needs to be modified
FIX: (#4251) preamble attribute was missing in SSHBatchComputingElement causing an error
in _submitJobHost()
*DMS
FIX: (#4248) SEManager correct order of calls at init
[v6r22p6]
*Core
CHANGE: (#4203) A VO with diracos extension, must be able to install the main diracos from DIRAC repository.
*TS
CHANGE: (#4218) do not set Job type 'hospital'
[v6r22p5]
*WMS
CHANGE: (#4217) Sandbox: Adding OwnerDN and OwnerGroup for DelayedExternalDeletion
*DMS
CHANGE: (#4202) relax permissions on getLFNForGUID
*Resources
FIX: (#4200) re-allow the use of the gsiftp cache in the SE
CHANGE: (#4206) Storage - improved treatment for the case without SpaceToken in SpaceOccupancy
[v6r22p4]
*Core
NEW: (#4181) Allow to install an extension of DIRACOS
*Docs
FIX: (#4187) Create a TransformationAgent Plugin -- more concrete
descriptions for adding a plugin to AllowedPlugins
[v6r22p3]
*WMS
CHANGE: (#4175) added function to get the number of Processors, using it in dirac-wms-get-wn-parameters (invoked by the pilots)
CHANGE: (#4175) changed port of PilotManager from 9129 to 9171
*docs
CHANGE: (#4174) update WebApp administrator docs
CHANGE: (#4175) removed mentions of MPIService
[v6r22p2]
*Core
FIX: (#4165) $Id$ keyword must be replaced only at the beginning of the file
*RSS
FIX: (#4169) PublisherHandler fix (UnboundLocalError)
*Docs
NEW: (#4167) /Operations/DataManagement - Added a link to the documentation on "Multi Protocol"
[v6r22p1]
*WMS
NEW: (#4147) added option for specifying a LocalCEType (which by default is "InProcess")
*Interfaces
NEW: (#4146) added setNumberOfProcessors method to Job() API
*Resources
NEW: (#4159) add dav(s) in the protocol lists of GFAL2_HTTPS
*tests
NEW: (#4154) Using variable for location of INSTALL_CFG_FILE (useful for extensions)
[v6r22]
*WorkloadManagementSystem
NEW: (#4045) PilotsHandler service (to interact with PilotAgentsDB)
CHANGE: (#4049) Pilot wrapper for pilot3: download files at runtime
CHANGE: (#4119) removed last bit looking into /Registry/VOMS/ section
CHANGE: (#4119) VOMS2CSAgent: mailFrom option is invalid, use MailFrom instead
FIX: (#4074) fixed PilotManager service name in ConfigTemplate.cfg
FIX: (#4100) use CAs to upload the pilot files to a dedicated web server using requests
FIX: (#4106) fixes for logging messages for Matcher
FIX: (#4106) fixes for logging messages for Optimizers
NEW: (#4136) added DIRACOS option (for SiteDirectors)
*ConfigurationSystem
NEW: (#4053) VOMS2CSSynchronizer/VOMS2CSAgent - enable automatic synchronization of the
Suspended user status with the VOMS database
CHANGE: (#4113) VOMS2CSSynchronizer: considering the case when no email is provided by VOMS
*Core
NEW: (#4053) AuthManager - interpret the Suspended user status considering suspended
users as anonymous visitors
CHANGE: (#4053) The use of CS.py module is replaced by the use of Registry.py and CSGlobals.py
*Interfaces
CHANGE: (#4098) removed dirac-admin-get-site-protocols.py as it could give potentially wrong results (use dirac-dms-protocol-matrix instead)
*Resources
NEW: (#4142) enable to get SE occupancy from plugin
NEW: (#4142) add occupancy plugin to retrieve the info from BDII
CHANGE: (#4142) GFAL2_SRM2Storage.getOccupancy() calls parent if SpaceToken is not given
*ResourceStatusSystem
CHANGE: clients: using DIRAC.Core.Base.Client as base
CHANGE: (#4098) SiteInspectorAgent runs only on sites with tokenOwner="rs_svc"
CHANGE: (#4098) remove SRM dependencies
CHANGE: (#4136) downtimeCommand will use the GOCServiceType only for SRM SEs
FIX: (#4139) only take the first endpoint for the SpaceOccupancy
*DataManagementSystem
CHANGE: (#4136) Moved methods from ResourceStatusSystem/CSHelpers to DMSHelpers
CHANGE: (#4138) FTS3 is now the default
*docs
NEW: (#4099) Instructions about setting up the DIRAC web server for pilot3
CHANGE: (#4119) added note on MultiProcessor jobs preparation
*test
FIX: (#4119) Not lhcb but dteam (for DIRAC certification)
FIX: (#4139) client_dms.sh not lhcb specific
CHANGE:(#4140) adapt transformation certification tests to dteam VO
[v6r21p16]
*Resources
FIX: (#4292) SSHComputingElement - define X509_USER_PROXY in case of gsissh access
*WMS
FIX: (#4292) SiteDirector - do not use keyword arguments when making setPilotStatus call
[v6r21p15]
*WMS
CHANGE: (#4214) Add an argument to the constructor of SandboxStoreClient for using in scripts
that cannot use the DB directly
*DMS
NEW: (#4171) ArchiveFiles Request Operation: to create a tarball out of a list of LFNs
NEW: (#4171) CheckMigration Request Operation to hold the request progress until the attached
LFNs are migrated to tape
NEW: (#4171) FCOnlyStorage StorageElement plugin to register LFNs without physical replica to
conserve LFN metadata when they are archived, for example
NEW: (#4171) dirac-dms-create-archive-request command to create a request to archive a list of
LFNs and remove their physical copies
NEW: (#4171) dirac-dms-create-moving-request command to move LFNs from a to b with optional
"CheckMigration" step. as it uses the ReplicateAndRegister operation, transfer via
FTS is also possible
FIX: (#4171) FileCatalogClient: add "addFileAncestors" to list of "write functions"
[v6r21p14]
*DMS
FIX: (#4192) dirac-dms-clean-directory correct usage message for list of arguments
FIX: (#4192) dirac-dms-clean-directory now properly prints error messages
FIX: (#4192) dirac-dms-clean-directory will now also clean empty directories
FIX: (#4192) FileCatalog.DirectoryMetadata: prevent error when removeMetadataDirectory is
called on empty list of directories, triggered when calling removeDirectory
on non-existing directory
FIX: (#4192) FileCatalog.DirectoryTreeBase: prevent maximum recursion depth exception
when calling remove directory with illegal path
*Resources
CHANGE: (#4191) Storages: catch specific DPM/Globus error code when creating existing directory
[v6r21p13]
*RSS
FIX: (#4173) only use the hostname of FTS servers in the RSS commands
[v6r21p12]
*WMS
FIX: (#4155) JobDB.getAttributesForJobList: Return S_ERROR if unknown attributes are requested.
Instead of potentially returning garbage a clear error message is returned.
[v6r21p11]
*Transformation
FIX: (#4144) fixed a logic bug in counting numberOfTasks in MCExtension which is expected
to limit the total number of tasks for MC transformations
*Accounting
FIX: (#4151) In AccountingDB.insertRecordThroughQueue fix bad dictionary key "0K"
[v6r21p10]
*Core
FIX: (#4133) dirac-install: correct location for ARC plugins with DIRACOS
*WMS
CHANGE: (#4136) JobStateUpdateHandler - restoring the job status to Running after hearbeat
*Docs
NEW: (#4134) Added /Operations/DataManagement parameters for protocols
[v6r21p9]
*Core
FIX: (#4130) correct symlinks in dirac-deploy
[v6r21p8]
*WMS
CHANGE: (#4111) Better logging in JobWrapper
CHANGE: (#4114) JobScheduling - allow both Tag and Tags option in the job JDL
*DMS
FIX: (#4101) FileManagerBase - use returned ID:LFN dict instead of the LFN list. Fixes the bug in
getReplicasByMetadata reported in #4058
*TransformationSystem
FIX: (#4112) TaskManager.py - testing if the request ID is correct was not done properly, test the numerical value
[v6r21p7]
*Core
FIX: (#4076) A certain module like WebAppDIRAC must be checked out from the code repository once.
*Resources
FIX: (#4078) Fix the exception when StorageElement objects are created with a list of plugins
*SMS
NEW: (#4086) StageMonitorAgent: new option StoragePlugins to limit the protocols used to contact storagelements for staged files.
*WMS
FIX: (#4093) better logging from services
*TS
CHANGE: (#4095) ConfigTemplate for RequestTaskAgent now contains all options
CHANGE: (#4096) the Broadcast TransformationPlugin no longer requires a SourceSE to be set. If it is set, the behaviour is unchanged
CHANGE: (#4096) dirac-transformation-replication: change default pluging back to Broadcast (reverts #4066)
*Docs:
CHANGE: (#4095) AdministratorGuide install TS tutorial: added options MonitorFiles and MonitorTasks for TaskAgents
[v6r21p6]
*CS
FIX: (#4064) Fix exception when calling dirac-admin-add-shifter with already existing values
*Core
NEW: (#4065) getIndexInList utility in List.py
*Resources
NEW: (#4065) add a SpaceReservation concept to storages
NEW: (#4065) add a getEndpoint method to StorageBase
*RSS
CHANGE: (#4065) CSHelpers.getStorageElementEndpoint returns the endpoint or non srm protocol
CHANGE: (#4065) add the SpaceReservation to the FreeDiskSpaceCommand result
*TS
FIX: (#4066) The dirac-transformation-replication script will now create valid transformations
given only the required arguments. Instead of the 'Broadcast' plugin, the 'Standard' plugin
is created if not SourceSE is given. If a value for the plugin argument is given, that will
be used.
*docs
CHANGE: (#4069) DIRAC installation procedure is updated taking account DIRACOS
CHANGE: (#4094) Pilots3 options: moved to /Operation/Pilot section
[v6r21p5]
*Core
NEW: (#4046) allow install_site to install DIRACOS
FIX: (#4047) dirac-deploy-scripts uses correct regex to find scripts
NEW: (#4047) dirac-deploy-scripts can use symplink instead of wrapper
CHANGE: (#4051) use debug level for logs in the ProcessPool
*RequestManagementSystem
CHANGE: (#4051) split log messages
*ResourceStatusSystem
FIX: (#4050) fix reporting from EmailAgent
CHANGE: (#4051) split log messages in static and dynamic parts
*Docs
CHANGE: (#4034) Add magic runs to setup DIRAC in example scripts, so they work out of the box.
NEW: (#4046) add a tuto for setting up a basic installation
NEW: (#4046) add a tuto for setting up two Dirac SEs
NEW: (#4046) add a tuto for setting up the DFC
NEW: (#4046) add a tuto for managing identities
NEW: (#4046) add a tuto for setting up the RMS
NEW: (#4046) add a tuto for doing DMS with TS
*ConfigurationSystem
CHANGE: (#4044) dirac-configure: forcing update (override, in fact) of CS list
*WorkloadManagementSystem
FIX: (#4052) SiteDirector - restore the logic of limiting the number of pilots to submit due to the
WaitingToRunningRatio option
FIX: (#4052) Matcher - if a pilot presents OwnerGroup parameter in its description, this is interpreted
as a pilot requirement to jobs and should not be overriden.
CHANGE: (#4027) Improve scalability of HTCondorCE jobs
*Accounting
CHANGE: (#4033) accounting clients use DIRAC.Core.Base.Client as base
*DataManagementSystem
FIX: (#4042) add exit handler for stored procedure
FIX: (#4048) correct the header of the CSV file generated by dirac-dms-protocol-matrix
*TransformationSystem
FIX: (#4038) TransformationCleaningAgent cancels the Request instead of removing them
*Resources
CHANGE: (#4048) SE: give preference to native plugins when generating third party URLS
[v6r21p4]
WorkloadManagementSystem
CHANGE: (#4008) Modification of utility function PilotCStoJSONSynchronizer. The modification
allows to add information to created json file about the DNs fields of users belonging
to 'lhcb_pilot' group. This information is needed for the second level authorization
used in the Pilot Logger architecture. Also, some basic unit tests are added.
*Docs
CHANGE: (#4028) update instructions to install and setup runit
*TransformationSystem
FIX: (#4022) when a site was requested inside the job workflow description, and BulkSubmission was used, such site was not considered.
*Resources
FIX: (#4006) Resources/MessageQueue: add a dedicated listener ReconnectListener
[v6r21p3]
*Core
FIX: (#4005) getDiracModules is removed, class member is used instead.
FIX: (#4013) Use getCAsLocation in order to avoid non-exist os.environ['X509_CERT_DIR']
*ConfigurationSystem
FIX: (#4004) BDII2CSAgent: fix for CEs with incomplete BDII info
*WorkloadManagementSystem
NEW: (#4016) JobAgent - added possibility to try out several CE descriptions when
getting jobs in one cycle
NEW: (#4016) Matcher - MultiProcessor tag is added to the resource description if appropriate
NEW: (#4016) JobScheduling - MultiProcessor tag is added to the job description if it
specifies multiple processor requirements
FIX: (#4018) JobMonitoring.getJobParameter cast to int
NEW: (#4019) added WMSAdministratorClient module, and using it throughout the code
*Resources/MessageQueue
CHANGE: (#4007) change the way of defining identifier format for MQ resources:
accepted values are 'Topics' or 'Queues'.
*DataManagementSystem
CHANGE: (#4017) DIP handler internally uses bytes instead of MB
NEW: (#4010) add dirac-dms-protocol-matrix script
CHANGE: (#4010) remove dirac-dms-add-files script
*Resources
NEW: (#4016) PoolComputingElement - getDescription returns a list of descriptions
with different requirements to jobs to be matched
CHANGE: (#4017) Standardize sizes returned by StoragePlugins in Bytes
CHANGE: (#4011) MQ: randomzied the broker list when putting message
[v6r21p2]
*Core
CHANGE: (#3992) dirac-install does not define REQUESTS_CA_BUNDLE in the bashrc anymore
NEW: (#3998) dirac-install if DIRACOS already installed and DIRACOS is not requested,
it will force to install it
CHANGE: (#3992) specify the ca location when calling requests
CHANGE: (#3991) MySQL class prints only debug logs
FIX: (#4003) dirac-install - if the DIRACOS version is not given then use the proper
release version
*WorkloadManagementSystem
CHANGE: (#3992) specify the ca location when calling requests
FIX: (#4002) Local protocols are retrieved as a list in InputDataResolution
*Interfaces
FIX: (#4000) Dirac.py - bug fixed: return value of getJobParameters changed that
should be taken into account by the clients
[v6r21p1]
*WorkloadManagementSystem
CHANGE: (#3989) JobDB.py - do not add default SubmitPool parameter to a job description
FIX: (#3989) dirac-admin-get-site-mask - show only sites in Active state
*DataManagementSystem
CHANGE: (#3985) FTS3DB: getActiveJobs, those jobs are now selected that have been monitored the longest time ago. Ensure better cycling through FTS Jobs
FIX: (#3987) check missing file with another string
[v6r21]
*Core
NEW: (#3921) DictCache - allow threadLocal cache
FIX: (#3936) DictCache - Fix exception upon delete
FIX: (#3922) allow Script.py to accommodate specific test calls with pytest
CHANGE: (#3940) dirac-install - instrument to support DiracOS
FIX: (#3945) set DIRACOS environment variable before souring diracosrc
CHANGE: (#3949) Removed unattended dirac-install-client.py
CHANGE: (#3950) File.py - do not follow links when getting files list or size
in directory via getGlobbedFiles and getGlobbedTotalSize
CHANGE: (#3969) Use EOS for installing DIRAC software
*FrameworkSystem
FIX: (#3968) removed the old logging
*ResourceStatusSystem
FIX: (#3921) fix logic of the RSSCache leading in expired keys
*Accounting
CHANGE: (#3933) Change the columns size of the FinalMinorStatus
*WorkloadManagementSystem
CHANGE: (#3923) Clean PYTHONPATH from *rc when installing DIRAC from the pilot
NEW: (#3941) JobDB: getJobParameters work also with list on job IDs
CHANGE: (#3941) JobCleaningAgent queries for job parameters in bulk
CHANGE: (#3941) Optimizers only set optimizers parameters (backported from v7r0)
CHANGE: (#3970) streamlining code in OptimizerModule. Also pep8 formatting (ignore white spaces for reviewing)
FIX: (#3976) fixed Banned Sites matching in TaskQueueDB
FIX: (#3970) when an optimizer agent was instantiating JobDB (via the base class) and the machine
was overloaded, the connection to the DB failed but this was not noticed and the agent was
not working until restarted after max cycles. Now testing JobDB is valid in OptimizerModule