2.8.2.4

    Release notes - hops - Version 2.8.2.4

Bug

• [HOPS-77] - Fix TestDataDirs test
• [HOPS-113] - Add support for /.reserved/.inodes/<inodeId> directory
• [HOPS-154] - Allow superusers to submit Yarn applications wtih SSL enabled
• [HOPS-155] - Store small files on the datanodes if the database disk tables are full.
• [HOPS-160] - Do not swallow FileNotFound Exception in NamenodeSelector
• [HOPS-161] - TestSmallFilesCreation.java takes ~30 min on slow machines.
• [HOPS-162] - RM and NM should materialize certificates in {RM,NM}_certLoc/UUID/
• [HOPS-168] - fix table creation script.
• [HOPS-170] - Slow deletionof very large directories
• [HOPS-172] - HOPS-86 removed the datanode upgrade system, we need to fix it
• [HOPS-174] - apply HDFS-4639
• [HOPS-175] - apply HDFS-4693
• [HOPS-176] - apply HDFS-4737
• [HOPS-183] - Old keystore password for Client is cached into the system
• [HOPS-184] - apply HDFS-4209
• [HOPS-185] - Containers requiring GPUs should not be able to reserve resources on NodeManagers with no GPUs.
• [HOPS-186] - CryptoMaterial buffers should be available for re-reading
• [HOPS-188] - apply HDFS-4334
• [HOPS-189] - apply HDFS-4296
• [HOPS-190] - apply HDFS-4339
• [HOPS-191] - apply HDFS-4346
• [HOPS-192] - apply HDFS-4243
• [HOPS-193] - apply HDFS-4340
• [HOPS-194] - apply HDFS-4502
• [HOPS-195] - apply HDFS-4745
• [HOPS-198] - apply HDFS-4768
• [HOPS-199] - apply HDFS-4305
• [HOPS-200] - apply HDFS-4434
• [HOPS-201] - Spark incompatible with our new constructor for ApplicationInitializationContext
• [HOPS-202] - update README
• [HOPS-203] - apply HDFS-4785
• [HOPS-204] - apply HDFS-4784
• [HOPS-206] - apply HDFS-4799
• [HOPS-207] - apply HDFS-4751
• [HOPS-208] - apply HDFS-4533
• [HOPS-209] - apply HDFS-4765
• [HOPS-210] - apply HDFS-3180
• [HOPS-215] - apply HDFS-4830
• [HOPS-216] - apply HDFS-4824
• [HOPS-217] - apply HDFS-3875
• [HOPS-218] - apply HDFS-4807
• [HOPS-219] - apply HDFS-4714
• [HOPS-220] - apply HDFS-4740
• [HOPS-221] - apply HDFS-4705
• [HOPS-222] - apply HDFS-4825
• [HOPS-223] - apply HDFS-4780
• [HOPS-224] - apply HDFS-4827
• [HOPS-225] - apply HDFS-4865
• [HOPS-228] - apply HDFS-4261
• [HOPS-229] - apply HDFS-4840
• [HOPS-230] - apply HDFS-4862
• [HOPS-231] - apply HDFS-4832
• [HOPS-232] - apply HDFS-4752
• [HOPS-233] - apply HDFS-4878
• [HOPS-234] - partly apply HDFS-4873
• [HOPS-235] - apply HDFS-3009
• [HOPS-236] - apply HDFS-3934
• [HOPS-237] - apply HDFS-4910
• [HOPS-238] - apply HDFS-4626
• [HOPS-239] - apply HDFS-4818
• [HOPS-240] - apply HDFS-4906
• [HOPS-242] - apply HDFS-4917
• [HOPS-244] - apply HDFS-4883
• [HOPS-247] - apply HDFS-4205
• [HOPS-248] - apply HDFS-4944
• [HOPS-249] - apply HDFS-4888
• [HOPS-250] - apply HDFS-4943
• [HOPS-253] - apply HDFS-4951
• [HOPS-254] - apply HDFS-4887
• [HOPS-255] - apply HDFS-4969
• [HOPS-257] - apply HDFS-4980
• [HOPS-264] - Add dependencies license document
• [HOPS-265] - Apply HDFS-4657
• [HOPS-266] - apply HDFS-4999
• [HOPS-267] - apply HDFS-4687
• [HOPS-268] - apply HDFS-4998
• [HOPS-276] - apply HDFS-5016
• [HOPS-278] - apply HDFS-5035
• [HOPS-282] - apply HDFS-5043
• [HOPS-284] - apply HDFS-5028
• [HOPS-288] - apply HDFS-5047
• [HOPS-289] - apply HDFS-5083
• [HOPS-290] - apply HDFS-4993
• [HOPS-293] - apply HDFS*4632
• [HOPS-294] - apply HDFS-5093
• [HOPS-295] - apply HDFS-2994
• [HOPS-296] - apply HDFS-5100
• [HOPS-297] - apply HDFS-5103
• [HOPS-298] - apply HDFS-5105
• [HOPS-299] - apply HDFS-5106
• [HOPS-306] - apply HDFS-5024
• [HOPS-307] - apply HDFS-5124
• [HOPS-310] - apply HDFS-5132
• [HOPS-311] - apply HDFS-5077
• [HOPS-313] - apply HDFS-5140
• [HOPS-314] - apply HDFS-5150
• [HOPS-317] - apply HDFS-4680
• [HOPS-318] - apply HDFS-5192
• [HOPS-321] - Get classloader's classpath once when running TLS tests
• [HOPS-322] - interferences between TransactionalRequestHandler and LightWeightRequestHandler
• [HOPS-323] - Inode locks acquirer fails when a client hold multiple leases.
• [HOPS-325] - apply HDFS-5219
• [HOPS-328] - apply HDFS-5251
• [HOPS-329] - apply HDFS-5228
• [HOPS-330] - apply HDFS-5265
• [HOPS-333] - apply HDFS-3735
• [HOPS-334] - apply HDFS-4512
• [HOPS-335] - apply HDFS-5255
• [HOPS-336] - datanode using 100% cpu
• [HOPS-337] - replication Index stuck in UnderReplicatedBlocks
• [HOPS-338] - apply HDFS-5279
• [HOPS-340] - apply HDFS-5306
• [HOPS-342] - Fix thread safety of CertificateLocalizationService
• [HOPS-343] - apply HDFS-5299
• [HOPS-344] - apply HDFS-5307
• [HOPS-345] - apply HDFS-5291
• [HOPS-346] - apply HDFS-5317
• [HOPS-347] - apply HDFS-5316
• [HOPS-348] - apply HDFS-4510
• [HOPS-352] - apply HDFS-5335
• [HOPS-354] - apply HDFS-5322
• [HOPS-355] - apply HDFS-5329
• [HOPS-357] - apply HDFS-5352
• [HOPS-358] - apply HDFS-5370
• [HOPS-359] - apply HDFS-5346
• [HOPS-362] - apply HDFS-5336
• [HOPS-363] - apply HDFS-5374
• [HOPS-364] - apply HDFS-5365
• [HOPS-365] - NullPointerException when trying to role back transaction
• [HOPS-366] - Datanode not doing block report after exception
• [HOPS-371] - apply HDFS-4511
• [HOPS-374] - Subtree rename incorrectly allowed due to wrong subtree access request since HOPS-5
• [HOPS-376] - apply HDFS-5400
• [HOPS-377] - apply HDFS-5403
• [HOPS-378] - apply HDFS-5363
• [HOPS-379] - apply HDFS-5341
• [HOPS-380] - apply HDFS-5257
• [HOPS-381] - apply HDFS-5413
• [HOPS-382] - apply HDFS-5436
• [HOPS-383] - apply HDFS-5432
• [HOPS-385] - apply HDFS-4633
• [HOPS-386] - apply HDFS-5456
• [HOPS-387] - apply HDFS-5325
• [HOPS-388] - apply HDFS-5488
• [HOPS-390] - apply HDFS-5440
• [HOPS-391] - apply HDFS-5075
• [HOPS-392] - apply HDFS-5487
• [HOPS-393] - apply HDFS-4995
• [HOPS-394] - apply HDFS-5506
• [HOPS-396] - Maximum GPU allocation gets set to 0
• [HOPS-397] - java.lang.ArrayIndexOutOfBoundsException when updating from 2.8.2.3 to 2.8.3.4
• [HOPS-399] - apply HDFS-5438
• [HOPS-400] - apply HDFS-5489
• [HOPS-401] - apply HDFS-5502
• [HOPS-403] - apply HDFS-5073
• [HOPS-404] - apply HDFS-1386
• [HOPS-407] - apply HDFS-4516
• [HOPS-408] - apply HDFS-5014
• [HOPS-409] - apply HDFS-3987
• [HOPS-414] - apply HDFS-5552
• [HOPS-415] - apply HDFS-5581
• [HOPS-417] - apply HDFS-5557
• [HOPS-418] - apply HDFS-5560
• [HOPS-419] - apply HDFS-5558
• [HOPS-420] - apply HDFS-4997
• [HOPS-421] - apply HDFS-5590
• [HOPS-423] - apply HDFS-5353
• [HOPS-424] - apply HDFS-4201
• [HOPS-427] - apply HDFS-5666
• [HOPS-428] - apply HDFS-5538
• [HOPS-429] - apply HDFS-5545
• [HOPS-430] - apply HDFS-5536
• [HOPS-453] - Hops users groups mapping does not fall back to OS groups
• [HOPS-458] - SubTree operation on a large directory with millions of immediate children can kill the database.
• [HOPS-461] - HDFS web UI not working with proxy servlet
• [HOPS-480] - BockInfoDataAccess.existsOnAnyStorage is doing a full table scan
• [HOPS-481] - INodeFile.getStoragePolicyID need readcommited lock on all parent
• [HOPS-483] - layoutversion for ACLs is wrong
• [HOPS-484] - BlockManager.processMisReplicatedBlocks is too slow
• [HOPS-625] - Node removal too slow
• [HOPS-628] - HOPS-625 not good enough
• [HOPS-630] - block reporting should log when failling
• [HOPS-631] - HOPS-628 fail when number of inodes "present" on datanode is greater than 4096
• [HOPS-636] - INodeLock.acquireINodeLocks does not work on multiple paths when one of the path does not exist

Sub-task

• [HOPS-303] - Add states to RMApp for generating and revoking certificates
• [HOPS-304] - Generate and revoke app certificates
• [HOPS-341] - Inject app keystores in containers' LocalResources

New Feature

• [HOPS-5] - Support for Access Control Lists (ACLs)
• [HOPS-86] - Heterogenous Storage
• [HOPS-147] - Reloadable key store manager
• [HOPS-178] - apply HDFS-4124
• [HOPS-226] - apply HDFS-3601
• [HOPS-227] - apply HDFS-3495
• [HOPS-259] - apply HDFS-4372
• [HOPS-260] - apply HDFS-4373
• [HOPS-261] - apply HDFS-4374
• [HOPS-300] - apply HDFS-5104
• [HOPS-301] - apply HDFS-2933
• [HOPS-302] - apply HDFS-4594
• [HOPS-315] - apply HDFS-5118
• [HOPS-331] - apply HDFS-5260
• [HOPS-370] - apply HDFS-6772

Improvement

• [HOPS-135] - Upgrade HopsFS to use NDB 7.5.8
• [HOPS-153] - [hops-hadoop-chef] Secure webhdfs REST API
• [HOPS-158] - Refactor HopsSSLSocketFactory and add check for Environment Variable set with crypto material
• [HOPS-159] - Add pull request template
• [HOPS-164] - fix hops schema updates for Flyway
• [HOPS-165] - need of a script to set up database for testing
• [HOPS-166] - change flyway storage engine from innodb to ndb
• [HOPS-167] - add commit messages format to pull request template
• [HOPS-169] - HOPS-86 partially implement the suport for archival we should finish it
• [HOPS-177] - apply HDFS-4129
• [HOPS-179] - apply HDFS-4152
• [HOPS-180] - apply HDFS-4206
• [HOPS-181] - apply HDFS-4215
• [HOPS-182] - apply HDFS-4151
• [HOPS-187] - Consult CRL in Server when RPC TLS is enabled
• [HOPS-196] - apply HDFS-4053
• [HOPS-197] - apply HDFS-4721
• [HOPS-205] - apply HDFS-4804
• [HOPS-211] - apply HDFS-347
• [HOPS-212] - apply HDFS-4661
• [HOPS-213] - apply HDFS-4698
• [HOPS-214] - apply HDFS-4787
• [HOPS-241] - apply HDFS-4461
• [HOPS-243] - apply HDFS-4914
• [HOPS-245] - Remove isConfigurationNeededForNormalUser check in HopsSSLSocketFactory
• [HOPS-246] - apply HDFS-4772
• [HOPS-251] - apply HDFS-4465
• [HOPS-252] - apply HDFS-4860
• [HOPS-256] - apply HDFS-4645
• [HOPS-258] - Remove unsupported SSL/TLS versions
• [HOPS-262] - apply HDFS-4912
• [HOPS-263] - apply HDFS-4903
• [HOPS-269] - apply HDFS-5007
• [HOPS-270] - apply HDFS-4278
• [HOPS-271] - apply HDFS-5010
• [HOPS-272] - apply HDFS-5008
• [HOPS-273] - apply HDFS-4974
• [HOPS-274] - apply HDFS-4817
• [HOPS-275] - apply HDFS-4979
• [HOPS-277] - apply HDFS-5004
• [HOPS-279] - apply HDFS-5027
• [HOPS-280] - apply HDFS-4525
• [HOPS-281] - apply HDFS-5025
• [HOPS-283] - apply HDFS-4513
• [HOPS-285] - apply HDFS-5061
• [HOPS-286] - apply HDFS-4905
• [HOPS-287] - apply HDFS-4926
• [HOPS-291] - Per Yarn application X.509 certificate
• [HOPS-292] - apply HDFS-5068
• [HOPS-305] - apply HDFS-5020
• [HOPS-308] - apply HDFS-3245
• [HOPS-309] - apply HDFS-5128
• [HOPS-312] - apply HDFS-5144
• [HOPS-316] - apply HDFS-4879
• [HOPS-319] - apply HDFS-5188
• [HOPS-320] - update ndb version to 7.5.9
• [HOPS-326] - apply HDFS-5240
• [HOPS-327] - apply HDFS-5139
• [HOPS-332] - apply HDFS-4517
• [HOPS-339] - apply HDFS-5308
• [HOPS-349] - apply HDFS-5323
• [HOPS-350] - Update ClusterJ and libndbclient to 7.5.10
• [HOPS-351] - Add profiles to enabling building on linux and mac machines
• [HOPS-353] - apply HDFS-5276
• [HOPS-356] - apply HDFS-5342
• [HOPS-360] - apply HDFS-5334
• [HOPS-361] - apply HDFS-5379
• [HOPS-372] - apply HDFS-5382
• [HOPS-375] - apply HDFS-4885
• [HOPS-384] - apply HDFS-4246
• [HOPS-389] - apply HDFS-5495
• [HOPS-402] - apply HDFS-5393
• [HOPS-405] - Block Report Needs refactoring
• [HOPS-410] - apply HDFS-5532
• [HOPS-411] - apply HDFS-5544
• [HOPS-412] - apply HDFS-5444
• [HOPS-413] - apply HDFS-5525
• [HOPS-422] - apply HDFS-4983
• [HOPS-425] - apply HDFS-5637
• [HOPS-426] - apply HDFS-5652