forked from ritaliu2010/hadoop-hdfs
-
Notifications
You must be signed in to change notification settings - Fork 0
/
CHANGES.txt
2389 lines (1507 loc) · 83.5 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Hadoop HDFS Change Log
Trunk (unreleased changes)
INCOMPATIBLE CHANGES
HDFS-1526. Dfs client name for a map/reduce task should be unique
among threads. (hairong)
HDFS-1536. Improve HDFS WebUI. (hairong)
NEW FEATURES
HDFS-1365. Federation: propose ClusterID and BlockPoolID format
(Tanping via boryas)
HDFS-1394. Federation: modify -format option for namenode to generated
new blockpool id and accept newcluster (boryas)
HDFS-1400. Federation: DataTransferProtocol uses ExtendedBlockPool to
include BlockPoolID in the protocol. (suresh)
HDFS-1428. Federation : add cluster ID and block pool ID into
Name node web UI(Tanping via boryas)
HDFS-1450. Federation: Introduce block pool ID into FSDatasetInterface.
(suresh)
HDFS-1632. Federation: data node storage structure changes and
introduce block pool storage. (Tanping via suresh)
HDFS-1634. Federation: Convert single threaded DataNode into
per BlockPool thread model.(boryas)
HDFS-1637. Federation: FSDataset in Datanode should be created after
initial handshake with namenode. (boryas and jitendra)
HDFS-1653. Federation: Block received message from datanode sends invalid
DatanodeRegistration. (Tanping via suresh)
HDFS-1645. Federation: DatanodeCommond.Finalize needs to include
BlockPoolId. (suresh)
HDFS-1638. Federation: DataNode.handleDiskError needs to inform
ALL namenodes if a disk failed (boryas)
HDFS-1647. Federation: Multiple namenode configuration. (jitendra)
HDFS-1639. Federation: Add block pool management to FSDataset. (suresh)
HDFS-1648. Federation: Only DataStorage must be locked using in_use.lock
and no locks must be associated with BlockPoolStorage. (Tanping via suresh)
HDFS-1641. Federation: Datanode fields that are no longer used should
be removed (boryas)
HDFS-1642. Federation: add Datanode.getDNRegistration(String bpid)
method (boryas)
HDFS-1643. Federation: remove namenode argument from DataNode
constructor (boryas)
HDFS-1657. Federation: Tests that corrupt block files fail due to changed
file path in federation. (suresh)
HDFS-1661. Federation: Remove unnecessary TODO:FEDERATION comments.
(jitendra)
HDFS-1660. Federation: Datanode doesn't start with two namenodes (boryas)
HDFS-1650. Federation: TestReplication fails. (Tanping via suresh)
HDFS-1651. Federation: Tests fail due to null pointer exception in
Datnode#shutdown() method. (Tanping via suresh)
HDFS-1649. Federation: Datanode command to refresh namenode list at
the datanode. (jitendra)
HDFS-1646. Federation: MiniDFSClsuter#waitActive() waits for ever
with the introduction of BPOfferService in datanode. (suresh)
HDFS-1659. Federation: BPOfferService exits after one iteration
incorrectly. (Tanping via suresh)
HDFS-1654. Federation: Fix TestDFSUpgrade and TestDFSRollback failures.
(suresh)
HDFS-1668. Federation: Datanodes sends block pool usage information
to the namenode in heartbeat. (suresh)
HDFS-1669. Federation: Fix TestHftpFileSystem failure. (suresh)
HDFS-1670. Federation: remove dnRegistration from Datanode (boryas)
HDFS-1662. Federation: fix unit test case, TestCheckpoint
and TestDataNodeMXBean (tanping via boryas)
HDFS-1671. Federation: shutdown in DataNode should be able to
shutdown individual BP threads as well as the whole DN (boryas).
HDFS-1663. Federation: Rename getPoolId() everywhere to
getBlockPoolId() (tanping via boryas)
HDFS-1652. FederationL Add support for multiple namenodes in
MiniDFSCluster. (suresh)
HDFS-1672. Federation: refactor stopDatanode(name) to work
with multiple Block Pools (boryas)
HDFS-1687. Federation: DirectoryScanner changes for
federation (Matt Foley via boryas)
HDFS-1626. Make BLOCK_INVALIDATE_LIMIT configurable. (szetszwo)
HDFS-1655. Federation: DatablockScanner should scan blocks for
all the block pools. (jitendra)
HDFS-1664. Federation: Add block pool storage usage to Namenode WebUI.
(Tanping via suresh)
HDFS-1674. Federation: Rename BlockPool class to BlockPoolSlice.
(jghoman, Tanping via suresh)
HDFS-1673. Federation: Datanode changes to track block token secret per
namenode. (suresh)
HDFS-1677. Federation: Fix TestFsck and TestListCorruptFileBlocks
failures. (Tanping via suresh)
HDFS-1678. Federation: Remove unnecessary #getBlockpool()
for NameNodeMXBean in FSNameSystem. (Tanping via Suresh)
HDFS-1688. Federation: Fix failures in fault injection tests,
TestDiskError, TestDatanodeRestart and TestDFSTartupVersions. (suresh)
HDFS-1696. Federation: when build version doesn't match -
datanode should wait (keep connecting) untill NN comes up
with the right version (boryas)
HDFS-1681. Balancer: support per pool and per node policies. (szetszwo)
HDFS-1695. Federation: Fix testOIV and TestDatanodeUtils
(jhoman and tanping via boryas)
HDFS:1699. Federation: Fix failure of TestBlockReport.
(Matt Foley via suresh)
HDFS-1698. Federation: TestOverReplicatedBlocks and TestWriteToReplica
failing. (jhoman and jitendra)
HDFS-1701. Federation: Fix TestHeartbeathandling.
(Erik Steffl and Tanping Wang via suresh)
HDFS-1693. Federation: Fix TestDFSStorageStateRecovery failure. (suresh)
HDFS-1694. Federation: SimulatedFSDataset changes to work with
federation and multiple block pools. (suresh)
HDFS-1689. Federation: Configuration for namenodes. (suresh and jitendra)
HDFS-1682. Change Balancer CLI for multiple namenodes and balancing
policy. (szetszwo)
HDFS-1697. Federation: fix TestBlockRecovery (boryas)
HDFS-1702. Federation: fix TestBackupNode and TestRefreshNamendoes
failures. (suresh)
HDFS-1706. Federation: TestFileAppend2, TestFileAppend3 and
TestBlockTokenWithDFS failing. (jitendra)
HDFS-1704. Federation: Add a tool that lists namenodes, secondary and
backup from configuration file. (suresh)
HDFS-1711. Federation: create method for updating machine name in
DataNode.java (boryas)
HDFS-1712. Federation: when looking up datanode we should use machineNmae
(in testOverReplicatedBlocks) (boryas)
HDFS-1709. Federation: Error "nnaddr url param is null" when clicking on a
node from NN Live Node Link. (jitendra)
HDFS-1714. Federation: refactor upgrade object in DataNode (boryas)
HDFS-1715. Federation: warning/error not generated when datanode sees
inconsistent/different Cluster ID between namenodes (boryas)
HDFS-1715. Federation: warning/error not generated when datanode sees
inconsistent/different Cluster ID between namenodes (boryas)
HDFS-1716. Federation: Add decommission tests for federated namenodes.
(suresh)
HDFS-1713. Federation: Prevent DataBlockScanner from running in tight loop.
(jitendra)
HDFS-1721. Federation: Configuration for principal names should not be
namenode specific. (jitendra)
HDFS-1717. Federation: FSDataset volumeMap access is not synchronized
correctly. (suresh)
HDFS-1722. Federation: Add flag to MiniDFSCluser to differentiate between
federation and non-federation modes. (boryas via suresh)
HDFS-1718. Federation: MiniDFSCluster#waitActive() bug causes some tests
to fail. (suresh)
HDFS-1719. Federation: Fix TestDFSRemove that fails intermittently.
(suresh)
HDFS-1720. Federation: FSVolumeSet volumes is not synchronized correctly.
(suresh)
HDFS-1700. Federation: fsck needs to work with federation changes.
(Matt Foley via suresh)
HDFS-1482. Add listCorruptFileBlocks to DistributedFileSystem.
(Patrick Kling via hairong)
HDFS-1448. Add a new tool Offline Edits Viewer (oev). (Erik Steffl
via szetszwo)
HDFS-1735. Federation: merge FSImage change in federation to
FSImage+NNStorage refactoring in trunk. (suresh)
HDFS-1737. Federation: Update the layout version for federation
changes. (suresh)
HDFS-1744. Federation: Add new layout version to offline image viewer
and edits viewer. (suresh)
HDFS-1745. Federation: Fix fault injection test failures. (suresh)
HDFS-1746. Federation: TestFileAppend3 fails intermittently. (jitendra)
HDFS-1703. Improve start/stop scripts and add decommission tool for
federation. (Tanping Wang, Erik Steffl via suresh)
HDFS-1749. Federation: TestListCorruptFileBlocks failing in federation
branch. (jitendra)
HDFS-1754. Federation: testFsck fails. (boryas)
HDFS-1755. Federation: The BPOfferService must always connect to namenode
as the login user. (jitendra)
HDFS-1675. Support transferring RBW between datanodes. (szetszwo)
HDFS-1791. Federation: Add command to delete block pool directories
from a datanode. (jitendra)
HDFS-1761. Add a new DataTransferProtocol operation, Op.TRANSFER_BLOCK,
for transferring RBW/Finalized with acknowledgement and without using RPC.
(szetszwo)
HDFS-1813. Federation: Authentication using BlockToken in RPC to datanode
fails. (jitendra)
HDFS-1630. Support fsedits checksum. (hairong)
HDFS-1606. Provide a stronger data guarantee in the write pipeline by
adding a new datanode when an existing datanode failed. (szetszwo)
HDFS-1442. Api to get delegation token in Hdfs class. (jitendra)
HDFS-1070. Speedup namenode image loading and saving by storing only
local file names. (hairong)
HDFS-1751. Intrinsic limits for HDFS files, directories (daryn via boryas).
HDFS-1873. Federation: Add cluster management web console.
(Tanping Wang via suresh)
HDFS 1911 HDFS tests for the newly added viewfs
HDFS-1814. Add "hdfs groups" command to query the server-side groups
resolved for a user. (Aaron T. Myers via todd)
HDFS-1914. Federation: namenode storage directories must be configurable
specific to name service. (suresh)
HDFS-1963. Create RPM and Debian packages for HDFS. Changes deployment
layout to be consistent across the binary tgz, rpm, and deb.
(Eric Yang via omalley)
HDFS-2058. Change Data Transfer wire protocol to use protocol buffers.
(todd)
IMPROVEMENTS
HDFS-1875. MiniDFSCluster hard-codes dfs.datanode.address to localhost
(Eric Payne via mattf)
HDFS-2019. Fix all the places where Java method File.list is used with
FileUtil.list API (Bharath Mundlapudi via mattf)
HDFS-1934. Fix NullPointerException when certain File APIs return null
(Bharath Mundlapudi via mattf)
HDFS-1510. Added test-patch.properties required by test-patch.sh (nigel)
HDFS-1628. Display full path in AccessControlException. (John George
via szetszwo)
HDFS-1707. Federation: Failure in browsing data on new namenodes.
(jitendra)
HDFS-1683. Test Balancer with multiple NameNodes. (szetszwo)
HDFS-1547. Improve decommission mechanism. (suresh)
HDFS-1588. Remove hardcoded strings for configuration keys, "dfs.hosts"
and "dfs.hosts.exlude". (Erik Steffl via suresh)
HDFS-1481. NameNode should validate fsimage before rolling. (hairong)
HDFS-1506. Refactor fsimage loading code. (hairong)
HDFS-1533. A more elegant FileSystem#listCorruptFileBlocks API
(HDFS portion) (Patrick Kling via hairong)
HDFS-1476. listCorruptFileBlocks should be functional while the
name node is in safe mode. (Patrick Kling via hairong)
HDFS-1534. Fix some incorrect logs in FSDirectory. (eli)
HDFS-1539. A config option for the datanode to fsycn a block file
when block is completely written. (dhruba)
HDFS-1335. HDFS side change of HADDOP-6904: RPC compatibility. (hairong)
HDFS-1557. Separate Storage from FSImage. (Ivan Kelly via jitendra)
HDFS-560 Enhancements/tuning to hadoop-hdfs/build.xml
HDFS-1629. Add a method to BlockPlacementPolicy for keeping the chosen
nodes in the output array. (szetszwo)
HDFS-1731. Allow using a file to exclude certain tests from build (todd)
HDFS-1736. Remove the dependency from DatanodeJspHelper to FsShell.
(Daryn Sharp via szetszwo)
HDFS-780. Revive TestFuseDFS. (eli)
HDFS-1445. Batch the calls in DataStorage to FileUtil.createHardLink().
(Matt Foley via jghoman)
HDFS-1763. Replace hard-coded option strings with variables from
DFSConfigKeys. (eli)
HDFS-1541. Not marking datanodes dead when namenode in safemode.
(hairong)
HDFS-1120. Make DataNode's block-to-device placement policy pluggable
(Harsh J Chouraria via todd)
HDFS-1785. In BlockReceiver and DataXceiver, clientName.length() is used
multiple times for determining whether the source is a client or a
datanode. (szetszwo)
HDFS-1789. Refactor frequently used codes from DFSOutputStream and
DataXceiver. (szetszwo)
HDFS-1767. Namenode ignores non-initial block report from datanodes
when in safemode during startup. (Matt Foley via suresh)
HDFS-1817. Move pipeline_Fi_[39-51] from TestFiDataTransferProtocol
to TestFiPipelineClose. (szetszwo)
HDFS-1760. In FSDirectory.getFullPathName(..), it is better to return "/"
for root directory instead of an empty string. (Daryn Sharp via szetszwo)
HDFS-1833. Reduce repeated string constructions and unnecessary fields,
and fix comments in BlockReceiver.PacketResponder. (szetszwo)
HDFS-1486. Generalize CLITest structure and interfaces to faciliate
upstream adoption (e.g. for web testing). (cos)
HDFS-1844. Move "fs -help" shell command tests from HDFS to COMMOM; see
also HADOOP-7230. (Daryn Sharp via szetszwo)
HDFS-1840. In DFSClient, terminate the lease renewing thread when all files
being written are closed for a grace period, and start a new thread when
new files are opened for write. (szetszwo)
HDFS-1854. make failure message more useful in
DFSTestUtil.waitReplication(). (Matt Foley via eli)
HDFS-1562. Add rack policy tests. (eli)
HDFS-1856. TestDatanodeBlockScanner waits forever, errs without giving
information. (Matt Foley via eli)
HDFS-1295. Improve namenode restart times by short-circuiting the
first block reports from datanodes. (Matt Foley via suresh)
Corrected merge error in DataNode.java. (Matt Foley)
HDFS-1843. Discover file not found early for file append.
(Bharath Mundlapudi via jitendra)
HDFS-1862. Improve test reliability of HDFS-1594. (Aaron T. Myers via eli)
HDFS-1846. Preallocate edit log with OP_INVALID instead of zero bytes
to ensure blocks are actually allocated. (Aaron T. Myers via todd)
HDFS-1741. Provide a minimal pom file to allow integration of HDFS into Sonar
analysis (cos)
HDFS-1870. Move and rename DFSClient.LeaseChecker to a seperated class
LeaseRenewer. (szetszwo)
HDFS-1866. Document dfs.datanode.max.transfer.threads in hdfs-default.xml
(Harsh J Chouraria via todd)
HDFS-1890. Improve the name, class and value type of the map
LeaseRenewer.pendingCreates. (szetszwo)
HDFS-1865. Share LeaseRenewer among DFSClients so that there is only a
LeaseRenewer thread per namenode per user. (szetszwo)
HDFS-1906. Remove logging exception stack trace in client logs when one of
the datanode targets to read from is not reachable. (suresh)
HDFS-1378. Edit log replay should track and report file offsets in case of
errors. (Aaron T. Myers and Todd Lipcon via todd)
HDFS-1917. Separate hdfs jars from common in ivy configuration. (Eric Yang
via szetszwo)
HDFS-1899. GenericTestUtils.formatNamenode should be moved to DFSTestUtil
(Ted Yu via todd)
HDFS-1117. Metrics 2.0 HDFS instrumentation. (Luke Lu via suresh)
HDFS-1946. HDFS part of HADOOP-7291. (eli)
HDFS-1945. Removed the deprecated fields in DataTransferProtocol.
(szetszwo)
HDFS-1730. Use DaemonFactory from common and delete it from HDFS.
(Tanping via suresh)
HDFS-1573. Add useful tracing information to Lease Renewer thread names
(todd)
HDFS-1939. In ivy.xml, test conf should not extend common conf.
(Eric Yang via szetszwo)
HDFS-1332 Include more information in exceptions and debug messages
when BlockPlacementPolicy cannot be satisfied. (Ted Yu via szetszwo)
HDFS-1958. Confirmation should be more lenient of user input when
formatting the NameNode. (todd)
HDFS-1905. Improve namenode -format command by not making -clusterId
parameter mandatory. (Bharath Mundlapudi via suresh)
HDFS-1877. Add a new test for concurrent read and write. (CW Chung
via szetszwo)
HDFS-1959. Better error message for missing namenode directory. (eli)
HDFS-1996. ivy: hdfs test jar should be independent to common test jar.
(Eric Yang via szetszwo)
HDFS-1812. TestHDFSCLI should clean up cluster in teardown method.
(Uma Maheswara Rao G via todd)
HDFS-1884. Improve TestDFSStorageStateRecovery to properly throw in the
case of errors. (Aaron T. Myers via todd)
HDFS-1727. fsck command should display command usage if user passes any
illegal argument. (Sravan Kumar via todd)
HDFS-1636. If dfs.name.dir points to an empty dir, namenode format
shouldn't require confirmation. (Harsh J Chouraria via todd)
HDFS-1966. Encapsulate individual DataTransferProtocol op headers.
(szetszwo)
HDFS-2024. Format TestWriteRead source codes. (CW Chung via szetszwo)
HDFS-1968. Enhance TestWriteRead to support position/sequential read,
append, truncate and verbose options. (CW Chung via szetszwo)
HDFS-1986. Add option to get http/https address from
DFSUtil#getInfoServer(). (Tanping via suresh)
HDFS-2029. In TestWriteRead, check visible length immediately after
openning the file and fix code style. (John George via szetszwo)
HDFS-2040. Only build libhdfs if a flag is passed. (eli)
HDFS-1586. Add InterfaceAudience and InterfaceStability annotations to
MiniDFSCluster. (suresh)
HDFS-2003. Separate FSEditLog reading logic from edit log memory state
building logic. (Ivan Kelly via todd)
HDFS-2066. Create a package and individual class files for
DataTransferProtocol. (szetszwo)
OPTIMIZATIONS
HDFS-1458. Improve checkpoint performance by avoiding unnecessary image
downloads and loading. (hairong)
HDFS-1601. Pipeline ACKs are sent as lots of tiny TCP packets (todd)
HDFS-1826. NameNode should save image to name directories in parallel
during upgrade. (Matt Foley via hairong)
HDFS-941. The DFS client should cache and reuse open sockets to datanodes
while performing reads. (bc Wong and Todd Lipcon via todd)
HDFS-2030. Improve usability of namenode -upgrade command.
(Bharath Mundlapudi via suresh)
HDFS-2056. Update fetchdt usage. (Tanping Wang via jitendra)
BUG FIXES
HDFS-2061. Two minor bugs in BlockManager block report processing. (mattf)
HDFS-1449. Fix test failures - ExtendedBlock must return
block file name in #getBlockName(). (suresh)
HDFS-1680. Fix TestBalancer. (szetszwo)
HDFS-1705. Balancer command throws NullPointerException. (suresh via
szetszwo)
HDFS-1559. Add missing UGM overrides to TestRefreshUserMappings
(Todd Lipcon via eli)
HDFS-1585. Fix build after HDFS-1547 (todd)
HDFS-1684. Balancer cannot start with with multiple namenodes. (szetszwo)
HDFS-1516. mvn-install is broken after 0.22 branch creation. (cos)
HDFS-1360. TestBlockRecovery should bind ephemeral ports.
(Todd Lipcon via hairong)
HDFS-1551. Fix pom templates dependency list (gkesavan)
HDFS-1509. A savenamespace command writes the fsimage and edits into
all configured directories. (dhruba)
HDFS-1540. Make Datanode handle errors from RPC calls to namenode
more elegantly. (dhruba)
HDFS-1463. Accesstime of a file is not updated in safeMode. (dhruba)
HDFS-863. Potential deadlock in TestOverReplicatedBlocks.
(Ken Goodhope via jghoman)
HDFS-1607. Fix referenced to misspelled method name getProtocolSigature
(todd)
HDFS-1610. Fix TestClientProtocolWithDelegationToken and TestBlockToken
on trunk after HADOOP-6904 (todd)
HDFS-1600. Fix release audit warnings on trunk. (todd)
HDFS-1691. Remove a duplicated static initializer for reading default
configurations in DFSck. (Alexey Diomin via szetszwo)
HDFS-1748. Balancer utilization classification is incomplete. (szetszwo)
HDFS-1738. change hdfs jmxget to return an empty string instead of
null when an attribute value is not available (tanping vi boryas)
HDFS-1757. Don't compile fuse-dfs by default. (eli)
HDFS-1770. TestFiRename fails due to invalid block size. (eli)
HDFS-1797. Fix new findbugs warning introduced by HDFS-1120 (todd)
HDFS-1611. Fix up some log messages in DFSClient and MBean registration
(Uma Maheswara Rao G via todd)
HDFS-1543. Reduce dev. cycle time by moving system testing artifacts from
default build and push to maven for HDFS (Luke Lu via cos)
HDFS-1818. TestHDFSCLI is failing on trunk after HADOOP-7202.
(Aaron T. Myers via todd)
HDFS-1828. TestBlocksWithNotEnoughRacks intermittently fails assert.
(Matt Foley via eli)
HDFS-1824. delay instantiation of file system object until it is
needed (linked to HADOOP-7207) (boryas)
HDFS-1831. Fix append bug in FileContext and implement CreateFlag
check (related to HADOOP-7223). (suresh)
HDFS-1594. When the disk becomes full Namenode is getting shutdown and
not able to recover. (Aaron T. Myers via eli)
HDFS-1822. Handle editlog opcode conflict with 0.20.203 during upgrade,
by throwing an error to indicate the editlog needs to be empty.
(suresh)
HDFS-1808. TestBalancer waits forever, errs without giving information.
(Matt Foley via eli)
HDFS-1829. TestNodeCount waits forever, errs without giving information.
(Matt Foley via eli)
HDFS-1860. when renewing/canceling DelegationToken over http we need to
pass exception information back to the caller.(boryas)
HDFS-1871. Mapreduce build fails due to MiniDFSCluster change from
HDFS-1052. (suresh)
HDFS-1876. One MiniDFSCluster constructor ignores numDataNodes parameter
(todd)
HDFS-1773. Do not show decommissioned datanodes, which are not in both
include and exclude lists, on web and JMX interfaces.
(Tanping Wang via szetszwo)
HDFS-1888. MiniDFSCluster#corruptBlockOnDatanodes() access must be
public. (suresh)
HDFS-1889. incorrect path in start/stop dfs script. (John George via eli)
HDFS-1891. Disable IPV6 for junit tests to fix TestBackupNode failure.
(suresh)
HDFS-1898. Tests failing on trunk due to use of NameNode.format.
(todd via eli)
HDFS-1902. Fix setrep path display for TestHDFSCLI. (Daryn Sharp
via szetszwo)
HDFS-1827. Fix timeout problem in TestBlockReplacement. (Matt Foley
via szetszwo)
HDFS-1908. Fix a NullPointerException in fi.DataTransferTestUtil.
(szetszwo)
HDFS-1912. Update tests for FsShell standardized error messages.
(Daryn Sharp via szetszwo)
HDFS-1903. Fix path display for rm/rmr in TestHDFSCLI and TestDFSShell.
(Daryn Sharp via szetszwo)
HDFS-1627. Fix NullPointerException in Secondary NameNode. (hairong)
HDFS-1928. Fix path display for touchz in TestHDFSCLI.
(Daryn Sharp via todd)
HDFS-1938. Fix ivy-retrieve-hdfs dependence in build.xml and aop.xml.
(Eric Yang via szetszwo)
HDFS-1929. TestEditLogFileOutputStream fails if running on same host as NN
(Aaron T. Myers via todd)
HDFS-1933. Update TestDFSShell for improved "test" shell command. (Daryn
Sharp via todd)
HDFS-1931. Update TestDFSShell for improved "du" shell command. (Daryn
Sharp via todd)
HDFS-1881. Federation: after taking snapshot the current directory
of datanode is empty. (Tanping Wang via suresh)
HDFS-1927. Fix a bug which causes ip=null in NameNode audit log.
(John George via szetszwo)
HDFS-1953. Federation: Change name node mxbean name in cluster web
console. (Tanping Wang via suresh)
HDFS-1922. Fix recurring failure of TestJMXGet (Luke Lu via todd)
HDFS-1371. One bad node can incorrectly flag many files as corrupt.
(Tanping Wang via jitendra)
HDFS-1943. Fail to start datanode while start-dfs.sh is executed by
root user. (Wei Yongjun via jghoman)
HDFS-1983. Fix path display for copy and rm commands in TestHDFSCLI and
TestDFSShell. (Daryn Sharp via todd)
HDFS-1999. Tests use deprecated configs. (Aaron T. Myers via eli)
HDFS-1592. Datanode startup doesn't honor volumes.tolerated.
(Bharath Mundlapudi via jitendra)
HDFS-1920. libhdfs does not build for ARM processors.
(Trevor Robinson via eli)
HDFS-1936. Layout version change from HDFS-1822 causes upgrade failure.
(suresh)
HDFS-2021. Update numBytesAcked before sending the ack in PacketResponder.
(John George via szetszwo)
HDFS-2020. Fix TestDFSUpgradeFromImage by removing the use of DataNode
as a singleton. (suresh via todd)
HDFS-2022. ant binary should build libhdfs. (Eric Yang via eli)
HDFS-2014. Change HDFS scripts to work in developer enviroment post
RPM packaging changes. (Eric Yang via suresh)
HDFS-1995. Federation: Minor bug fixes and modification cluster web UI.
(Tanping Wang via suresh)
HDFS-1907. Fix position read for reading still-being-written file in
DFSInputStream. (John George via szetszwo)
HDFS-1923. In TestFiDataTransferProtocol2, reduce random sleep time period
and increase the number of datanodes. (szetszwo)
HDFS-1149. Lease reassignment should be persisted to the edit log.
(Aaron T. Myers via todd)
HDFS-1998. Federation: Make refresh-namenodes.sh refresh all the
namenode. (Tanping Wang via suresh)
HDFS-2041. OP_CONCAT_DELETE doesn't properly restore modification time
of the concatenated file when edit logs are replayed. (todd)
HDFS-2063. libhdfs test is broken. (Eric Yang via eli)
HDFS-2067. Bump DATA_TRANSFER_VERSION constant in trunk after introduction
of protocol buffers in the protocol. (szetszwo via todd)
HDFS-2069. Incorrect default trash interval value in the docs.
(Harsh J Chouraria via eli)
Release 0.22.0 - Unreleased
INCOMPATIBLE CHANGES
HDFS-1825. Remove thriftfs contrib. (nigel via eli)
NEW FEATURES
HDFS-992. Re-factor block access token implementation to conform to the
generic Token interface in Common (Kan Zhang and Jitendra Pandey via jghoman)
HDFS-599. Allow NameNode to have a seprate port for service requests from
client requests. (Dmytro Molkov via hairong)
HDFS-1004. Update NN to support Kerberized SSL from HADOOP-6584.
(jghoman and Kan Zhang via jghoman)
HDFS-1005. Fsck security. (borya and Kan Zhang via jghoman)
HDFS-1006. getImage/putImage http requests should be https for the case
of security enabled. (borya and jghoman via jghoman)
HDFS-1033. In secure clusters, NN and SNN should verify that the remote
principal during image and edits transfer. (jghoman)
HDFS-1023. Allow http server to start as regular principal if https
principal not defined. (jghoman)
HDFS-1150. Verify datanodes' identities to clients in secure clusters.
(jghoman)
HDFS-1330. Make RPCs to DataNodes timeout. (hairong)
HDFS-202. HDFS support of listLocatedStatus introduced in HADOOP-6870.
HDFS piggyback block locations to each file status when listing a
directory. (hairong)
HDFS-1359. Add BlockPoolID to Block. (suresh)
HDFS-1361. Add -fileStatus operation to NNThroughputBenchmark. (shv)
HDFS-1435. Provide an option to store fsimage compressed. (hairong)
HDFS-903. Support fsimage validation through MD5 checksum. (hairong)
HDFS-1457. Provide an option to throttle image transmission between
pimary and secondary NameNodes. (Yifei Lu and hairong via hairong)
HDFS-1164. TestHdfsProxy is failing. (Todd Lipcon via cos)
HDFS-811. Add metrics, failure reporting and additional tests for HDFS-457.
(eli)
HDFS-895. Allow hflush/sync to occur in parallel with new writes
to the file. (Todd Lipcon via hairong)
HDFS-528. Add ability for safemode to wait for a minimum number of
live datanodes (Todd Lipcon via eli)
IMPROVEMENTS
HDFS-1304. Add a new unit test for HftpFileSystem.open(..). (szetszwo)
HDFS-1096. fix for prev. commit. (boryas)
HDFS-1096. allow dfsadmin/mradmin refresh of superuser proxy group
mappings (boryas)
HDFS-1146. Javadoc for getDelegationTokenSecretManager in FSNamesystem (jnp via boryas)
HDFS-1132. Refactor TestFileStatus (Eli Collins via cos)
HDFS-1163. normalize property names for JT/NN kerberos principal
names in configuration (from HADOOP 6633) (boryas)
HDFS-1003. authorization checks for inter-server protocol
(based on HADOOP-6600) (boryas)
HDFS-1061. Memory footprint optimization for INodeFile object.
(Bharath Mundlapudi via jghoman)
HDFS-1079. Throw exceptions as specified by the AbstractFileSystem
in HDFS implemenation and protocols. (suresh)
HDFS-1112. Edit log buffer should not grow unfoundedly. (hairong)
HDFS-1119. Introduce a GSet interface to BlocksMap. (szetszwo)
HDFS-1184. Replace tabs in code with spaces. (Jeff Ames via jghoman)
HDFS-1185. Remove duplicate now() functions in DataNode, FSNamesysetm.
(Jeff Ames via jghoman)
HDFS-1183. Remove some duplicate code in NamenodeJspHelper.java.
(Jeff Ames via jghoman)
HDFS-1190. Remove unused getNamenode() method from DataNode.
(Jeff Ames via jghoman)
HDFS-1110. Reuses objects for commonly used file names in namenode to
reduce the heap usage. (suresh)
HDFS-752. Add interfaces classification to to HDFS source code. (suresh)
HDFS-947. An Hftp read request is redirected to a datanode that has
the most replicas of the blocks in the file. (Dmytro Molkov via dhruba)
HDFS-1272. Fixes to take care of the changes in HADOOP-6845.
(Jitendra Pandey via ddas)
HDFS-1298 - Add support in HDFS for new statistics added in FileSystem
to track the file system operations. (suresh)
HDFS-1201. The HDFS component for HADOOP-6632.
(Kan Zhang & Jitendra Pandey via ddas)
HDFS-1307 Add start time, end time and total time taken for FSCK to
FSCK report (suresh)
HDFS-1302. The HDFS side of the changes corresponding to HADOOP-6861.
(Jitendra Pandey & Owen O'Malley via ddas)
HDFS-1315. Add fsck event to audit log and remove other audit log events
corresponding to FSCK listStatus and open calls. (suresh)
HDFS-1178. The NameNode servlets should not use RPC to connect to the
NameNode. (Kan Zhang via jghoman)
HDFS-1130. Adds dfs.cluster.administrator ACL configuration that can
be used to control who can view the default hdfs servlets. (ddas)
HDFS-1297. Fix some comments. (Jeff Ames via jghoman)
HDFS-330. Datanode Web UIs should provide robots.txt.
(Allen Wittenauer via jghoman)
HDFS-881. Refactor DataNode Packet header into DataTransferProtocol.
(Todd Lipcon via jghoman)
HDFS-1036. docs for fetchdt
HDFS-1318. Add JMX interface for read access to namenode and datanode
web UI information. (Tanping Wang via suresh).
HDFS-1356. Provide information as to whether or not security is
enabled on web interface for NameNode (boryas)
HDFS-1205. FSDatasetAsyncDiskService should name its threads.
(Todd Lipcon via eli)
HDFS-1111. Introduce getCorruptFileBlocks() for fsck. (Sriram Rao via shv)
HDFS-1395. Add @Override to FSDataset methods that implement
FSDatasetInterface methods. (suresh)
HDFS-1383. Improve the error messages when using hftp://. (szetszwo)
HDFS-1093. Change the FSNamesystem lock to a read/write lock. (dhruba)
HDFS-1407. Change DataTransferProtocol methods to use Block instead
of individual elements of Block. (suresh)
HDFS-1417. Add @Override to SimulatedFSDataset methods that implement
FSDatasetInterface methods. (suresh)
HDFS-1426. Remove unused method BlockInfo#listCount. (hairong)
HDFS-1472. Allow programmatic access to fsck output.
(Ramkumar Vadali via dhruba)
HADOOP-7007. Update the hudson-test-patch ant target to work with the
latest test-patch.sh script (gkesavan)
HDFS-1462. Refactor edit log loading to a separate class from edit log writing.
(Todd Lipcon via eli)
HDFS-1485. Fix typo in BlockPlacementPolicy. (Jingguo Yao via shv)
HDFS-1035. Generate Eclipse's .classpath file from Ivy config. (nigel)
HDFS-1408. Herriot NN and DN clients should vend statistics. (cos)
HDFS-1491 Update Hdfs to match the change of methods from protected to public
in AbstractFileSystem (Hadoop-6903) (sanjay)
HDFS-1160. Improve some FSDataset warnings and comments. (eli)
HDFS-556. Provide info on failed volumes in the web ui. (eli)
HDFS-697. Enable asserts for tests by default. (eli)
HDFS-1187. Modify fetchdt to allow renewing and canceling token.
(Owen O'Malley and Kan Zhang via jghoman)
HDFS-1387. Update HDFS permissions guide for security. (Todd Lipcon via eli)
HDFS-455. Make NN and DN handle in a intuitive way comma-separated
configuration strings. (Michele Catasta via eli)
HDFS-1071. savenamespace should write the fsimage to all configured
fs.name.dir in parallel (Dmytro Molkov via jghoman)
HDFS-1055. Improve thread naming for DataXceivers.
(Todd Lipcon and Ramkumar Vadali via eli).
HDFS-718. Configuration parameter to prevent accidental formatting of
HDFS filesystem. (Andrew Ryan via jghoman)
HDFS-1500. TestOfflineImageViewer failing on trunk. (Todd Lipcon
via hairong)
HDFS-1483. DFSClient.getBlockLocations should indicate if corresponding
blocks are corrupt. (Patrick Kling via hairong)
HDFS-259. Remove intentionally corrupt 0.13 directory layout creation.
(Todd Lipcon via eli)
HDFS-1513. Fix a number of warnings. (eli)
HDFS-1473. Refactor storage management into separate classes than fsimage
file reading/writing. (Todd Lipcon via eli)
HDFS-1582. Remove auto-generated native build files. (rvs via eli)
HDFS-1456. Provide builder for constructing instances of MiniDFSCluster.
(jghoman)
HDFS-1861. Rename dfs.datanode.max.xcievers and bump its default value.
(eli)
HDFS-1052. HDFS Federation - Merge of umbrella jira changes from
HDFS-1052 branch into trunk.
HDFS-1835. DataNode should not depend on SHA1PRNG secure random generator
to generate a storage ID. (John Carrino via todd)
HDFS-1947. DFSClient should use mapreduce.task.attempt.id. (eli)
HDFS-1957. Add documentation for HFTP. (Ari Rabkin via todd)
HDFS-1454. Update the documentation to reflect that clients don't write
blocks to local disk before copying to HDFS. (Harsh J Chouraria via todd)
HDFS-1980. Move build/webapps deeper in the build directory heirarchy
to aid eclipse users. (todd)
HDFS-1619. Remove AC_TYPE* from the libhdfs. (Roman Shaposhnik via eli)
HDFS-1948 Forward port 'hdfs-1520 lightweight namenode operation to
trigger lease recovery' (stack)
HDFS-1954. Improved corrupt files warning on NameNode web UI.
(Patrick Hunt via shv)
HDFS-1409. BackupNode registration throwing
UnsupportedActionException("register") instead of "journal".
(Ching-Shen Chen via shv)
OPTIMIZATIONS
HDFS-1140. Speedup INode.getPathComponents. (Dmytro Molkov via shv)
HDFS-1081. Performance regression in