diff options
| author | Andrew McDermott <andrew.mcdermott@linaro.org> | 2014-02-12 16:59:54 +0000 |
|---|---|---|
| committer | Andrew McDermott <andrew.mcdermott@linaro.org> | 2014-02-12 16:59:54 +0000 |
| commit | 86a9909c577bdd641e7483f56ffbec998ea421eb (patch) | |
| tree | 3f67536f5008b7d434d0a25e2d1aa97d9179b74b /aarch64/share/doc/hadoop/mapreduce | |
Initial import
Signed-off-by: Andrew McDermott <andrew.mcdermott@linaro.org>
Diffstat (limited to 'aarch64/share/doc/hadoop/mapreduce')
| -rw-r--r-- | aarch64/share/doc/hadoop/mapreduce/CHANGES.txt | 6904 | ||||
| -rw-r--r-- | aarch64/share/doc/hadoop/mapreduce/LICENSE.txt | 244 | ||||
| -rw-r--r-- | aarch64/share/doc/hadoop/mapreduce/NOTICE.txt | 2 |
3 files changed, 7150 insertions, 0 deletions
diff --git a/aarch64/share/doc/hadoop/mapreduce/CHANGES.txt b/aarch64/share/doc/hadoop/mapreduce/CHANGES.txt new file mode 100644 index 0000000..2c328ce --- /dev/null +++ b/aarch64/share/doc/hadoop/mapreduce/CHANGES.txt @@ -0,0 +1,6904 @@ +Hadoop MapReduce Change Log + +Release 2.2.0 - 2013-10-13 + + INCOMPATIBLE CHANGES + + NEW FEATURES + + IMPROVEMENTS + + OPTIMIZATIONS + + BUG FIXES + MAPREDUCE-5504. mapred queue -info inconsistent with types (Kousuke Saruta + via tgraves) + + MAPREDUCE-5488. Changed MR client to keep trying to reach the application + when it sees that on attempt's AM is down. (Jian He via vinodkv) + + MAPREDUCE-5515. Fixed MR AM's webapp to depend on a new config + mapreduce.ssl.enabled to enable https and disabling it by default as MR AM + needs to set up its own certificates etc and not depend on clusters'. + (Omkar Vinit Joshi via vinodkv) + + MAPREDUCE-5505. Clients should be notified job finished only after job + successfully unregistered (Zhijie Shen via bikas) + + MAPREDUCE-5503. Fixed a test issue in TestMRJobClient. (Jian He via vinodkv) + + MAPREDUCE-5170. Fixed a wrong log message in CombineFileInputFormat class. + (Sangjin Lee via vinodkv) + + MAPREDUCE-5525. Increase timeout of TestDFSIO.testAppend and + TestMRJobsWithHistoryService.testJobHistoryData. (Chuan Liu via cnauroth) + + MAPREDUCE-5513. ConcurrentModificationException in JobControl (Robert + Parker via jlowe) + + MAPREDUCE-5531. Fix compat with hadoop-1 in mapreduce.(TaskID, + TaskAttemptID) by re-introducing missing constructors. (Robert Kanter via + acmurthy) + + MAPREDUCE-5545. org.apache.hadoop.mapred.TestTaskAttemptListenerImpl.testCommitWindow + times out (Robert Kanter via jlowe) + + MAPREDUCE-5529. Fix compat with hadoop-1 in mapred.TotalOrderPartitioner + by re-introducing (get,set)PartitionFile which takes in JobConf. (Robert + Kanter via acmurthy) + + MAPREDUCE-5538. Fixed MR AppMaster to send job-notification URL only after + the job is really done - a bug caused by MAPREDUCE-5505. (Zhijie Shen via + vinodkv) + + MAPREDUCE-5551. Fix compat with hadoop-1 in + SequenceFileAsBinaryOutputFormat.WritableValueBytes by re-introducing + missing constructors. (Zhijie Shen via acmurthy) + + MAPREDUCE-5544. JobClient#getJob loads job conf twice. (Sandy Ryza) + + MAPREDUCE-5536. Fixed MR AM and JHS to respect + mapreduce.jobhistory.webapp.https.address. (Omkar Vinit Joshi via vinodkv) + + MAPREDUCE-5530. Fix compat with hadoop-1 in + mapred.lib.CombinFileInputFormat by re-introducing + isSplittable(FileSystem, Path) api and ensuring semantic compatibility. + (Robert Kanter via acmurthy) + + MAPREDUCE-5459. Update documentation on how to run MRv1 examples on YARN. + (Zhijie Shen via acmurthy) + + MAPREDUCE-5554. hdfs-site.xml included in hadoop-mapreduce-client-jobclient + tests jar is breaking tests for downstream components (Robert Kanter via + Sandy Ryza) + + MAPREDUCE-5489. MR jobs hangs as it does not use the node-blacklisting + feature in RM requests (Zhijie Shen via bikas) + + MAPREDUCE-5442. $HADOOP_MAPRED_HOME/$HADOOP_CONF_DIR setting not working on + Windows. (Yingda Chen via cnauroth) + + MAPREDUCE-5533. Fixed MR speculation code to track any TaskAttempts that + aren't heart-beating for a while, so that we can aggressively speculate + instead of waiting for task-timeout (Xuan Gong via vinodkv) + + MAPREDUCE-5562. Fixed MR App Master to perform pending tasks like staging-dir + cleanup, sending job-end notification correctly when unregister with RM + fails. (Zhijie Shen via vinodkv) + +Release 2.1.1-beta - 2013-09-23 + + INCOMPATIBLE CHANGES + + NEW FEATURES + + IMPROVEMENTS + + MAPREDUCE-5478. TeraInputFormat unnecessarily defines its own FileSplit + subclass (Sandy Ryza) + + MAPREDUCE-5497. Changed MRAppMaster to sleep only after doing everything else + but just before ClientService to avoid race conditions during RM restart. + (Jian He via vinodkv) + + MAPREDUCE-5379. Include token tracking ids in jobconf. (kkambatl via tucu) + + MAPREDUCE-5523. Added separate configuration properties for https for JHS + without which even when https is enabled, it starts on http port itself. + (Omkar Vinit Joshi via vinodkv) + + OPTIMIZATIONS + + MAPREDUCE-5446. TestJobHistoryEvents and TestJobHistoryParsing have race + conditions (jlowe via kihwal) + + MAPREDUCE-5462. In map-side sort, swap entire meta entries instead of + indexes for better cache performance. (Sandy Ryza) + + MAPREDUCE-1981. Improve getSplits performance by using listLocatedStatus + (Hairong Kuang and Jason Lowe via jlowe) + + BUG FIXES + + MAPREDUCE-5385. Fixed a bug with JobContext getCacheFiles API. (Omkar Vinit + Joshi via vinodkv) + + MAPREDUCE-5428. HistoryFileManager doesn't stop threads when service is + stopped (Karthik Kambatla via jlowe) + + MAPREDUCE-5251. Reducer should not implicate map attempt if it has + insufficient space to fetch map output (Ashwin Shankar via jlowe) + + MAPREDUCE-5317. Stale files left behind for failed jobs (Ravi Prakash via + jlowe) + + MAPREDUCE-5358. MRAppMaster throws invalid transitions for JobImpl + (Devaraj K via jlowe) + + MAPREDUCE-3193. FileInputFormat doesn't read files recursively in the + input path dir (Devaraj K via jlowe) + + MAPREDUCE-5440. TestCopyCommitter Fails on JDK7 (Robert Parker via jlowe) + + MAPREDUCE-5367. Local jobs all use same local working directory + (Sandy Ryza) + + MAPREDUCE-5425. Junit in TestJobHistoryServer failing in jdk 7 (Robert + Parker via jlowe) + + MAPREDUCE-5454. TestDFSIO fails intermittently on JDK7 (Karthik Kambatla + via Sandy Ryza) + + MAPREDUCE-5001. LocalJobRunner has race condition resulting in job + failures (Sandy Ryza via jlowe) + + MAPREDUCE-5466. Changed MR AM to not promote history files of intermediate + AMs in case they are exiting because of errors and thus help history-server + pick up the right history file for the last successful AM. (Jian He via + vinodkv) + + MAPREDUCE-5468. Fix MR AM recovery for map-only jobs. (vinodkv via + acmurthy) + + MAPREDUCE-5470. LocalJobRunner does not work on Windows. (Sandy Ryza via + cnauroth) + + MAPREDUCE-5476. Changed MR AM recovery code to cleanup staging-directory + only after unregistering from the RM. (Jian He via vinodkv) + + MAPREDUCE-5483. revert MAPREDUCE-5357. (rkanter via tucu) + + MAPREDUCE-5441. Changed MR AM to return RUNNING state if exiting when RM + commands to reboot, so that client can continue to track the overall job. + (Jian He via vinodkv) + + MAPREDUCE-5475. MRClientService does not verify ACLs properly (jlowe) + + MAPREDUCE-5414. TestTaskAttempt fails in JDK7 with NPE (Nemon Lou via + devaraj) + + MAPREDUCE-5020. Compile failure with JDK8 (Trevor Robinson via tgraves) + + MAPREDUCE-5164. mapred job and queue commands omit HADOOP_CLIENT_OPTS + (Nemon Lou via devaraj) + + MAPREDUCE-5493. Cleanup in-memory & on-disk segments to prevent leak on + shuffle completion. (jlowe via acmurthy) + +Release 2.1.0-beta - 2013-08-22 + + INCOMPATIBLE CHANGES + + MAPREDUCE-4067. Changed MRClientProtocol api to throw IOException only (Xuan + Gong via vinodkv) + + MAPREDUCE-5234. Change mapred.TaskReport and mapreduce.TaskReport for binary + compatibility with mapred in 1.x but incompatible with 0.23.x. (Mayank Bansal + via vinodkv) + + MAPREDUCE-5156. Change hadoop examples ProgramDriver to be able to run + 1.x examples jar on top of YARN. This change breaks 0.23.x direct usages of + ProgramDriver. (Zhijie Shen via vinodkv) + + MAPREDUCE-5233. Add methods that are changed or removed from JobControl.Job + when compared to 1.x. This breaks 0.23.x users of one API in Job. (Mayank + Bansal via vinodkv) + + MAPREDUCE-5237. Add methods that were removed from ClusterStatus back into + 2.x so as to be compatible with 1.x. Incompatible as + ClusterStatus.UNINITIALIZED_MEMORY_VALUE is a long now and so breaks 0.23.x + but it shouldn't be a big deal in reality. (Zhijie Shen via vinodkv) + + NEW FEATURES + + HADOOP-8562. Enhancements to support Hadoop on Windows Server and Windows + Azure environments. (See breakdown of tasks below for subtasks and + contributors) + + IMPROVEMENTS + + MAPREDUCE-3008. Improvements to cumulative CPU emulation for short running + tasks in Gridmix. (amarrk via tgraves) + + MAPREDUCE-5033. mapred shell script should respect usage flags + (--help -help -h). (Andrew Wang via atm) + + MAPREDUCE-4892. Modify CombineFileInputFormat to not skew input slits' + allocation on small clusters. (Bikas Saha via vinodkv) + + MAPREDUCE-4990. Construct debug strings conditionally in + ShuffleHandler.Shuffle#sendMapOutput(). (kkambatl via tucu) + + MAPREDUCE-4875. coverage fixing for org.apache.hadoop.mapred + (Aleksey Gorshkov via bobby) + + MAPREDUCE-5129. Allow tags to JobHistory for deeper analytics. (billie via + acmurthy) + + MAPREDUCE-3787. [Gridmix] Optimize job monitoring and STRESS mode for + faster job submission. (amarrk via tgraves) + + MAPREDUCE-5079. Changes job recovery to restore state directly from job + history, instaed of simulating state machine events. + (Jason Lowe and Robert Parker via sseth) + + MAPREDUCE-4981. Add WordMean, WordMedian, WordStandardDeviation + to ExamplesDriver. (Plamen Jeliazkov via shv) + + MAPREUDUCE-5059. Change average merge time on Job overview page to be the + time delta between the end of the shuffle and the start of the reduce. + (Omkar Vinit Joshi via vinodkv) + + MAPREDUCE-4985. Add compression option to TestDFSIO usage. + (Plamen Jeliazkov via shv) + + MAPREDUCE-5152. Make MR App to simply pass through the container from RM + instead of extracting and populating information itself to start any + container. (vinodkv) + + MAPREDUCE-5175. Updated MR App to not set envs that will be set by NMs + anyways after YARN-561. (Xuan Gong via vinodkv) + + MAPREDUCE-5069. add concrete common implementations of + CombineFileInputFormat (Sangjin Lee via bobby) + + MAPREDUCE-5145. Changed default max-attempts to be more than one for MR jobs + inline with YARN. (Zhijie Shen via vinodkv) + + MAPREDUCE-5036. Default shuffle handler port should not be 8080. + (Sandy Ryza via tomwhite) + + MAPREDUCE-5159. Change ValueAggregatorJob to add APIs which can support + binary compatibility with hadoop-1 examples. (Zhijie Shen via vinodkv) + + MAPREDUCE-5157. Bring back old sampler related code so that we can support + binary compatibility with hadoop-1 sorter example. (Zhijie Shen via vinodkv) + + MAPREDUCE-5222. Bring back some methods and constants in Jobclient for + binary compatibility with mapred in 1.x. (Karthik Kambatla via vinodkv) + + MAPREDUCE-5235. Bring back old fields and exceptions in Counters for + binary compatibility with mapred in 1.x. (Mayank Bansal via vinodkv) + + MAPREDUCE-5246. Specify application-type at the time of job submission after + YARN-563. (Mayank Bansal via vinodkv) + + MAPREDUCE-5230. Bring back NLineInputFormat.createFileSplit for binary + compatibility with mapred in 1.x (Mayank Bansal via vinodkv) + + MAPREDUCE-5270. Migrated MR app from using BuilderUtil factory methods to + individual record factory methods. (Jian He via vinodkv) + + MAPREDUCE-5263. Bring back old methods and fields in + filecache.DistributedCache for binary compatibility with mapred in 1.x. + (Zhijie Shen via vinodkv) + + MAPREDUCE-5228. Bring back FileInputFormat.Counter and + FileOuputFormat.Counter for binary compatibility with 1.x mapred APIs. + (Mayank Bansal via vinodkv) + + MAPREDUCE-5176. Add annotation for tagging tasks as responsive to + preemption. (Carlo Curino, cdouglas) + + MAPREDUCE-5275. Bring back a couple of APIs in mapreduce.security.TokenCache + for binary compatibility with 1.x mapreduce APIs. (Mayank Bansal via vinodkv) + + MAPREDUCE-5231. Bring back a constructor in mapred's + DBInputFormat.DBRecordReader for binary compatibility with 1.x mapred APIs. + (Zhijie Shen via vinodkv) + + MAPREDUCE-5273. Bring back a couple of protected variables in mapred and + mapreduce CombineFileRecordReader for binary compatibility with 1.x APIs. + (Mayank Bansal via vinodkv) + + MAPREDUCE-5280. Bring back removed constructor and a method in mapreduce + ClusterMetrics for binary compatibility with 1.x APIs. (Mayank Bansal via + vinodkv) + + MAPREDUCE-5289. Updated MR App to use Token directly after YARN-717. (Jian He + via vinodkv) + + MAPREDUCE-5229. Put back FileOutputCommitter.TEMP_DIR_NAME in mapreduce for + binary compatibility with 1.x APIs. (Zhijie Shen via vinodkv) + + MAPREDUCE-5274. Bring back SecureShuffleUtils.toHex in mapreduce for binary + compatibility with 1.x APIs. (Mayank Bansal via vinodkv) + + MAPREDUCE-5300. Fix backward incompatibility for + o.a.h.mapreduce.filecache.DistributedCache. (Zhijie Shen via acmurthy) + + MAPREDUCE-5283. Over 10 different tests have near identical + implementations of AppContext (Sandy Ryza via jlowe) + + MAPREDUCE-5199. Removing ApplicationTokens file as it is no longer needed. + (Daryn Sharp via vinodkv) + + MAPREDUCE-5192. Allow for alternate resolutions of TaskCompletionEvents. + (cdouglas via acmurthy) + + MAPREDUCE-5184. Document compatibility for MapReduce applications in + hadoop-2 vis-a-vis hadoop-1. (Zhijie Shen via acmurthy) + + MAPREDUCE-5194. Heed interrupts during Fetcher shutdown. (cdouglas) + + MAPREDUCE-5326. Added version to shuffle header. (Zhijie Shen via + acmurthy) + + MAPREDUCE-5333. Add test that verifies MRAM works correctly when sending + requests with non-normalized capabilities. (ywskycn via tucu) + + MAPREDUCE-5398. MR changes for YARN-513 (Jian He via bikas) + + OPTIMIZATIONS + + MAPREDUCE-4974. Optimising the LineRecordReader initialize() method + (Gelesh via bobby) + + MAPREDUCE-5268. Improve history server startup performance (Karthik + Kambatla via jlowe) + + MAPREDUCE-5352. Optimize node local splits generated by + CombineFileInputFormat. (sseth) + + BUG FIXES + + MAPREDUCE-4671. AM does not tell the RM about container requests which are + no longer needed. (Bikas Saha via sseth) + + MAPREDUCE-4994. -jt generic command line option does not work. (sandyr via tucu) + + MAPREDUCE-5000. Fixes getCounters when speculating by fixing the selection + of the best attempt for a task. (Jason Lowe via sseth) + + MAPREDUCE-4994. Addendum fixing testcases failures. (sandyr via tucu) + + MAPREDUCE-4846. Some JobQueueInfo methods are public in MR1 but protected + in MR2. (Sandy Ryza via tomwhite) + + MAPREDUCE-5013. mapred.JobStatus compatibility: MR2 missing constructors + from MR1. (Sandy Ryza via tomwhite) + + MAPREDUCE-4951. Container preemption interpreted as task failure. + (Sandy Ryza via tomwhite) + + MAPREDUCE-5008. Merger progress miscounts with respect to EOF_MARKER. + (Sandy Ryza via tomwhite) + + MAPREDUCE-4693. History server should include counters for failed tasks. + (Xuan Gong via sseth) + + MAPREDUCE-4896. mapred queue -info spits out ugly exception when queue does + not exist. (sandyr via tucu) + + MAPREDUCE-3685. Fix bugs in MergeManager to ensure compression codec is + appropriately used and that on-disk segments are correctly sorted on + file-size. (Anty Rao and Ravi Prakash via acmurthy) + + MAPREDUCE-4571. TestHsWebServicesJobs fails on jdk7. (tgraves via tucu) + + MAPREDUCE-4716. TestHsWebServicesJobsQuery.testJobsQueryStateInvalid + fails with jdk7. (tgraves via tucu) + + MAPREDUCE-5075. DistCp leaks input file handles since ThrottledInputStream + does not close the wrapped InputStream. (Chris Nauroth via szetszwo) + + MAPREDUCE-3872. Fix an event handling races in ContainerLauncherImpl. + (Robert Kanter via sseth) + + MAPREDUCE-5062. Fix MR AM to read max-retries from the RM. (Zhijie Shen via + vinodkv) + + MAPREDUCE-3829. [Gridmix] Gridmix should give better error message when + input data directory already exists and -generate opton is + given.(ravigummadi via tgraves) + + MAPREDUCE-2722. [Gridmix] Gridmix simulated job's map's hdfsBytesRead + counter is wrong when compressed input is used.(ravigummadi via tgraves) + + MAPREDUCE-3953. [Gridmix] Gridmix throws NPE and does not simulate a + job if the trace contains null taskStatus for a task. (ravigummadi via + tgraves) + + MAPREDUCE-4087. [Gridmix] GenerateDistCacheData job of Gridmix can + become slow in some cases (ravigummadi via tgraves). + + MAPREDUCE-5077. Remove mapreduce.util.ResourceCalculatorPlugin and related + code. (Karthik Kambatla via sseth) + + MAPREDUCE-4083. [Gridmix] NPE in cpu emulation. (amarrk via tgraves) + + MAPREDUCE-4100. [Gridmix] Bug fixed in compression emulation feature for + map only jobs. (amarrk via tgraves) + + MAPREDUCE-4356. [Rumen] Provide access to the method + ParsedTask.obtainTaskAttempts(). (ravigummadi via tgraves) + + MAPREDUCE-4149. [Rumen] Rumen fails to parse certain counter + strings. (ravigummadi via tgraves) + + MAPREDUCE-3757. [Rumen] Fixed Rumen Folder to adjust shuffleFinished and + sortFinished times when needed. (Ravi Gummadi via tgraves) + + MAPREDUCE-5138. Fix LocalDistributedCacheManager after YARN-112. (Omkar Vinit + Joshi via vinodkv) + + MAPREDUCE-5086. MR app master deletes staging dir when sent a reboot + command from the RM. (Jian He via jlowe) + + MAPREDUCE-5113. Streaming input/output types are ignored with java + mapper/reducer. (sandyr via tucu) + + MAPREDUCE-5098. Fix findbugs warnings in gridmix. (kkambatl via tucu) + + MAPREDUCE-5137. AM web UI: clicking on Map Task results in 500 error + (Thomas Graves via jlowe) + + MAPREDUCE-5136. TestJobImpl->testJobNoTasks fails with IBM JAVA (Amir + Sanjar via jlowe) + + MAPREDUCE-5139. Update MR AM to use the modified startContainer API after + YARN-486. (Xuan Gong via vinodkv) + + MAPREDUCE-5151. Update MR AM to use standard exit codes from the API after + YARN-444. (Sandy Ryza via vinodkv) + + MAPREDUCE-5140. MR part of YARN-514 (Zhijie Shen via bikas) + + MAPREDUCE-5128. mapred-default.xml is missing a bunch of history server + configs. (sandyr via tucu) + + MAPREDUCE-4898. FileOutputFormat.checkOutputSpecs and + FileOutputFormat.setOutputPath incompatible with MR1. (rkanter via tucu) + + MAPREDUCE-5078. TestMRAppMaster fails on Windows due to mismatched path + separators. (Chris Nauroth via sseth) + + MAPREDUCE-4932. mapreduce.job#getTaskCompletionEvents incompatible with + Hadoop 1. (rkanter via tucu) + + MAPREDUCE-5163. Update MR App to not use API utility methods for collections + after YARN-441. (Xuan Gong via vinodkv) + + MAPREDUCE-5066. Added a timeout for the job.end.notification.url. (Ivan + Mitic via acmurthy) + + MAPREDUCE-5146. application classloader may be used too early to load + classes. (Sangjin Lee via tomwhite) + + MAPREDUCE-4737. Ensure that mapreduce APIs are semantically consistent + with mapred API w.r.t Mapper.cleanup and Reducer.cleanup; in the sense that + cleanup is now called even if there is an error. The old mapred API + already ensures that Mapper.close and Reducer.close are invoked during + error handling. Note that it is an incompatible change, however end-users + can override Mapper.run and Reducer.run to get the old (inconsistent) + behaviour. (acmurthy) + + MAPREDUCE-5166. Fix ConcurrentModificationException due to insufficient + synchronization on updates to task Counters. (Sandy Ryza via acmurthy) + + MAPREDUCE-5181. RMCommunicator should not use AMToken from the env. + (Vinod Kumar Vavilapalli via sseth) + + MAPREDUCE-5178. Update MR App to set progress in ApplicationReport after + YARN-577. (Hitesh Shah via vinodkv) + + MAPREDUCE-5167. Update MR App after YARN-562 to use the new builder API + for the container. (Jian He via vinodkv) + + MAPREDUCE-5179. Fix unit test in TestHSWebServices which fails when + versionInfo has parantheses like when running on a git checkout. (Hitesh Shah + via vinodkv) + + MAPREDUCE-5193. A few MR tests use block sizes which are smaller than the + default minimum block size. (Andrew Wang via atm) + + MAPREDUCE-5205. Fixed MR App to load tokens correctly. (vinodkv) + + MAPREDUCE-5204. Handling YarnRemoteException separately from IOException in + MR app after YARN-629. (Xuan Gong via vinodkv) + + MAPREDUCE-5209. Fix units in a ShuffleScheduler log message. + (Tsuyoshi OZAWA via cdouglas) + + MAPREDUCE-5212. Handling YarnRemoteException separately from IOException in + MR App's use of ClientRMProtocol after YARN-631. (Xuan Gong via vinodkv) + + MAPREDUCE-5226. Handling YarnRemoteException separately from IOException in + MR App's use of AMRMProtocol after YARN-630. (Xuan Gong via vinodkv) + + MAPREDUCE-4942. mapreduce.Job has a bunch of methods that throw + InterruptedException so its incompatible with MR1. (rkanter via tucu) + + MAPREDUCE-5239. Updated MR App to reflect YarnRemoteException changes after + YARN-634. (Siddharth Seth via vinodkv) + + MAPREDUCE-5208. Modified ShuffleHandler to use SecureIOUtils for reading + local files. (Omkar Vinit Joshi via vinodkv) + + MAPREDUCE-5220. Setter methods in TaskCompletionEvent are public in MR1 and + protected in MR2. (sandyr via tucu) + + MAPREDUCE-5240. Fix a bug in MRAppMaster because of which OutputCommitter + could not access credentials set by the user. (vinodkv) + + MAPREDUCE-5244. Two functions changed their visibility in JobStatus. + (zjshen via tucu) + + MAPREDUCE-4927. Historyserver 500 error due to NPE when accessing specific + counters page for failed job. (Ashwin Shankar via jlowe) + + MAPREDUCE-5257. Fix issues in TestContainerLauncherImpl after YARN-617. + (Omkar Vinit Joshi via vinodkv) + + MAPREDUCE-5282. Updating MR App to use immutable ApplicationID after + YARN-716. (Siddharth Seth via vinodkv) + + MAPREDUCE-5286. Change MapReduce to use ContainerTokenIdentifier instead + of the entire Container in the startContainer call - YARN-684. + (Vinod Kumar Vavilapalli via sseth) + + MAPREDUCE-5299. Fix backward incompatibility for TaskCompletionEvent by + adding back setTaskID. (Zhijie Shen via acmurthy) + + MAPREDUCE-5296. Fix backward incompatibility for JobControl by adding the + omitted addJob. (Zhijie Shen via acmurthy) + + MAPREDUCE-5245. Added back constants to JobConf to fix incompatibilities. + (Zhijie Shen via acmurthy) + + MAPREDUCE-5297. Updated MR App since BuilderUtils is no longer public + after YARN-748. (Jian He via vinodkv) + + MAPREDUCE-5301. Updated MR code to work with YARN-635 changes of renaming + YarnRemoteException to YarnException. (Siddharth Seth via vinodkv) + + MAPREDUCE-5308. Shuffling to memory can get out-of-sync when fetching + multiple compressed map outputs (Nathan Roberts via jlowe) + + MAPREDUCE-5315. DistCp reports success even on failure. (mithun and jlowe + via daryn) + + MAPREDUCE-5259. TestTaskLog fails on Windows because of path separators + missmatch. (Ivan Mitic via cnauroth) + + MAPREDUCE-4019. -list-attempt-ids is not working (Ashwin Shankar, + Devaraj K, and B Anil Kumar via jlowe) + + MAPREDUCE-5334. Fix failing unit tests - TestContainerLauncher, + TestContainerLauncherImpl. (Vinod Kumar Vavilapalli via sseth) + + MAPREDUCE-5325. MR changes related to YARN-727. ClientRMProtocol.getAllApplications + should accept ApplicationType as a parameter. (Xuan Gong via hitesh) + + MAPREDUCE-5291. Change MR App to use updated property names in + container-log4j.properties. (Zhijie Shen via sseth) + + MAPREDUCE-5303. Changed MR app after moving ProtoBase to package impl.pb via + YARN-724. (Jian He via vinodkv) + + MAPREDUCE-5312. TestRMNMInfo is failing. (sandyr via tucu) + + MAPREDUCE-5304. mapreduce.Job killTask/failTask/getTaskCompletionEvents + methods have incompatible signature changes. (kkambatl via tucu) + + MAPREDUCE-5298. Moved MapReduce services to YARN-530 stricter lifecycle. + (Steve Loughran via vinodkv) + + MAPREDUCE-5319. Set user.name in job.xml. (Xuan Gong via acmurthy) + + MAPREDUCE-5310. MRAM should not normalize allocation request capabilities. + (tucu) + + MAPREDUCE-5213. Re-assess TokenCache methods marked @Private. + (kkambatl via tucu) + + MAPREDUCE-5412. Update MR app to use multiple containers API of + ContainerManager after YARN-926. (Jian He via vinodkv) + + MAPREDUCE-5421. Fixed TestNonExistentJob failure after YARN-873. (Junping Du + via vinodkv) + + MAPREDUCE-5419. TestSlive is getting FileNotFound Exception (Robert Parker + via jlowe) + + MAPREDUCE-5399. Unnecessary Configuration instantiation in IFileInputStream + slows down merge. (Stanislav Barton via Sandy Ryza) + + BREAKDOWN OF HADOOP-8562 SUBTASKS + + MAPREDUCE-4739. Some MapReduce tests fail to find winutils. + (Chris Nauroth via suresh) + + MAPREDUCE-4780. MapReduce distribution build fails on Windows. + (Chris Nauroth via suresh) + + MAPREDUCE-4790. MapReduce build script would be more readable using abspath. + (Chris Nauroth via suresh) + + MAPREDUCE-4869. Fix TestMapReduceChildJVM. (Chris Nauroth via acmurthy) + + MAPREDUCE-4870. Fix TestMRJobsWithHistoryService. (Chris Nauroth via acmurthy) + + MAPREDUCE-4983. Fixed various platform specific assumptions in various tests, + so that they can pass on Windows too. (Chris Nauroth via vinodkv) + + HADOOP-9372. Fix bad timeout annotations on tests. + (Arpit Agarwal via suresh) + + MAPREDUCE-4885. Streaming tests have multiple failures on Windows. (Chris + Nauroth via bikas) + + MAPREDUCE-5177. Use common utils FileUtil#setReadable/Writable/Executable & + FileUtil#canRead/Write/Execute. (Ivan Mitic via suresh) + + MAPREDUCE-5349. TestClusterMapReduceTestCase and TestJobName fail on Windows + in branch-2. (Chuan Liu via cnauroth) + + MAPREDUCE-5355. MiniMRYarnCluster with localFs does not work on Windows. + (Chuan Liu via cnauroth) + + MAPREDUCE-5359. JobHistory should not use File.separator to match timestamp + in path. (Chuan Liu via cnauroth) + + MAPREDUCE-5357. Job staging directory owner checking could fail on Windows. + (Chuan Liu via cnauroth) + + MAPREDUCE-5360. TestMRJobClient fails on Windows due to path format. + (Chuan Liu via cnauroth) + + MAPREDUCE-5366. TestMRAsyncDiskService fails on Windows. (Chuan Liu via + cnauroth) + + MAPREDUCE-5187. Create mapreduce command scripts on Windows. (Chuan Liu via + cnauroth) + + MAPREDUCE-4374. Fix child task environment variable config and add support + for Windows. (Chuan Liu via cnauroth) + +Release 2.0.5-alpha - 06/06/2013 + + INCOMPATIBLE CHANGES + + NEW FEATURES + + IMPROVEMENTS + + OPTIMIZATIONS + + BUG FIXES + + MAPREDUCE-5240 inside of FileOutputCommitter the initialized Credentials cache + appears to be empty. (vinodkv) + +Release 2.0.4-alpha - 2013-04-25 + + INCOMPATIBLE CHANGES + + NEW FEATURES + + IMPROVEMENTS + + OPTIMIZATIONS + + BUG FIXES + + MAPREDUCE-5006. Fix failing streaming tests due to MAPREDUCE-4994. + (Sandy Ryza via tomwhite) + + MAPREDUCE-5088. MR Client gets an renewer token exception while Oozie is + submitting a job (Daryn Sharp via cos) + + MAPREDUCE-5117. Changed MRClientProtocolPBClientImpl to be closeable and thus + fix failures in renewal of HistoryServer's delegations tokens. (Siddharth + Seth via vinodkv) + + MAPREDUCE-5083. MiniMRCluster should use a random component when creating an + actual cluster (Siddharth Seth via hitesh) + + MAPREDUCE-5094. Disabled memory monitoring by default in MiniMRYarnCluster + to avoid some downstream tests failing. (Siddharth Seth via vinodkv) + +Release 2.0.3-alpha - 2013-02-06 + + INCOMPATIBLE CHANGES + + MAPREDUCE-4123. Remove the 'mapred groups' command, which is no longer + supported. (Devaraj K via sseth) + + MAPREDUCE-4938. Use token request messages defined in hadoop common. + (suresh) + + NEW FEATURES + + MAPREDUCE-4520. Added support for MapReduce applications to request for + CPU cores along-with memory post YARN-2. (acmurthy) + + MAPREDUCE-4810. Added new admin command options for MR AM. (Jerry Chen via + vinodkv) + + MAPREDUCE-4049. Experimental api to allow for alternate shuffle plugins. + (Avner BenHanoch via acmurthy) + + MAPREDUCE-4807. Allow MapOutputBuffer to be pluggable. (masokan via tucu) + + MAPREDUCE-4808. Refactor MapOutput and MergeManager to facilitate reuse + by Shuffle implementations. (masokan via tucu) + + IMPROVEMENTS + + MAPREDUCE-3678. The Map tasks logs should have the value of input + split it processed. (harsh) + + MAPREDUCE-4616. Improve javadoc for MultipleOutputs. (Tony Burton via + acmurthy) + + HADOOP-8911. CRLF characters in source and text files. + (Raja Aluri via suresh) + + MAPREDUCE-4703. Add the ability to start the MiniMRClientCluster using + the configurations used before it is being stopped. (ahmed.radwan via tucu) + + MAPREDUCE-4845. ClusterStatus.getMaxMemory() and getUsedMemory() exist in + MR1 but not MR2. (Sandy Ryza via tomwhite) + + MAPREDUCE-4899. Implemented a MR specific plugin for tracking finished + applications that YARN's ResourceManager doesn't keep track off anymore + (Derek Dagit via vinodkv) + + MAPREDUCE-4920. Use security token protobuf definition from hadoop common. + (Suresh Srinivas via vinodkv) + + MAPREDUCE-4907. TrackerDistributedCacheManager issues too many getFileStatus + calls. (sandyr via tucu) + + MAPREDUCE-4949. Enable multiple pi jobs to run in parallel. (sandyr via tucu) + + MAPREDUCE-4809. Change visibility of classes for pluggable sort changes. + (masokan via tucu) + + MAPREDUCE-4838. Add additional fields like Locality, Avataar to the + JobHistory logs. (Zhijie Shen via sseth) + + MAPREDUCE-4971. Minor extensibility enhancements to Counters & + FileOutputFormat. (Arun C Murthy via sseth) + + MAPREDUCE-4977. Documentation for pluggable shuffle and pluggable sort. + (tucu) + + OPTIMIZATIONS + + MAPREDUCE-4893. Fixed MR ApplicationMaster to do optimal assignment of + containers to get maximum locality. (Bikas Saha via vinodkv) + + BUG FIXES + + MAPREDUCE-4272. SortedRanges.Range#compareTo is not spec compliant. + (Yu Gao via llu) + + MAPREDUCE-4607. Race condition in ReduceTask completion can result in Task + being incorrectly failed. (Bikas Saha via tomwhite) + + MAPREDUCE-4646. Fixed MR framework to send diagnostic information correctly + to clients in case of failed jobs also. (Jason Lowe via vinodkv) + + MAPREDUCE-4674. Hadoop examples secondarysort has a typo + "secondarysrot" in the usage. (Robert Justice via eli) + + MAPREDUCE-4681. Fix unit tests broken by HDFS-3910. (acmurthy) + + MAPREDUCE-4712. mr-jobhistory-daemon.sh doesn't accept --config + (Vinod Kumar Vavilapalli via tgraves) + + MAPREDUCE-4654. TestDistCp is ignored. (Sandy Ryza via tomwhite) + + MAPREDUCE-4637. Handle TaskAttempt diagnostic updates while in the NEW and + UNASSIGNED states. (Mayank Bansal via sseth) + + MAPREDUCE-1806. CombineFileInputFormat does not work with paths not on default FS. (Gera Shegalov via tucu) + + MAPREDUCE-4777. In TestIFile, testIFileReaderWithCodec relies on + testIFileWriterWithCodec. (Sandy Ryza via tomwhite) + + MAPREDUCE-4800. Cleanup o.a.h.mapred.MapTaskStatus - remove unused + code. (kkambatl via tucu) + + MAPREDUCE-4861. Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal. + (kkambatl via tucu) + + MAPREDUCE-4856. TestJobOutputCommitter uses same directory as + TestJobCleanup. (Sandy Ryza via tomwhite) + + MAPREDUCE-4895. Fix compilation failure of org.apache.hadoop.mapred. + gridmix.TestResourceUsageEmulators (Dennis Y via tgraves) + + MAPREDUCE-4278. Cannot run two local jobs in parallel from the same + gateway. (Sandy Ryza via tomwhite) + + MAPREDUCE-1700. User supplied dependencies may conflict with MapReduce + system JARs. (tomwhite) + + MAPREDUCE-4936. JobImpl uber checks for cpu are wrong (Arun C Murthy via + jlowe) + + MAPREDUCE-4924. flakey test: org.apache.hadoop.mapred.TestClusterMRNotification.testMR. + (rkanter via tucu) + + MAPREDUCE-4923. Add toString method to TaggedInputSplit. (sandyr via tucu) + + MAPREDUCE-4948. Fix a failing unit test TestYARNRunner.testHistoryServerToken. + (Junping Du via sseth) + + MAPREDUCE-4803. Remove duplicate copy of TestIndexCache. (Mariappan Asokan + via sseth) + + MAPREDUCE-2264. Job status exceeds 100% in some cases. + (devaraj.k and sandyr via tucu) + + MAPREDUCE-4969. TestKeyValueTextInputFormat test fails with Open JDK 7. + (Arpit Agarwal via suresh) + + MAPREDUCE-4884. Streaming tests fail to start MiniMRCluster due to missing + queue configuration. (Chris Nauroth via suresh) + + MAPREDUCE-4953. HadoopPipes misuses fprintf. (Andy Isaacson via atm) + +Release 2.0.2-alpha - 2012-09-07 + + INCOMPATIBLE CHANGES + + NEW FEATURES + + MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process + command-line. (ahmed via tucu) + + MAPREDUCE-4417. add support for encrypted shuffle (tucu) + + MAPREDUCE-4355. Add RunningJob.getJobStatus() (kkambatl via tucu) + + MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu) + + MAPREDUCE-4438. Add a simple, generic client to run 'easy' AMs in YARN. + (Bikas Saha via acmurthy) + + IMPROVEMENTS + + MAPREDUCE-4157. ResourceManager should not kill apps that are well behaved + (Jason Lowe via bobby) + + MAPREDUCE-4511. Add IFile readahead (ahmed via tucu) + + MAPREDUCE-4408. allow jobs to set a JAR that is in the distributed cached + (rkanter via tucu) + + MAPREDUCE-4440. Changed SchedulerApp and SchedulerNode to be a minimal + interface to allow schedulers to maintain their own. (acmurthy) + + MAPREDUCE-4146. Support limits on task status string length and number of + block locations in branch-2. (Ahmed Radwan via tomwhite) + + MAPREDUCE-3871. Allow symlinking in LocalJobRunner DistributedCache. + (tomwhite) + + MAPREDUCE-3921. MR AM should act on node health status changes. + (Bikas Saha via sseth) + + MAPREDUCE-4355. Add RunningJob.getJobStatus() (kkambatl via tucu) + + MAPREDUCE-4427. Added an 'unmanaged' mode for AMs so as to ease + development of new applications. (Bikas Saha via acmurthy) + + MAPREDUCE-3289. Make use of fadvise in the NM's shuffle handler. + (Todd Lipcon and Siddharth Seth via sseth) + + MAPREDUCE-4580. Change MapReduce to use the yarn-client module. + (Vinod Kumar Vavilapalli via sseth) + + MAPREDUCE-4579. Split TestTaskAttempt into two so as to pass tests on + jdk7. (Thomas Graves via vinodkv) + + MAPREDUCE-4638. MR AM supplies MapReduce jars in classpath rather than + rely on YARN. (acmurthy) + + BUG FIXES + + MAPREDUCE-4422. YARN_APPLICATION_CLASSPATH needs a documented default value in + YarnConfiguration. (ahmed via tucu) + + MAPREDUCE-4406. Users should be able to specify the MiniCluster ResourceManager + and JobHistoryServer ports. (ahmed via tucu) + + MAPREDUCE-4407. Add hadoop-yarn-server-tests-<version>-tests.jar to hadoop dist + package. (ahmed via tucu) + + MAPREDUCE-4465. Update description of yarn.nodemanager.address property. + (bowang via tucu) + + MAPREDUCE-4342. Distributed Cache gives inconsistent result if cache files + get deleted from tasktracker. (mayank_bansal via tucu) + + MAPREDUCE-4498. Remove hsqldb jar from Hadoop runtime classpath. (rkanter via tucu) + + MAPREDUCE-4494. TestFifoScheduler failing with Metrics source QueueMetrics,q0=default + already exists!. (ahmed.radwan via tucu) + + MAPREDUCE-4484. Incorrect IS_MINI_YARN_CLUSTER property name in YarnConfiguration. + (ahmed.radwan via tucu) + + MAPREDUCE-4562. Support for "FileSystemCounter" legacy counter group name + for compatibility reasons is creating incorrect counter name. + (Jarek Jarcec Cecho via tomwhite) + + MAPREDUCE-4068. Jars in lib subdirectory of the submittable JAR are not added to the + classpath (rkanter via tucu) + + MAPREDUCE-4577. HDFS-3672 broke + TestCombineFileInputFormat.testMissingBlocks() test (atm) + + MAPREDUCE-4470. Fix TestCombineFileInputFormat.testForEmptyFile (ikatsov via tucu) + + MAPREDUCE-4608. hadoop-mapreduce-client is missing some dependencies. + (tucu via tomwhite) + + MAPREDUCE-4610. Support deprecated mapreduce.job.counters.limit property in + MR2. (tomwhite) + + MAPREDUCE-4629. Remove JobHistory.DEBUG_MODE (Karthik Kambatla via bobby) + + MAPREDUCE-4642. MiniMRClientClusterFactory should not use job.setJar() (rkanter via tucu) + + MAPREDUCE-4148. MapReduce should not have a compile-time dependency on + HDFS. (tomwhite) + + MAPREDUCE-4250. hadoop-config.sh missing variable exports, causes Yarn + jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) + + MAPREDUCE-4002. MultiFileWordCount job fails if the input path is not + from default file system. (Bhallamudi Venkata Siva Kamesh via todd) + + MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta. + (todd via bobby) + + MAPREDUCE-4262. NM gives wrong log message saying "Connected to + ResourceManager" before trying to connect. (Devaraj K via tgraves) + + MAPREDUCE-4276. Allow setting yarn.nodemanager.delete.debug-delay-sec + property to "-1" for easier container debugging. (ahmed via tucu) + + MAPREDUCE-4224. TestFifoScheduler throws + org.apache.hadoop.metrics2.MetricsException (Devaraj K via tgraves) + + MAPREDUCE-3493. Add the default mapreduce.shuffle.port property + to mapred-default.xml (Madhukara Phatak via harsh) + + MAPREDUCE-4307. TeraInputFormat calls FileSystem.getDefaultBlockSize() + without a Path - Failure when using ViewFileSystem. (Ahmed Radwan via eli) + + MAPREDUCE-4313. TestTokenCache doesn't compile due + TokenCache.getDelegationToken compilation error (bobby) + + MAPREDUCE-3873. Fixed NodeManagers' decommissioning at RM to accept IP + addresses also. (xieguiming via vinodkv) + + MAPREDUCE-4306. Fix distributed shell to work with users other than the one + running the daemons. (Ahmed Radwan via sseth) + + MAPREDUCE-4031. Prevent a Node Manager hang during shutdown. + (Devaraj K via sseth) + + MAPREDUCE-4336. Distributed Shell fails when used with the CapacityScheduler + (ahmed via tucu) + + MAPREDUCE-4290. Fix Yarn Applicaiton Status to MR JobState conversion. + (Devaraj K via sseth) + + MAPREDUCE-2289. Permissions race can make getStagingDir fail on local filesystem + (ahmed via tucu) + + MAPREDUCE-4372. Deadlock in Resource Manager (Devaraj K via bobby) + + MAPREDUCE-4376. TestClusterMRNotification times out (Kihwal Lee via bobby) + + MAPREDUCE-4383. HadoopPipes.cc needs to include unistd.h. + (Andy Isaacson via eli) + + MAPREDUCE-2739. Update installation docs (remove YarnClientFactory) (bowang via tucu) + + MAPREDUCE-3993. Graceful handling of codec errors during decompression + (kkambatl via tucu) + + MAPREDUCE-4416. Some tests fail if Clover is enabled (Kihwal Lee via bobby) + + MAPREDUCE-4441. Fix build issue caused by MR-3451 (kkambatl via tucu) + + HADOOP-8499. Lower min.user.id to 500 for the tests. + (Colin Patrick McCabe via eli) + + MAPREDUCE-4395. Possible NPE at ClientDistributedCacheManager + #determineTimestamps (Bhallamudi via bobby) + + MAPREDUCE-4380. Empty Userlogs directory is getting created under logs + directory (Devaraj K via bobby) + + MAPREDUCE-4649. Ensure MapReduce JobHistory Daemon doens't assume + HADOOP_YARN_HOME and HADOOP_MAPRED_HOME are the same. (vinodkv via + acmurthy) + +Release 2.0.0-alpha - 05-23-2012 + + INCOMPATIBLE CHANGES + + MAPREDUCE-3545. Remove Avro RPC. (suresh) + + NEW FEATURES + + IMPROVEMENTS + + MAPREDUCE-2887. Due to HADOOP-7524, change RPC to allow multiple protocols + including multuple versions of the same protocol (Sanjay Radia) + + MAPREDUCE-2934. MR portion of HADOOP-7607 - Simplify the RPC proxy cleanup + process (atm) + + HADOOP-7862. MR changes to work with HADOOP 7862: Move the support for + multiple protocols to lower layer so that Writable, PB and Avro can all + use it (Sanjay Radia) + + MAPREDUCE-3909 Javadoc the Service interfaces (stevel) + + MAPREDUCE-3885. Avoid an unnecessary copy for all requests/responses in + MRs ProtoOverHadoopRpcEngine. (Devaraj Das via sseth) + + MAPREDUCE-3935. Annotate Counters.Counter and Counters.Group as @Public. + (tomwhite) + + MAPREDUCE-3991. Streaming FAQ has some wrong instructions about input files + splitting. (harsh) + + MAPREDUCE-3773. Add queue metrics with buckets for job run times. (omalley + via acmurthy) + + MAPREDUCE-3970. Add ServiceOperations class to aid working with Services + (stevel) + + MAPREDUCE-3353. Add a channel between RM and AM to get information on + nodes. (Bikas Saha via acmurthy) + + MAPREDUCE-3955. Change MR to use ProtobufRpcEngine from hadoop-common + instead of ProtoOverHadoopRpcEngine. (Jitendra Nath Pandey via sseth) + + MAPREDUCE-4103. Fix HA docs for changes to shell command fencer args (todd) + + MAPREDUCE-4093. Improve RM WebApp start up when proxy address is not set + (Devaraj K vai bobby) + + MAPREDUCE-4138. Reduce memory usage of counters due to non-static nested + classes. (tomwhite) + + MAPREDUCE-3883. Document yarn.nodemanager.delete.debug-delay-sec + configuration property (Eugene Koontz via tgraves) + + MAPREDUCE-4219. make default container-executor.conf.dir be a path + relative to the container-executor binary. (rvs via tucu) + + MAPREDUCE-4205. retrofit all JVM shutdown hooks to use ShutdownHookManager + (tucu) + + HADOOP-8285 MR changes for Use ProtoBuf for RpcPayLoadHeader (sanjay radia) + + MAPREDUCE-2220. Fix new API FileOutputFormat-related typos in + mapred-default.xml (Rui Kubo via harsh) + + MAPREDUCE-3907. Document entries mapred-default.xml for the + jobhistory server. (Eugene Koontz via harsh) + + MAPREDUCE-3906. Fix inconsistency in documentation regarding + mapreduce.jobhistory.principal. (Eugene Koontz via harsh) + + MAPREDUCE-4432. Confusing warning message when GenericOptionsParser + is not used. (Gabriel Reid via harsh) + + OPTIMIZATIONS + + BUG FIXES + + MAPREDUCE-3740. Fixed broken mapreduce compilation after the patch for + HADOOP-7965. (Devaraj K via vinodkv) + + MAPREDUCE-3818. Fixed broken compilation in TestSubmitJob after the patch + for HDFS-2895. (Suresh Srinivas via vinodkv) + + MAPREDUCE-2942. TestNMAuditLogger.testNMAuditLoggerWithIP failing (Thomas + Graves via mahadev) + + MAPREDUCE-3933. Failures because MALLOC_ARENA_MAX is not set (ahmed via tucu) + + MAPREDUCE-3728. ShuffleHandler can't access results when configured in a + secure mode (ahmed via tucu) + + MAPREDUCE-3952. In MR2, when Total input paths to process == 1, + CombinefileInputFormat.getSplits() returns 0 split. (zhenxiao via tucu) + + MAPREDUCE-3578. Starting nodemanager as root gives "Unknown -jvm option" + (tomwhite) + + MAPREDUCE-3348. Fixed a bug in MR client to redirect to JobHistoryServer + correctly when RM forgets the app. (Devaraj K via vinodkv) + + MAPREDUCE-3974. TestSubmitJob in MR1 tests doesn't compile after HDFS-162 + merge. (atm) + + MAPREDUCE-4007. JobClient getJob(JobID) should return NULL if the job + does not exist (for backwards compatibility) (tucu) + + MAPREDUCE-3431 NPE in Resource Manager shutdown. (stevel) + + MAPREDUCE-4010. TestWritableJobConf fails on trunk (tucu via bobby) + + MAPREDUCE-3992. Reduce fetcher doesn't verify HTTP status code of response + (todd) + + MAPREDUCE-4066. Use default value when fetching MR_AM_STAGING_DIR + (xieguiming via harsh) + + MAPREDUCE-3377. Added a unit test to ensure OutputCommitter.checkOutputSpecs + is called prior to copying job.xml. (Jane Chen via acmurthy) + + MAPREDUCE-4081. TestMROutputFormat.java does not compile (Jason Lowe via + bobby) + + MAPREDUCE-4082. hadoop-mapreduce-client-app's mrapp-generated-classpath + file should not be in the module JAR (tucu) + + MAPREDUCE-3916. various issues with running yarn proxyserver (devaraj via tucu) + + MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu) + + MAPREDUCE-4098. TestMRApps testSetClasspath fails (tucu) + + MAPREDUCE-4097. tools testcases fail because missing mrapp-generated-classpath + file in classpath (rvs via tucu) + + MAPREDUCE-4113. Fix tests org.apache.hadoop.mapred.TestClusterMRNotification + (Devaraj K via bobby) + + MAPREDUCE-4112. Fix tests org.apache.hadoop.mapred.TestClusterMapReduceTestCase + (Devaraj K via bobby) + + MAPREDUCE-4111. Fix tests in org.apache.hadoop.mapred.TestJobName (Devaraj + K via bobby) + + MAPREDUCE-4110. Fix tests in org.apache.hadoop.mapred.TestMiniMRClasspath & + org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers (Devaraj K via + bobby) + + MAPREDUCE-4105. Yarn RackResolver ignores rack configurations. + (Ahmed Radwan via tomwhite) + + MAPREDUCE-3869. Fix classpath for DistributedShell application. (Devaraj K + via sseth) + + MAPREDUCE-4057. Update RAID for the HA and fsdataset changes. (Devaraj K + via szetszwo) + + MAPREDUCE-4076. Stream job fails with ZipException when use yarn jar + command (Devaraj K via bobby) + + MAPREDUCE-4108. Fix tests in org.apache.hadoop.util.TestRunJar + (Devaraj K via tgraves) + + MAPREDUCE-4107. Fix tests in org.apache.hadoop.ipc.TestSocketFactory + (Devaraj K via tgraves) + + MAPREDUCE-4147. YARN should not have a compile-time dependency on HDFS. + (tomwhite) + + MAPREDUCE-4008. ResourceManager throws MetricsException on start up + saying QueueMetrics MBean already exists (Devaraj K via tgraves) + + MAPREDUCE-3867. MiniMRYarn/MiniYarn uses fixed ports (tucu) + + MAPREDUCE-4141. clover integration broken, also mapreduce poms are + pulling in clover as a dependency. (phunt via tucu) + + MAPREDUCE-4193. broken doc link for yarn-default.xml in site.xml. + (phunt via tomwhite) + + MAPREDUCE-4202. TestYarnClientProtocolProvider is broken (Daryn Sharp via + bobby) + + MAPREDUCE-3173. MRV2 UI doesn't work properly without internet (Devaraj K + via bobby) + + MAPREDUCE-3958. RM: Remove RMNodeState and replace it with NodeState + (Bikas Saha via bobby) + + MAPREDUCE-4231. Update RAID to use the new BlockCollection interface. + (szetszwo) + + MAPREDUCE-4483. 2.0 build does not work (John George via bobby) + + MAPREDUCE-4444. nodemanager fails to start when one of the local-dirs is + bad (Jason Lowe via bobby) + +Release 0.23.9 - UNRELEASED + + INCOMPATIBLE CHANGES + + NEW FEATURES + + IMPROVEMENTS + + OPTIMIZATIONS + + MAPREDUCE-5268. Improve history server startup performance (Karthik + Kambatla via jlowe) + + BUG FIXES + + MAPREDUCE-5308. Shuffling to memory can get out-of-sync when fetching + multiple compressed map outputs (Nathan Roberts via jlowe) + + MAPREDUCE-5315. DistCp reports success even on failure. (mithun and jlowe + via daryn) + + MAPREDUCE-4019. -list-attempt-ids is not working (Ashwin Shankar, + Devaraj K, and B Anil Kumar via jlowe) + +Release 0.23.8 - 2013-06-05 + + INCOMPATIBLE CHANGES + + NEW FEATURES + + IMPROVEMENTS + + MAPREDUCE-5065. DistCp should skip checksum comparisons if block-sizes + are different on source/target (Mithun Radhakrishnan via kihwal) + + OPTIMIZATIONS + + BUG FIXES + + MAPREDUCE-5015. Coverage fix for org.apache.hadoop.mapreduce.tools.CLI + (Aleksey Gorshkov via tgraves) + + MAPREDUCE-5147. Maven build should create + hadoop-mapreduce-client-app-VERSION.jar directly (Robert Parker via tgraves) + + MAPREDUCE-4927. Historyserver 500 error due to NPE when accessing specific + counters page for failed job. (Ashwin Shankar via jlowe) + +Release 0.23.7 - 2013-04-18 + + INCOMPATIBLE CHANGES + + NEW FEATURES + + IMPROVEMENTS + + MAPREDUCE-4905. test org.apache.hadoop.mapred.pipes + (Aleksey Gorshkov via bobby) + + MAPREDUCE-4989. JSONify DataTables input data for Attempts page (Ravi + Prakash via jlowe) + + MAPREDUCE-5027. Shuffle does not limit number of outstanding connections + (Robert Parker via jeagles) + + MAPREDUCE-4972. Coverage fixing for org.apache.hadoop.mapreduce.jobhistory + (Aleksey Gorshkov via bobby) + + OPTIMIZATIONS + + MAPREDUCE-4946. Fix a performance problem for large jobs by reducing the + number of map completion event type conversions. (Jason Lowe via sseth) + + MAPREDUCE-4822. Unnecessary conversions in History Events. (Chu Tong via + jlowe) + + BUG FIXES + + MAPREDUCE-4458. Warn if java.library.path is used for AM or Task + (Robert Parker via jeagles) + + MAPREDUCE-4992. AM hangs in RecoveryService when recovering tasks with + speculative attempts (Robert Parker via jlowe) + + MAPREDUCE-5009. Killing the Task Attempt slated for commit does not clear + the value from the Task commitAttempt member (Robert Parker via jeagles) + + MAPREDUCE-4871. AM uses mapreduce.jobtracker.split.metainfo.maxsize but + mapred-default has mapreduce.job.split.metainfo.maxsize (Jason Lowe via + jeagles) + + MAPREDUCE-4794. DefaultSpeculator generates error messages on normal + shutdown (Jason Lowe via jeagles) + + MAPREDUCE-5043. Fetch failure processing can cause AM event queue to + backup and eventually OOM (Jason Lowe via bobby) + + MAPREDUCE-5023. History Server Web Services missing Job Counters (Ravi + Prakash via tgraves) + + MAPREDUCE-5060. Fetch failures that time out only count against the first + map task (Robert Joseph Evans via jlowe) + + MAPREDUCE-5042. Reducer unable to fetch for a map task that was recovered + (Jason Lowe via bobby) + + MAPREDUCE-5053. java.lang.InternalError from decompression codec cause + reducer to fail (Robert Parker via jeagles) + + MAPREDUCE-4991. coverage for gridmix (Aleksey Gorshkov via tgraves) + + MAPREDUCE-5007. fix coverage org.apache.hadoop.mapreduce.v2.hs (Aleksey + Gorshkov via tgraves) + + MAPREDUCE-5137. AM web UI: clicking on Map Task results in 500 error + (Thomas Graves via jlowe) + +Release 0.23.6 - 2013-02-06 + + INCOMPATIBLE CHANGES + + NEW FEATURES + + IMPROVEMENTS + + MAPREDUCE-4811. JobHistoryServer should show when it was started in WebUI + About page (Ravi Prakash via jlowe) + + OPTIMIZATIONS + + BUG FIXES + + MAPREDUCE-4802. Takes a long time to load the task list on the AM for + large jobs (Ravi Prakash via bobby) + + MAPREDUCE-4764. repair TestBinaryTokenFile (Ivan A. Veselovsky via bobby) + + MAPREDUCE-4825. JobImpl.finished doesn't expect ERROR as a final job state + (jlowe via bobby) + + MAPREDUCE-4817. Hardcoded task ping timeout kills tasks localizing large + amounts of data (tgraves) + + MAPREDUCE-4836. Elapsed time for running tasks on AM web UI tasks page is 0 + (Ravi Prakash via jeagles) + + MAPREDUCE-4842. Shuffle race can hang reducer (Mariappan Asokan via jlowe) + + MAPREDUCE-4833. Task can get stuck in FAIL_CONTAINER_CLEANUP (Robert + Parker via jlowe) + + MAPREDUCE-4793. Problem with adding resources when using both -files and + -file to hadoop streaming (jlowe) + + MAPREDUCE-4890. Invalid TaskImpl state transitions when task fails while + speculating (jlowe) + + MAPREDUCE-4902. Fix typo "receievd" should be "received" in log output + (Albert Chu via jlowe) + + MAPREDUCE-4813. AM timing out during job commit (jlowe via bobby) + + MAPREDUCE-4279. getClusterStatus() fails with null pointer exception when + running jobs in local mode (Devaraj K via bobby) + + MAPREDUCE-4832. MR AM can get in a split brain situation (jlowe) + + MAPREDUCE-4894. Renewal / cancellation of JobHistory tokens (Siddharth + Seth via tgraves) + + MAPREDUCE-4819. AM can rerun job after reporting final job status to the + client (bobby and Bikas Saha via bobby) + + MAPREDUCE-4913. TestMRAppMaster#testMRAppMasterMissingStaging occasionally + exits (Jason Lowe via tgraves) + + MAPREDUCE-4848. TaskAttemptContext cast error during AM recovery (Jerry + Chen via jlowe) + + MAPREDUCE-4921. JobClient should acquire HS token with RM principal + (daryn via bobby) + + MAPREDUCE-4934. Maven RAT plugin is not checking all source files (tgraves) + + MAPREDUCE-4678. Running the Pentomino example with defaults throws + java.lang.NegativeArraySizeException (Chris McConnell via harsh) + + MAPREDUCE-4925. The pentomino option parser may be buggy. + (Karthik Kambatla via harsh) + +Release 0.23.5 - 2012-11-28 + + INCOMPATIBLE CHANGES + + NEW FEATURES + + IMPROVEMENTS + + MAPREDUCE-4596. Split StateMachine state from states seen by MRClientProtocol + for Job, Task and TaskAttempt. (Siddarth Seth via vinodkv) + + MAPREDUCE-4752. Reduce MR AM memory usage through String Interning (Robert + Evans via tgraves) + + MAPREDUCE-4266. remove Ant remnants from MR (tgraves via bobby) + + MAPREDUCE-4666. JVM metrics for history server (jlowe via jeagles) + + OPTIMIZATIONS + + MAPREDUCE-4720. Browser thinks History Server main page JS is taking too + long (Ravi Prakash via bobby) + + BUG FIXES + + MAPREDUCE-4554. Job Credentials are not transmitted if security is turned + off (Benoy Antony via bobby) + + MAPREDUCE-4705. Fix a bug in job history lookup, which makes older jobs + inaccessible despite the presence of a valid history file. (Jason Lowe + via sseth) + + MAPREDUCE-4521. mapreduce.user.classpath.first incompatibility with 0.20/1.x + (Ravi Prakash via bobby) + + MAPREDUCE-4721. Task startup time in JHS is same as job startup time. + (Ravi Prakash via bobby) + + MAPREDUCE-4479. Fix parameter order in assertEquals() in + TestCombineInputFileFormat.java (Mariappan Asokan via bobby) + + MAPREDUCE-4733. Reducer can fail to make progress during shuffle if too many + reducers complete consecutively. (Jason Lowe via vinodkv) + + MAPREDUCE-4740. only .jars can be added to the Distributed Cache + classpath. (Robert Joseph Evans via jlowe) + + MAPREDUCE-4229. Intern counter names in the JT (Miomir Boljanovic and bobby via daryn) + + MAPREDUCE-4741. WARN and ERROR messages logged during normal AM shutdown. + (Vinod Kumar Vavilapalli via jlowe) + + MAPREDUCE-4730. Fix Reducer's EventFetcher to scale the map-completion + requests slowly to avoid HADOOP-8942. (Jason Lowe via vinodkv) + + MAPREDUCE-4748. Invalid event: T_ATTEMPT_SUCCEEDED at SUCCEEDED. (jlowe) + + MAPREDUCE-4724. job history web ui applications page should be sorted to + display last app first (tgraves via bobby) + + MAPREDUCE-4746. The MR Application Master does not have a config to set + environment variables (Rob Parker via bobby) + + MAPREDUCE-4729. job history UI not showing all job attempts. (Vinod + Kumar Vavilapalli via jlowe) + + MAPREDUCE-4763 repair test TestUmbilicalProtocolWithJobToken (Ivan A. + Veselovsky via bobby) + + MAPREDUCE-4771. KeyFieldBasedPartitioner not partitioning properly when + configured (jlowe via bobby) + + MAPREDUCE-4772. Fetch failures can take way too long for a map to be + restarted (bobby) + + MAPREDUCE-4782. NLineInputFormat skips first line of last InputSplit + (Mark Fuhs via bobby) + + MAPREDUCE-4774. JobImpl does not handle asynchronous task events in FAILED + state (jlowe via bobby) + + MAPREDUCE-4751. AM stuck in KILL_WAIT for days (vinodkv via bobby) + + MAPREDUCE-4787. TestJobMonitorAndPrint is broken (Rob Parker via bobby) + + MAPREDUCE-4425. Speculation + Fetch failures can lead to a hung job (jlowe + via bobby) + + MAPREDUCE-4786. Job End Notification retry interval is 5 milliseconds by + default (Ravi Prakash via bobby) + + MAPREDUCE-4517. Too many INFO messages written out during AM to RM heartbeat + (Jason Lowe via tgraves) + + MAPREDUCE-4797. LocalContainerAllocator can loop forever trying to contact + the RM (jlowe via bobby) + + MAPREDUCE-4801. ShuffleHandler can generate large logs due to prematurely + closed channels (jlowe via bobby) + +Release 0.23.4 + + INCOMPATIBLE CHANGES + + NEW FEATURES + + IMPROVEMENTS + + MAPREDUCE-2786. Add compression option for TestDFSIO. + (Plamen Jeliazkov via shv) + + MAPREDUCE-4645. Provide a random seed to Slive to make the sequence + of file names deterministic. (Ravi Prakash via shv) + + MAPREDUCE-4651. Benchmarking random reads with DFSIO. (shv) + + OPTIMIZATIONS + + BUG FIXES + + MAPREDUCE-4647. We should only unjar jobjar if there is a lib directory + in it. (Robert Evans via tgraves) + + MAPREDUCE-4691. Historyserver can report "Unknown job" after RM says job + has completed (Robert Joseph Evans via jlowe) + + MAPREDUCE-4689. JobClient.getMapTaskReports on failed job results in NPE + (jlowe via bobby) + +Release 0.23.3 + + INCOMPATIBLE CHANGES + + MAPREDUCE-4072. User set java.library.path seems to overwrite default + creating problems native lib loading (Anupam Seth via bobby) + + MAPREDUCE-3812. Lower default allocation sizes, fix allocation + configurations and document them (Harsh J via bobby) + + NEW FEATURES + + IMPROVEMENTS + + MAPREDUCE-4059. The history server should have a separate pluggable + storage/query interface. (Robert Evans via tgraves) + + MAPREDUCE-3942. Randomize master key generation for + ApplicationTokenSecretManager and roll it every so often. (Vinod Kumar + Vavilapalli via sseth) + + MAPREDUCE-4151. RM scheduler web page should filter apps to those that + are relevant to scheduling (Jason Lowe via tgraves) + + MAPREDUCE-4134. Remove references of mapred.child.ulimit etc. since they + are not being used any more (Ravi Prakash via bobby) + + MAPREDUCE-3972. Fix locking and exception issues in JobHistory server. + (Robert Joseph Evans via sseth) + + MAPREDUCE-4161. create sockets consistently (Daryn Sharp via bobby) + + MAPREDUCE-4079. Allow MR AppMaster to limit ephemeral port range. + (bobby via tgraves) + + MAPREDUCE-4210. Expose listener address for WebApp (Daryn Sharp via bobby) + + MAPREDUCE-4162. Correctly set token service (Daryn Sharp via bobby) + + MAPREDUCE-4301. Dedupe some strings in MRAM for memory savings + (bobby via tgraves) + + MAPREDUCE-4267. mavenize pipes (tgraves via bobby) + + MAPREDUCE-4375. Show Configuration Tracability in MR UI (bobby + via tgraves) + + MAPREDUCE-4569. Fixed TestHsWebServicesJobsQuery to pass on JDK7 by not + depending on test order. (Thomas Graves via vinodkv) + + OPTIMIZATIONS + + MAPREDUCE-3850. Avoid redundant calls for tokens in TokenCache (Daryn + Sharp via bobby) + + BUG FIXES + + MAPREDUCE-4092. commitJob Exception does not fail job (Jon Eagles via + bobby) + + MAPREDUCE-4089. Hung Tasks never time out. (Robert Evans via tgraves) + + MAPREDUCE-4024. RM webservices can't query on finalStatus (Tom Graves + via bobby) + + MAPREDUCE-4060. Multiple SLF4J binding warning (Jason Lowe via bobby) + + MAPREDUCE-3983. TestTTResourceReporting can fail, and should just be + deleted (Ravi Prakash via bobby) + + MAPREDUCE-4012 Hadoop Job setup error leaves no useful info to users + (when LinuxTaskController is used). (tgraves) + + MAPREDUCE-4062. AM Launcher thread can hang forever (tgraves via bobby) + + MAPREDUCE-3988. mapreduce.job.local.dir doesn't point to a single directory + on a node. (Eric Payne via bobby) + + MAPREDUCE-3999. Tracking link gives an error if the AppMaster hasn't + started yet (Ravi Prakash via bobby) + + MAPREDUCE-4020. Web services returns incorrect JSON for deep queue tree + (Anupam Seth via tgraves) + + MAPREDUCE-3672. Killed maps shouldn't be counted towards + JobCounter.NUM_FAILED_MAPS. (Anupam Seth via tgraves) + + MAPREDUCE-3682 Tracker URL says AM tasks run on localhost. + (Ravi Prakash via tgraves) + + MAPREDUCE-3082. Archive command take wrong path for input file with current + directory (John George via bobby) + + MAPREDUCE-3650. testGetTokensForHftpFS() fails (Ravi Prakash via bobby) + + MAPREDUCE-3621. TestDBJob and TestDataDrivenDBInputFormat ant tests fail + (Ravi Prakash via tgraves) + + MAPREDUCE-4073. CS assigns multiple off-switch containers when using + multi-level-queues (Siddharth Seth via bobby) + + MAPREDUCE-4051. Remove the empty hadoop-mapreduce-project/assembly/all.xml + file (Ravi Prakash via bobby) + + MAPREDUCE-4117. mapred job -status throws NullPointerException (Devaraj K + via bobby) + + MAPREDUCE-4099. ApplicationMaster may fail to remove staging directory + (Jason Lowe via bobby) + + MAPREDUCE-4017. Add jobname to jobsummary log (tgraves and Koji Noguchi + via bobby) + + MAPREDUCE-4040. History links should use hostname rather than IP address. + (Bhallamudi Venkata Siva Kamesh via sseth) + + MAPREDUCE-4099 amendment. ApplicationMaster will remove staging directory + after the history service is stopped. (Jason Lowe via sseth) + + MAPREDUCE-3932. Fix the TaskAttempt state machine to handle + CONTIANER_LAUNCHED and CONTIANER_LAUNCH_FAILED events in additional + states. (Robert Joseph Evans via sseth) + + MAPREDUCE-4140. mapreduce classes incorrectly importing + "clover.org.apache.*" classes. (Patrick Hunt via tomwhite) + + MAPREDUCE-4050. For tasks without assigned containers, changes the node + text on the UI to N/A instead of a link to null. (Bhallamudi Venkata Siva + Kamesh via sseth) + + MAPREDUCE-4128. AM Recovery expects all attempts of a completed task to + also be completed. (Bikas Saha via bobby) + + MAPREDUCE-4144. Fix a NPE in the ResourceManager when handling node + updates. (Jason Lowe via sseth) + + MAPREDUCE-4156. ant build fails compiling JobInProgress (tgraves) + + MAPREDUCE-4160. some mrv1 ant tests fail with timeout - due to 4156 + (tgraves) + + MAPREDUCE-4074. Client continuously retries to RM When RM goes down + before launching Application Master (xieguiming via tgraves) + + MAPREDUCE-4159. Job is running in Uber mode after setting + "mapreduce.job.ubertask.maxreduces" to zero (Devaraj K via bobby) + + MAPREDUCE-4165. Committing is misspelled as commiting in task logs + (John Eagles via bobby) + + MAPREDUCE-4129. Lots of unneeded counters log messages (Ahmed Radwan via + bobby) + + MAPREDUCE-3947. yarn.app.mapreduce.am.resource.mb not documented + (Devaraj K via bobby) + + MAPREDUCE-4190. Improve web UI for task attempts userlog link (Tom Graves + via bobby) + + MAPREDUCE-4133. MR over viewfs is broken (John George via bobby) + + MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection + (Jonathan Eagles via bobby) + + MAPREDUCE-3613. web service calls header contains 2 content types + (tgraves) + + MAPREDUCE-4169. Container Logs appear in unsorted order (Jonathan Eagles + via bobby) + + MAPREDUCE-4189. TestContainerManagerSecurity is failing (Devaraj K via + bobby) + + MAPREDUCE-4209. junit dependency in hadoop-mapreduce-client is missing + scope test (Radim Kolar via bobby) + + MAPREDUCE-4206. Sorting by Last Health-Update on the RM nodes page sorts + does not work correctly (Jonathon Eagles via tgraves) + + MAPREDUCE-4212. TestJobClientGetJob sometimes fails + (Daryn Sharp via tgraves) + + MAPREDUCE-4211. Error conditions (missing appid, appid not found) are + masked in the RM app page (Jonathan Eagles via bobby) + + MAPREDUCE-4163. consistently set the bind address (Daryn Sharp via bobby) + + MAPREDUCE-4048. NullPointerException exception while accessing the + Application Master UI (Devaraj K via bobby) + + MAPREDUCE-4220. RM apps page starttime/endtime sorts are incorrect + (Jonathan Eagles via bobby) + + MAPREDUCE-4226. ConcurrentModificationException in FileSystemCounterGroup. + (tomwhite) + + MAPREDUCE-4215. RM app page shows 500 error on appid parse error + (Jonathon Eagles via tgraves) + + MAPREDUCE-4237. TestNodeStatusUpdater can fail if localhost has a domain + associated with it (bobby) + + MAPREDUCE-4233. NPE can happen in RMNMNodeInfo. (bobby) + + MAPREDUCE-4238. mavenize data_join. (tgraves) + + MAPREDUCE-4102. job counters not available in Jobhistory webui for + killed jobs (Bhallamudi Venkata Siva Kamesh via tgraves) + + MAPREDUCE-3543. Mavenize Gridmix. (tgraves) + + MAPREDUCE-4197. Include the hsqldb jar in the hadoop-mapreduce tar + file (Ravi Prakash via tgraves) + + MAPREDUCE-4269. documentation: Gridmix has javadoc warnings in + StressJobFactory (Jonathon Eagles via tgraves). + + MAPREDUCE-3870. Invalid App Metrics + (Bhallamudi Venkata Siva Kamesh via tgraves). + + MAPREDUCE-4152. map task left hanging after AM dies trying to connect to RM + (Tom Graves via bobby) + + MAPREDUCE-4297. Usersmap file in gridmix should not fail on empty lines + (Ravi Prakash via bobby) + + MAPREDUCE-4302. NM goes down if error encountered during log aggregation + (Daryn Sharp via bobby) + + MAPREDUCE-3350. Per-app RM page should have the list of application-attempts + like on the app JHS page (Jonathon Eagles via tgraves) + + MAPREDUCE-3842. Stop webpages from automatic refreshing (tgraves) + + MAPREDUCE-3927. Shuffle hang when set map.failures.percent + (Bhallamudi Venkata Siva Kamesh via tgraves) + + MAPREDUCE-4311. Capacity scheduler.xml does not accept decimal values for + capacity and maximum-capacity settings (Karthik Kambatla via tgraves) + + MAPREDUCE-4341. add types to capacity scheduler properties documentation + (Karthik Kambatla via tgraves) + + MAPREDUCE-4270. Move the data_join test classes to the correct path. + (Thomas Graves via sseth) + + MAPREDUCE-3889. job client tries to use /tasklog interface, but that + doesn't exist anymore (Devaraj K via bobby) + + MAPREDUCE-4320. gridmix mainClass wrong in pom.xml (tgraves) + + MAPREDUCE-4295. RM crashes due to DNS issue (tgraves) + + MAPREDUCE-4228. mapreduce.job.reduce.slowstart.completedmaps is not working + properly (Jason Lowe via bobby) + + MAPREDUCE-4392. Counters.makeCompactString() changed behavior from 0.20 + (Jason Lowe via bobby) + + MAPREDUCE-4384. Race conditions in IndexCache (Kihwal Lee via tgraves) + + MAPREDUCE-4387. RM gets fatal error and exits during TestRM + (Kihwal Lee via tgraves) + + MAPREDUCE-4379. Node Manager throws java.lang.OutOfMemoryError: Java heap + space due to org.apache.hadoop.fs.LocalDirAllocator.contexts (Devaraj K + via bobby) + + MAPREDUCE-4402. TestFileInputFormat fails intermittently (Jason Lowe via + bobby) + + MAPREDUCE-4300. OOM in AM can turn it into a zombie. (Robert Evans via + tgraves) + + MAPREDUCE-4252. MR2 job never completes with 1 pending task (Tom White via + bobby) + + MAPREDUCE-3940. ContainerTokens should have an expiry interval. (Siddharth + Seth and Vinod Kumar Vavilapalli via vinodkv) + + MAPREDUCE-4419. ./mapred queue -info <queuename> -showJobs displays all + the jobs irrespective of <queuename> (Devaraj K via bobby) + + MAPREDUCE-4299. Terasort hangs with MR2 FifoScheduler (Tom White via + bobby) + + MAPREDUCE-4437. Race in MR ApplicationMaster can cause reducers to never be + scheduled (Jason Lowe via bobby) + + MAPREDUCE-4449. Incorrect MR_HISTORY_STORAGE property name in JHAdminConfig + (Ahmed Radwan via bobby) + + MAPREDUCE-4283. Display tail of aggregated logs by default (Jason Lowe via + bobby) + + MAPREDUCE-4448. Fix NM crash during app cleanup if aggregation didn't + init. (Jason Lowe via daryn) + + MAPREDUCE-3893. allow capacity scheduler configs maximum-applications and + maximum-am-resource-percent configurable on a per queue basis (tgraves via + bobby) + + MAPREDUCE-4467. IndexCache failures due to missing synchronization + (Kihwal Lee via tgraves) + + MAPREDUCE-4423. Potential infinite fetching of map output (Robert Evans + via tgraves) + + MAPREDUCE-4456. LocalDistributedCacheManager can get an + ArrayIndexOutOfBounds when creating symlinks (Robert Evans via tgraves) + + MAPREDUCE-4496. AM logs link is missing user name (Jason Lowe via bobby) + + MAPREDUCE-4493. Distibuted Cache Compatability Issues (Robert Evans + via tgraves) + + MAPREDUCE-4492. Configuring total queue capacity between 100.5 and 99.5 at + perticular level is sucessfull (Mayank Bansal via bobby) + + MAPREDUCE-4457. mr job invalid transition TA_TOO_MANY_FETCH_FAILURE at + FAILED (Robert Evans via tgraves) + + MAPREDUCE-4234. SortValidator.java is incompatible with multi-user or + parallel use (due to a /tmp file with static name) (Robert Evans via + jeagles) + + MAPREDUCE-4504. SortValidator writes to wrong directory (Robert Evans + via tgraves) + + MAPREDUCE-4503. Should throw InvalidJobConfException if duplicates found in + cacheArchives or cacheFiles (Robert Evans via jeagles) + + MAPREDUCE-3782. teragen terasort jobs fail when using webhdfs:// (Jason + Lowe via bobby) + + MAPREDUCE-4053. Counters group names deprecation is wrong, iterating over + group names deprecated names don't show up (Robert Evans via tgraves) + + MAPREDUCE-3506. Calling getPriority on JobInfo after parsing a history log + with JobHistoryParser throws a NullPointerException (Jason Lowe via bobby) + + MAPREDUCE-4570. ProcfsBasedProcessTree#constructProcessInfo() prints a + warning if procfsDir/<pid>/stat is not found. (Ahmed Radwan via bobby) + + MAPREDUCE-4600. TestTokenCache.java from MRV1 no longer compiles (daryn + via bobby) + + MAPREDUCE-4612. job summary file permissions not set when its created + (tgraves via bobby) + + MAPREDUCE-4614. Simplify debugging a job's tokens (daryn via bobby) + + MAPREDUCE-4611. MR AM dies badly when Node is decommissioned (Robert + Evans via tgraves) + + MAPREDUCE-4604. In mapred-default, mapreduce.map.maxattempts & + mapreduce.reduce.maxattempts defaults are set to 4 as well as + mapreduce.job.maxtaskfailures.per.tracker. (Ravi Prakash via jeagles) + + MAPREDUCE-4633. history server doesn't set permissions on all subdirs + (tgraves via bobby) + + MAPREDUCE-4641. Exception in commitJob marks job as successful in job + history (Jason Lowe via bobby) + + MAPREDUCE-4549. Distributed cache conflicts breaks backwards compatability + (Robert Evans via tucu) + +Release 0.23.2 - UNRELEASED + + INCOMPATIBLE CHANGES + + NEW FEATURES + + IMPROVEMENTS + + MAPREDUCE-3849. Change TokenCache's reading of the binary token file + (Daryn Sharp via bobby) + + MAPREDUCE-3854. Fixed and reenabled tests related to MR child JVM's + environmental variables in TestMiniMRChildTask. (Tom White via vinodkv) + + MAPREDUCE-3877 Add a test to formalise the current state transitions + of the yarn lifecycle. (stevel) + + MAPREDUCE-3866. Fixed the bin/yarn script to not print the command line + unnecessarily. (vinodkv) + + MAPREDUCE-3730. Modified RM to allow restarted NMs to be able to join the + cluster without waiting for expiry. (Jason Lowe via vinodkv) + + MAPREDUCE-2793. Corrected AppIDs, JobIDs, TaskAttemptIDs to be of correct + format on the web pages. (Bikas Saha via vinodkv) + + MAPREDUCE-3614. Fixed MR AM to close history file quickly and send a correct + final state to the RM when it is killed. (Ravi Prakash via vinodkv) + + MAPREDUCE-3497. Added docs for YARN CLI. (tgraves via acmurthy) + + MAPREDUCE-3954. Added new envs to separate heap size for different daemons + started via bin scripts. (Robert Joseph Evans via vinodkv) + + MAPREDUCE-4025. AM can crash if task attempt reports bogus progress value + (Jason Lowe via bobby) + + MAPREDUCE-4034. Unable to view task logs on history server with + mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) + + MAPREDUCE-4043. Secret keys set in Credentials are not seen by tasks + (Jason Lowe via bobby) + + MAPREDUCE-3989. Cap space usage of default log4j rolling policy. + (Patrick Hunt via eli) + + OPTIMIZATIONS + + MAPREDUCE-3901. Modified JobHistory records in YARN to lazily load job and + task reports so as to improve UI response times. (Siddarth Seth via vinodkv) + + MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to + minimize counter names lookup time. (Siddarth Seth via vinodkv) + + MAPREDUCE-3944. Change the history jobs/ webservice to return partial job + info for a significant performance improvement. (Robert Joseph Evans via + sseth) + + BUG FIXES + + MAPREDUCE-3918 proc_historyserver no longer in command line arguments for + HistoryServer (Jon Eagles via bobby) + + MAPREDUCE-3862. Nodemanager can appear to hang on shutdown due to lingering + DeletionService threads (Jason Lowe via bobby) + + MAPREDUCE-3680. FifoScheduler web service rest API can print out invalid + JSON. (B Anil Kumar via tgraves) + + MAPREDUCE-3852. Test TestLinuxResourceCalculatorPlugin failing. (Thomas + Graves via mahadev) + + MAPREDUCE-3864. Fix cluster setup docs for correct SecondaryNameNode + HTTPS parameters. (todd) + + MAPREDUCE-3583. Change pid to String and stime to BigInteger in order to + avoid NumberFormatException caused by overflow. (Zhihong Yu via szetszwo) + + MAPREDUCE-3634. Fixed all daemons to crash instead of hanging around when + their EventHandlers get exceptions. (vinodkv) + + MAPREDUCE-3798. Fixed failing TestJobCleanup.testCusomCleanup() and moved it + to the maven build. (Ravi Prakash via vinodkv) + + MAPREDUCE-3884. PWD should be first in the classpath of MR tasks (tucu) + + MAPREDUCE-3878. Null user on filtered jobhistory job page (Jonathon Eagles + via tgraves) + + MAPREDUCE-3738. MM can hang during shutdown if AppLogAggregatorImpl thread + dies unexpectedly (Jason Lowe via sseth) + + MAPREDUCE-3904 Job history produced with mapreduce.cluster.acls.enabled + false can not be viewed with mapreduce.cluster.acls.enabled true + (Jonathon Eagles via tgraves) + + MAPREDUCE-3910. Fixed a bug in CapacityScheduler LeafQueue which was causing + app-submission to fail. (John George via vinodkv) + + MAPREDUCE-3686. Fixed two bugs in Counters because of which web app displays + zero counter values for framework counters. (Bhallamudi Venkata Siva Kamesh + via vinodkv) + + MAPREDUCE-3913. RM application webpage is unresponsive after 2000 jobs + (Jason Lowe via tgraves) + + MAPREDUCE-3922. Fixed build to not compile 32bit container-executor binary + by default on all platforms. (Hitesh Shah via vinodkv) + + MAPREDUCE-3790. Broken pipe on streaming job can lead to truncated output for + a successful job (Jason Lowe via bobby) + + MAPREDUCE-3816. capacity scheduler web ui bar graphs for used capacity wrong + (tgraves via bobby) + + MAPREDUCE-3930. Fixed an NPE while accessing the AM page/webservice for a + task attempt without an assigned container. (Robert Joseph Evans via + sseth) + + MAPREDUCE-3931. Changed PB implementation of LocalResource to take locks + so that race conditions don't fail tasks by inadvertantly changing the + timestamps. (Siddarth Seth via vinodkv) + + MAPREDUCE-3687. If AM dies before it returns new tracking URL, proxy + redirects to http://N/A/ and doesn't return error code (Ravi Prakash via + bobby) + + MAPREDUCE-3920. Revise yarn default port number selection + (Dave Thompson via tgraves) + + MAPREDUCE-3903. Add support for mapreduce admin users. (Thomas Graves via + sseth) + + MAPREDUCE-3706. Fix circular redirect error in job-attempts page. (bobby + via acmurthy) + + MAPREDUCE-3896. Add user information to the delegation token issued by the + history server. (Vinod Kumar Vavilapalli via sseth) + + MAPREDUCE-3792. Fix "bin/mapred job -list" to display all jobs instead of + only the jobs owned by the user. (Jason Lowe via vinodkv) + + MAPREDUCE-3929. Fixed output of 'bin/mapred queue -showacl' command to + clarify ACLs for users. (John George via acmurthy) + + MAPREDUCE-3960. Fix web-proxy to forward request to AM with configured + hostname or IP. (tgraves via acmurthy) + + MAPREDUCE-3897. Fixed computation of maxActiveAppsPerUser for queues by + using capacity and not max-capacity since we are already scaling it by + userLimitFactor. (Eric Payne via acmurthy) + + MAPREDUCE-3009. Fixed node link on JobHistory webapp. (chackaravarthy via + vinodkv) + + MAPREDUCE-3964. ResourceManager does not have JVM metrics (Jason Lowe via + bobby) + + MAPREDUCE-3034. Ensure NodeManager reboots itself on direction from + ResourceManager. (Devaraj K & Eric Payne via acmurthy) + + MAPREDUCE-3976. TestRMContainerAllocator failing (Jason Lowe via bobby) + + MAPREDUCE-3961. Map/ReduceSlotMillis computation incorrect (Siddharth Seth + via bobby) + + MAPREDUCE-3977. LogAggregationService leaks log aggregator objects + (Jason Lowe via bobby) + + MAPREDUCE-3975. Default value not set for Configuration parameter + mapreduce.job.local.dir (Eric Payne via bobby) + + MAPREDUCE-3982. Fixed FileOutputCommitter to not err out for an 'empty-job' + whose tasks don't write any outputs. (Robert Joseph Evans via vinodkv) + + MAPREDUCE-4005. Fixes broken AM container logs URL on ResourceManager + Application Info page. (Jason Lowe via sseth) + + MAPREDUCE-4006. History server container log web UI sometimes combines + stderr/stdout/syslog contents together (Siddharth Seth via tgraves) + + MAPREDUCE-4061. RM only has 1 AM launcher thread (tgraves via bobby) + +Release 0.23.1 - 2012-02-17 + + NEW FEATURES + + MAPREDUCE-778. Rumen Anonymizer. (Amar Kamat and Chris Douglas via amarrk) + + MAPREDUCE-3121. NodeManager should handle disk-failures (Ravi Gummadi via mahadev) + + MAPREDUCE-2863. Support web services for YARN and MR components. (Thomas + Graves via vinodkv) + + MAPREDUCE-3251. Network ACLs can prevent some clients to talk to MR ApplicationMaster. + (Anupam Seth via mahadev) + + MAPREDUCE-778. Rumen Anonymizer. (Amar Kamat and Chris Douglas via amarrk) + + IMPROVEMENTS + + MAPREDUCE-3481. [Gridmix] Improve Gridmix STRESS mode. (amarrk) + + MAPREDUCE-3597. [Rumen] Rumen should provide APIs to access all the + job-history related information. + + MAPREDUCE-3375. [Gridmix] Memory Emulation system tests. + (Vinay Thota via amarrk) + + MAPREDUCE-3840. JobEndNotifier doesn't use the proxyToUse during connecting + (Ravi Prakash via bobby) + + MAPREDUCE-3736. Variable substitution depth too large for fs.default.name + causes jobs to fail (ahmed via tucu). + + MAPREDUCE-2733. [Gridmix] Gridmix3 cpu emulation system tests. + (Vinay Thota via amarrk) + + MAPREDUCE-3297. Moved log related components into yarn-common so that + HistoryServer and clients can use them without depending on the + yarn-server-nodemanager module. (Siddharth Seth via vinodkv) + + MAPREDUCE-3336. Replaced guice internal.Preconditions api usage with the + public Preconditions API. (Thomas Graves via vinodkv) + + MAPREDUCE-3280. Removed the unnecessary job user-name configuration in + mapred-site.xml. (vinodkv) + + MAPREDUCE-3370. Fixed MiniMRYarnCluster and related tests to not use + a hard-coded path for the mr-app jar. (Ahmed Radwan via vinodkv) + + MAPREDUCE-3325. Improvements to CapacityScheduler doc. (Thomas Graves + via mahadev) + + MAPREDUCE-3341. Enhance logging of initalized queue limit values. + (Anupam Seth via mahadev) + + MAPREDUCE-3243. Invalid tracking URL for streaming jobs (Jonathan Eagles + via mahadev) + + MAPREDUCE-3331. Improvement to single node cluster setup documentation for + 0.23 (Anupam Seth via mahadev) + + MAPREDUCE-3102. Changed NodeManager to fail fast when LinuxContainerExecutor + has wrong configuration or permissions. (Hitesh Shah via vinodkv) + + MAPREDUCE-3415. improve MiniMRYarnCluster & DistributedShell JAR + resolution. (tucu) + + MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides + client APIs cross MR1 and MR2. (Ahmed via tucu) + + MAPREDUCE-3373. Hadoop scripts unconditionally source + "$bin"/../libexec/hadoop-config.sh. (Bruno Mahé via tomwhite) + + MAPREDUCE-3372. HADOOP_PREFIX cannot be overridden. + (Bruno Mahé via tomwhite) + + MAPREDUCE-3411. Performance Upgrade for jQuery (Jonathan Eagles via + mahadev) + + MAPREDUCE-3371. Review and improve the yarn-api javadocs. (Ravi Prakash + via mahadev) + + MAPREDUCE-3238. Small cleanup in SchedulerApp. (Todd Lipcon via mahadev) + + MAPREDUCE-3413. RM web ui applications not sorted in any order by default. + (Jonathan Eagles via mahadev) + + MAPREDUCE-3045. Fixed UI filters to not filter on hidden title-numeric + sort fields. (Jonathan Eagles via sseth) + + MAPREDUCE-3448. TestCombineOutputCollector javac unchecked warning on mocked + generics (Jonathan Eagles via mahadev) + + MAPREDUCE-3169 amendment. Deprecate MiniMRCluster. (Ahmed Radwan via + sseth) + + MAPREDUCE-3369. Migrate MR1 tests to run on MR2 using the new interfaces + introduced in MAPREDUCE-3169. (Ahmed Radwan via tomwhite) + + MAPREDUCE-3518. mapred queue -info <queue> -showJobs throws NPE. + (Jonathan Eagles via mahadev) + + MAPREDUCE-3391. Making a trivial change to correct a log message in + DistributedShell app's AM. (Subroto Sanyal via vinodkv) + + MAPREDUCE-3547. Added a bunch of unit tests for the the RM/NM webservices. + (Thomas Graves via acmurthy) + + MAPREDUCE-3610. Remove use of the 'dfs.block.size' config for default block + size fetching. Use FS#getDefaultBlocksize instead. (Sho Shimauchi via harsh) + + MAPREDUCE-3478. Cannot build against ZooKeeper 3.4.0. (Tom White via mahadev) + + MAPREDUCE-3528. Fixed TaskHeartBeatHandler to use a new configuration + for the thread loop interval separate from task-timeout configuration + property. (Siddharth Seth via vinodkv) + + MAPREDUCE-3312. Modified MR AM to not send a stop-container request for + a container that isn't launched at all. (Robert Joseph Evans via vinodkv) + + MAPREDUCE-3382. Enhanced MR AM to use a proxy to ping the job-end + notification URL. (Ravi Prakash via vinodkv) + + MAPREDUCE-3299. Added AMInfo table to the MR AM job pages to list all the + job-attempts when AM restarts and recovers. (Jonathan Eagles via vinodkv) + + MAPREDUCE-3251. Network ACLs can prevent some clients to talk to MR AM. + Improved the earlier patch to not to JobHistoryServer repeatedly. + (Anupam Seth via vinodkv) + + MAPREDUCE-3553. Add support for data returned when exceptions thrown from web + service apis to be in either xml or in JSON. (Thomas Graves via mahadev) + + MAPREDUCE-3641. Making CapacityScheduler more conservative so as to + assign only one off-switch container in a single scheduling + iteration. (Arun C Murthy via vinodkv) + + MAPREDUCE-3692. yarn-resourcemanager out and log files can get big. (eli) + + MAPREDUCE-3710. Improved FileInputFormat to return better locality for the + last split. (Siddarth Seth via vinodkv) + + MAPREDUCE-2765. DistCp Rewrite. (Mithun Radhakrishnan via mahadev) + + MAPREDUCE-3737. The Web Application Proxy's is not documented very well. + (Robert Evans via mahadev) + + MAPREDUCE-3699. Increased RPC handlers for all YARN servers to reasonable + values for working at scale. (Hitesh Shah via vinodkv) + + MAPREDUCE-3693. Added mapreduce.admin.user.env to mapred-default.xml. + (Roman Shapshonik via acmurthy) + + MAPREDUCE-3732. Modified CapacityScheduler to use only users with pending + requests for computing user-limits. (Arun C Murthy via vinodkv) + + MAPREDUCE-3679. AM logs and others should not automatically refresh after every 1 + second. (Vinod KV via mahadev) + + MAPREDUCE-3754. Modified RM UI to filter applications based on state of the + applications. (vinodkv) + + MAPREDUCE-3774. Moved yarn-default.xml to hadoop-yarn-common from + hadoop-server-common. (Mahadev Konar via vinodkv) + + MAPREDUCE-3771. Un-deprecated the old mapred apis, port of MAPREDUCE-1735. + (acmurthy) + + MAPREDUCE-3784. Fixed CapacityScheduler so that maxActiveApplications and + maxActiveApplicationsPerUser per queue are not too low for small + clusters. (Arun C Murthy via vinodkv) + + OPTIMIZATIONS + + MAPREDUCE-3567. Extraneous JobConf objects in AM heap. (Vinod Kumar + Vavilapalli via sseth) + + MAPREDUCE-3399. Modifying ContainerLocalizer to send a heartbeat to NM + immediately after downloading a resource instead of always waiting for a + second. (Siddarth Seth via vinodkv) + + MAPREDUCE-3568. Optimized Job's progress calculations in MR AM. (vinodkv) + + MAPREDUCE-3569. TaskAttemptListener holds a global lock for all + task-updates. (Vinod Kumar Vavilapalli via sseth) + + MAPREDUCE-3511. Removed a multitude of cloned/duplicate counters in the AM + thereby reducing the AM heap size and preventing full GCs. (vinodkv) + + MAPREDUCE-3618. Fixed TaskHeartbeatHandler to not hold a global lock for all + task-updates. (Siddarth Seth via vinodkv) + + MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush + for every event slowing down AM. (Siddarth Seth via vinodkv) + + MAPREDUCE-3718. Change default AM heartbeat interval to 1 second. (Hitesh + Shah via sseth) + + MAPREDUCE-3360. Added information about lost/rebooted/decommissioned nodes + on the webapps. (Bhallamudi Venkata Siva Kamesh and Jason Lowe via vinodkv) + + MAPREDUCE-3756. Made single shuffle limit configurable. (Hitesh Shah via + acmurthy) + + MAPREDUCE-3811. Made jobclient-to-AM retries configurable. (sseth via + acmurthy) + + BUG FIXES + + MAPREDUCE-2784. [Gridmix] Bug fixes in ExecutionSummarizer and + ResourceUsageMatcher. (amarrk) + + MAPREDUCE-3194. "mapred mradmin" command is broken in mrv2 + (Jason Lowe via bobby) + + MAPREDUCE-3462. Fix Gridmix JUnit testcase failures. + (Ravi Prakash and Ravi Gummadi via amarrk) + + MAPREDUCE-2950. [Rumen] Fixed TestUserResolve. (Ravi Gummadi via amarrk) + + MAPREDUCE-3412. Fix 'ant docs'. (amarrk) + + MAPREDUCE-3346 [Rumen] LoggedTaskAttempt#getHostName() returns null. + (amarrk) + + MAPREDUCE-3221. Reenabled the previously ignored test in TestSubmitJob + and fixed bugs in it. (Devaraj K via vinodkv) + + MAPREDUCE-3215. Reenabled and fixed bugs in the failing test + TestNoJobSetupCleanup. (Hitesh Shah via vinodkv) + + MAPREDUCE-3219. Reenabled and fixed bugs in the failing test + TestDelegationToken. (Hitesh Shah via vinodkv) + + MAPREDUCE-3217. Reenabled and fixed bugs in the failing ant test + TestAuditLogger. (Devaraj K via vinodkv) + + MAPREDUCE-3291. App fail to launch due to delegation token not + found in cache (Robert Evans via mahadev) + + MAPREDUCE-3344. o.a.h.mapreduce.Reducer since 0.21 blindly casts to + ReduceContext.ValueIterator. (Brock Noland via tomwhite) + + MAPREDUCE-3342. Fixed JobHistoryServer to also show the job's queue + name. (Jonathan Eagles via vinodkv) + + MAPREDUCE-3345. Fixed a race condition in ResourceManager that was causing + TestContainerManagerSecurity to fail sometimes. (Hitesh Shah via vinodkv) + + MAPREDUCE-3368. Fixed test compilation. (Hitesh Shah via vinodkv) + + MAPREDUCE-3333. Fixed bugs in ContainerLauncher of MR AppMaster due to + which per-container connections to NodeManager were lingering long enough + to hit the ulimits on number of processes. (vinodkv) + + MAPREDUCE-3392. Fixed Cluster's getDelegationToken's API to return null + when there isn't a supported token. (John George via vinodkv) + + MAPREDUCE-3379. Fixed LocalResourceTracker in NodeManager to remove deleted + cache entries correctly. (Siddharth Seth via vinodkv) + + MAPREDUCE-3324. Not All HttpServer tools links (stacks,logs,config,metrics) are + accessible through all UI servers (Jonathan Eagles via mahadev) + + MAPREDUCE-3355. Fixed MR AM's ContainerLauncher to handle node-command + timeouts correctly. (vinodkv) + + MAPREDUCE-3407. Fixed pom files to refer to the correct MR app-jar needed + by the integration tests. (Hitesh Shah via vinodkv) + + MAPREDUCE-3437. Fix examples pom to refer to the correct 0.23 snapshot + version. (Jonathan Eagles via todd) + + MAPREDUCE-3434. Nightly build broken (Hitesh Shah via mahadev) + + MAPREDUCE-3447. mapreduce examples not working (mahadev) + + MAPREDUCE-3444. trunk/0.23 builds broken (Hitesh Shah via mahadev) + + MAPREDUCE-3454. [Gridmix] TestDistCacheEmulation is broken (Hitesh Shah + via mahadev) + + MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger + (Bruno Mahe via mahadev) + + MAPREDUCE-3329. Fixed CapacityScheduler to ensure maximum-capacity cannot + be lesser than capacity for any queue. (acmurthy) + + MAPREDUCE-3464. mapreduce jsp pages missing DOCTYPE. (Dave Vronay via mattf) + + MAPREDUCE-3265. Removed debug logs during job submission to LOG.debug to + cut down noise. (acmurthy) + + MAPREDUCE-3468. Changed ant based infrastructure to use 0.23.1 version. + (sseth via acmurthy) + + MAPREDUCE-3433. Finding counters by legacy group name returns empty + counters. (tomwhite) + + MAPREDUCE-3450. NM port info no longer available in JobHistory. + (Siddharth Seth via mahadev) + + MAPREDUCE-3477. Hadoop site documentation cannot be built anymore. + (jeagles via tucu) + + MAPREDUCE-3488. Streaming jobs are failing because the main class + isnt set in the pom files. (mahadev) + + MAPREDUCE-3463. Second AM fails to recover properly when first AM is killed with + java.lang.IllegalArgumentException causing lost job. (Siddharth Seth via mahadev) + + MAPREDUCE-3452. fifoscheduler web ui page always shows 0% used for the queue. + (Jonathan Eagles via mahadev) + + MAPREDUCE-3443. JobClient and Job should function in the context of the + UGI which created them. (Mahadev Konar via sseth) + + MAPREDUCE-3460. MR AM can hang if containers are allocated on a node + blacklisted by the AM. (Hitesh Shah and Robert Joseph Evans via sseth) + + MAPREDUCE-3453. RM web ui application details page shows RM cluster about + information. (Jonathan Eagles via sseth) + + MAPREDUCE-3479. JobClient#getJob cannot find local jobs. (tomwhite) + + MAPREDUCE-3500. MRJobConfig creates an LD_LIBRARY_PATH using the platform ARCH. (tucu) + + MAPREDUCE-3456. $HADOOP_PREFIX/bin/yarn should set defaults for + $HADOOP_*_HOME (Eric Payne via mahadev) + + MAPREDUCE-3458. Fix findbugs warnings in hadoop-examples. (Devaraj K + via mahadev) + + MAPREDUCE-3485. DISKS_FAILED -101 error code should be defined in same location as + ABORTED_CONTAINER_EXIT_STATUS. (Ravi Gummadi via mahadev) + + MAPREDUCE-3389. MRApps loads the 'mrapp-generated-classpath' file with + classpath from the build machine. (tucu) + + MAPREDUCE-3496. Fixed client to print queue acls in consistent order. + (Jonathan Eagles via acmurthy) + + MAPREDUCE-3147. Handle leaf queues with the same name properly. + (Ravi Prakash via mahadev) + + MAPREDUCE-3327. RM web ui scheduler link doesn't show correct max value + for queues (Anupam Seth via mahadev) + + MAPREDUCE-3513. Capacity Scheduler web UI has a spelling mistake for Memory. + (chackaravarthy via mahadev) + + MAPREDUCE-3519. Fixed a deadlock in NodeManager LocalDirectories's handling + service. (Ravi Gummadi via vinodkv) + + MAPREDUCE-3527. Fix minor API incompatibilities between 1.0 and 0.23. + (tomwhite) + + MAPREDUCE-3328. mapred queue -list output inconsistent and missing child + queues. (Ravi Prakash via mahadev) + + MAPREDUCE-3510. Capacity Scheduler inherited ACLs not displayed by mapred queue + -showacls (Jonathan Eagles via mahadev) + + MAPREDUCE-3537. Fix race condition in DefaultContainerExecutor which led + to container localization occuring in wrong directories. (acmurthy) + + MAPREDUCE-3542. Support "FileSystemCounter" legacy counter group name for + compatibility. (tomwhite) + + MAPREDUCE-3426. Fixed MR AM in uber mode to write map intermediate outputs + in the correct directory to work properly in secure mode. (Hitesh Shah via + vinodkv) + + MAPREDUCE-3544. gridmix build is broken, requires hadoop-archives to be + added as ivy dependency. (tucu) + + MAPREDUCE-3557. MR1 test fail to compile because of missing hadoop-archives + dependency. (tucu) + + MAPREDUCE-3541. Fix broken TestJobQueueClient test. (Ravi Prakash via + mahadev) + + MAPREDUCE-3398. Fixed log aggregation to work correctly in secure mode. + (Siddharth Seth via vinodkv) + + MAPREDUCE-3530. Fixed an NPE occuring during scheduling in the + ResourceManager. (Arun C Murthy via vinodkv) + + MAPREDUCE-3484. Fixed JobEndNotifier to not get interrupted before completing + all its retries. (Ravi Prakash via vinodkv) + + MAPREDUCE-3531. Fixed a race in ContainerTokenSecretManager. (Robert Joseph + Evans via sseth) + + MAPREDUCE-3560. TestRMNodeTransitions is failing on trunk. + (Siddharth Seth via mahadev) + + MAPREDUCE-3487. Fixed JobHistory web-UI to display links to single task's + counters' page. (Jason Lowe via vinodkv) + + MAPREDUCE-3564. Fixed failures in TestStagingCleanup and TestJobEndNotifier + tests. (Siddharth Seth via vinodkv) + + MAPREDUCE-3422. Counter display names are not being picked up. (Jonathan + Eagles via sseth) + + MAPREDUCE-3366. Mapreduce component should use consistent directory structure + layout as HDFS/common (Eric Yang via mahadev) + + MAPREDUCE-3387. Fixed AM's tracking URL to always go through the proxy, even + before the job started, so that it works properly with oozie throughout + the job execution. (Robert Joseph Evans via vinodkv) + + MAPREDUCE-3579. ConverterUtils shouldn't include a port in a path from a url + without a port. (atm via harsh) + + MAPREDUCE-3563. Fixed LocalJobRunner to work correctly with new mapreduce + apis. (acmurthy) + + MAPREDUCE-3376. Fixed Task to ensure it passes reporter to combiners using + old MR api. (Subroto Sanyal via acmurthy) + + MAPREDUCE-3339. Fixed MR AM to stop considering node blacklisting after the + number of nodes blacklisted crosses a threshold. (Siddharth Seth via vinodkv) + + MAPREDUCE-3588. Fixed bin/yarn which was broken by MAPREDUCE-3366 so that + yarn daemons can start. (Arun C Murthy via vinodkv) + + MAPREDUCE-3349. Log rack-name in JobHistory for unsuccessful tasks. (Amar + Kamat and Devaraj K via sseth) + + MAPREDUCE-3586. Modified CompositeService to avoid duplicate stop operations + thereby solving race conditions in MR AM shutdown. (vinodkv) + + MAPREDUCE-3604. Fixed streaming to use new mapreduce.framework.name to + check for local mode. (acmurthy) + + MAPREDUCE-3521. Fixed streaming to ensure it doesn't silently ignore + unknown arguments. (Robert Evans via acmurthy) + + MAPREDUCE-3522. Ensure queues inherit ACLs from parent if they aren't + explicitly specified. (Jonathan Eagles via acmurthy) + + MAPREDUCE-3608. Fixed compile issue with MAPREDUCE-3522. (mahadev via + acmurthy) + + MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce + ramp up. (Sharad Agarwal and Arun C Murthy via vinodkv) + + MAPREDUCE-3529. TokenCache does not cache viewfs credentials correctly + (sseth) + + MAPREDUCE-3595. Add missing TestCounters#testCounterValue test from branch + 1 to 0.23 (Tom White via sseth) + + MAPREDUCE-3566. Fixed MR AM to construct CLC only once across all tasks. + (vinodkv via acmurthy) + + MAPREDUCE-3572. Moved AM event dispatcher to a separate thread for + performance reasons. (vinodkv via acmurthy) + + MAPREDUCE-3615. Fix some ant test failures. (Thomas Graves via sseth) + + MAPREDUCE-1744. DistributedCache creates its own FileSytem instance when + adding a file/archive to the path. (Dick King via tucu) + + MAPREDUCE-3326. Added detailed information about queue's to the + CapacityScheduler web-ui. (Jason Lowe via acmurthy) + + MAPREDUCE-3548. Added more unit tests for MR AM & JHS web-services. + (Thomas Graves via acmurthy) + + MAPREDUCE-3617. Removed wrong default value for + yarn.resourcemanager.principal and yarn.nodemanager.principal. (Jonathan + Eagles via acmurthy) + + MAPREDUCE-3624. Remove unnecessary dependency on JDK's tools.jar. (mahadev + via acmurthy) + + MAPREDUCE-3616. Thread pool for launching containers in MR AM not + expanding as expected. (vinodkv via sseth) + + MAPREDUCE-3639. Fixed TokenCache to work with absent FileSystem canonical + service-names. (Siddharth Seth via vinodkv) + + MAPREDUCE-3380. Token infrastructure for running clients which are not kerberos + authenticated. (mahadev) + + MAPREDUCE-3648. TestJobConf failing. (Thomas Graves via mahadev) + + MAPREDUCE-3651. TestQueueManagerRefresh fails. (Thomas Graves via mahadev) + + MAPREDUCE-3645. TestJobHistory fails. (Thomas Graves via mahadev) + + MAPREDUCE-3652. org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization + fails. (Thomas Graves via mahadev) + + MAPREDUCE-3625. CapacityScheduler web-ui display of queue's used capacity is broken. + (Jason Lowe via mahadev) + + MAPREDUCE-3596. Fix scheduler to handle cleaned up containers, which NMs + may subsequently report as running. (Vinod Kumar Vavilapalli via sseth) + + MAPREDUCE-3656. Fixed a race condition in MR AM which is failing the sort + benchmark consistently. (Siddarth Seth via vinodkv) + + MAPREDUCE-3532. Modified NM to report correct http address when an ephemeral + web port is configured. (Bhallamudi Venkata Siva Kamesh via vinodkv) + + MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable + speculating either maps or reduces. (Eric Payne via vinodkv) + + MAPREDUCE-3664. Federation Documentation has incorrect configuration example. + (Brandon Li via jitendra) + + MAPREDUCE-3649. Job End notification gives an error on calling back. + (Ravi Prakash via mahadev) + + MAPREDUCE-3657. State machine visualize build fails. (Jason Lowe + via mahadev) + + MAPREDUCE-2450. Fixed a corner case with interrupted communication threads + leading to a long timeout in Task. (Rajesh Balamohan via acmurthy) + + MAPREDUCE-3669. Allow clients to talk to MR HistoryServer using both + delegation tokens and kerberos. (mahadev via acmurthy) + + MAPREDUCE-3684. LocalDistributedCacheManager does not shut down its thread + pool (tomwhite) + + MAPREDUCE-3582. Move successfully passing MR1 tests to MR2 maven tree. + (ahmed via tucu) + + MAPREDUCE-3698. Client cannot talk to the history server in secure mode. + (mahadev) + + MAPREDUCE-3689. RM web UI doesn't handle newline in job name. + (Thomas Graves via mahadev) + + MAPREDUCE-3701. Delete HadoopYarnRPC from 0.23 branch. + (mahadev) + + MAPREDUCE-3549. write api documentation for web service apis for RM, NM, + mapreduce app master, and job history server (Thomas Graves via mahadev) + + MAPREDUCE-3705. ant build fails on 0.23 branch. (Thomas Graves via + mahadev) + + MAPREDUCE-3691. webservices add support to compress response. + (Thomas Graves via mahadev) + + MAPREDUCE-3702. internal server error trying access application master + via proxy with filter enabled (Thomas Graves via mahadev) + + MAPREDUCE-3646. Remove redundant URL info from "mapred job" output. + (Jonathan Eagles via mahadev) + + MAPREDUCE-3681. Fixed computation of queue's usedCapacity. (acmurthy) + + MAPREDUCE-3505. yarn APPLICATION_CLASSPATH needs to be overridable. + (ahmed via tucu) + + MAPREDUCE-3714. Fixed EventFetcher and Fetcher threads to shut-down properly + so that reducers don't hang in corner cases. (vinodkv) + + MAPREDUCE-3712. The mapreduce tar does not contain the hadoop-mapreduce-client- + jobclient-tests.jar. (mahadev) + + MAPREDUCE-3717. JobClient test jar has missing files to run all the test programs. + (mahadev) + + MAPREDUCE-3630. Fixes a NullPointer exception while running TeraGen - if a + map is asked to generate 0 records. (Mahadev Konar via sseth) + + MAPREDUCE-3683. Fixed maxCapacity of queues to be product of parent + maxCapacities. (acmurthy) + + MAPREDUCE-3713. Fixed the way head-room is allocated to applications by + CapacityScheduler so that it deducts current-usage per user and not + per-application. (Arun C Murthy via vinodkv) + + MAPREDUCE-3721. Fixed a race in shuffle which caused reduces to hang. + (sseth via acmurthy) + + MAPREDUCE-3733. Add Apache License Header to hadoop-distcp/pom.xml. + (mahadev) + + MAPREDUCE-3735. Add distcp jar to the distribution (tar). + (mahadev) + + MAPREDUCE-3720. Changed bin/mapred job -list to not print job-specific + information not available at RM. (vinodkv via acmurthy) + + MAPREDUCE-3742. "yarn logs" command fails with ClassNotFoundException. + (Jason Lowe via mahadev) + + MAPREDUCE-3703. ResourceManager should provide node lists in JMX output. + (Eric Payne via mahadev) + + MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use + java.io.File.createTempFile to create temporary files as part of their + tasks. (Jonathan Eagles via vinodkv) + + MAPREDUCE-3748. Changed a log in CapacityScheduler.nodeUpdate to debug. + (ramya via acmurthy) + + MAPREDUCE-3764. Fixed resource usage metrics for queues and users. + (acmurthy) + + MAPREDUCE-3749. ConcurrentModificationException in counter groups. + (tomwhite) + + MAPREDUCE-3762. Fixed default CapacityScheduler configs. (mahadev via + acmurthy) + + MAPREDUCE-3499. New MiniMR does not setup proxyuser configuration + correctly, thus tests using doAs do not work. (johnvijoe via tucu) + + MAPREDUCE-3696. MR job via oozie does not work on hadoop 23. + (John George via mahadev) + + MAPREDUCE-3427. Fix streaming unit tests broken after mavenization. + (Hitesh Shah via acmurthy) + + MAPREDUCE-3640. Allow AMRecovery to work with partial JobHistory files. + (Arun C Murthy via sseth) + + MAPREDUCE-3752. Modified application limits to include queue max-capacities + besides the usual user limits. (Arun C Murthy via vinodkv) + + MAPREDUCE-3744. Fix the yarn logs command line. Improve error messages for + mapred job -logs. (Jason Lowe via sseth) + + MAPREDUCE-3780. Fixed a bug where applications killed before getting + activated were not getting cleaned up properly. (Hitesh Shah via acmurthy) + + MAPREDUCE-3708. Metrics: Incorrect Apps Submitted Count (Bhallamudi via + mahadev) + + MAPREDUCE-3727. jobtoken location property in jobconf refers to wrong + jobtoken file (tucu) + + MAPREDUCE-3711. Fixed MR AM recovery so that only single selected task + output is recovered and thus reduce the unnecessarily bloated recovery + time. (Robert Joseph Evans via vinodkv) + + MAPREDUCE-3760. Changed active nodes list to not contain unhealthy nodes + on the webUI and metrics. (vinodkv) + + MAPREDUCE-3417. Fixed job-access-controls to work with MR AM and + JobHistoryServer web-apps. (Jonathan Eagles via vinodkv) + + MAPREDUCE-3803. Fix broken build of raid contrib due to HDFS-2864. + (Ravi Prakash via suresh) + + MAPREDUCE-3791. can't build site in hadoop-yarn-server-common. + (mahadev) + + MAPREDUCE-3723. TestAMWebServicesJobs & TestHSWebServicesJobs + incorrectly asserting tests (Bhallamudi Venkata Siva Kamesh + via mahadev) + + MAPREDUCE-3795. "job -status" command line output is malformed. + (vinodkv via mahadev) + + MAPREDUCE-3759. ClassCastException thrown in -list-active-trackers when + there are a few unhealthy nodes (vinodkv via mahadev) + + MAPREDUCE-3775. Change MiniYarnCluster to escape special chars in testname. + (Hitesh Shah via mahadev) + + MAPREDUCE-3765. FifoScheduler does not respect yarn.scheduler.fifo.minimum- + allocation-mb setting (Hitesh Shah via mahadev) + + MAPREDUCE-3747. Initialize queue metrics upfront and added start/finish + time to RM Web-UI. (acmurthy) + + MAPREDUCE-3814. Fixed MRV1 compilation. (Arun C Murthy via vinodkv) + + MAPREDUCE-3810. Performance tweaks - reduced logging in AM and defined + hascode/equals for ResourceRequest & Priority. (vinodkv via acmurthy) + + MAPREDUCE-3813. Added a cache for resolved racks. (vinodkv via acmurthy) + + MAPREDUCE-3808. Fixed an NPE in FileOutputCommitter for jobs with maps + but no reduces. (Robert Joseph Evans via vinodkv) + + MAPREDUCE-3804. yarn webapp interface vulnerable to cross scripting attacks + (Dave Thompson via bobby) + + MAPREDUCE-3354. Changed scripts so that jobhistory server is started by + bin/mapred instead of bin/yarn. (Jonathan Eagles via acmurthy) + + MAPREDUCE-3809. Ensure that there is no needless sleep in Task at the end + of the task. (sseth via acmurthy) + + MAPREDUCE-3794. Support mapred.Task.Counter and mapred.JobInProgress.Counter + enums for compatibility (Tom White via mahadev) + + MAPREDUCE-3697. Support binary compatibility for Counters after + MAPREDUCE-901. (mahadev via acmurthy) + + MAPREDUCE-3817. Fixed bin/mapred to allow running of distcp and archive + jobs. (Arpit Gupta via acmurthy) + + MAPREDUCE-3709. TestDistributedShell is failing. (Hitesh Shah via + mahadev) + + MAPREDUCE-3436. JobHistory webapp address should use the host configured + in the jobhistory address. (Ahmed Radwan via sseth) + + MAPREDUCE-3815. Fixed MR AM to always use hostnames and never IPs when + requesting containers so that scheduler can give off data local containers + correctly. (Siddarth Seth via vinodkv) + + MAPREDUCE-3833. Fixed a bug in reinitiaziling of queues. (Jason Lowe via + acmurthy) + + MAPREDUCE-3826. Fixed a bug in RM web-ui which broke sorting. (Jonathan + Eagles via acmurthy) + + MAPREDUCE-3823. Ensure counters are calculated only once after a job + finishes. (Vinod Kumar Vavilapalli via sseth) + + MAPREDUCE-3827. Changed Counters to use ConcurrentSkipListMap for + performance. (vinodkv via acmurthy) + + MAPREDUCE-3822. Changed FS counter computation to use all occurences of + the same FS scheme, instead of randomly using one. (Mahadev Konar via + sseth) + + MAPREDUCE-3834. Changed MR AM to not add the same rack entry multiple times + into the container request table when multiple hosts for a split happen to + be on the same rack. (Siddarth Seth via vinodkv) + + MAPREDUCE-3828. Ensure that urls in single-node mode are correct. (sseth + via acmurthy) + + MAPREDUCE-3770. Zombie.getJobConf() results into NPE. (amarrk) + + MAPREDUCE-3843. Job summary log file found missing on the RM host + (Anupam Seth via tgraves) + + MAPREDUCE-3846. Addressed MR AM hanging issues during AM restart and then + the recovery. (vinodkv) + + MAPREDUCE-3802. Added test to validate that AM can crash multiple times and + still can recover successfully after MAPREDUCE-3846. (vinodkv) + + MAPREDUCE-3858. Task attempt failure during commit results in task never completing. + (Tom White via mahadev) + + MAPREDUCE-3856. Instances of RunningJob class givs incorrect job tracking + urls when mutiple jobs are submitted from same client jvm. (Eric Payne via + sseth) + + MAPREDUCE-3880. Changed LCE binary to be 32-bit. (acmurthy) + +Release 0.23.0 - 2011-11-01 + + INCOMPATIBLE CHANGES + + MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of + JobTrackerStatus. (tomwhite) + + MAPREDUCE-2430. Remove mrunit contrib. (nigel via eli) + + MAPREDUCE-2606. Remove IsolationRunner. (Alejandro Abdelnur via eli) + + NEW FEATURES + + MAPREDUCE-2682. Add "mapred classpath" command to print classpath + for MR applications. (vinodkv via acmurthy) + + MAPREDUCE-2107. [Gridmix] Total heap usage emulation in Gridmix. + (Amar Kamat and Ravi Gummadi via amarrk) + + MAPREDUCE-2106. [Gridmix] Cumulative CPU usage emulation in Gridmix. + (amarrk) + + MAPREDUCE-2543. [Gridmix] High-Ram feature emulation in Gridmix. (amarrk) + + MAPREDUCE-2408. [Gridmix] Compression emulation in Gridmix. (amarrk) + + MAPREDUCE-2473. Add "mapred groups" command to query the server-side groups + resolved for a user. (Aaron T. Myers via todd) + + MAPREDUCE-461. Enable ServicePlugins for the JobTracker. + (Fredrik Hedberg via tomwhite) + + MAPREDUCE-2521. Create RPM and Debian packages for MapReduce. Changes + deployment layout to be consistent across the binary tgz, rpm, and deb. + (Eric Yang via omalley) + + MAPREDUCE-2323. Add metrics to the fair scheduler. (todd) + + MAPREDUCE-2037. Capture intermediate progress, CPU and memory usage for + tasks. (Dick King via acmurthy) + + MAPREDUCE-279. MapReduce 2.0. Merging MR-279 branch into trunk. Contributed by + Arun C Murthy, Christopher Douglas, Devaraj Das, Greg Roelofs, Jeffrey + Naisbitt, Josh Wills, Jonathan Eagles, Krishna Ramachandran, Luke Lu, Mahadev + Konar, Robert Evans, Sharad Agarwal, Siddharth Seth, Thomas Graves, and Vinod + Kumar Vavilapalli. + + MAPREDUCE-2930. Added the ability to be able to generate graphs from the + state-machine definitions. (Binglin Chang via vinodkv) + + MAPREDUCE-2719. Add a simple, DistributedShell, application to illustrate + alternate frameworks on YARN. (Hitesh Shah via acmurthy) + + MAPREDUCE-3104. Implemented Application-acls. (vinodkv) + + MAPREDUCE-2708. Designed and implemented MR Application Master recovery to + make MR AMs resume their progress after restart. (Sharad Agarwal via vinodkv) + + MAPREDUCE-2858. Added a WebApp Proxy for applications. (Robert Evans via + acmurthy) + + IMPROVEMENTS + + MAPREDUCE-2187. Reporter sends progress during sort/merge. (Anupam Seth via + acmurthy) + + MAPREDUCE-2365. Add counters to track bytes (read,written) via + File(Input,Output)Format. (Siddharth Seth via acmurthy) + + MAPREDUCE-2680. Display queue name in job client CLI. (acmurthy) + + MAPREDUCE-2679. Minor changes to sync trunk with MR-279 branch. (acmurthy) + + MAPREDUCE-2400. Remove Cluster's dependency on JobTracker via a + ServiceProvider for the actual implementation. (tomwhite via acmurthy) + + MAPREDUCE-2596. [Gridmix] Summarize Gridmix runs. (amarrk) + + MAPREDUCE-2563. [Gridmix] Add High-Ram emulation system tests to + Gridmix. (Vinay Kumar Thota via amarrk) + + MAPREDUCE-2104. [Rumen] Add Cpu, Memory and Heap usages to + TraceBuilder's output. (amarrk) + + MAPREDUCE-2554. [Gridmix] Add distributed cache emulation system tests + to Gridmix. (Vinay Kumar Thota via amarrk) + + MAPREDUCE-2543. [Gridmix] High-Ram feature emulation testcase. (amarrk) + + MAPREDUCE-2469. Task counters should also report the total heap usage of + the task. (Ravi Gummadi and Amar Ramesh Kamat via amarrk) + + MAPREDUCE-2544. [Gridmix] Add compression emulation system tests to + Gridmix. (Vinay Kumar Thota via amarrk) + + MAPREDUCE-2517. [Gridmix] Add system tests to Gridmix. + (Vinay Kumar Thota via amarrk) + + MAPREDUCE-2492. The new MapReduce API should make available task's + progress to the task. (amarrk) + + MAPREDUCE-2153. Bring in more job configuration properties in to the trace + file. (Rajesh Balamohan via amarrk) + + MAPREDUCE-1461. Feature to instruct rumen-folder utility to skip jobs worth + of specific duration. (Rajesh Balamohan via amarrk) + + MAPREDUCE-2172. Added test-patch.properties required by test-patch.sh + (nigel) + + MAPREDUCE-2156. Raid-aware FSCK. (Patrick Kling via dhruba) + + MAPREDUCE-2215. A more elegant FileSystem#listCorruptFileBlocks API + (RAID changes) (Patrick Kling via hairong) + + MAPREDUCE-1831. BlockPlacement policy for HDFS-RAID. + (Scott Chen via dhruba) + + MAPREDUCE-1906. Lower minimum heartbeat interval for TaskTracker + (Scott Carey and Todd Lipcon via todd) + + MAPREDUCE-1382. MRAsyncDiscService should tolerate missing local.dir. + (Zheng Shao and tomwhite via tomwhite) + + MAPREDUCE-2263. MapReduce side of HADOOP-6904: RPC compatibility. + (hairong) + + MAPREDUCE-1706. Log RAID recoveries on HDFS. (schen) + + MAPREDUCE-2334. Update BlockPlacementPolicyRaid for the new method + in BlockPlacementPolicy. (szetszwo) + + MAPREDUCE-2254. Allow setting of end-of-record delimiter for + TextInputFormat (Ahmed Radwan via todd) + + MAPREDUCE-1927. Unit test for HADOOP-6835 (concatenated gzip support). + (Greg Roelofs via tomwhite) + + MAPREDUCE-2206. The task-cleanup tasks should be optional. (schen) + + MAPREDUCE-2225. MultipleOutputs should not require the use of 'Writable'. + (Harsh J Chouraria via tomwhite) + + MAPREDUCE-1811. Job.monitorAndPrintJob() should print status of the job + at completion. (Harsh J Chouraria via tomwhite) + + MAPREDUCE-993. bin/hadoop job -events <jobid> <from-event-#> <#-of-events> + help message is confusing. (Harsh J Chouraria via tomwhite) + + MAPREDUCE-2302. Add static factory methods in GaloisField. (schen) + + MAPREDUCE-2351. mapred.job.tracker.history.completed.location should + support an arbitrary filesystem URI. (tomwhite) + + MAPREDUCE-2239. BlockPlacementPolicyRaid should call getBlockLocations + only when necessary. (schen) + + MAPREDUCE-2331. Add coverage of task graph servlet to fair scheduler system + test. (todd) + + MAPREDUCE-2367. Allow using a file to exclude certain tests from build. + (todd) + + MAPREDUCE-2202. Generalize CLITest structure and interfaces to faciliate + upstream adoption (e.g. for web or system testing). (cos) + + MAPREDUCE-2420. JobTracker should be able to renew delegation token over + HTTP (Boris Shkolnik via jitendra) + + MAPREDUCE-2474. Add docs to the new API Partitioner on how to access the + Job Configuration. (Harsh J Chouraria via todd) + + MAPREDUCE-2475. Disable IPV6 for junit tests. (suresh srinivas via mahadev) + + MAPREDUCE-2422. Removed unused internal methods from DistributedCache. + (tomwhite) + + MAPREDUCE-2456. Log the reduce taskID and associated TaskTrackers with + failed fetch notifications in the JobTracker log. + (Jeffrey Naisbitt via cdouglas) + + MAPREDUCE-869. Documentation for config to set map/reduce task environment + (Alejandro Abdelnur via todd) + + MAPREDUCE-2410. Add entry to streaming FAQ about how streaming reducers + receive keys. (Harsh J Chouraria via todd) + + MAPREDUCE-2499. MR part of HADOOP-7291. (eli) + + MAPREDUCE-2497. Missing spaces in error messages. (eli) + + MAPREDUCE-2502. JobSubmitter should use mapreduce.job.maps instead of + its deprecated equivalent. (eli via todd) + + MAPREDUCE-2381. JobTracker instrumentation not consistent about error + handling. (Philip Zeyliger via tomwhite) + + MAPREDUCE-2449. Allow for command line arguments when performing + "Run on Hadoop" action in Eclipse plugin. (Jeff Zemerick via todd) + + MAPREDUCE-2483. Remove duplication of jars between Hadoop subprojects + from build artifacts. (Eric Yang via omalley) + + MAPREDUCE-2372. TaskLogAppender mechanism shouldn't be set up in + log4j.properties (todd) + + MAPREDUCE-2516. Rename webinterface.private.actions to + mapreduce.jobtracker.webinterface.trusted (Ari Rabkin via todd) + + MAPREDUCE-2459. Cache HAR filesystem metadata. (Mac Yang via mahadev) + + HADOOP-7259. Contrib modules should include the build.properties from + the enclosing hadoop directory. (omalley) + + MAPREDUCE-2494. Order distributed cache deletions by LRU. (Robert Joseph + Evans via cdouglas) + + MAPREDUCE-2452. Makes the cancellation of delegation tokens happen in a + separate thread. (ddas) + + HADOOP-7106. Reorganize project SVN layout to "unsplit" the projects. + (todd, nigel) + + MAPREDUCE-2249. Check the reflexive property of Counters objects when + comparing equality. (Devaraj K via todd) + + MAPREDUCE-2623. Update ClusterMapReduceTestCase to use + MiniDFSCluster.Builder (Harsh J Chouraria via eli) + + MAPREDUCE-2602. Allow setting of end-of-record delimiter for + TextInputFormat for the old API. (Ahmed Radwan via todd) + + MAPREDUCE-2705. Permits parallel multiple task launches. + (Thomas Graves via ddas) + + MAPREDUCE-2489. Jobsplits with random hostnames can make the queue + unusable (jeffrey naisbit via mahadev) + + MAPREDUCE-2854. update INSTALL with config necessary run mapred on yarn. + (thomas graves via mahadev) + + MAPREDUCE-2701. app/Job.java needs UGI for the user that launched it. + (Robert Evans via mahadev) + + MAPREDUCE-2652. Enabled multiple NMs to be runnable on a single node by + making shuffle service port to be truely configurable. (Robert Evans via + vinodkv) + + MAPREDUCE-2735. Add an applications summary log to ResourceManager. + (Thomas Graves via acmurthy) + + MAPREDUCE-2697. Enhance CapacityScheduler to cap concurrently running + applications per-queue & per-user. (acmurthy) + Configuration changes: + add yarn.capacity-scheduler.maximum-am-resource-percent + + MAPREDUCE-2774. Add startup message to ResourceManager & NodeManager on + startup. (Venu Gopala Rao via acmurthy) + + MAPREDUCE-2655. Add audit logs to ResourceManager and NodeManager. (Thomas + Graves via acmurthy) + + MAPREDUCE-2864. Normalize configuration variable names for YARN. (Robert + Evans via acmurthy) + + MAPREDUCE-2690. Web-page for FifoScheduler. (Eric Payne via acmurthy) + + MAPREDUCE-2711. Update TestBlockPlacementPolicyRaid for the new namesystem + and block management APIs. (szetszwo) + + MAPREDUCE-2933. Change allocate call to return ContainerStatus for + completed containers rather than Container. (acmurthy) + + MAPREDUCE-2675. Reformat JobHistory Server main page to be more + useful. (Robert Joseph Evans via vinodkv). + + MAPREDUCE-2896. Simplify all apis to in + org.apache.hadoop.yarn.api.records.* to be get/set only. Added javadocs to + all public records. (acmurthy) + + MAPREDUCE-2676. MR-279: JobHistory Job page needs reformatted. (Robert Evans via + mahadev) + + MAPREDUCE-2899. Replace major parts of ApplicationSubmissionContext with a + ContainerLaunchContext (Arun Murthy via mahadev) + + MAPREDUCE-2966. Added ShutDown hooks for MRV2 processes so that they can + gracefully exit. (Abhijit Suresh Shingate via vinodkv) + + MAPREDUCE-2672. MR-279: JobHistory Server needs Analysis this job. + (Robert Evans via mahadev) + + MAPREDUCE-2965. Streamlined the methods hashCode(), equals(), compareTo() + and toString() for all IDs. (Siddharth Seth via vinodkv) + + MAPREDUCE-2726. Added job-file to the AM and JobHistoryServer web + interfaces. (Jeffrey Naisbitt via vinodkv) + + MAPREDUCE-2880. Improve classpath-construction for mapreduce AM and + containers. (Arun C Murthy via vinodkv) + + MAPREDUCE-3055. Simplified ApplicationAttemptId passing to + ApplicationMaster via environment variable. (vinodkv) + + MAPREDUCE-3092. Removed a special comparator for JobIDs in JobHistory as + JobIDs are already comparable. (Devaraj K via vinodkv) + + MAPREDUCE-3099. Add docs for setting up a single node MRv2 cluster. + (mahadev) + + MAPREDUCE-3001. Added task-specific counters to AppMaster and JobHistory + web-UIs. (Robert Joseph Evans via vinodkv) + + MAPREDUCE-3098. Fixed RM and MR AM to report YarnApplicationState and + application's FinalStatus separately. (Hitesh Shah via vinodkv) + + MAPREDUCE-2889. Added documentation for writing new YARN applications. + (Hitesh Shah via acmurthy) + + MAPREDUCE-3134. Added documentation the CapacityScheduler. (acmurthy) + + MAPREDUCE-3013. Removed YarnConfiguration.YARN_SECURITY_INFO and its usage + as it doesn't affect security any more. (vinodkv) + + MAPREDUCE-2907. Changed log level for various messages in ResourceManager + from INFO to DEBUG. (Ravi Prakash via vinodkv) + + MAPREDUCE-2702. Added a new API in OutputCommitter for recovering + the outputs of tasks from a crashed job so as to support MR Application + Master recovery. (Sharad Agarwal and Arun C Murthy via vinodkv) + + MAPREDUCE-2738. Added the missing cluster level statistics on the RM web + UI. (Robert Joseph Evans via vinodkv) + + MAPREDUCE-2988. Reenabled TestLinuxContainerExecutor reflecting the + current NodeManager code. (Robert Joseph Evans via vinodkv) + + MAPREDUCE-3161. Improved some javadocs and fixed some typos in + YARN. (Todd Lipcon via vinodkv) + + MAPREDUCE-3148. Ported MAPREDUCE-2702 to old mapred api for aiding task + recovery. (acmurthy) + + MAPREDUCE-3133. Running a set of methods in a Single Test Class. + (Jonathan Eagles via mahadev) + + MAPREDUCE-3059. QueueMetrics do not have metrics for aggregate + containers-allocated and aggregate containers-released. + (Devaraj K via mahadev) + + MAPREDUCE-3187. Add names for various unnamed threads in MR2. + (Todd Lipcon and Siddharth Seth via mahadev) + + MAPREDUCE-3136. Added documentation for setting up Hadoop clusters in both + non-secure and secure mode for both HDFS & YARN. (acmurthy) + + MAPREDUCE-3068. Added a whitelist of environment variables for containers + from the NodeManager and set MALLOC_ARENA_MAX for all daemons and + containers. (Chris Riccomini via acmurthy) + + MAPREDUCE-3144. Augmented JobHistory with the information needed for + serving aggregated logs. (Siddharth Seth via vinodkv) + + MAPREDUCE-3163. JobClient spews errors when killing MR2 job. + (mahadev) + + MAPREDUCE-3239. Use new createSocketAddr API in MRv2 to give better + error messages on misconfig (Todd Lipcon via mahadev) + + MAPREDUCE-2747. Cleaned up LinuxContainerExecutor binary sources and changed + the configuration to use yarn names. (Robert Joseph Evans via vinodkv) + + MAPREDUCE-3205. Fix memory specifications to be physical rather than + virtual, allowing for a ratio between the two to be configurable. (todd + via acmurthy) + + MAPREDUCE-2986. Fixed MiniYARNCluster to support multiple NodeManagers. + (Anupam Seth via vinodkv) + + MAPREDUCE-2736. Remove unused contrib components dependent on MR1. (eli) + + MAPREDUCE-2989. Modified JobHistory to link to task and AM logs from the + JobHistoryServer. (Siddharth Seth via vinodkv) + + MAPREDUCE-3014. Rename and invert logic of '-cbuild' profile to 'native' and off + by default. (tucu) + + MAPREDUCE-3171. normalize nodemanager native code compilation with common/hdfs + native. (tucu) + + MAPREDUCE-3146. Added a MR specific command line to dump logs for a + given TaskAttemptID. (Siddharth Seth via vinodkv) + + MAPREDUCE-3275. Added documentation for AM WebApp Proxy. (Robert Evans via + acmurthy) + + MAPREDUCE-3322. Added a better index.html and an brief overview of YARN + architecture. (acmurthy) + + OPTIMIZATIONS + + MAPREDUCE-2026. Make JobTracker.getJobCounters() and + JobInProgress.getCounters() aquire locks in a shorter time period. + (Joydeep Sen Sarma via schen) + + MAPREDUCE-2740. MultipleOutputs in new API creates needless + TaskAttemptContexts. (todd) + + MAPREDUCE-901. Efficient framework counters. (llu via acmurthy) + + MAPREDUCE-2629. Workaround a JVM class loading quirk which prevents + JIT compilation of inner classes methods in ReduceContextImpl. + + BUG FIXES + + MAPREDUCE-2603. Disable High-Ram emulation in system tests. + (Vinay Kumar Thota via amarrk) + + MAPREDUCE-2539. Fixed NPE in getMapTaskReports in JobClient. (Robert Evans via + acmurthy) + + MAPREDUCE-1978. Rumen TraceBuilder should provide recursive + input folder scanning. + + MAPREDUCE-2416. Remove the restriction of specifying group names in + users-list file for Gridmix in RoundRobinUserResolver mode. + + MAPREDUCE-2417. Fix Gridmix in RoundRobinUserResolver mode to + map testing/proxy users to unique users in a trace. + + MAPREDUCE-2307. Exception thrown in Jobtracker logs, when the Scheduler + configured is FairScheduler. (Devaraj K via matei) + + MAPREDUCE-2199. build is broken 0.22 branch creation. (cos) + + MAPREDUCE-1752. Implement getFileBlockLocations in HarFilesystem. + (Patrick Kling via dhruba) + + MAPREDUCE-2155. RaidNode should optionally use the mapreduce jobs to + fix missing blocks. (Patrick Kling via dhruba) + + MAPREDUCE-1334. Fix TestIndexUpdater by ignoring _SUCCESS file in HDFS. + (Kay Kay via yhemanth) + + MAPREDUCE-2232. Add missing methods to TestMapredGroupMappingServiceRefresh. + (Todd Lipcon via eli) + + MAPREDUCE-2271. Fix TestSetupTaskScheduling failure on trunk. + (Liyin Liang via todd) + + MAPREDUCE-2290. Fix compilation error in TestTaskCommit. (eli) + + MAPREDUCE-2294. Fix compilation error in mumak. (todd) + + MAPREDUCE-2300. Fix TestUmbilicalProtocolWithJobToken on trunk after + HADOOP-6904. (todd) + + MAPREDUCE-2296. Fix references to misspelled method name + getProtocolSigature (todd) + + MAPREDUCE-2311. Fix TestFairScheduler failure (schen) + + MAPREDUCE-1996. API: Reducer.reduce() method detail misstatement. + (Harsh J Chouraria via tomwhite) + + MAPREDUCE-2203. Wrong javadoc for TaskRunner's appendJobJarClasspaths + method. (Jingguo Yao via tomwhite) + + MAPREDUCE-2074. Task should fail when symlink creation fails. + (Priyo Mustafi via tomwhite) + + MAPREDUCE-1242. Chain APIs error misleading. + (Harsh J Chouraria via tomwhite) + + MAPREDUCE-2379. Adds missing DistributedCache configurations in + mapred-default.xml (Todd Lipcon via amareshwari) + + MAPREDUCE-2348. Disable mumak tests on trunk since they currently time out + (todd) + + MAPREDUCE-2395. TestBlockFixer timing out on trunk. (Ramkumar Vadali via + todd) + + MAPREDUCE-2426. Make TestFairSchedulerSystem fail with more verbose output + (todd) + + MAPREDUCE-2448. NoSuchMethodError: + org.apache.hadoop.hdfs.TestDatanodeBlockScanner.corruptReplica(..) (eli) + + MAPREDUCE-2460. Fix flaky test TestFairSchedulerSystem. (todd) + + MAPREDUCE-2451. Log the details from health check script at the + JobTracker. (Thomas Graves via cdouglas) + + MAPREDUCE-2467. HDFS-1052 changes break the raid contrib module in + MapReduce. (suresh srinivas via mahadev) + + MAPREDUCE-2258. IFile reader closes stream and compressor in wrong order. + (todd via tomwhite) + + MAPREDUCE-2518. The t flag is missing in distcp help message. (Wei Yongjun + via szetszwo) + + MAPREDUCE-2514. Fix typo in TaskTracker ReinitTrackerAction log message. + (Jonathan Eagles via cdouglas) + + MAPREDUCE-2490. Add logging to graylist and blacklist activity to aid + diagnosis of related issues. (Jonathan Eagles via cdouglas) + + MAPREDUCE-2495. exit() the TaskTracker when the distributed cache cleanup + thread dies. (Robert Joseph Evans via cdouglas) + + MAPREDUCE-2470. Fix NPE in RunningJobs::getCounters. (Robert Joseph Evans + via cdouglas) + + MAPREDUCE-2536. Update FsShell -mv command usage in TestMRCLI. (Daryn + Sharp via szetszwo) + + MAPREDUCE-2529. Add support for regex-based shuffle metric counting + exceptions. (Thomas Graves via cdouglas) + + MAPREDUCE-2559. ant binary fails due to missing c++ lib dir. (eli) + + MAPREDUCE-2573. Fix new findbugs warning introduced by MAPREDUCE-2494. + (Robert Joseph Evans via todd) + + MAPREDUCE-2581. Spelling errors in log messages. (Tim Sell via eli) + + MAPREDUCE-2588. Change raid to the new DataTransferProtocol API. (szetszwo) + + MAPREDUCE-2576. Typo in comment in SimulatorLaunchTaskAction.java. + (Tim Sell via jghoman) + + MAPREDUCE-2550. Fix bin/mapred to work properly from within a source + checkout (Eric Yang via todd) + + MAPREDUCE-2620. Update RAID for HDFS-2087. (szetszwo) + + MAPREDUCE-2624. Update RAID for HDFS-2107. (szetszwo) + + MAPREDUCE-2670. Fixing spelling mistake in FairSchedulerServlet.java. (eli) + + MAPREDUCE-2710. Update JobSubmitter.printTokens(..) for HDFS-2161. + (szetszwo) + + MAPREDUCE-2409. DistributedCache maps files and archives to the same path, + despite semantic incompatibility. (Siddharth Seth via cdouglas) + + MAPREDUCE-2575. TestMiniMRDFSCaching fails if test.build.dir is set + to something other than build/test (Thomas Graves via mahadev) + + MAPREDUCE-2622. Remove the last remaining reference to the deprecated + configuration "io.sort.mb". (Harsh J Chouraria via todd) + + MAPREDUCE-2732. Remove directly accessing FSNamesystem.LOG from + TestCopyFiles and TestDistCh. (szetszwo) + + MAPREDUCE-2463. Job history files are not moved to done folder when job + history location is hdfs. (Devaraj K via szetszwo) + + MAPREDUCE-2243. Close streams propely in a finally-block to avoid leakage + in CompletedJobStatusStore, TaskLog, EventWriter and TotalOrderPartitioner. + (Devaraj K via szetszwo) + + MAPREDUCE-2741. Make ant build system work with hadoop-common JAR + generated by Maven. (Alejandro Abdelnur via tomwhite) + + MAPREDUCE-2760. mapreduce.jobtracker.split.metainfo.maxsize typoed + in mapred-default.xml. (todd via eli) + + MAPREDUCE-2797. Update mapreduce tests and RAID for HDFS-2239. (szetszwo) + + MAPREDUCE-2805. Update RAID for HDFS-2241. (szetszwo) + + MAPREDUCE-2837. Ported bug fixes from y-merge to prepare for MAPREDUCE-279 + merge. (acmurthy) + + MAPREDUCE-2541. Fixed a race condition in IndexCache.removeMap. (Binglin + Chang via acmurthy) + + MAPREDUCE-2458. Rename sanitized pom.xml in build directory to work around IDE + bug (Luke Lu via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Clear application notification if sent once + to NodeManager (mahadev) + + MAPREDUCE-2433. YARNApplicationConstants hard code app master jar version (Luke + Lu via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Implement restart for resource manager + phase 1 - Helper classes to store and restore the data structures. (mahadev) + + MAPREDUCE-2414. Change MRv2 to use generic interfaces. (Siddharth Seth via + acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Implement health-checks for the node - + server side(ResourceManager) changes. (vinodkv) + + MAPREDUCE-2405: Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2) + (Greg Roelofs via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Implementing Containers' memory monitoring. + (vinodkv) + + MAPREDUCE-2440. Name clashes in TypeConverter (luke via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Add fail count to the command line of the + application master. (mahadev) + + MAPREDUCE-2424. Polish uber-AppMaster: add uber-AM counters and GUI indicators. + (Greg Roelofs via mahadev) + + MAPREDUCE-2405. Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2). + (Greg Roelofs and Sharad Agarwal via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix YarnRemoteException to give more + details. (Siddharth Seth via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. WebApp for Job History (Krishna + Ramachandran via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Install sanitized poms for downstream + sanity (Luke Lu via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Add HistoryCleanerService to Job History + server. (Krishna Ramachandran via sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Implement 'bin/mapred job -list' and + 'bin/mapred job -list-active-trackers'. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Implement 'bin/mapred queue [-info + [-showJobs]] [-list] and enhanced 'bin/mapred job -list' to show queue and + ApplicationMaster information. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Fixed computation of user-limits at + runtime. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Added functionality to refresh queues at + runtime via the 'bin/yarn rmadmin' command. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Added functionality to stop/start queues. + (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Introducing web-UI for NodeManager and + linking it from RM UI. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Fix race condition in TestJobHistoryEvents + and TestJobHistoryParsing. (sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Add Containers' logs' view to NM UI and + link it from AM UI. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Add ACLs for queues and command-line + utilities for viewing them. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Recovery of MR Application Master from + failures. (sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Added support High-RAM applications in + CapacityScheduler. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Completing the ZooKeeper Store for + ResourceManager state. (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Reorient container localization to be + per-container rather than per-application. (cdouglas) + + MAPREDUCE-279. Fix in MR-279 branch. Fix file creation in + JobHistoryEventHandler. (sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Disable ContainerMonitoring for non-linux + systems. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Fix container launch w/ inconsistent + credential file naming. (cdouglas) + + MAPREDUCE-2434. Metrics for ResourceManager. (Luke Lu via acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. RM Restart Phase 2 - Completed the recovery + of components in the RM (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix to send finish application event only + when the application is finished (mahadev) + + MAPREDUCE-2462. Write job conf along with JobHistory, other minor improvements. + (Siddharth Seth via sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Implement 'delay scheduling' for better + locality in CapacityScheduler and improved high-ram applications. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Implement Job Acls in MR Application + Master. (sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Moving userlogs out of container work-dir + into a separate directory structure. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Completing RM Restart. Completed Phase 3 of + making sure events are logged and restored (mahadev) + + MAPREDUCE-2468. Add metrics for NM Shuffle. (Luke Lu via cdouglas) + + MAPREDUCE-279. Fix in MR-279 branch. Adding user log handling for YARN. Making + NM put the user-logs on DFS and providing log-dump tools. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Fixing three tight-loops in RM that are + causing high cpu-usage. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Replacing FileContext usage with FileSystem + to work around security authentication issues with FileContext against a secure + DFS. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Client reconnect to restarted AM. (sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Fix refreshProxy in ClientServiceDelegate. + (sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Fix Null Pointer in TestUberAM. (sharad) + + MAPREDUCE-2478. Improve history server. (Siddharth Seth via sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Improved TestJobHistoryEvents and + TestJobHistoryParsing. (sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Fix NM to use multiple disks for local + files and the userlogs. (vinodkv) + + MAPREDUCE-2480: MR App should not depend on hard coded version of shuffle (luke + lu via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Propagate error back to client in case of a + job submission failure (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix assembly to add mapreduce shell scripts + to the assembly package. (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix TestQueueMetrics. (Luke Lu via sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Removal of stale application-log dirs from + NM local disks. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Add license header and minor cleanup in + history server. (Siddharth Seth via sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Minor fix for install instructions. + (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix a race in MR task that was causing MR + containers to overwrite each other's job.xml. Also fix leaking attempt-dirs in + app-local-dir. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Adding valid state to ASM on a finish when + its already completed and also disble UberAM. (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fixed CS user limits. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Fixed reservation's bad interaction with + delay scheduling in CS. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Support mapreduce old (0.20) APIs. (sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Support fail-fast for MR jobs. (ddas) + + MAPREDUCE-279. Fix in MR-279 branch. Fix for clearing container requests on an + AM failure and add tostring methods to taskids and taskattemptids for better + grep support. (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Speed up communication between MR AM and RM + by relying on a new config rather than AM_EXPIRY_INTERVAL which is too large. + (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Fix calculation of maximum capacity to use + parent's absolute-capacity rather than the leaf queue's absolute-capacity. + (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Fixing a bug in JobIDPbImpl that's causing + AM to crash randomly. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Fix calculation of maximum capacity in + ParentQueue to use its parent's absolute-capacity rather than its own + absolute-capacity. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Launching bin/yarn and bin/mapred only + *once* in AM for constructing classpaths to avoid multiple forks and huge vmem + usage by AM. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Fix CapacityScheduler to release unused + reservations on application completion. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Fix CapacityScheduler (LeafQueue) to not + allocate DATA_LOCAL containers when they are not required on the rack. + (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Makes uber-task disabled by default (ddas) + + MAPREDUCE-279. Fix in MR-279 branch. Make logging and memory for AM configurable + for the user via command line (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fixing a bug in previous patch (r1103657). + Now bin/yarn truly shouldn't be launched multiple times in a single AM. + (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Fixing a bug to do with setting the staging + dir. (ddas) + + MAPREDUCE-279. Fix in MR-279 branch. Fixed Composite Service to shutdown + services if an error occurs on starting any one of those (mahadev & chris) + + MAPREDUCE-279. Fix in MR-279 branch. Fix the tests to use jvm fork mode to avoid + errors in shutting down services (sidharth seth) + + MAPREDUCE-2500. PB factories are not thread safe (Siddharth Seth via mahadev) + + MAPREDUCE-2504. race in JobHistoryEventHandler stop (Siddharth Seth via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix job hang if the AM launch fails. + (mahadev) + + MAPREDUCE-2509. Fix NPE in UI for pending attempts. (Luke Lu via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Add junit jar to lib in assembly (mahadev + and luke) + + MAPREDUCE-279. Fix in MR-279 branch. Distributed cache bug fix to pass Terasort. + (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Fix null pointer exception in kill task + attempt (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Refactored RMContainerAllocator to release + unused containers. (sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Changed Scheduler to return available limit + to AM in the allocate api. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Fix nodemanager expiry to not throw OOM. + (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Use DefaultContainerExecutor for + integration tests. (cdouglas) + + MAPREDUCE-279. Fix in MR-279 branch. Fix NPE in test case (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix for regression on the scheduling of + reduces before maps are done (ddas) + + MAPREDUCE-279. Fix in MR-279 branch. Fix distributed-cache related bugs. + (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Ensure unused containers released by AM are + correctly counted for queue-capacity. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Fix TestRuntimeEstimators (Siddharth Seth + via ddas) + + MAPREDUCE-279. Fix in MR-279 branch. Fix queue refresh to correctly record newly + added queues in CapacityScheduler. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Added metrics for tracking reservations in + CapacityScheduler. (Luke Lu via acmurthy) + + MAPREDUCE-2522. Security for JobHistory service. (Siddharth Seth via mahadev) + + MAPREDUCE-2534. Fix CI breaking hard coded version in jobclient pom. (Luke Lu + via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Add public cache. (cdouglas) + + MAPREDUCE-279. Fix in MR-279 branch. Made number of RPC server threads + configurable. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Added acl check for RMAdmin. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Adding job kill for any state that the job + is in with access control. (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Add debug statements for AM not launching + (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fixing race condition leader to hung jobs + in scheduler negotiator (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Add debug config for delaying delete of + local files. (cdouglas) + + MAPREDUCE-2527. Metrics for MRAppMaster (Luke lu via mahadev) + + MAPREDUCE-2532. Metrics for NodeManager (Luke Lu via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fixed an NPE during handling of unnecessary + reservations in CS. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Fix for public dist cache to work with non + default hdfs (mahadev &ddas) + + MAPREDUCE-279. Fix in MR-279 branch. Making streaming -file option work. Also + minor fixes for successful compilation of contrib tests. (vinodkv) + + MAPREDUCE-2536. Backporting changes to MR-279. + + MAPREDUCE-279. Fix in MR-279 branch. Bugfix for using user staging directory for + history files (Siddharth Seth via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. First fix for making basic speculative + execution work (ddas) + + MAPREDUCE-279. Fix in MR-279 branch. Fixes for TestFail/Kill (ddas) + + MAPREDUCE-279. Fix in MR-279 branch. Set correct version of avro-maven-plugin + that is available in apache maven repositories. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Fixing and reneabling + TestContainerTokenSecretManager. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Cleaning up configuration constants in + mapreduce modules. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Fixing NPE on speculator in MRAppMaster and + making job-history optional in tests to make test goal succeed. (vinodk and + sharadag). + + MAPREDUCE-279. Fix in MR-279 branch. Fixed NPE in CS by checking Application + state before scheduling and fixing synchronization in CS. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Making pipes work with YARN. Changed pipes + to get log-locations from an environmental variable. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Ensure 'lost' NodeManagers are dealt + appropriately, the containers are released correctly. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Adding some more logging for AM expiry logs + (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Reduce ramp up and zero maps support. + (sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Allowing hdfs calls from streaming/pipes + tasks. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Added ability to decommission nodes and + completed RM administration tools to achieve parity with JobTracker. (acmurthy) + + MAPREDUCE-2551. Added JobSummaryLog. (Siddharth Seth via acmurthy) + + MAPREDUCE-2552. Fixed NPE in CompletedJob in JobHistoryServer. (Siddharth Seth + via acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Fix reduce slow start. (sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Fixed TestFifoScheduler. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Fix build issue for using yarn.version + instead of hadoop-mapred.version (mahadev and giri) + + MAPREDUCE-279. Fix in MR-279 branch. Fixes in the handling of KILL events in the + SUCCEEDED state for tasks in the application master (ddas) + + MAPREDUCE-279. Fix in MR-279 branch. Fix for NPE in TestRMNMRPCResponseId. + (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix a deadlock in the resourcemanager. + (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. NodeStatus.getNodeHealthStatus().setBlah + broken (Siddharth Seth) + + MAPREDUCE-279. Fix in MR-279 branch. Fix another NPE in TestRMNMRPCResponseId. + (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix for NPE in TestNMExpiry (siddharth + seth) + + MAPREDUCE-279. Fix in MR-279 branch. Making each node aggregate all its + user-logs to a separate hdfs file. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Fix calculation of max-capacity for a + queue, also fixed a bug in registration of NodeManagers. (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. More cleaning up constants, removing stale + code, and making conspicuous the envs that apps depend on to be provided by + YARN. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Fix container size rounding in AM and + headroom in RM. (acmurthy and sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Disable Job acls until fixed (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix to report job status if the application + is KILLED/FAILED. (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix a corner case in headroom computation - + now reservations are taken into account and headroom is computed much later to + account for allocations/reservations. (acmurthy) + + MAPREDUCE-2537. The RM writes its log to + yarn-mapred-resourcemanager-<RM_Host>.out (Robert Evans via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix logging for showing the state of job + (FAILED/KILLED/SUCCEEDED) when it completes (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Re-enabled TestCapacityScheduler. + (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Support for min and max container capacity. + (acmurthy and sharad) + + MAPREDUCE-2531. Fixed jobcontrol to downgrade JobID. (Robert Evans via acmurthy) + + MAPREDUCE-2539. Fixed NPE in getMapTaskReports in JobClient. (Robert Evans via + acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Fixing the wrong config key used in + JobHistory that prevented configuring move-thread interval. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Fixed inconsistency in QueueACL enums. + (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Fix various issues with Web UI's. (Luke Lu) + + MAPREDUCE-279. Fix in MR-279 branch. Fix class cast exception in Task abort for + old mapreduce apis. (sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Add deletion of distributed cache + resources. (cdouglas) + + MAPREDUCE-279. Fix in MR-279 branch. Disable aggregation of logs onto DFS till + JobHistoryServer starts serving logs. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Cleanup redundant code in TaskAttemptImpl. + (sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Work around broken signaling in public + cache. (cdouglas) + + MAPREDUCE-2566. YarnConfiguration should reloadConfiguration if instantiated + with a non YarnConfiguration object. (Siddharth Seth) + + MAPREDUCE-279. Fix in MR-279 branch. Fully resolve paths when launching + containers. (Siddharth Seth) + + MAPREDUCE-279. Fix in MR-279 branch. Re-enabling Uber-AM feature. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Fixed deadlock during expiring NMs. + (acmurthy) + + MAPREDUCE-279. Fix in MR-279 branch. Solving NPEs during + ContainerManager#StopContainer. Also removing the unused + ContainerManager#CleanupContainer api. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Remove retries in dist cache so that NM's + do not shutdown (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix classpath construction for Task. + (vinodkv via sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Sending Job diagnostics from AM to RM and + redirect to history-url on job completion. (vinodkv and sharadag) + + MAPREDUCE-279. Fix in MR-279 branch. Added clover in pom dependency. (sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Intermittent TestMRApp failures on faster + Linux desktop. (Luke lu via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Including source files in release + distribution (Luke Lu via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Intermittent TestMRApp failures on faster + Linux desktop (part 2) (Luke lu via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Disable Uber AM. (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Added few job diagnostic messages. (sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Add ability to includes src files in + assembly target for maven (Luke Lu via mahadev) + + MAPREDUCE-2582. Cleanup JobHistory event generation.(Siddharth Seth via sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Fix rounding off problem in reduce ramp up. + (sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Fix more rounding off problems in reduce + ramp up. Also fix a bug preventing the application of the cap on reduce ramp-up. + (Sharad Agarwal via vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Fix to exclude images dir into the tar + distribution (Luke Lu via gkesavan) + + MAPREDUCE-279. Fix in MR-279 branch. Changes a couple of usages of FileContext + to FileSystem in YarnRunner to handle distributed cache path resolutions on + non-default filesystems. (ddas) + + MAPREDUCE-279. Fix in MR-279 branch. Display failed/killed attempts of the task + on MR AM UI separately. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Miscellaneous UI fixes + source code + formatting for MR JobHistoryEventHandler. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Fixing broken link to logs for container on + NM web UI. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Fixing the bug which was causing FAILED + jobs to be displayed as COMPLETED on the RM UI. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Job level node blacklisting. (sharad) + + MAPREDUCE-279. Fix in MR-279 branch. Fix NPE in history event handling + (Siddharth Seth via mahadev) + + MAPREDUCE-2569. Ensure root queue allocated 100% capacity. (Jonathan Eagles via + cdouglas) + + MAPREDUCE-279. Fix in MR-279 branch. Fix ClassCastException in JobHistoryServer + for certain jobs. (Siddharth Seth via llu) + + MAPREDUCE-279. Fix in MR-279 branch. Changes for invoking rack resolution in the + RM and in the AM (ddas) + + MAPREDUCE-279. Fix in MR-279 branch. Fix concurrent modification exception in + the Capacity Scheduler (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix stopContainer for setsid challenged + platforms. (llu) + + MAPREDUCE-2587. Generate yarn version for UI. (Thomas Graves via lluts page to + the history server UI. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Bug fix to set correct state on containers + so as to avoid duplicate containers from RM to AM. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Hack until MAPREDUCE-2365 is fixed to make + PIG work with MRV2. (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Changes a couple of usages of FileContext + to FileSystem in TaskAttemptImpl to handle distributed cache path resolutions on + non-default filesystems. (ddas) + + MAPREDUCE-279. Fix in MR-279 branch. Fix NPE when requesting attempts for + completed jobs. (Siddharth Seth via llu) + + MAPREDUCE-279. Fix in MR-279 branch. Improve logging for AM when requesting + containers to show the right ask and release fields (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix race condition between multiple + localizers on a single node. (cdouglas via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix RM app start/finish time and + diagnostics. (llu) + + MAPREDUCE-279. Fix in MR-279 branch. Fix to schedule reduces irrespective of the + headroom when all maps are done so as to avoid stall in reduce-scheduling when + slow-start is disabled. (Sharad Agarwal via vinodkv). + + MAPREDUCE-279. Fix in MR-279 branch. Disabling locality-wait in + CapacityScheduler for now to prevent uber-slow scheduling for apps with no + data-locality constraints (sleep-job like). (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Fixing scheduling deadlock in AM because of + incorrect headRoom values from RM. The bug happens when AM releases containers + and RM decrements current memory usage twice for all those containers. (vinodkv) + + MAPREDUCE-2611. Fix counters, finish times etc. in job history. (Siddharth Seth + via llu) + + MAPREDUCE-279. Fix in MR-279 branch. Fix for ConcurrentModification exception + while iterating through tokens in a UGI in ContainerLauncherImpl. (ddas) + + MAPREDUCE-279. Fix in MR-279 branch. Fix for NPE in YarnChild that was causing + lots of tasks to fail. (vinodkv) + + MAPREDUCE-2615. Make killJob go through AM and fix JobSummaryLog. (Siddharth + Seth via llu) + + MAPREDUCE-279. Fix in MR-279 branch. Fix class cast exception in release + reserved containers in capacity scheduler (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix diagnostics display for more than 100 + apps in RM. (llu) + + MAPREDUCE-279. Fix in MR-279 branch. Fix some invalid transitions in the RM. + (vinodkv via ddas) + + MAPREDUCE-2618. Fix NPE in 0 map 0 reduce jobs. (Jeffrey Naisbitt via llu) + + MAPREDUCE-2625. Add version info to nodemanager info page. (Jonathan Eagles via + llu) + + MAPREDUCE-279. Fix in MR-279 branch. (1) Faster retries from AM to HistoryServer + (2) Correct diagnostics for containers. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Major ASM cleanup. Streamlining classes, + interface and events. (vinodkv) + + MAPREDUCE-279. Fix in MR-279 branch. Reinstate resolve path fixes for viewfs. + (Siddharth Seth via llu) + + MAPREDUCE-2633. Add a getCounter(Enum) method to the Counters record. (Josh + Wills via sharad) + + MAPREDUCE-2645. Updates to MRv2 INSTALL and README documentation. (Josh Wills + via vinodkv) + + MAPREDUCE-2628. Add compiled on date to NM and RM info/about page. + + MAPREDUCE-2400. Remove Cluster's dependency on JobTracker via a ServiceProvider + for the actual implementation. (tomwhite via acmurthy) + + MAPREDUCE-2663. Refactoring StateMachineFactory inner classes. (ahmed radwan via + mahadev) + + MAPREDUCE-2678. minimum-user-limit-percent no longer honored. (naisbitt via + mahadev) + + MAPREDUCE-2630. refreshQueues leads to NPEs when used w/FifoScheduler. (Josh + Wills via mahadev) + + MAPREDUCE-2644. NodeManager fails to create containers when NM_LOG_DIR is not + explicitly set in the Configuration. (Josh Wills via vinodkv) + + MAPREDUCE-2661. Fix TaskImpl to not access MapTaskImpl. (Ahmed Radwan via + sharad) + + HADOOP-6929. Backport changes to MR-279 (mahadev and owen) + + HADOOP-6929. Making Security Info abstract and not an interface (mahadev) + + MAPREDUCE-2667. mapred job -kill leaves application in RUNNING state (thomas + graves via mahadev) + + MAPREDUCE-2664. Implement JobCounters for Mtions as asynchronous. (vinodkv, + sharad and acmurthy) + + MAPREDUCE-2773. server.api.records.NodeHealthStatus renamed but not updated in + client NodeHealthStatus.java (Thomas Graves via mahadev) + + MAPREDUCE-2772. Fix MR-279 build after common mavenization. (Robert Joseph Evans + via llu) + + MAPREDUCE-2772. Fix MR-279 build after common mavenization, part 2. (Thomas + Graves via llu) + + MAPREDUCE-279. Fix in MR-279 branch. Harmonize slf4j versions. (llu) + + MAPREDUCE-279. Fix in MR-279 branch. Fix NPE in FifoScheduler. (mahadev) + + MAPREDUCE-2776. Fix some of the yarn findbug warnings. (Siddharth Seth via + mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix findbugs warnings in mr-client modules, + part 1 (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix findbugs warnings in mr-client modules + part 2 (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix findbugs warnings in mr-client modules + part 3 (mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix the poms to enable 0.23 snapshots for + hdfs/common from apache nightly builds (gkesavan) + + MAPREDUCE-279. Fix in MR-279 branch. Fix ivy conf to work with the hadoop common + trunk maven build changes. (Giridharan Kesavan) + + MAPREDUCE-279. Fix in MR-279 branch. Patch for findbugs warnings in Resource + Manager (Siddharth Seth via mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fix for running ant targets to use the + right set of common/test jars (gkesavan via mahadev) + + MAPREDUCE-2782. Unit tests for CapacityScheduler. (acmurthy) + + MAPREDUCE-2706. Log job submission failures. (Jeffrey Naisbitt via acmurthy) + + MAPREDUCE-2781. mr279 RM application finishtime not set (Thomas Graves via + mahadev) + + MAPREDUCE-279. Fix in MR-279 branch. Fixed CS locality wait factor. (acmurthy) + + MAPREDUCE-2808. pull MAPREDUCE-2797 into mr279 branch (Thomas Graves via + mahadev) + + MAPREDUCE-2639. Bug fixes in speculate.DataStatistics. (Josh Wills via acmurthy) + + MAPREDUCE-2839. Fixed TokenCache to get delegation tokens using both new and old + apis. (Siddharth Seth via acmurthy) + + MAPREDUCE-2727. Fix divide-by-zero error in SleepJob for sleepCount equals 0. + (Jeffrey Naisbitt via acmurthy) + + MAPREDUCE-2839. Fixed TokenCache to get delegation tokens using both new + and old apis. (Siddharth Seth via acmurthy) + + MAPREDUCE-2727. Fix divide-by-zero error in SleepJob for sleepCount equals + 0. (Jeffrey Naisbitt via acmurthy) + + MAPREDUCE-2860. Fix log4j logging in the maven test cases. (mahadev) + + MAPREDUCE-2867. Remove Unused TestApplicaitonCleanup in resourcemanager/applicationsmanager. + (mahadev) + + MAPREDUCE-2868. ant build broken in hadoop-mapreduce dir (mahadev, giri and arun via mahadev) + + MAPREDUCE-2649. Handling of finished applications in RM. (Thomas Graves + via acmurthy) + + MAPREDUCE-2838. Fix MapReduce builds to use new hadoop-common test jars. + (gkesavan via acmurthy) + + MAPREDUCE-2859. Fix eclipse plugin contrib module compilation (gkesavan) + + MAPREDUCE-2846. Fix missing synchronization in the task log management. + (omalley) + + MAPREDUCE-2807. Fix AM restart and client redirection. (sharad) + + MAPREDUCE-2877. Add missing Apache license header in some files in MR + and also add the rat plugin to the poms. (mahadev) + + MAPREDUCE-2796. Set start times for MR applications for clients to see. + (Devaraj K via acmurthy) + + MAPREDUCE-2879. Fix version for MR-279 to 0.23.0. (acmurthy) + + MAPREDUCE-2881. Fix to include log4j 1.2.16 depenency (gkesavan) + + MAPREDUCE-2885. Fix mapred-config.sh to look for hadoop-config.sh in + HADOOP_COMMON_HOME/libexec. (acmurthy) + + MAPREDUCE-2893. Remove duplicate entry of YarnClientProtocolProvider in + ClientProtocolProvider services file. (Liang-Chi Hsieh via acmurthy) + + MAPREDUCE-2891. Javadoc for AMRMProtocol and related records. (acmurthy) + + MAPREDUCE-2898. Javadoc for ContainerManager protocol and related records. + (acmurthy) + + MAPREDUCE-2904. Fixed bin/yarn to correctly include HDFS jars and + clean up of stale refs to pre-mavenized Hadoop Common and HDFS. + (Sharad Agarwal and Arun C. Murthy via acmurthy) + + MAPREDUCE-2737. Update the progress of jobs on client side. (Siddharth Seth + and Mahadev Konar via mahadev) + + MAPREDUCE-2886. Fix Javadoc warnings in MapReduce. (mahadev) + + MAPREDUCE-2897. Javadoc for ClientRMProtocol protocol and related records. + (acmurthy) + + MAPREDUCE-2916. Ivy build for MRv1 fails with bad organization for + common daemon. (mahadev) + + MAPREDUCE-2917. Fixed corner case in container reservation which led to + starvation and hung jobs. (acmurthy) + + MAPREDUCE-2756. Better error handling in JobControl for failed jobs. + (Robert Evans via acmurthy) + + MAPREDUCE-2716. MRReliabilityTest job fails because of missing + job-file. (Jeffrey Naisbitt via vinodkv) + + MAPREDUCE-2882. TestLineRecordReader depends on ant jars. (todd) + + MAPREDUCE-2687. Fix NodeManager to use the right version of + LocalDirAllocator.getLocalPathToWrite. (mahadev & acmurthy) + + MAPREDUCE-2800. Set final progress for tasks to ensure all task information + is correctly logged to JobHistory. (Siddharth Seth via acmurthy) + + MAPREDUCE-2938. Log application submission failure in CapacityScheduler. + (acmurthy) + + MAPREDUCE-2948. Hadoop streaming test failure, post MR-2767 (mahadev) + + MAPREDUCE-2908. Fix all findbugs warnings. (vinodkv via acmurthy) + + MAPREDUCE-2947. Fixed race condition in AuxiliaryServices. (vinodkv via + acmurthy) + + MAPREDUCE-2844. Fixed display of nodes in UI. (Ravi Teja Ch N V via + acmurthy) + + MAPREDUCE-2677. Fixed 404 for some links from HistoryServer. (Robert Evans + via acmurthy) + + MAPREDUCE-2937. Ensure reason for application failure is displayed to the + user. (mahadev via acmurthy) + + MAPREDUCE-2953. Fix a race condition on submission which caused client to + incorrectly assume application was gone by making submission synchronous + for RMAppManager. (Thomas Graves via acmurthy) + + MAPREDUCE-2963. Fix hang in TestMRJobs. (Siddharth Seth via acmurthy) + + MAPREDUCE-2954. Fixed a deadlock in NM caused due to wrong synchronization + in protocol buffer records. (Siddharth Seth via vinodkv) + + MAPREDUCE-2975. Fixed YARNRunner to use YarnConfiguration rather than + Configuration. (mahadev via acmurthy) + + MAPREDUCE-2971. ant build mapreduce fails protected access jc.displayJobList + (jobs) (Thomas Graves via mahadev) + + MAPREDUCE-2691. Finishing up the cleanup of distributed cache file resources + and related tests. (Siddharth Seth via vinodkv) + + MAPREDUCE-2749. Ensure NM registers with RM after starting all its services + correctly. (Thomas Graves via acmurthy) + + MAPREDUCE-2979. Removed the needless ClientProtocolProvider configuration + from the hadoop-mapreduce-client-core module. (Siddharth Seth via vinodkv) + + MAPREDUCE-2985. Fixed findbugs warnings in ResourceLocalizationService. + (Thomas Graves via acmurthy) + + MAPREDUCE-2874. Fix formatting of ApplicationId in web-ui. (Eric Payne via + acmurthy) + + MAPREDUCE-2995. Better handling of expired containers in MapReduce + ApplicationMaster. (vinodkv via acmurthy) + + MAPREDUCE-2995. Fixed race condition in ContainerLauncher. (vinodkv via + acmurthy) + + MAPREDUCE-2949. Fixed NodeManager to shut-down correctly if a service + startup fails. (Ravi Teja via vinodkv) + + MAPREDUCE-3005. Fix both FifoScheduler and CapacityScheduler to correctly + enforce locality constraints. (acmurthy) + + MAPREDUCE-3007. Fixed Yarn Mapreduce client to be able to connect to + JobHistoryServer in secure mode. (vinodkv) + + MAPREDUCE-2987. Fixed display of logged user on RM Web-UI. (Thomas Graves + via acmurthy) + + MAPREDUCE-3006. Fixed MapReduce AM to exit only after properly writing out + history file. (vinodkv) + + MAPREDUCE-2925. Fixed Yarn+MR client code to behave saner with completed + jobs. (Devaraj K via vinodkv) + + MAPREDUCE-3030. Fixed a bug in NodeId.equals() that was causing RM to + reject all NMs. (Devaraj K via vinodkv) + + MAPREDUCE-3042. Fixed default ResourceTracker address. (Chris Riccomini + via acmurthy) + + MAPREDUCE-3038. job history server not starting because conf() missing + HsController (Jeffrey Naisbitt via mahadev) + + MAPREDUCE-3004. Fix ReduceTask to not assume 'local' mode in YARN. (Hitesh + Shah via acmurthy) + + MAPREDUCE-3017. The Web UI shows FINISHED for killed/successful/failed jobs. + (mahadev) + + MAPREDUCE-3040. Fixed extra copy of Configuration in + YarnClientProtocolProvider and ensured MiniMRYarnCluster sets JobHistory + configuration for tests. (acmurthy) + + MAPREDUCE-3018. Fixed -file option for streaming. (mahadev via acmurthy) + + MAPREDUCE-3036. Fixed metrics for reserved resources in CS. (Robert Evans + via acmurthy) + + MAPREDUCE-2998. Fixed a bug in TaskAttemptImpl which caused it to fork + bin/mapred too many times. (vinodkv via acmurthy) + + MAPREDUCE-3023. Fixed clients to display queue state correctly. (Ravi + Prakash via acmurthy) + + MAPREDUCE-2970. Fixed NPEs in corner cases with different configurations + for mapreduce.framework.name. (Venu Gopala Rao via vinodkv) + + MAPREDUCE-3062. Fixed default RMAdmin address. (Chris Riccomini + via acmurthy) + + MAPREDUCE-3066. Fixed default ResourceTracker address for the NodeManager. + (Chris Riccomini via acmurthy) + + MAPREDUCE-3044. Pipes jobs stuck without making progress. (mahadev) + + MAPREDUCE-2754. Fixed MR AM stdout, stderr and syslog to redirect to + correct log-files. (Ravi Teja Ch N V via vinodkv) + + MAPREDUCE-3073. Fixed build issues in MR1. (mahadev via acmurthy) + + MAPREDUCE-2691. Increase threadpool size for launching containers in + MapReduce ApplicationMaster. (vinodkv via acmurthy) + + + MAPREDUCE-2990. Fixed display of NodeHealthStatus. (Subroto Sanyal via + acmurthy) + + MAPREDUCE-3053. Better diagnostic message for unknown methods in ProtoBuf + RPCs. (vinodkv via acmurthy) + + MAPREDUCE-2952. Fixed ResourceManager/MR-client to consume diagnostics + for AM failures in a couple of corner cases. (Arun C Murthy via vinodkv) + + MAPREDUCE-3064. 27 unit test failures with Invalid + "mapreduce.jobtracker.address" configuration value for + JobTracker: "local" (Venu Gopala Rao via mahadev) + + MAPREDUCE-3090. Fix MR AM to use ApplicationAttemptId rather than + (ApplicationId, startCount) consistently. (acmurthy) + + MAPREDUCE-2646. Fixed AMRMProtocol to return containers based on + priority. (Sharad Agarwal and Arun C Murthy via vinodkv) + + MAPREDUCE-3031. Proper handling of killed containers to prevent stuck + containers/AMs on an external kill signal. (Siddharth Seth via vinodkv) + + MAPREDUCE-2984. Better error message for displaying completed containers. + (Devaraj K via acmurthy) + + MAPREDUCE-3071. app master configuration web UI link under the Job menu + opens up application menu. (thomas graves via mahadev) + + MAPREDUCE-3067. Ensure exit-code is set correctly for containers. (Hitesh + Shah via acmurthy) + + MAPREDUCE-2999. Fix YARN webapp framework to properly filter servlet + paths. (Thomas Graves via vinodkv) + + MAPREDUCE-3095. fairscheduler ivy including wrong version for hdfs. + (John George via mahadev) + + MAPREDUCE-3054. Unable to kill submitted jobs. (mahadev) + + MAPREDUCE-3021. Change base urls for RM web-ui. (Thomas Graves via + acmurthy) + + MAPREDUCE-3041. Fixed ClientRMProtocol to provide min/max resource + capabilities along-with new ApplicationId for application submission. + (Hitesh Shah via acmurthy) + + MAPREDUCE-2843. Fixed the node-table to be completely displayed and making + node entries on RM UI to be sortable. (Abhijit Suresh Shingate via vinodkv) + + MAPREDUCE-3110. Fixed TestRPC failure. (vinodkv) + + MAPREDUCE-3078. Ensure MapReduce AM reports progress correctly for + displaying on the RM Web-UI. (vinodkv via acmurthy) + + MAPREDUCE-3114. Fixed invalid ApplicationURL on RM WebUI. (Subroto Sanyal + via vinodkv) + + MAPREDUCE-2791. Added missing info on 'job -status' output. (Devaraj K via + acmurthy) + + MAPREDUCE-2996. Add uber-ness information to JobHistory. (Jonathan Eagles + via acmurthy) + + MAPREDUCE-3050. Add ability to get resource usage information for + applications and nodes. (Robert Evans via acmurthy) + + MAPREDUCE-3113. Ensure bin/yarn and bin/yarn-daemon.sh identify the root + of the install properly. (Xie Xianshan via acmurthy) + + MAPREDUCE-3137. Fix broken merge of MAPREDUCE-2179. (Hitesh Shah via + acmurthy) + + MAPREDUCE-2792. Replace usage of node ip-addresses with hostnames. + (vinodkv via acmurthy) + + MAPREDUCE-3112. Fixed recursive sourcing of HADOOP_OPTS environment + variable. (Eric Yang) + + MAPREDUCE-3056. Changed the default staging directory to not include + user.name to prevent issues with non-secure mode. (Devaraj K via vinodkv) + + MAPREDUCE-2913. Fixed TestMRJobs.testFailingMapper to assert the correct + TaskCompletionEventStatus. (Jonathan Eagles via vinodkv) + + MAPREDUCE-2794. [MR-279] Incorrect metrics value for AvailableGB per + queue per user. (John George via mahadev) + + MAPREDUCE-2783. Fixing RM web-UI to show no tracking-URL when AM + crashes. (Eric Payne via vinodkv) + + MAPREDUCE-3141. Fix the broken MRAppMaster to work over YARN in security + mode.(vinodkv) + + MAPREDUCE-2751. Modified NodeManager to stop leaving around local files + after application finishes. (Siddharth Seth via vinodkv) + + MAPREDUCE-3033. Ensure Master interface pays attention to classic v/s yarn + frameworks. (Hitesh Shah via acmurthy) + + MAPREDUCE-2802. Ensure JobHistory filenames have jobId. (Jonathan Eagles + via acmurthy) + + MAPREDUCE-2876. Use a different config for ContainerAllocationExpirer. + (Anupam Seth via acmurthy) + + MAPREDUCE-3153. Fix TestFileOutputCommitter which was broken by + MAPREDUCE-2702. (mahadev via acmurthy) + + MAPREDUCE-3123. Fix NM to quote symlink names to escape special + characters. (Hitesh Shah via acmurthy) + + MAPREDUCE-3154. Fix JobSubmitter to check for output specs before copying + job submission files to fail fast. (Abhijit Suresh Shingate via acmurthy) + + MAPREDUCE-3158. Fix test failures in MRv1 due to default framework being + set to yarn. (Hitesh Shah via acmurthy) + + MAPREDUCE-3167. container-executor is not being packaged with the assembly + target. (mahadev) + + MAPREDUCE-3020. Fixed TaskAttemptImpl to log the correct node-address for + a finished Reduce task. (Chackaravarthy via vinodkv) + + MAPREDUCE-2668. Fixed AuxServices to send a signal on application-finish + to all the services. (Thomas Graves via vinodkv) + + MAPREDUCE-3126. Fixed a corner case in CapacityScheduler where headroom + wasn't updated on changes to cluster size. (acmurthy) + + MAPREDUCE-3140. Fixed the invalid JobHistory URL for failed + applications. (Subroto Sanyal via vinodkv) + + MAPREDUCE-3125. Modified TaskImpl to consider only non-failed, non-killed + task-attempts for obtaining task's progress. (Hitesh Shah via vinodkv) + + MAPREDUCE-2666. Retrieve shuffle port number from JobHistory on MR AM + restart. (Jonathan Eagles via acmurthy) + + MAPREDUCE-2789. Complete schedulingInfo on CLI. (Eric Payne via acmurthy) + + MAPREDUCE-3170. Fixed job output commit for deep hierarchies. (Hitesh Shah + via acmurthy) + + MAPREDUCE-3124. Fixed location of native libs i.e. libhadoop.so for + containers. (John George via acmurthy) + + MAPREDUCE-3057. Job History Server goes of OutOfMemory with 1200 Jobs + and Heap Size set to 10 GB. (Eric Payne via mahadev) + + MAPREDUCE-2840. mr279 TestUberAM.testSleepJob test fails. (jonathan eagles + via mahadev) + + MAPREDUCE-3190. Ensure bin/yarn fails early with a clear error message + when HADOOP_COMMON_HOME or HADOOP_HDFS_HOME are not set. (todd & acmurthy + via acmurthy) + + MAPREDUCE-3189. Add link decoration back to MR2's CSS. (Todd Lipcon via + mahadev) + + MAPREDUCE-3127. Changed default value of yarn.resourcemanager.acl.enable + to true and added some more documentation. (acmurthy) + + MAPREDUCE-3032. Fixed TaskAttemptImpl so that JobHistory can have error + information about failed tasks. (Devaraj K via vinodkv) + + MAPREDUCE-3196. TestLinuxContainerExecutorWithMocks fails on Mac OSX. + (Arun Murthy via mahadev) + + MAPREDUCE-3197. TestMRClientService failing on building clean checkout of + branch 0.23 (mahadev) + + MAPREDUCE-2762. Cleanup MR staging directory on completion. (mahadev via + acmurthy) + + MAPREDUCE-3165. Ensure logging options are set correctly for MR AM and + tasks. (todd via acmurthy) + + MAPREDUCE-3203. Fix some javac warnings in MRAppMaster. (mahadev) + + MAPREDUCE-3162. Separated application-init and container-init event types + in NodeManager's Application state machine. (Todd Lipcon via vinodkv) + + MAPREDUCE-3176. Fixed ant mapreduce tests that are timing out because + of wrong framework name. (Hitesh Shah via vinodkv) + + MAPREDUCE-3181. Fixed MapReduce runtime to load yarn-default.xml and + yarn-site.xml. (acmurthy) + + MAPREDUCE-2788. Normalize resource requests in FifoScheduler + appropriately. (Ahmed Radwan via acmurthy) + + MAPREDUCE-2693. Fix NPE in job-blacklisting. (Hitesh Shah via acmurthy) + + MAPREDUCE-3208. Fix NPE task/container log appenders. (liangzhwa via + acmurthy) + + MAPREDUCE-3212. Fix usage/help message for bin/yarn. (Bhallamudi Venkata + Siva Kamesh via acmurthy) + + MAPREDUCE-3179. Ensure failed tests exit with right error code. (Jonathan + Eagles via acmurthy) + + MAPREDUCE-3188. Ensure correct shutdown in services. (todd via acmurthy) + + MAPREDUCE-3226. Fix shutdown of fetcher threads. (vinodkv via acmurthy) + + MAPREDUCE-3070. Fix NodeManager to use ephemeral ports by default. + (Devaraj K via acmurthy) + + MAPREDUCE-3242. Trunk compilation broken with bad interaction from + MAPREDUCE-3070 and MAPREDUCE-3239. (mahadev) + + MAPREDUCE-3058. Fixed MR YarnChild to report failure when task throws an + error and thus prevent a hanging task and job. (vinodkv) + + MAPREDUCE-3087. Fixed the mapreduce classpath to correctly include the + generated-classpath file needed for tests. (Ravi Prakash via vinodkv) + + MAPREDUCE-3233. Fixed a bug in MR Job so as to be able to restart the + application on AM crash. (Mahadev Konar via vinodkv) + + MAPREDUCE-3028. Added job-end notification support. (Ravi Prakash via + acmurthy) + + MAPREDUCE-3249. Ensure shuffle-port is correctly used duringMR AM recovery. + (vinodkv via acmurthy) + + MAPREDUCE-3252. Fix map tasks to not rewrite data an extra time when + map output fits in spill buffer. (todd) + + MAPREDUCE-3159. Ensure DefaultContainerExecutor doesn't delete application + directories during app-init. (todd via acmurthy) + + MAPREDUCE-2746. Yarn servers can't communicate with each other with + hadoop.security.authorization set to true (acmurthy via mahadev) + + MAPREDUCE-2821. Added missing fields (resourcePerMap & resourcePerReduce) + to JobSummary logs. (mahadev via acmurthy) + + MAPREDUCE-3253. Fixed ContextFactory to clone JobContext correctly. + (acmurthy) + + MAPREDUCE-3263. Fixed the MAPREDUCE-3028 commit which broke MR1. (Hitesh + Shah via acmurthy) + + MAPREDUCE-3269. Fixed log4j properties to correctly set logging options + for JobHistoryServer vis-a-vis JobSummary logs. (mahadev via acmurthy) + + MAPREDUCE-3250. When AM restarts, client keeps reconnecting to the new AM + and prints a lots of logs. (vinodkv via mahadev) + + MAPREDUCE-3254. Fixed streaming to set the job.jar by using the right + JobConf ctor. (acmurthy) + + MAPREDUCE-3264. mapreduce.job.user.name needs to be set automatically. + (acmurthy via mahadev) + + MAPREDUCE-3175. Add authorization to admin web-pages such as /stacks, /jmx + etc. (Jonathan Eagles via acmurthy) + + MAPREDUCE-3257. Added authorization checks for the protocol between + ResourceManager and ApplicationMaster. (vinodkv via acmurthy) + + MAPREDUCE-3259. Added java.library.path of NodeManager to + ContainerLocalizer in LinuxContainerExecutor. (Kihwal Lee via acmurthy) + + MAPREDUCE-3279. Fixed TestJobHistoryParsing which assumed user name to be + mapred all the time. (Siddharth Seth via acmurthy) + + MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its + containers (process-trees) irrespective of whether the container succeeded, + or killed. Contributed by Hitesh Shah. + + MAPREDUCE-3281. Fixed a bug in TestLinuxContainerExecutorWithMocks. (vinodkv) + + MAPREDUCE-3228. Fixed MR AM to timeout RPCs to bad NodeManagers. (vinodkv + via acmurthy) + + MAPREDUCE-3284. Moved JobQueueClient to hadoop-mapreduce-client-core. + (acmurthy) + + MAPREDUCE-3282. bin/mapred job -list throws exception. (acmurthy via + mahadev) + + MAPREDUCE-3186. User jobs are getting hanged if the Resource manager + process goes down and comes up while job is getting executed. + (Eric Payne via mahadev) + + MAPREDUCE-3285. Tests on branch-0.23 failing (Siddharth Seth via mahadev) + + MAPREDUCE-3258. Fixed AM & JobHistory web-ui to display counters properly. + (Siddharth Seth via acmurthy) + + MAPREDUCE-3290. Fixed a NPE in ClientRMService. (acmurthy) + + MAPREDUCE-3185. RM Web UI does not sort the columns in some cases. + (Jonathan Eagles via mahadev) + + MAPREDUCE-3292. In secure mode job submission fails with Provider + org.apache.hadoop.mapreduce.security.token.JobTokenIndentifier$Renewer + not found. (mahadev) + + MAPREDUCE-3296. Fixed the remaining nine FindBugs warnings. (vinodkv) + + MAPREDUCE-2775. Fixed ResourceManager and NodeManager to force a + decommissioned node to shutdown. (Devaraj K via vinodkv) + + MAPREDUCE-3304. Fixed intermittent test failure due to a race in + TestRMContainerAllocator#testBlackListedNodes. (Ravi Prakash via acmurthy) + + MAPREDUCE-3306. Fixed a bug in NodeManager ApplicationImpl that was causing + NodeManager to crash. (vinodkv) + + MAPREDUCE-3295. TestAMAuthorization failing on branch 0.23. (vinodkv via mahadev) + + MAPREDUCE-3183. hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml + missing license header. (Hitesh Shah via tucu). + + MAPREDUCE-3003. Publish MR JARs to Maven snapshot repository. (tucu) + + MAPREDUCE-3199. Fixed pom files to include correct log4j configuration for + tests. (vinodkv) + + MAPREDUCE-3204. mvn site:site fails on MapReduce. (tucu) + + MAPREDUCE-3248. Fixed log4j properties. (vinodkv via acmurthy) + + MAPREDUCE-3256. Added authorization checks for the protocol between + NodeManager and ApplicationMaster. (vinodkv via acmurthy) + + MAPREDUCE-3274. Fixed a race condition in MRAppMaster that was causing a + task-scheduling deadlock. (Robert Joseph Evans via vinodkv) + + MAPREDUCE-3171 merge from trunk reverted changes from MAPREDUCE-2747 MAPREDUCE-3240. + + MAPREDUCE-3313. Fixed initialization of ClusterMetrics which was failing + TestResourceTrackerService sometimes. (Hitesh Shah via vinodkv) + + MAPREDUCE-2766. Fixed NM to set secure permissions for files and directories + in distributed-cache. (Hitesh Shah via vinodkv) + + MAPREDUCE-2696. Fixed NodeManager to cleanup logs in a thread when logs' + aggregation is not enabled. (Siddharth Seth via vinodkv) + + MAPREDUCE-3262. Fixed Container's state-machine in NodeManager to handle + a couple of events in failure states correctly. (Hitesh Shah and Siddharth + Seth via vinodkv) + + MAPREDUCE-3035. Fixed MR JobHistory to ensure rack information is present. + (chakravarthy via acmurthy) + + MAPREDUCE-3321. Disabled a few MR tests for 0.23. (Hitesh Shah via + acmurthy) + + MAPREDUCE-3220. Fixed TestCombineOutputCollector. (Devaraj K via acmurthy) + + MAPREDUCE-3103. Implement Job ACLs for MRAppMaster. + (mahadev) + + MAPREDUCE-3241. [Rumen] Fix Rumen to ignore the AMStartedEvent. (amarrk) + + MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying + on current history file name format. (Ravi Gummadi via amarrk) + + MAPREDUCE-3157. [Rumen] Fix TraceBuilder to handle 0.20 history file + names also. (Ravi Gummadi via amarrk) + + MAPREDUCE-3081. Fix vaidya startup script. (gkesavan via suhas). + + MAPREDUCE-2764. Fix renewal of dfs delegation tokens. (Owen via jitendra) + + MAPREDUCE-3192. Fix Javadoc warning in JobClient.java and Cluster.java. + (jitendra) + + MAPREDUCE-3237. Move LocalJobRunner to hadoop-mapreduce-client-core. + (tomwhite via acmurthy) + + MAPREDUCE-3316. Rebooted link is not working properly. + (Bhallamudi Venkata Siva Kamesh via mahadev) + + MAPREDUCE-3317. Rumen TraceBuilder is emiting null as hostname. + (Ravi Gummadi via mahadev) + + MAPREDUCE-3332. contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/ + Sender#writeBlock related to checksum handling (Hitesh Shah via mahadev) + + MAPREDUCE-3337. Added missing license headers. (acmurthy) + +Release 0.22.1 - Unreleased + + INCOMPATIBLE CHANGES + + NEW FEATURES + + IMPROVEMENTS + + OPTIMIZATIONS + + BUG FIXES + + MAPREDUCE-3837. Job tracker is not able to recover jobs after crash. + (Mayank Bansal via shv) + +Release 0.22.0 - 2011-11-29 + + INCOMPATIBLE CHANGES + + MAPREDUCE-1866. Removes deprecated class + org.apache.hadoop.streaming.UTF8ByteArrayUtils. (amareshwari) + + MAPREDUCE-1664. Changes the behaviour of the combination of job-acls + when they function together with queue-acls. (Ravi Gummadi via vinodkv) + + MAPREDUCE-2994. Fixed a bug in ApplicationID parsing that affects RM + UI. (Devaraj K via vinodkv) + + MAPREDUCE-1788. o.a.h.mapreduce.Job shouldn't make a copy of the JobConf. + (Arun Murthy via mahadev) + + NEW FEATURES + + MAPREDUCE-1804. Stress-test tool for HDFS introduced in HDFS-708. + (Joshua Harlow via shv) + + MAPREDUCE-220. Collect cpu and memory statistics per task. (Scott Chen via + acmurthy) + + MAPREDUCE-1970. Reed-Solomon code implementation for HDFS RAID. + (Scott Chen via dhruba) + + MAPREDUCE-2169. Integrated Reed-Solomon code with RaidNode. (Ramkumar + Vadali via schen) + + MAPREDUCE-2936. Contrib Raid compilation broken after HDFS-1620. (vinodkv) + + IMPROVEMENTS + + MAPREDUCE-2141. Add an "extra data" field to Task for use by Mesos. (matei) + + MAPREDUCE-2140. Regenerate fair scheduler design doc PDF. (matei) + + MAPREDUCE-1546. Redirect all job pages to corresponding history page + if job is not in JT memory. (Scott Chen via sharad) + + MAPREDUCE-1092. Enable assertions for unit tests. (Eli Collins via + cdouglas) + + MAPREDUCE-1680. Add a metric recording JobTracker heartbeats processed. + (Dick King via cdouglas) + + MAPREDUCE-1761. FairScheduler allows separate configuration of node + and rack locality wait time (Scott Chen via dhruba) + + MAPREDUCE-1539. authorization checks for inter-server protocol + (based on HADOOP-6600) (Boris Shkolnik via shv) + + MAPREDUCE-1798. Names the configuration keys for the Kerberos + principals better. (Boris Shkolnik via ddas) + + MAPREDUCE-1773. streaming doesn't support jobclient.output.filter. + (Amareshwari Sriramadasu via vinodkv) + + MAPREDUCE-1785. Add streaming config option for not emitting the key. + (Eli Collins via sharad) + + MAPREDUCE-572. If #link is missing from uri format of -cacheArchive + then streaming does not throw error. (Amareshwari Sriramadasu via + vinodkv) + + MAPREDUCE-1545. Add timestamps for first task type launched in job summary. + (Luke Lu via cdouglas) + + MAPREDUCE-1543. Add an audit log for authentication events. (Amar Kamat and + Luke Lu via cdouglas) + + MAPREDUCE-1762. Add ability to set values of task counters. (Scott Chen via + cdouglas) + + MAPREDUCE-1533. Reduce overhead of logging and string manipulation during + heartbeat processing. (Amar Kamat and Dick King via cdouglas) + + MAPREDUCE-1516. JobTracker issues delegation tokens only if the user's + authentication is Kerberos. (Jitendra Pandey via ddas) + + MAPREDUCE-647. Update distcp forrest documentation to reflect the changes + of HADOOP-5472, MAPREDUCE-642 and HADOOP-5620. (Rodrigo Schmidt via + szetszwo) + + MAPREDUCE-1851. Documents configuration parameters in streaming. + (amareshwari) + + MAPREDUCE-1868. Add a read and connection timeout to JobClient while + pulling tasklogs. (Krishna Ramachandran via acmurthy) + + MAPREDUCE-1778. Ensure failure to setup CompletedJobStatusStore is not + silently ignored by the JobTracker. (Krishna Ramachandran via acmurthy) + + MAPREDUCE-1850. Includes job submit host information (name and ip) in + jobconf and jobdetails display (Krishna Ramachandran via amareshwari) + + MAPREDUCE-1893. Slive with multiple reducers. (shv) + + MAPREDUCE-1248. Fixes redudant memory copying in StreamKeyValUtil. + (Ruibang He via amareshwari) + + MAPREDUCE-1840. Enhancements to Gridmix benchmark simulating user + diversity, queue replay, and task duration for JobTracker load testing. + Also includes compatibility with security enhancements, and scalability + improvements. (Amar Kamat, Rahul Singh, Hong Tang, and cdouglas) + + MAPREDUCE-1848. Put number of speculative, data local, rack local + tasks in JobTracker metrics. (Scott Chen via dhruba) + + MAPREDUCE-1935. Makes the Distcp to work in a secure environment. + (Boris Shkolnik via ddas) + + MAPREDUCE-1945. The MapReduce component for HADOOP-6632. + (Kan Zhang & Jitendra Pandey via ddas) + + MAPREDUCE-1936. Modify Gridmix3 to support more tunable parameters for + stress submission and sleep jobs. (Hong Tang via cdouglas) + + MAPREDUCE-1733. Makes pipes applications secure. (Jitendra Pandey via ddas) + + MAPREDUCE-1566. Adds a configuration attribute using which job clients can + specify a credentials file. The tokens from there will be passed to the job. + (Jitendra Pandey and Owen O'Malley via ddas) + + MAPREDUCE-1624. Documents the job credentials and associated details to do + with delegation tokens (on the client side). + (Jitendra Pandey and Devaraj Das via ddas) + + MAPREDUCE-1834. TestSimulatorDeterministicReplay timesout on trunk. + (Hong Tang via mahadev) + + MAPREDUCE-1993. Fixes test failure + TestTrackerDistributedCacheManagerWithLinuxTaskController. (Devaraj Das + via amareshwari) + + MAPREDUCE-1523. Making Mumak work with Capacity-Scheduler (Anirban Das + via mahadev) + + MAPREDUCE-1920. Enables completed jobstatus store by default. (Tom White + via amareshwari) + + MAPREDUCE-1881. Improve TaskTrackerInstrumentation to enable collection of + advanced metrics. (Matei Zaharia via acmurthy) + + MAPREDUCE-1548. Hadoop archives preserve times and other properties from + original files. (Rodrigo Schmidt via dhruba) + + MAPREDUCE-1517. Supports streaming job to run in the background. (Bochun Bai + via amareshwari) + + MAPREDUCE-1819. RaidNode is now smarter in submitting Raid jobs. (Ramkumar + Vadali via schen) + + MAPREDUCE-2132. A command line option in RaidShell to fix blocks using raid + + MAPREDUCE-2147. Remove redundant lines in JobInProgress ctor. + (Harsh J Chouraria via cutting) + + HADOOP-7007. Update the hudson-test-patch ant target to work with the + latest test-patch.sh script (gkesavan) + + MAPREDUCE-1818. RaidNode can specify scheduling parameters. (Ramkumar + Vadali via schen) + + MAPREDUCE-2051. Contribute a fair scheduler preemption system test. + (Todd Lipcon via tomwhite) + + MAPREDUCE-1892. RaidNode can allow layered policies more efficiently. + (Ramkumar Vadali via schen) + + MAPREDUCE-1592. Generate Eclipse's .classpath file from Ivy config. + (tomwhite via nigel) + + MAPREDUCE-2073. TestTrackerDistributedCacheManager should be up-front + about requirements on build environment. (Todd Lipcon via tomwhite) + + MAPREDUCE-2093. Herriot JT and TT clients should vend statistics. (cos) + + MAPREDUCE-2167. Faster directory traversal for raid node. (Ramkumar Vadali + via schen) + + MAPREDUCE-1931. Gridmix forrest documentation . (Ranjit Mathew via vinodkv). + + MAPREDUCE-2184. Port DistRaid.java to new mapreduce API. (Ramkumar Vadali + via schen) + + MAPREDUCE-1878. Add MRUnit documentation. (Aaron Kimball via tomwhite) + + MAPREDUCE-2180. Add coverage of fair scheduler servlet to system test (todd) + + MAPREDUCE-2250. Fix logging in raid code. (Ramkumar Vadali via schen) + + MAPREDUCE-2260. Remove auto-generated native build files. (rvs via eli) + + MAPREDUCE-2314. configure files that are generated as part of the released + tarball need to have executable bit set (rvs via cos) + + MAPREDUCE-1159. Limit Job name in web UI to be 80 char long. (Harsh J + Chouraria via szetszwo) + + MAPREDUCE-2337. Remove dependence of public MapReduce API on classes in + server package. (tomwhite) + + MAPREDUCE-2383. Improve documentation of DistributedCache methods (Harsh J + Chouraria via todd) + + MAPREDUCE-2222. Ivy resolve force mode should be turned off by default. + (Luke Lu via tomwhite) + + MAPREDUCE-2103. task-controller shouldn't require o-r permissions. + (todd via eli) + + MAPREDUCE-2505. Explain how to use ACLs in the fair scheduler. + (matei via eli) + + MAPREDUCE-3138. Add a utility to help applications bridge changes in + Context Objects APIs due to MAPREDUCE-954. (omalley via acmurthy) + + OPTIMIZATIONS + + MAPREDUCE-1354. Enhancements to JobTracker for better performance and + scalability. (Arun C. Murthy & Richard King via acmurthy) + + MAPREDUCE-1829. JobInProgress.findSpeculativeTask should use min() to + find the candidate instead of sort(). (Scott Chen via vinodkv) + + BUG FIXES + + MAPREDUCE-1845. FairScheduler.tasksToPreempt() can return negative number. + (Scott Chen via matei) + + MAPREDUCE-1707. TaskRunner can get NPE in getting ugi from TaskTracker. + (Vinod Kumar Vavilapalli) + + MAPREDUCE-1532. Ensures that delegation tokens is obtained as the + actual user when the proxy-user is used for submitting jobs. Also + refactors the DelegationTokenToRenew class. (ddas) + + MAPREDUCE-1558. Fixes MRAdmin to look up the principal of the + JobTracker and use that in the RefreshUserToGroupsMapping protocol and + RefreshAuthorizationPolicyProtocol. (Boris Shkolnik via ddas) + + MAPREDUCE-1662. Remove unused methods from TaskRunner. (Amareshwari + Sriramadasu via cdouglas) + + MAPREDUCE-1617. Use IPv4 stack for unit tests. (Amar Kamat and Luke Lu via + cdouglas) + + MAPREDUCE-1599. Fixes MRBench so that it reuses tokens across jobs + correctly. (Jitendra Nath Pandey via ddas) + + MAPREDUCE-1836. Refresh for proxy superuser config (mr part for HDFS-1096). + (Boris Shkolnik via shv) + + MAPREDUCE-1505. Create RPC client on job submission, not in cstr of Job + instance. (Dick King via cdouglas) + + MAPREDUCE-1813. NPE in PipeMapred.MRErrorThread. (Ravi Gummadi via vinodkv) + + MAPREDUCE-1225. Fixes DistributedCache to check if the file is fresh or not, + for the first localization also. (Zhong Wang via amareshwari) + + MAPREDUCE-1559. Fixes the token renewer to use the JobTracker's + credentials for talking to the NameNode. (ddas) + + MAPREDUCE-1492. Delete obsolete har files used on the parity files + of hdfs raid. (Rodrigo Schmidt via dhruba) + + MAPREDUCE-1857. Removes unused configuration parameters in streaming. + (amareshwari) + + MAPREDUCE-1887. MRAsyncDiskService now properly absolutizes volume root + paths. (Aaron Kimball via zshao) + + MAPREDUCE-1863. Fix NPE in Rumen when processing null CDF for failed task + attempts. (Amar Kamat via cdouglas) + + MAPREDUCE-1864. Removes uninitialized/unused variables in + org.apache.hadoop.streaming.PipeMapRed. (amareshwari) + + MAPREDUCE-1888. Fixes Streaming to override output key and value types, + only if mapper/reducer is a command. (Ravi Gummadi via amareshwari) + + MAPREDUCE-577. Fixes duplicate records in StreamXmlRecordReader. + (Ravi Gummadi via amareshwari) + + MAPREDUCE-1894. Fixed a bug in DistributedRaidFileSystem.readFully() + that was causing it to loop infinitely. (Ramkumar Vadali via dhruba) + + MAPREDUCE-1838. Reduce the time needed for raiding a bunch of files + by randomly assigning files to map tasks. (Ramkumar Vadali via dhruba) + + MAPREDUCE-1820. Fix InputSampler to clone sampled keys. (Alex Kozlov via + cdouglas) + + MAPREDUCE-1528. Incorporates the changes to the credentials API done in + HADOOP-6845. Also, introduces Credentials in JobConf, and in JobContext. + (Jitendra Pandey and Arun Murthy via ddas) + + MAPREDUCE-1865. Rumen should also support jobhistory files generated using + trunk. (Amar Kamat via amareshwari) + + MAPREDUCE-1621. Fixes NPE in TextOutputReader.getLastOutput if it has never + read any output. (amareshwari) + + MAPREDUCE-1911. Fixes errors in -info message in streaming. (amareshwari) + + MAPREDUCE-1772. Corrects errors in streaming documentation in forrest. + (amareshwari) + + MAPREDUCE-1925. Fix failing TestRumenJobTraces. (Ravi Gummadi via cdouglas) + + MAPREDUCE-1718. Fixes a bug in the construction of jobconf key for the + mapping that the tasks use at runtime for looking up delegation tokens. + (Boris Shkolnik via ddas) + + MAPREDUCE-1701. Fixes a problem to do with exception handling in + delegation-token renewals. (Boris Shkolnik via ddas) + + MAPREDUCE-1686. Fixes StreamUtil.goodClassOrNull to find classes without + package names. (Paul Burkhardt via amareshwari) + + MAPREDUCE-1288. Fixes TrackerDistributedCacheManager to take into account + the owner of the localized file in the mapping from cache URIs to + CacheStatus objects. (ddas) + + MAPREDUCE-1982. Fixes Rumen's TraceBuilder to extract job name from either + of configuration properties "mapreduce.job.name" and "mapred.job.name". + (Ravi Gummadi via amareshwari) + + MAPREDUCE-1958. The MapReduce part corresponding to the HADOOP-6873. + (Boris Shkolnik & Owen O'Malley via ddas) + + MAPREDUCE-1900. TaskTracker and JobTracker closes FileSystems, opened on + behalf of users that it no longer requires. (Kan Zhang and ddas via ddas) + + MAPREDUCE-1992. Fixes a problem to do with bringing up the JobTracker in + unsecure mode. (Kan Zhang via ddas) + + MAPREDUCE-1999. Fixes ClientProtocol to use the correct + DelegationTokenSelector. (Jitendra Pandey via ddas) + + MAPREDUCE-1780. AccessControlList.toString() is used for serialization of + ACL in JobStatus.java. (Ravi Gummadi via vinodkv) + + MAPREDUCE-1961. Fix ConcurrentModificationException in Gridmix during + shutdown. (Hong Tang via cdouglas) + + MAPREDUCE-2000. Fix parsing of JobHistory lines in Rumen when quotes are + escaped. (Hong Tang via cdouglas) + + MAPREDUCE-2022. Fixes compilation errors in TestSubmitJob. (amareshwari) + + MAPREDUCE-1670. RAID policies should not scan their own destination path. + (Ramkumar Vadali via dhruba) + + MAPREDUCE-1668. RaidNode Hars a directory only if all its parity files + have been created. (Ramkumar Vadali via dhruba) + + MAPREDUCE-2021. Fixes duplicate hostnames in CombineFileInputFormat's + split locations. (amareshwari) + + MAPREDUCE-1375. Fixes flaky test TestFileArgs. (Todd Lipcon via + amareshwari) + + MAPREDUCE-2023. TestDFSIO should not stop reading if curSize != bufferSize. + (Hong Tang via szetszwo) + + MAPREDUCE-2031. Fixes test failures TestTaskLauncher and + TestTaskTrackerLocalization. (Ravi Gummadi via amareshwari) + + MAPREDUCE-2046. Fixes CombineFileInputFormat to allow splits with size + less than DFS block size. (dhruba borthakur via amareshwari) + + MAPREDUCE-1975. Fixes unnecessary InterruptedException log in gridmix. + (Ravi Gummadi via amareshwari) + + MAPREDUCE-1597. Fixes CombineFileInputFormat to work with non-splittable + files. (amareshwari) + + MAPREDUCE-2032. Fixes TestJobCleanup to cleanup test directory in + tearDown. (Dick King via amareshwari) + + MAPREDUCE-1979. Fixes "Output directory already exists" error in gridmix + when gridmix.output.directory is not defined. (Ravi Gummadi via + amareshwari) + + MAPREDUCE-1918. Adds documentation to Rumen. (Amar Kamat via amareshwari) + + MAPREDUCE-2078. Fixes TraceBuilder to generate traces when a globbed job + history path is given. (Amar Kamat via amareshwari) + + MAPREDUCE-1989. Fixes error message in gridmix when user resolver is set + and no user list is given. (Ravi Gummadi via amareshwari) + + MAPREDUCE-2067. Distinct minicluster services (e.g. NN and JT) overwrite + each other's service policies. (Aaron T. Myers via tomwhite) + + MAPREDUCE-2029. DistributedRaidFileSystem removes itself from FileSystem + cache when it is closed. (Ramkumar Vadali via dhruba) + + MAPREDUCE-1816. HAR files used for RAID parity-bite have configurable + partfile size. (Ramkumar Vadali via dhruba) + + MAPREDUCE-2082. Fixes Pipes to create the jobtoken file in the right + place. (Jitendra Pandey via ddas) + + MAPREDUCE-2095. Fixes Gridmix to run from compressed traces. (Ranjit + Mathew via amareshwari) + + MAPREDUCE-1908. DistributedRaidFileSystem now handles ChecksumException + correctly. (Ramkumar Vadali via schen) + + MAPREDUCE-2126. JobQueueJobInProgressListener's javadoc is inconsistent + with source code. (Jingguo Yao via tomwhite) + + MAPREDUCE-2143. HarFileSystem is able to handle spaces in pathnames. + (Ramkumar Vadali via dhruba) + + MAPREDUCE-1867. Remove unused methods in + org.apache.hadoop.streaming.StreamUtil. (amareshwari via tomwhite) + + MAPREDUCE-2146. Raid does not affect access time of a source file. + (Ramkumar Vadali via dhruba) + + MAPREDUCE-2150. RaidNode periodically fixes corrupt blocks. (Ramkumar Vadali via + schen) + + MAPREDUCE-2099. RaidNode recreates outdated parity HARs. (Ramkumar Vadali + via schen) + + MAPREDUCE-2173. Fix race condition in TestBlockFixer that was + causing intermittent failure (Patrick Kling via dhruba) + + MAPREDUCE-2142. Refactor RaidNode so that the map-reduce component is + clearly separated out. (Patrick Kling via dhruba) + + MAPREDUCE-2179. Fix RaidBlockSender compilation failure. (Ramkumar Vadali + via schen) + + MAPREDUCE-2034. TestSubmitJob triggers NPE instead of permissions error. + (Todd Lipcon via tomwhite) + + MAPREDUCE-2195. New property for local conf directory in + system-test-mapreduce.xml file. (cos) + + MAPREDUCE-1783. FairScheduler initializes tasks only when the job can be + run. (Ramkumar Vadali via schen) + + MAPREDUCE-2224. Fix synchronization bugs in JvmManager. (todd) + + MAPREDUCE-714. JobConf.findContainingJar unescapes unnecessarily on linux (todd) + + MAPREDUCE-2096. Secure local filesystem IO from symlink vulnerabilities (todd) + + MAPREDUCE-2234. If Localizer can't create task log directory, it should fail + on the spot. (todd) + + MAPREDUCE-2219. JobTracker should not try to remove mapred.system.dir + during startup. (todd) + + MAPREDUCE-2207. Task-cleanup task should not be scheduled on the node that + the task just failed. (Liyin Liang via schen) + + MAPREDUCE-2084. Remove deprecate annotation for package file. The package + classes themselves are already deprecated. This removes an Eclipse error. + (tomwhite via nigel) + + MAPREDUCE-2248. DistributedRaidFileSystem should unraid only the corrupt + block (Ramkumar Vadali via schen) + + MAPREDUCE-1085. For tasks, "ulimit -v -1" is being run when user doesn't + specify a ulimit (todd) + + MAPREDUCE-2282. Fix TestMRServerPorts for the changes in + TestHDFSServerPorts. (shv via szetszwo) + + MAPREDUCE-2238. Fix permissions handling to avoid leaving undeletable + directories in local dirs. (todd) + + MAPREDUCE-2277. TestCapacitySchedulerWithJobTracker needs to wait for jobs + to complete before testing status. (todd) + + MAPREDUCE-2253. Servlets should specify content type (todd) + + MAPREDUCE-2283. Add timeout for Raid Tests (Ramkumar Vadali via schen) + + MAPREDUCE-1754. Replace mapred.persmissions.supergroup with an + acl : mapreduce.cluster.administrators (Amareshwari Sriramadasu via shv) + + MAPREDUCE-2256. FairScheduler fairshare preemption from multiple pools may + preempt all tasks from one pool causing that pool to go below fairshare. + (Priyo Mustafi via shv) + + MAPREDUCE-2281. MR part of HADOOP-6642. (Chris Douglas, Po Cheung via shv) + + MAPREDUCE-2200. TestUmbilicalProtocolWithJobToken is failing without Krb + evironment: needs to be conditional. (cos) + + MAPREDUCE-2077. Resolve name clash in the deprecated + o.a.h.util.MemoryCalculatorPlugin (Luke Lu via shv) + + MAPREDUCE-2188. The new API MultithreadedMapper doesn't initialize + RecordReader. (Owen O'Malley via shv) + + MAPREDUCE-1915. Fix IndexOutOfBoundsException in IndexCache. + (Priyo Mustafi via shv) + + MAPREDUCE-1974. Fix multiple preemtions of the same task in FairScheduler. + (Scott Chen via shv) + + MAPREDUCE-2304. Fix TestMRCLI to allow hostname with a hyphen (-). + (Priyo Mustafi via shv) + + MAPREDUCE-1825. jobqueue_details.jsp and FairSchedulerServelet should not + call finishedMaps and finishedReduces when job is not initialized. + (Scott Chen via shv) + + MAPREDUCE-2285. MiniMRCluster does not start after ant test-patch + (todd) + + MAPREDUCE-2315. javadoc is failing in nightly build (todd) + + MAPREDUCE-2054. Hierarchical queue implementation broke dynamic queue + addition in Dynamic Scheduler. (Thomas Sandholm via tomwhite) + + MAPREDUCE-2272. Job ACL file should not be executable + (Harsh J Chouraria via todd) + + MAPREDUCE-2241. ClusterWithLinuxTaskController should accept relative path + on the command line. (todd) + + MAPREDUCE-2251. Remove unused mapreduce.job.userhistorylocation config. + (Harsh J Chouraria via todd) + + MAPREDUCE-2284. TestLocalRunner.testMultiMaps times out (todd) + + MAPREDUCE-2336. Tool-related packages should be in the Tool javadoc group. + (tomwhite) + + MAPREDUCE-2394. JUnit output format doesn't propagate into raid contrib + build. (todd) + + MAPREDUCE-2392. TaskTracker shutdown in the tests sometimes take 60s. + (tomwhite) + + MAPREDUCE-2437. SLive uses only part* files to generating the final report. + (shv) + + MAPREDUCE-2428. start-mapred.sh script fails if HADOOP_HOME is not set. + (tomwhite via eli) + + MAPREDUCE-2445. Fix TestMiniMRWithDFSWithDistinctUsers to be a valid test. + (todd) + + MAPREDUCE-2457. Job submission should inject group.name on the JobTracker + (Alejandro Abdelnur via todd) + + MAPREDUCE-2472. Extra whitespace in mapred.child.java.opts breaks JVM + initialization. (Aaron T. Myers via todd) + + MAPREDUCE-2222. Ivy resolve force mode should be turned off by default. + (Luke Lu via tomwhite) + + MAPREDUCE-2486. Incorrect snapshot dependency published in .pom files + (todd) + + MAPREDUCE-2327. MapTask doesn't need to put username information in + SpillRecord. (todd via tomwhite) + + MAPREDUCE-2515. MapReduce code references some deprecated options + (Ari Rabkin via todd) + + MAPREDUCE-2487. ChainReducer uses MAPPER_BY_VALUE instead of + REDUCER_BY_VALUE. (Devaraj K via todd) + + MAPREDUCE-2185. Fix infinite loop at creating splits using + CombineFileInputFormat. (Ramkumar Vadali via schen) + + MAPREDUCE-2571. CombineFileInputFormat.getSplits throws a + java.lang.ArrayStoreException. (Bochun Bai via todd) + + MAPREDUCE-2767. Remove Linux task-controller. (Milind Bhandarkar via shv) + + MAPREDUCE-2991. queueinfo.jsp fails to show queue status for Capacity + scheduler if queue names contain special symbols. (Priyo Mustafi via shv) + + MAPREDUCE-2531. Fixed jobcontrol to downgrade JobID. (Robert Evans via + acmurthy) + + MAPREDUCE-3139. SlivePartitioner generates negative partitions. (jghoman) + +Release 0.21.1 - Unreleased + + NEW FEATURES + + MAPREDUCE-2040. Forrest Documentation for Dynamic Priority Scheduler. + (Thomas Sandholm via tomwhite) + + BUG FIXES + + MAPREDUCE-1897. trunk build broken on compile-mapred-test (cos) + + MAPREDUCE-1280. Update Eclipse plugin to the new eclipse.jdt API. + (Alex Kozlov via szetszwo) + + MAPREDUCE-1984. herriot TestCluster fails because exclusion is not there + (Balaji Rajagopalan via cos) + + MAPREDUCE-2090. Clover build doesn't generate per-test coverage. (cos) + + MAPREDUCE-2134. ant binary-system is broken in mapreduce project. (cos) + + MAPREDUCE-1905. Fixes Context.setStatus() and progress() apis. + (amareshwari) + + MAPREDUCE-1809. Ant build changes for Streaming system tests in contrib + projects. (Vinay Kumar Thota via amareshwari) + + MAPREDUCE-2223. TestMRCLI might fail on Ubuntu with default /etc/hosts + (cos) + + MAPREDUCE-2228. Remove java5 dependencies from build. (cos) + + MAPREDUCE-1929. Allow artifacts to be published to the staging Apache Nexus + Maven Repository. (tomwhite) + + MAPREDUCE-2317. Fix a NPE in HadoopArchives. (Devaraj K via szetszwo) + + MAPREDUCE-2127. mapreduce trunk builds are filing on hudson. + (Bruno Mahé via eli) + + MAPREDUCE-2779. JobSplitWriter.java can't handle large job.split file. + (Ming Ma via shv) + +Release 0.21.0 - 2010-08-13 + + INCOMPATIBLE CHANGES + + MAPREDUCE-516. Fix the starvation problem in the Capacity Scheduler + when running High RAM Jobs. (Arun Murthy via yhemanth) + + MAPREDUCE-358. Change org.apache.hadoop.examples. AggregateWordCount + and org.apache.hadoop.examples.AggregateWordHistogram to use new + mapreduce api. (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-245. Change Job and jobcontrol classes to use the List interface + rather than ArrayList in APIs. (Tom White via cdouglas) + + MAPREDUCE-766. Enhanced list-blacklisted-trackers to display reasons + for blacklisting a node. (Sreekanth Ramakrishnan via yhemanth) + + MAPREDUCE-817. Add a cache for retired jobs with minimal job info and + provide a way to access history file url. (sharad) + + MAPREDUCE-711. Moved Distributed Cache from Common to Map/Reduce + project. (Vinod Kumar Vavilapalli via yhemanth) + + MAPREDUCE-895. Per the contract elucidated in HADOOP-6201, throw + FileNotFoundException from FileSystem::listStatus rather than returning + null. (Jakob Homan via cdouglas) + + MAPREDUCE-479. Provide full task id to map output servlet rather than the + reduce id, only. (Jiaqi Tan via cdouglas) + + MAPREDUCE-873. Simplify job recovery. Incomplete jobs are resubmitted on + jobtracker restart. Removes a public constructor in JobInProgress. (sharad) + + HADOOP-6230. Moved process tree and memory calculator related classes from + Common to Map/Reduce. (Vinod Kumar Vavilapalli via yhemanth) + + MAPREDUCE-157. Refactor job history APIs and change the history format to + JSON. (Jothi Padmanabhan via sharad) + + MAPREDUCE-849. Rename configuration properties. (Amareshwari Sriramadasu + via sharad) + + MAPREDUCE-1287. Only call the partitioner with more than one reducer. + (cdouglas) + + MAPREDUCE-1385. Use the new UserGroupInformation from HADOOP-6299. + (ddas via omalley) + + MAPREDUCE-1493. Authorization for job-history pages. (vinodkv) + + MAPREDUCE-1607. Task controller may not set permissions for a + task cleanup attempt's log directory (Amareshwari Sriramadasu via vinodkv) + + MAPREDUCE-1683. Remove JNI calls from ClusterStatus cstr. (Arun Murthy and + Luke Lu via cdouglas) + + MAPREDUCE-1855. Makes the refresh methods (for groups and proxy users) + independent of the client side configuration. (Boris Shkolnik via ddas) + + NEW FEATURES + + MAPREDUCE-1774. Large-scale Automated Framework (Sharad Agarwal, Sreekanth + Ramakrishnan, Konstantin Boudnik, at all via cos) + + MAPREDUCE-706. Support for FIFO pools in the fair scheduler. + (Matei Zaharia) + + MAPREDUCE-546. Provide sample fair scheduler config file in conf/ and use + it by default if no other config file is specified. (Matei Zaharia) + + MAPREDUCE-551. Preemption support in the Fair Scheduler. (Matei Zaharia) + + MAPREDUCE-567. Add a new example MR that always fails. (Philip Zeyliger + via tomwhite) + + MAPREDUCE-211. Provides ability to run a health check script on the + tasktracker nodes and blacklist nodes if they are unhealthy. + (Sreekanth Ramakrishnan via yhemanth) + + MAPREDUCE-637. Add an example, distbbp, which able to compute the n th bit + of Pi for some large n. (szetszwo) + + MAPREDUCE-532. Provide a way to limit the number of used slots + per queue in the capacity scheduler. + (Rahul Kumar Singh via yhemanth) + + MAPREDUCE-467. Provide ability to collect statistics about total tasks + and succeeded tasks in different time windows. (sharad) + + MAPREDUCE-740. Log a job-summary at the end of a job, while allowing it + to be configured to use a custom appender if desired. (acmurthy) + + MAPREDUCE-814. Provide a way to configure completed job history files + to be on HDFS. (sharad) + + MAPREDUCE-800. MRUnit should support the new API. (Aaron Kimball via + tomwhite) + + MAPREDUCE-798. MRUnit should be able to test a succession of MapReduce + passes. (Aaron Kimball via tomwhite) + + MAPREDUCE-768. Provide an option to dump jobtracker configuration in JSON + format to standard output. (V.V.Chaitanya Krishna via yhemanth) + + MAPREDUCE-824. Add support for a hierarchy of queues in the capacity + scheduler. (Rahul Kumar Singh via yhemanth) + + MAPREDUCE-751. Add Rumen, a tool for extracting statistics from job tracker + logs and generating job traces for simulation and analysis. (Dick King via + cdouglas) + + MAPREDUCE-830. Add support for splittable compression to TextInputFormats. + (Abdul Qadeer via cdouglas) + + MAPREDUCE-861. Add support for hierarchical queues in the Map/Reduce + framework. (Rahul Kumar Singh via yhemanth) + + MAPREDUCE-776. Add Gridmix, a benchmark processing Rumen traces to simulate + a measured mix of jobs on a cluster. (cdouglas) + + MAPREDUCE-862. Enhance JobTracker UI to display hierarchical queues. + (V.V.Chaitanya Krishna via yhemanth) + + MAPREDUCE-777. Brand new apis to track and query jobs as a + replacement for JobClient. (Amareshwari Sriramadasu via acmurthy) + + MAPREDUCE-775. Add native and streaming support for Vertica as an input + or output format taking advantage of parallel read and write properties of + the DBMS. (Omer Trajman via ddas) + + MAPREDUCE-679. XML-based metrics as JSP servlet for JobTracker. + (Aaron Kimball via tomwhite) + + MAPREDUCE-980. Modify JobHistory to use Avro for serialization. (cutting) + + MAPREDUCE-728. Add Mumak, a Hadoop map/reduce simulator. (Arun C Murthy, + Tamas Sarlos, Anirban Dasgupta, Guanying Wang, and Hong Tang via cdouglas) + + MAPREDUCE-1383. Automates fetching of delegation tokens in File*Formats + Distributed Cache and Distcp. Also, provides a config + mapreduce.job.hdfs-servers that the jobs can populate with a comma + separated list of namenodes. The job client automatically fetches + delegation tokens from those namenodes. (Boris Shkolnik via ddas) + + MAPREDUCE-698. Per-pool task limits for the fair scheduler. + (Kevin Peterson via matei) + + MAPREDUCE-1026. Does mutual authentication of the shuffle + transfers using a shared JobTracker generated key. + (Boris Shkolnik via ddas) + + MAPREDUCE-744. Introduces the notion of a public distributed cache. + (Devaraj Das) + + MAPREDUCE-1338. Introduces the notion of token cache using which + tokens and secrets can be sent by the Job client to the JobTracker. + (Boris Shkolnik via ddas) + + HDFS-503. This patch implements an optional layer over HDFS that + implements offline erasure-coding. It can be used to reduce the + total storage requirements of HDFS. (dhruba) + + MAPREDUCE-1432. Adds hooks in the jobtracker and tasktracker + for loading the tokens in the user's ugi. This is required + for the copying of files from the hdfs. (ddas) + + MAPREDUCE-1335. Adds SASL Kerberos/Digest authentication in MapReduce. + (Kan Zhang via ddas) + + MAPREDUCE-1464. Makes a compatible change in JobTokenIdentifier to + account for HADOOP-6510. (Jitendra Nath Pandey via ddas) + + MAPREDUCE-1433. Add a delegation token for MapReduce. (omalley) + + MAPREDUCE-1307. Introduces the Job level ACLs feature. + (Vinod Kumar Vavilapalli via ddas) + + MAPREDUCE-1430. JobTracker automatically renews delegation tokens for jobs. + (Boris Shkolnik via ddas) + + MAPREDUCE-1455. Introduces job-level authorization for mapreduce servlets. + (Ravi Gummadi via vinodkv) + + IMPROVEMENTS + + MAPREDUCE-463. Makes job setup and cleanup tasks as optional. + (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-502. Allow jobtracker to be configured with zero completed jobs + in memory. (Amar Kamat via sharad) + + MAPREDUCE-416. Moves the history file to a "done" folder whenever a job + completes. (Amar Kamat via ddas) + + MAPREDUCE-646. Increase srcfilelist replication number in dictcp job. + (Ravi Gummadi via szetszwo) + + HADOOP-6106. Updated hadoop-core and test jars from hudson trunk + build #12. (Giridharan Kesavan) + + MAPREDUCE-642. A option to distcp that allows preserving the full + source path of a file in the specified destination directory. + (Rodrigo Schmidt via dhruba) + + MAPREDUCE-686. Move TestSpeculativeExecution.Fake* into a separate class + so that it can be used by other tests. (Jothi Padmanabhan via sharad) + + MAPREDUCE-625. Modify TestTaskLimits to improve execution time. + (Jothi Padmanabhan via sharad) + + MAPREDUCE-465. Deprecate o.a.h.mapred.lib.MultithreadedMapRunner and add + test for o.a.h.mapreduce.lib.MultithreadedMapper. + (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-701. Improves the runtime of the TestRackAwareTaskPlacement + by making it a unit test. (Jothi Padmanabhan via ddas) + + MAPREDUCE-371. Change KeyFieldBasedComparator and KeyFieldBasedPartitioner + to use new api. (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-623. Resolve javac warnings in mapreduce. (Jothi Padmanabhan + via sharad) + + MAPREDUCE-655. Change KeyValueLineRecordReader and KeyValueTextInputFormat + to use new mapreduce api. (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-632. Merge TestCustomOutputCommitter with + TestCommandLineJobSubmission. (Jothi Padmanabhan via sharad) + + MAPREDUCE-627. Improves execution time of TestTrackerBlacklistAcrossJobs. + (Jothi Padmanabhan via ddas) + + MAPREDUCE-630. Improves execution time of TestKillCompletedJob. + (Jothi Padmanabhan via ddas) + + MAPREDUCE-626. Improves the execution time of TestLostTracker. + (Jothi Padmanabhan via ddas) + + MAPREDUCE-353. Makes the shuffle read and connection timeouts + configurable. (Ravi Gummadi via ddas) + + MAPREDUCE-739. Allow relative paths to be created in archives. (Mahadev + Konar via cdouglas) + + MAPREDUCE-772. Merge HADOOP-4010 changes to LineRecordReader into mapreduce + package. (Abdul Qadeer via cdouglas) + + MAPREDUCE-785. Separate sub-test of TestReduceFetch to be included in + MR-670. (Jothi Padmanabhan via cdouglas) + + MAPREDUCE-784. Modify TestUserDefinedCounters to use LocalJobRunner + instead of MiniMR. (Jothi Padmanabhan via sharad) + + HADOOP-6160. Fix releaseaudit target to run on specific directories. + (gkesavan) + + MAPREDUCE-782. Use PureJavaCrc32 in SpillRecord. (Todd Lipcon via + szetszwo) + + MAPREDUCE-369. Change org.apache.hadoop.mapred.lib.MultipleInputs to + use new api. (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-373. Change org.apache.hadoop.mapred.lib.FieldSelectionMapReduce + to use new api. (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-628. Improves the execution time of TestJobInProgress. + (Jothi Padmanabhan via ddas) + + MAPREDUCE-793. Creates a new test that consolidates a few tests to + include in the commit-test list. (Jothi Padmanabhan via ddas) + + MAPREDUCE-797. Adds combiner support to MRUnit MapReduceDriver. + (Aaron Kimball via johan) + + MAPREDUCE-656. Change org.apache.hadoop.mapred.SequenceFile* classes + to use new mapreduce api. (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-670. Creates ant target for 10 mins patch test build. + (Jothi Padmanabhan via gkesavan) + + MAPREDUCE-375. Change org.apache.hadoop.mapred.lib.NLineInputFormat + and org.apache.hadoop.mapred.MapFileOutputFormat to use new api. + (Amareshwari Sriramadasu via ddas) + + MAPREDUCE-779. Added node health failure counts into + JobTrackerStatistics. (Sreekanth Ramakrishnan via yhemanth) + + MAPREDUCE-842. Setup secure permissions for localized job files, + intermediate outputs and log files on tasktrackers. + (Vinod Kumar Vavilapalli via yhemanth) + + MAPREDUCE-478. Allow map and reduce jvm parameters, environment variables + and ulimit to be set separately. + Configuration changes: + add mapred.map.child.java.opts + add mapred.reduce.child.java.opts + add mapred.map.child.env + add mapred.reduce.child.ulimit + add mapred.map.child.env + add mapred.reduce.child.ulimit + deprecated mapred.child.java.opts + deprecated mapred.child.env + deprecated mapred.child.ulimit + (acmurthy) + + MAPREDUCE-767. Remove the dependence on the CLI 2.0 snapshot. + (Amar Kamat via omalley) + + MAPREDUCE-712. Minor efficiency tweaks to RandomTextWriter. (cdouglas) + + MAPREDUCE-870. Remove the job retire thread and the associated + config parameters. (sharad) + + MAPREDUCE-874. Rename the PiEstimator example to QuasiMonteCarlo. + (szetszwo) + + MAPREDUCE-336. Allow logging level of map/reduce tasks to be configurable. + Configuration changes: + add mapred.map.child.log.level + add mapred.reduce.child.log.level + (acmurthy) + + MAPREDUCE-355. Update mapred.join package to use the new API. (Amareshwari + Sriramadasu via cdouglas) + + HADOOP-6184. Updated hadoop common and test jars to get the new API + in Configuration for dumping in JSON format from Hudson trunk build #68. + (yhemanth) + + MAPREDUCE-476. Extend DistributedCache to work locally (LocalJobRunner). + (Philip Zeyliger via tomwhite) + + MAPREDUCE-825. JobClient completion poll interval of 5s causes slow tests + in local mode. (Aaron Kimball via tomwhite) + + MAPREDUCE-910. Support counters in MRUnit. (Aaron Kimball via cdouglas) + + MAPREDUCE-788. Update gridmix2 to use the new API (Amareshwari Sriramadasu + via cdouglas) + + MAPREDUCE-875. Make DBRecordReader execute queries lazily. (Aaron Kimball + via enis) + + MAPREDUCE-318. Modularizes the shuffle code. (Jothi Padmanabhan and + Arun Murthy via ddas) + + MAPREDUCE-936. Allow a load difference for fairshare scheduler. + (Zheng Shao via dhruba) + + MAPREDUCE-370. Update MultipleOutputs to use the API, merge funcitonality + of MultipleOutputFormat. (Amareshwari Sriramadasu via cdouglas) + + MAPREDUCE-898. Changes DistributedCache to use the new API. + (Amareshwari Sriramadasu via ddas) + + MAPREDUCE-144. Includes dump of the process tree in task diagnostics when + a task is killed due to exceeding memory limits. + (Vinod Kumar Vavilapalli via yhemanth) + + MAPREDUCE-945. Modifies MRBench and TestMapRed to use ToolRunner so that + options such as queue name can be passed via command line. + (Sreekanth Ramakrishnan via yhemanth) + + MAPREDUCE-963. Deprecate o.a.h.mapred.FileAlreadyExistsException and + replace it with o.a.h.fs.FileAlreadyExistsException. (Boris Shkolnik + via szetszwo) + + MAPREDUCE-960. Remove an unnecessary intermediate copy and obsolete API + from KeyValueLineRecordReader. (cdouglas) + + MAPREDUCE-930. Modify Rumen to resolve paths in the canonical way, rather + than defaulting to the local filesystem. (cdouglas) + + MAPREDUCE-944. Extend the LoadManager API of the fair-share scheduler + to support regulating tasks for a job based on resources currently in use + by that job. (dhruba) + + MAPREDUCE-973. Move FailJob and SleepJob from examples to test. (cdouglas + via omalley) + + MAPREDUCE-966. Modify Rumen to clean up interfaces and simplify integration + with other tools. (Hong Tang via cdouglas) + + MAPREDUCE-856. Setup secure permissions for distributed cache files. + (Vinod Kumar Vavilapalli via yhemanth) + + MAPREDUCE-885. More efficient SQL queries for DBInputFormat. (Aaron Kimball + via enis) + + MAPREDUCE-284. Enables ipc.client.tcpnodelay in Tasktracker's Child. + (Ravi Gummadi via sharad) + + MAPREDUCE-916. Split the documentation to match the project split. + (Corinne Chandel via omalley) + + MAPREDUCE-649. Validate a copy by comparing the source and destination + checksums in distcp. Also adds an intra-task retry mechanism for errors + detected during the copy. (Ravi Gummadi via cdouglas) + + MAPREDUCE-654. Add a -dryrun option to distcp printing a summary of the + file data to be copied, without actually performing the copy. (Ravi Gummadi + via cdouglas) + + MAPREDUCE-664. Display the number of files deleted by distcp when the + -delete option is specified. (Ravi Gummadi via cdouglas) + + MAPREDUCE-781. Let the name of distcp jobs be configurable. (Venkatesh S + via cdouglas) + + MAPREDUCE-975. Add an API in job client to get the history file url for + a given job id. (sharad) + + MAPREDUCE-905. Add Eclipse launch tasks for MapReduce. (Philip Zeyliger + via tomwhite) + + MAPREDUCE-277. Makes job history counters available on the job history + viewers. (Jothi Padmanabhan via ddas) + + MAPREDUCE-893. Provides an ability to refresh queue configuration + without restarting the JobTracker. + (Vinod Kumar Vavilapalli and Rahul Kumar Singh via yhemanth) + + MAPREDUCE-1011. Add build.properties to svn and git ignore. (omalley) + + MAPREDUCE-954. Change Map-Reduce context objects to be interfaces. + (acmurthy) + + MAPREDUCE-639. Change Terasort example to reflect the 2009 updates. + (omalley) + + MAPREDUCE-1063. Document gridmix benchmark. (cdouglas) + + MAPREDUCE-931. Use built-in interpolation classes for making up task + runtimes in Rumen. (Dick King via cdouglas) + + MAPREDUCE-1012. Mark Context interfaces as public evolving. (Tom White via + cdouglas) + + MAPREDUCE-971. Document use of distcp when copying to s3, managing timeouts + in particular. (Aaron Kimball via cdouglas) + + HDFS-663. DFSIO for append. (shv) + + HDFS-641. Move all of the components that depend on map/reduce to + map/reduce. (omalley) + + HADOOP-5107. Use Maven ant tasks to publish artifacts. (Giridharan Kesavan + via omalley) + + MAPREDUCE-1229. Allow customization of job submission policy in Mumak. + (Hong Tang via cdouglas) + + MAPREDUCE-1317. Reduce the memory footprint of Rumen objects by interning + host Strings. (Hong Tang via cdouglas) + + MAPREDUCE-1097. Add support for Vertica 3.5 to its contrib module. (Omer + Trajman via cdouglas) + + MAPREDUCE-1627. HadoopArchives should not uses a method in DistCp. + (szetszwo) + + MAPREDUCE-1198. Alternatively schedule different types of tasks in + fair share scheduler. (Scott Chen via matei) + + MAPREDUCE-707. Provide a jobconf property for explicitly assigning a job to + a pool in the Fair Scheduler. (Alan Heirich via matei) + + MAPREDUCE-947. Added commitJob and abortJob apis to OutputCommitter. + Enhanced FileOutputCommitter to create a _SUCCESS file for successful + jobs. (Amar Kamat & Jothi Padmanabhan via acmurthy) + + MAPREDUCE-1103. Added more metrics to Jobtracker. (sharad) + + MAPREDUCE-1048. Add occupied/reserved slot usage summary on jobtracker UI. + (Amareshwari Sriramadasu and Hemanth Yamijala via sharad) + + MAPREDUCE-1090. Modified log statement in TaskMemoryManagerThread to + include task attempt id. (yhemanth) + + MAPREDUCE-1189. Reduce ivy console output to ovservable level (cos) + + MAPREDUCE-1167. ProcfsBasedProcessTree collects rss memory information. + (Scott Chen via dhruba) + + MAPREDUCE-1231. Added a new DistCp option, -skipcrccheck, so that the CRC + check during setup can be skipped. (Jothi Padmanabhan via szetszwo) + + MAPREDUCE-1190. Add package documentation for BBP example. + (Tsz Wo (Nicholas) Sze via cdouglas) + + MAPREDUCE-1119. When tasks fail to report status, show tasks's stack dump + before killing. (Aaron Kimball via tomwhite) + + MAPREDUCE-1185. Redirect running job url to history url if job is already + retired. (Amareshwari Sriramadasu and Sharad Agarwal via sharad) + + MAPREDUCE-1050. Introduce a mock object testing framework. (tomwhite) + + MAPREDUCE-1084. Implementing aspects development and fault injeciton + framework for MapReduce. (Sreekanth Ramakrishnan via cos) + + MAPREDUCE-1209. Move common specific part of the test TestReflectionUtils + out of mapred into common. (Todd Lipcon via tomwhite) + + MAPREDUCE-967. TaskTracker does not need to fully unjar job jars. + (Todd Lipcon via tomwhite) + + MAPREDUCE-1083. Changes in MapReduce so that group information of users + can be refreshed in the JobTracker via command line. + (Boris Shkolnik via ddas) + + MAPREDUCE-181. Changes the job submission process to be secure. + (Devaraj Das) + + MAPREDUCE-1250. Refactors the JobToken to use Common's Token interface. + (Kan Zhang via ddas) + + MAPREDUCE-896. Enhance tasktracker to cleanup files that might have + been created by user tasks with non-writable permissions. + (Ravi Gummadi via yhemanth) + + MAPREDUCE-372. Change org.apache.hadoop.mapred.lib.ChainMapper/Reducer + to use new mapreduce api. (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-1295. Add a tool in Rumen for folding and manipulating job + traces. (Dick King via cdouglas) + + MAPREDUCE-1302. TrackerDistributedCacheManager deletes file + asynchronously, thus reducing task initialization delays. + (Zheng Shao via dhruba) + + MAPREDUCE-1218. TaskTrackers send cpu and memory usage of + node to JobTracker. (Scott Chen via dhruba) + + MAPREDUCE-847. Fix Releaseaudit warning count to zero + (Giridharan Kesavan) + + MAPREDUCE-1337. Use generics in StreamJob to improve readability of that + class. (Kay Kay via cdouglas) + + MAPREDUCE-361. Port terasort example to the new mapreduce API. (Amareshwari + Sriramadasu via cdouglas) + + MAPREDUCE-1367. LocalJobRunner should support parallel mapper execution. + (Aaron Kimball via tomwhite) + + MAPREDUCE-64. Eliminate io.sort.record.percent from MapTask configuration. + (cdouglas) + + MAPREDUCE-1440. Replace the long user name in MapReduce with the local + name. (omalley) + + MAPREDUCE-1470. Move delegation tokens from HDFS to Common so that + MapReduce can use them too. (omalley) + + MAPREDUCE-1425. Reduce memory usage by archive. (mahadev via szetszwo) + + MAPREDUCE-1441. Trim whitespace from directory lists pulled from the + configuration. (Todd Lipcon via cdouglas) + + MAPREDUCE-1309. Refactor Rumen trace generator to improve code structure + and add extensible support for log formats. (Dick King via cdouglas) + + MAPREDUCE-1503. Delegation token renewing and cancelling should provide + meaningful exceptions when there are failures instead of returning + false. (omalley) + + HADOOP-6579. Upgrade commons-codec library to 1.4. (omalley) + + MAPREDUCE-1423. Improve performance of CombineFileInputFormat when multiple + pools are configured. (Dhruba Borthakur via zshao) + + MAPREDUCE-1454. Quote user supplied strings in Tracker servlets. (cdouglas) + + MAPREDUCE-1408. Add customizable job submission policies to Gridmix. (Rahul + Singh via cdouglas) + + MAPREDUCE-1527. Better warning logged when mapred.queue.names is + overshadowed by mapred-queues.xml. (Hong Tang via acmurthy) + + MAPREDUCE-1403. Save the size and number of distributed cache artifacts in + the configuration. (Arun Murthy via cdouglas) + + MAPREDUCE-1482. Truncate state string and diagnostic information in + TaskStatus. (Amar Kamat via szetszwo) + + MAPREDUCE-1593. [Rumen] Improvements to random seed generation (tamas via + mahadev) + + MAPREDUCE-1460. Oracle support in DataDrivenDBInputFormat. + (Aaron Kimball via tomwhite) + + MAPREDUCE-1569. Pass configuration through mocked contexts in MRUnit. + (Chris White via cdouglas) + + MAPREDUCE-1590. Move HarFileSystem from Hadoop Common to Mapreduce tools. + (mahadev) + + MAPREDUCE-1629. Get rid of fakeBlockLocations() on HarFileSystem, since + it's not used (mahadev) + + MAPREDUCE-1489. DataDrivenDBInputFormat should not query the database + when generating only one split. (Aaron Kimball via tomwhite) + + MAPREDUCE-1514. Add documentation on replication, permissions, new options, + limitations and internals of har. (mahadev via szetszwo) + + MAPREDUCE-1428. Make block size and the size of archive created files + configurable. (mahadev via szetszwo) + + MAPREDUCE-1656. JobStory should provide queue info. (hong via mahadev) + + MAPREDUCE-1466. Record number of files processed in FileInputFormat in the + Configuration for offline analysis. (Luke Lu and Arun Murthy via cdouglas) + + MAPREDUCE-1538. TrackerDistributedCacheManager manages the + number of files. (Scott Chen via dhruba) + + MAPREDUCE-1673. Scripts to start and stop RaidNode. + (Rodrigo Schmidt via dhruba) + + MAPREDUCE-1659. RaidNode writes temp files on configured tmp directory and + add random numbers to their names to avoid conflicts + (Rodrigo Schmidt via dhruba) + + MAPREDUCE-1221. Allow admins to control physical memory limits per-task + and per-node. (Scott Chen via acmurthy) + + MAPREDUCE-1065. Update mapred tutorial to use the new API. (Aaron Kimball + via cdouglas) + + MAPREDUCE-1304. Add a task counter tracking time spent in GC. (Aaron + Kimball via cdouglas) + + MAPREDUCE-1570. Add grouping comparators to MRUnit. (Chris White via + cdouglas) + + MAPREDUCE-1650. Exclude Private elements from generated MapReduce + Javadoc. (tomwhite) + + MAPREDUCE-1625. Improve grouping of packages in Javadoc. (tomwhite) + + MAPREDUCE-1417. Forrest documentation should be updated to reflect + the changes in MAPREDUCE-744. (Ravi Gummadi via vinodkv) + + MAPREDUCE-1568. TrackerDistributedCacheManager should clean up cache + in a background thread. (Scott Chen via zshao) + + MAPREDUCE-1749. Move configuration strings out of JobContext so that it + can be made public stable. (omalley) + + MAPREDUCE-1623. Apply audience and stability notations to Hadoop + Map-Reduce. (tomwhite via acmurthy) + + MAPREDUCE-1751. Change MapReduce to depend on Hadoop 'common' artifacts + instead of 'core'. (tomwhite) + + MAPREDUCE-1535. Replace usage of FileStatus#isDir(). (Eli Collins via + tomwhite) + + MAPREDUCE-1832. Allow file sizes less than 1MB in DFSIO benchmark. (shv) + + MAPREDUCE-1404. Move Cluster-Setup and Single-Node-Setup Docs from + MapReduce to Common. (tomwhite) + + MAPREDUCE-1697. Document the behavior of -file option and deprecate it + in favour of -files option in streaming. (Amareshwari Sriramadasu + via vinodkv) + + MAPREDUCE-1033. Resolve location of scripts and configuration files after + project split. (tomwhite) + + MAPREDUCE-1018. Document changes to the memory management and scheduling + model. (Hemanth Yamijala via vinodkv) + + MAPREDUCE-1896. [Herriot] New property for multi user list. (Vinay Thota + via cos) + + MAPREDUCE-1812. New properties for suspend and resume process. (Vinay + Thota via cos) + + OPTIMIZATIONS + + MAPREDUCE-270. Fix the tasktracker to optionally send an out-of-band + heartbeat on task-completion for better job-latency. (acmurthy) + Configuration changes: + add mapreduce.tasktracker.outofband.heartbeat + + MAPREDUCE-1186. Modified code in distributed cache to set permissions + only on required set of localized paths. + (Amareshwari Sriramadasu via yhemanth) + + MAPREDUCE-1501. FileInputFormat supports multi-level, recursive + directory listing. (Zheng Shao via dhruba) + + MAPREDUCE-1556. upgrade to Avro 1.3.0. (cutting via tomwhite) + + MAPREDUCE-1613. Install/deploy source jars to Maven repo + (Patrick Angeles via ddas) + + MAPREDUCE-1610. Forrest documentation should be updated to reflect + the changes in MAPREDUCE-856. (Ravi Gummadi via vinodkv) + + MAPREDUCE-1853. Adds caching for TaskAttemptContext in MultipleOutputs. + (Torsten Curdt via amareshwari) + + BUG FIXES + + MAPREDUCE-878. Rename fair scheduler design doc to + fair-scheduler-design-doc.tex and add Apache license header (matei) + + HADOOP-4687. MapReduce is split from Hadoop Core. It is a subproject under + Hadoop (Owen O'Malley) + + HADOOP-6096. Fix Eclipse project and classpath files following project + split. (tomwhite) + + MAPREDUCE-419. Reconcile mapred.userlog.limit.kb defaults in configuration + and code. (Philip Zeyliger via cdouglas) + + MAPREDUCE-2. Fixes a bug in KeyFieldBasedPartitioner in handling empty + keys. (Amar Kamat via sharad) + + MAPREDUCE-130. Delete the jobconf copy from the log directory of the + JobTracker when the job is retired. (Amar Kamat via sharad) + + MAPREDUCE-657. Fix hardcoded filesystem problem in CompletedJobStatusStore. + (Amar Kamat via sharad) + + MAPREDUCE-179. Update progress in new RecordReaders. (cdouglas) + + MAPREDUCE-658. Replace NPE in distcp with a meaningful error message when + the source path does not exist. (Ravi Gummadi via cdouglas) + + MAPREDUCE-671. Update ignore list to include untracked, generated + build artifacts and config files. (cdouglas) + + MAPREDUCE-433. Use more reliable counters in TestReduceFetch. (cdouglas) + + MAPREDUCE-124. Fix a bug in failure handling of abort task of + OutputCommiter. (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-694. Fix to add jsp-api jars to capacity-scheduler classpath. + (Giridharan Kesavan) + + MAPREDUCE-702. Fix eclipse-plugin jar target (Giridharan Kesavan) + + MAPREDUCE-522. Replace TestQueueCapacities with simpler test case to + test integration between capacity scheduler and MR framework. + (Sreekanth Ramakrishnan via yhemanth) + + MAPREDUCE-683. Fixes an initialization problem in the JobHistory. + The initialization of JobHistoryFilesManager is now done in the + JobHistory.init call. (Amar Kamat via ddas) + + MAPREDUCE-708. Fixes a bug to allow updating the reason for + blacklisting a node on the JobTracker UI. + (Sreekanth Ramakrishnan via yhemanth) + + MAPREDUCE-709. Fixes message displayed for a blacklisted node where + the reason for blacklisting is due to the health check script + timing out. (Sreekanth Ramakrishnan via yhemanth) + + MAPREDUCE-676. Existing diagnostic rules fail for MAP ONLY jobs. + (Suhas Gogate via tomwhite) + + MAPREDUCE-722. Fixes a bug with tasktracker reservations for + high memory jobs in capacity scheduler. + (Vinod Kumar Vavilapalli via yhemanth) + + HADOOP-6090. Updates gridmix script to use new mapreduce api output + format. (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-732. Removed spurious log statements in the node + blacklisting logic. (Sreekanth Ramakrishnan via yhemanth) + + MAPREDUCE-734. Fix a ConcurrentModificationException in unreserving + unused reservations for a job when it completes. + (Arun Murthy and Sreekanth Ramakrishnan via yhemanth) + + MAPREDUCE-733. Fix a RuntimeException while unreserving trackers + that are blacklisted for a job. + (Arun Murthy and Sreekanth Ramakrishnan via yhemanth) + + MAPREDUCE-677. Fix timeout in TestNodeRefresh. (Amar Kamat via + sharad) + + MAPREDUCE-153. Fix timeout in TestJobInProgressListener. (Amar + Kamat via sharad) + + MAPREDUCE-742. Fix output messages and java comments in the Pi related + examples. (szetszwo) + + MAPREDUCE-565. Fix partitioner to work with new API. (Owen O'Malley via + cdouglas) + + MAPREDUCE-680. Fix so MRUnit can handle reuse of Writable objects. + (Aaron Kimball via johan) + + MAPREDUCE-18. Puts some checks for cross checking whether a reduce + task gets the correct shuffle data. (Ravi Gummadi via ddas) + + MAPREDUCE-771. Fix scheduling of setup and cleanup tasks to use + free slots instead of tasks for scheduling. (yhemanth) + + MAPREDUCE-717. Fixes some corner case issues in speculative + execution heuristics. (Devaraj Das) + + MAPREDUCE-716. Make DBInputFormat work with Oracle. (Aaron Kimball + via tomwhite) + + MAPREDUCE-735. Fixes a problem in the KeyFieldHelper to do with + the end index for some inputs (Amar Kamat via ddas) + + MAPREDUCE-682. Removes reservations on tasktrackers which are + blacklisted. (Sreekanth Ramakrishnan via yhemanth) + + MAPREDUCE-743. Fixes a problem to do with progress reporting + in the map phase. (Ravi Gummadi via ddas) + + MAPREDUCE-765. Eliminate the deprecated warnings introduced by H-5438. + (He Yongqiang via szetszwo) + + MAPREDUCE-383. Fix a bug in Pipes combiner due to bytes count not + getting reset after the spill. (Christian Kunz via sharad) + + MAPREDUCE-809. Fix job-summary logs to correctly record status of FAILED + and KILLED jobs. (acmurthy) + + MAPREDUCE-792. Fix unchecked warnings in DBInputFormat. (Aaron Kimball + via szetszwo) + + MAPREDUCE-760. Fix a timing issue in TestNodeRefresh. (Amar Kamat via + sharad) + + MAPREDUCE-40. Keep memory management backwards compatible for job + configuration parameters and limits. (Rahul Kumar Singh via yhemanth) + + MAPREDUCE-587. Fixes a OOM issue in TestStreamingExitStatus. + (Amar Kamat via ddas) + + MAPREDUCE-408. Fixes an assertion problem in TestKillSubProcesses + (Ravi Gummadi via ddas) + + MAPREDUCE-659. Fix gridmix2 compilation. (Giridharan Kesavan) + + MAPREDUCE-796. Fixes a ClassCastException in an exception log in + MultiThreadedMapRunner. (Amar Kamat via ddas) + + MAPREDUCE-808. Fixes a serialization problem in TypedBytes. + (Klaas Bosteels via ddas) + + MAPREDUCE-845. Fix a findbugs heap size problem in build.xml and add + a new property findbugs.heap.size. (Lee Tucker via szetszwo) + + MAPREDUCE-838. Fixes a problem in the way commit of task outputs + happens. The bug was that even if commit failed, the task would + be declared as successful. (Amareshwari Sriramadasu via ddas) + + MAPREDUCE-813. Updates Streaming and M/R tutorial documents. + (Corinne Chandel via ddas) + + MAPREDUCE-805. Fixes some deadlocks in the JobTracker due to the fact + the JobTracker lock hierarchy wasn't maintained in some JobInProgress + method calls. (Amar Kamat via ddas) + + MAPREDUCE-799. Fixes so all of the MRUnit self-tests run. + (Aaron Kimball via johan) + + MAPREDUCE-848. Fixes a problem to do with TestCapacityScheduler + failing (Amar Kamat via ddas) + + MAPREDUCE-840. DBInputFormat leaves open transaction. + (Aaron Kimball via tomwhite) + + MAPREDUCE-859. Adds Avro and its dependencies required by Hadoop + common. (Ravi Gummadi via sharad) + + MAPREDUCE-867. Fix ivy conf to look for avro jar from maven repo. + (Giridharan Kesavan) + + MAPREDUCE-877. Added avro as a dependency to contrib ivy settings. + (Tsz Wo (Nicholas) Sze via yhemanth) + + MAPREDUCE-852. In build.xml, remove the Main-Class, which is incorrectly + set in tools, and rename the target "tools-jar" to "tools". (szetszwo) + + MAPREDUCE-773. Sends progress reports for compressed gzip inputs in maps. + Fixes a native direct buffer leak in LineRecordReader classes. + (Hong Tang and ddas) + + MAPREDUCE-832. Reduce number of warning messages printed when + deprecated memory variables are used. (Rahul Kumar Singh via yhemanth) + + MAPREDUCE-745. Fixes a testcase problem to do with generation of JobTracker + IDs. (Amar Kamat via ddas) + + MAPREDUCE-834. Enables memory management on tasktrackers when old + memory management parameters are used in configuration. + (Sreekanth Ramakrishnan via yhemanth) + + MAPREDUCE-818. Fixes Counters#getGroup API. (Amareshwari Sriramadasu + via sharad) + + MAPREDUCE-807. Handles the AccessControlException during the deletion of + mapred.system.dir in the JobTracker. The JobTracker will bail out if it + encounters such an exception. (Amar Kamat via ddas) + + MAPREDUCE-430. Fix a bug related to task getting stuck in case of + OOM error. (Amar Kamat via ddas) + + MAPREDUCE-871. Fix ownership of Job/Task local files to have correct + group ownership according to the egid of the tasktracker. + (Vinod Kumar Vavilapalli via yhemanth) + + MAPREDUCE-911. Fix a bug in TestTaskFail related to speculative + execution. (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-687. Fix an assertion in TestMiniMRMapRedDebugScript. + (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-924. Fixes the TestPipes testcase to use Tool. + (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-903. Add Avro jar to eclipse classpath. + (Philip Zeyliger via tomwhite) + + MAPREDUCE-943. Removes a testcase in TestNodeRefresh that doesn't make + sense in the new Job recovery model. (Amar Kamat via ddas) + + MAPREDUCE-764. TypedBytesInput's readRaw() does not preserve custom type + codes. (Klaas Bosteels via tomwhite) + + HADOOP-6243. Fixes a NullPointerException in handling deprecated keys. + (Sreekanth Ramakrishnan via yhemanth) + + MAPREDUCE-968. NPE in distcp encountered when placing _logs directory on + S3FileSystem. (Aaron Kimball via tomwhite) + + MAPREDUCE-826. harchive doesn't use ToolRunner / harchive returns 0 even + if the job fails with exception (koji Noguchi via mahadev) + + MAPREDUCE-839. unit test TestMiniMRChildTask fails on mac os-x (hong tang + via mahadev) + + MAPREDUCE-112. Add counters for reduce input, output records to the new API. + (Jothi Padmanabhan via cdouglas) + + MAPREDUCE-648. Fix two distcp bugs: (1) it should not launch a job if all + src paths are directories, and (2) it does not skip copying when updating + a single file. (Ravi Gummadi via szetszwo) + + MAPREDUCE-946. Fix a regression in LineRecordReader where the + maxBytesToConsume parameter is not set correctly. (cdouglas) + + MAPREDUCE-977. Missing jackson jars from Eclipse template. (tomwhite) + + MAPREDUCE-988. Fix a packaging issue in the contrib modules. (Hong Tang via + cdouglas) + + MAPREDUCE-971. distcp does not always remove distcp.tmp.dir. (Aaron Kimball + via tomwhite) + + MAPREDUCE-995. Fix a bug in JobHistory where tasks completing after the job + is closed cause a NPE. (Jothi Padmanabhan via cdouglas) + + MAPREDUCE-953. Fix QueueManager to dump queue configuration in JSON format. + (V.V. Chaitanya Krishna via yhemanth) + + MAPREDUCE-645. Prevent distcp from running a job when the destination is a + file, but the source is not. (Ravi Gummadi via cdouglas) + + MAPREDUCE-1002. Flushed writer in JobQueueClient so queue information is + printed correctly. (V.V. Chaitanya Krishna via yhemanth) + + MAPREDUCE-1003. Fix compilation problem in eclipse plugin when + eclipse.home is set. (Ravi Gummadi via yhemanth) + + MAPREDUCE-941. Vaidya script fails on Solaris. (Chad Metcalf + via tomwhite) + + MAPREDUCE-912. Add and standardize Apache license headers. (Chad Metcalf + via cdouglas) + + MAPREDUCE-1022. Fix compilation of vertica testcases. (Vinod Kumar + Vavilapalli via acmurthy) + + MAPREDUCE-1000. Handle corrupt history files in JobHistory.initDone(). + (Jothi Padmanabhan via sharad) + + MAPREDUCE-1028. Fixed number of slots occupied by cleanup tasks to one + irrespective of slot size for the job. + (Ravi Gummadi via yhemanth) + + MAPREDUCE-964. Fixed start and finish times of TaskStatus to be + consistent, thereby fixing inconsistencies in metering tasks. + (Sreekanth Ramakrishnan via yhemanth) + + MAPREDUCE-1076. Deprecate ClusterStatus and add javadoc in ClusterMetrics. + (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-979. Fixed JobConf APIs related to memory parameters to return + values of new configuration variables when deprecated variables are + disabled. (Sreekanth Ramakrishnan via yhemanth) + + MAPREDUCE-1030. Modified scheduling algorithm to return a map and reduce + task per heartbeat in the capacity scheduler. + (Rahul Kumar Singh via yhemanth) + + MAPREDUCE-1071. Use DataInputStream rather than FSDataInputStream in the + JobHistory EventReader. (Hong Tang via cdouglas) + + MAPREDUCE-986. Fix Rumen to work with truncated task lines. (Dick King via + cdouglas) + + MAPREDUCE-1029. Fix failing TestCopyFiles by restoring the unzipping of + HDFS webapps from the hdfs jar. (Aaron Kimball and Jothi Padmanabhan via + cdouglas) + + MAPREDUCE-769. Make findbugs and javac warnings to zero. + (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-1104. Initialize RecoveryManager in JobTracker cstr called by + Mumak. (Hong Tang via cdouglas) + + MAPREDUCE-1061. Add unit test validating byte specifications for gridmix + jobs. (cdouglas) + + MAPREDUCE-1077. Fix Rumen so that truncated tasks do not mark the job as + successful. (Dick King via cdouglas) + + MAPREDUCE-1041. Make TaskInProgress::taskStatuses map package-private. + (Jothi Padmanabhan via cdouglas) + + MAPREDUCE-1070. Prevent a deadlock in the fair scheduler servlet. + (Todd Lipcon via cdouglas) + + MAPREDUCE-1086. Setup Hadoop logging environment for tasks to point to + task related parameters. (Ravi Gummadi via yhemanth) + + MAPREDUCE-1105. Remove max limit configuration in capacity scheduler in + favor of max capacity percentage thus allowing the limit to go over + queue capacity. (Rahul Kumar Singh via yhemanth) + + MAPREDUCE-1016. Make the job history log format JSON. (cutting) + + MAPREDUCE-1038. Weave Mumak aspects only if related files have changed. + (Aaron Kimball via cdouglas) + + MAPREDUCE-1163. Remove unused, hard-coded paths from libhdfs. (Allen + Wittenauer via cdouglas) + + MAPREDUCE-962. Fix a NullPointerException while killing task process + trees. (Ravi Gummadi via yhemanth) + + MAPREDUCE-1177. Correct setup/cleanup inversion in + JobTracker::getTaskReports. (Vinod Kumar Vavilapalli via cdouglas) + + MAPREDUCE-1178. Fix ClassCastException in MultipleInputs by adding + a DelegatingRecordReader. (Amareshwari Sriramadasu and Jay Booth + via sharad) + + MAPREDUCE-1068. Fix streaming job to show proper message if file is + is not present. (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-1147. Add map output counters to new API. (Amar Kamat via + cdouglas) + + MAPREDUCE-915. The debug scripts are run as the job user. (ddas) + + MAPREDUCE-1007. Fix NPE in CapacityTaskScheduler.getJobs(). + (V.V.Chaitanya Krishna via sharad) + + MAPREDUCE-28. Refactor TestQueueManager and fix default ACLs. + (V.V.Chaitanya Krishna and Rahul K Singh via sharad) + + MAPREDUCE-1182. Fix overflow in reduce causing allocations to exceed the + configured threshold. (cdouglas) + + MAPREDUCE-1239. Fix contrib components build dependencies. + (Giridharan Kesavan and omalley) + + MAPREDUCE-787. Fix JobSubmitter to honor user given symlink path. + (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-1249. Update config default value for socket read timeout to + match code default. (Amareshwari Sriramadasu via cdouglas) + + MAPREDUCE-1161. Remove ineffective synchronization in NotificationTestCase. + (Owen O'Malley via cdouglas) + + MAPREDUCE-1244. Fix eclipse-plugin's build dependencies. (gkesavan) + + MAPREDUCE-1075. Fix JobTracker to not throw an NPE for a non-existent + queue. (V.V.Chaitanya Krishna via yhemanth) + + MAPREDUCE-754. Fix NPE in expiry thread when a TT is lost. (Amar Kamat + via sharad) + + MAPREDUCE-1074. Document Reducer mark/reset functionality. (Jothi + Padmanabhan via cdouglas) + + MAPREDUCE-1267. Fix typo in mapred-default.xml. (Todd Lipcon via cdouglas) + + MAPREDUCE-952. Remove inadvertently reintroduced Task.Counter enum. (Jothi + Padmanabhan via cdouglas) + + MAPREDUCE-1230. Fix handling of null records in VerticaInputFormat. (Omer + Trajman via cdouglas) + + MAPREDUCE-1171. Allow shuffle retries and read-error reporting to be + configurable. (Amareshwari Sriramadasu via acmurthy) + + MAPREDUCE-879. Fix broken unit test TestTaskTrackerLocalization on MacOS. + (Sreekanth Ramakrishnan via yhemanth) + + MAPREDUCE-1124. Fix imprecise byte counts in Gridmix. (cdouglas) + + MAPREDUCE-1222. Add an option to exclude numeric IP addresses in topologies + processed by Mumak. (Hong Tang via cdouglas) + + MAPREDUCE-1284. Fix fts_open() call in task-controller that was failing + LinuxTaskController unit tests. (Ravi Gummadi via yhemanth) + + MAPREDUCE-1143. Fix running task counters to be updated correctly + when speculative attempts are running for a TIP. + (Rahul Kumar Singh via yhemanth) + + MAPREDUCE-1241. Use a default queue configuration in JobTracker when + mapred-queues.xml is unavailable. (Todd Lipcon via cdouglas) + + MAPREDUCE-1301. Fix set up of permission checking script used in + localization tests. (Amareshwari Sriramadasu via yhemanth) + + MAPREDUCE-1286. Remove quoting from client opts in TaskRunner. (Yuri + Pradkin via cdouglas) + + MAPREDUCE-1059. Use distcp.bytes.per.map when adding sync markers in + distcp. (Aaron Kimball via cdouglas) + + MAPREDUCE-1009. Update forrest documentation describing hierarchical + queues. (Vinod Kumar Vavilapalli via yhemanth) + + MAPREDUCE-1342. Fixed deadlock in global blacklisting of tasktrackers. + (Amareshwari Sriramadasu via acmurthy) + + MAPREDUCE-1316. Fixes a memory leak of TaskInProgress instances in + the jobtracker. (Amar Kamat via yhemanth) + + MAPREDUCE-1359. TypedBytes TestIO doesn't mkdir its test dir first. + (Anatoli Fomenko via cos) + + MAPREDUCE-1314. Correct errant mapreduce.x.mapreduce.x replacements from + bulk change. (Amareshwari Sriramadasu via cdouglas) + + MAPREDUCE-1365. Restore accidentally renamed test in + TestTaskTrackerBloacklisting. (Amareshwari Sriramadasu via cdouglas) + + MAPREDUCE-1406. Fix spelling of JobContext.MAP_COMBINE_MIN_SPILLS. + (cdouglas) + + MAPREDUCE-1369. JUnit tests should never depend on anything in conf + (Anatoli Fomenko via cos) + + MAPREDUCE-1412. Fix timer granularity issue causing failures in + TestTaskTrackerBlacklisting. (cdouglas) + + MAPREDUCE-1448. Respect --config option in Mumak script. (Hong Tang via + cdouglas) + + MAPREDUCE-1251. c++ utils doesn't compile. (Eli Collins via tomwhite) + + MAPREDUCE-1522. FileInputFormat may use the default FileSystem for the + input path. (Tsz Wo (Nicholas), SZE via cdouglas) + + MAPREDUCE-1407. Update javadoc in mapreduce.{Mapper,Reducer} to match + actual usage. (Benoit Sigoure via cdouglas) + + MAPREDUCE-1258. Fix fair scheduler event log not logging job info. + (matei) + + MAPREDUCE-1089. Fix NPE in fair scheduler preemption when tasks are + scheduled but not running. (Todd Lipcon via matei) + + MAPREDUCE-1014. Fix the libraries for common and hdfs. (omalley) + + MAPREDUCE-1111. JT Jetty UI not working if we run mumak.sh + off packaged distribution directory. (hong tang via mahadev) + + MAPREDUCE-1133. Eclipse .classpath template has outdated jar files and is + missing some new ones. (cos) + + MAPREDUCE-1098. Fixed the distributed-cache to not do i/o while holding a + global lock. (Amareshwari Sriramadasu via acmurthy) + + MAPREDUCE-1158. Fix JT running maps and running reduces metrics. + (sharad) + + MAPREDUCE-1160. Reduce verbosity of log lines in some Map/Reduce classes + to avoid filling up jobtracker logs on a busy cluster. + (Ravi Gummadi and Hong Tang via yhemanth) + + MAPREDUCE-1153. Fix tasktracker metrics when trackers are decommissioned. + (sharad) + + MAPREDUCE-1128. Fix MRUnit to prohibit iterating over values twice. (Aaron + Kimball via cdouglas) + + MAPREDUCE-665. Move libhdfs to HDFS subproject. (Eli Collins via dhruba) + + MAPREDUCE-1196. Fix FileOutputCommitter to use the deprecated cleanupJob + api correctly. (acmurthy) + + MAPREDUCE-1244. Fix eclipse-plugin's build dependencies. (gkesavan) + + MAPREDUCE-1140. Fix DistributedCache to not decrement reference counts for + unreferenced files in error conditions. + (Amareshwari Sriramadasu via yhemanth) + + MAPREDUCE-1245. Fix TestFairScheduler failures by instantiating lightweight + Jobtracker. (sharad) + + MAPREDUCE-1260. Update Eclipse configuration to match changes to Ivy + configuration. (Edwin Chan via cos) + + MAPREDUCE-1152. Distinguish between failed and killed tasks in + JobTrackerInstrumentation. (Sharad Agarwal via cdouglas) + + MAPREDUCE-1285. In DistCp.deleteNonexisting(..), get class from the + parameter instead of using FileStatus.class. (Peter Romianowski via + szetszwo) + + MAPREDUCE-1294. Build fails to pull latest hadoop-core-* artifacts (cos) + + MAPREDUCE-1213. TaskTrackers restart is faster because it deletes + distributed cache directory asynchronously. (Zheng Shao via dhruba) + + MAPREDUCE-1265. The task attempt error log prints the name of the + tasktracker machine. (Scott Chen via dhruba) + + MAPREDUCE-1201. ProcfsBasedProcessTree collects CPU usage information. + (Scott Chen via dhruba) + + MAPREDUCE-1326. fi tests don't use fi-site.xml (cos) + + MAPREDUCE-1165. Replace non-portable function name with C99 equivalent. + (Allen Wittenauer via cdouglas) + + MAPREDUCE-1331. Fixes a typo in a testcase (Devaraj Das) + + MAPREDUCE-1293. AutoInputFormat doesn't work with non-default FileSystems. + (Andrew Hitchcock via tomwhite) + + MAPREDUCE-1131. Using profilers other than hprof can cause JobClient to + report job failure. (Aaron Kimball via tomwhite) + + MAPREDUCE-1155. Streaming tests swallow exceptions. + (Todd Lipcon via tomwhite) + + MAPREDUCE-1212. Mapreduce contrib project ivy dependencies are not included + in binary target. (Aaron Kimball via tomwhite) + + MAPREDUCE-1388. Move the HDFS RAID package from HDFS to MAPREDUCE. + (Eli Collins via dhruba) + + MAPREDUCE-1322. Defines default value for staging directory to be user + based fixing a failing streaming test. + (Devaraj Das and Amareshwari Sriramadasu via yhemanth) + + MAPREDUCE-899. Modified LinuxTaskController to check that task-controller + has right permissions and ownership before performing any actions. + (Amareshwari Sriramadasu via yhemanth) + + MAPREDUCE-1443. DBInputFormat can leak connections. + (Aaron Kimball via tomwhite) + + MAPREDUCE-1457. Fixes JobTracker to get the FileSystem object within + getStagingAreaDir within a privileged block. Fixes Child.java to use the + appropriate UGIs while getting the TaskUmbilicalProtocol proxy and + while executing the task. (Jakob Homan via ddas) + + MAPREDUCE-1399. The archive command shows a null error message (nicholas + via mahadev) + + MAPREDUCE-1305. Improve efficiency of distcp -delete. (Peter Romianowski + via cdouglas) + + MAPREDUCE-1474. Update forrest documentation for Hadoop Archives. (Mahadev + Konar via cdouglas) + + MAPREDUCE-1400. Use tr rather than sed to effect literal substitution in + the build script. (Allen Wittenauer via cdouglas) + + MAPREDUCE-1358. Avoid false positives in OutputLogFilter. (Todd Lipcon via + cdouglas) + + MAPREDUCE-1490. Fix a NullPointerException that could occur during + instantiation and initialization of the DistributedRaidFileSystem. + (Rodrigo Schmidt via dhruba) + + MAPREDUCE-1476. Fix the M/R framework to not call commit for special + tasks like job setup/cleanup and task cleanup. + (Amareshwari Sriramadasu via yhemanth) + + MAPREDUCE-1398. Fix TaskLauncher to stop waiting for slots on a TIP that + is killed / failed. + (Amareshwari Sriramadasu via yhemanth) + + MAPREDUCE-1491. The parity files created by the RAID are combined + using Hadoop Archive Files (HAR). (Rodrigo Schmidt via dhruba) + + MAPREDUCE-1378. URL encode link in jobhistory.jsp to avoid errors caused by + unescaped characters. (E. Sammer via cdouglas) + + MAPREDUCE-1519. RaidNode fails to create new parity file + if an older version already exists. (Rodrigo Schmidt via dhruba) + + MAPREDUCE-1537. Fixes a compilation problem in a testcase after commit + HDFS-984. (Jitendra Nath Pandey via ddas) + + MAPREDUCE-1537. The patch makes the job client call the getDelegationToken + only when security is enabled. (Jitendra Nath Pandey via ddas) + + MAPREDUCE-1510. RAID should regenerate parity files if they get deleted. + (Rodrigo Schmidt via dhruba) + + MAPREDUCE-1421. Fix the LinuxTaskController tests failing on trunk after + the commit of MAPREDUCE-1385. (Amareshwari Sriramadasu via vinodkv) + + MAPREDUCE-1520. Fix TestMiniMRLocalFS failure caused by regression in + getting user working dir. (Amareshwari Sriramadasu via cdouglas) + + MAPREDUCE-1512. RAID uses HarFileSystem directly instead of + FileSystem.get (Rodrigo Schmidt via dhruba) + + MAPREDUCE-1435. Fix symlink handling in task work directory when + cleaning up, essentially to avoid following links. + (Ravi Gummadi via yhemanth) + + MAPREDUCE-1518. RaidNode does not run the deletion check on the + directory that stores the parity files. (Rodrigo Schmidt via dhruba) + + MAPREDUCE-1573. TestStreamingAsDifferentUser fails if run as tt_user. + (Ravi Gummadi via vinodkv) + + MAPREDUCE-927. Cleanup of task-logs should happen in TaskTracker instead + of the Child. (Amareshwari Sriramadasu via vinodkv) + + MAPREDUCE-1578. Decouple HadoopArchives vesrion from HarFileSystem version. + (Rodrigo Schmidt via szetszwo) + + MAPREDUCE-1422. Fix cleanup of localized job directory to work if files + with non-deletable permissions are created within it. + (Amar Kamat via yhemanth) + + MAPREDUCE-1306. Randomize the arrival of heartbeat responses in Mumak. + (Tamas Sarlos via cdouglas) + + MAPREDUCE-1579. archive: check and possibly replace the space charater + in source paths. (szetszwo) + + MAPREDUCE-1536. DataDrivenDBInputFormat does not split date columns correctly. + (Aaron Kimball via enis) + + MAPREDUCE-890. After HADOOP-4491, the user who started mapred system is + not able to run job. (Ravi Gummadi via vinodkv) + + MAPREDUCE-1615. Fix compilation of TestSubmitJob. (cdouglas) + + MAPREDUCE-1508. Protect against NPE in TestMultipleLevelCaching. (Aaron + Kimball via cdouglas) + + MAPREDUCE-1497. Suppress spurious findbugs warning about IndexCache + synchronization. (Amareshwari Sriramadasu via cdouglas) + + MAPREDUCE-1420. Fix TestTTResourceReporting failure. (Scott Chen via + cdouglas) + + MAPREDUCE-1480. Correctly initialize child RecordReaders in + CombineFileInputFormat. (Aaron Kimball via cdouglas) + + MAPREDUCE-1348. Fix block forensics packaging. (Tom White via cdouglas) + + MAPREDUCE-1628. HarFileSystem shows incorrect replication numbers and + permissions. (szetszwo via mahadev) + + MAPREDUCE-1602. Fix the error message for the case that src does not + exist. (szetszwo) + + MAPREDUCE-1585. Create Hadoop Archives version 2 with filenames + URL-encoded (rodrigo via mahadev) + + MAPREDUCE-1523. Sometimes rumen trace generator fails to extract the job + finish time. (dick king via mahadev) + + MAPREDUCE-1635. ResourceEstimator does not work after MAPREDUCE-842. + (Amareshwari Sriramadasu via vinodkv) + + MAPREDUCE-889. binary communication formats added to Streaming by + HADOOP-1722 should be documented. (Klaas Bosteels via tomwhite) + + MAPREDUCE-1031. ant tar target doens't seem to compile tests in contrib + projects. (Aaron Kimball via tomwhite) + + MAPREDUCE-1692. Removed unused testcase TestStreamedMerge. + (Sreekanth Ramakrishnan and Amareshwari Sriramadasu via yhemanth) + + MAPREDUCE-1062. Fix ReliabilityTest to work with retired jobs. (Sreekanth + Ramakrishnan via cdouglas) + + MAPREDUCE-1409. IOExceptions thrown from FIleOutputCommitter::abortTask + should cause the task to fail. (Amareshwari Sriramadasu via cdouglas) + + MAPREDUCE-1695. Include capacity scheduler in findbugs and javadoc-dev + targets and also fix existing warnings. (Hong Tang via yhemanth) + + MAPREDUCE-1494. Ensure TestJobDirCleanup verifies the correct paths. + (Amareshwari Sriramadasu via cdouglas) + + MAPREDUCE-1622. Include missing slf4j dependencies. (cdouglas) + + MAPREDUCE-1515. Accept java5.home from build.properties, not only from the + command line when building forrest docs. (Al Thompson via cdouglas) + + MAPREDUCE-1618. Add missing javadoc to JobStatus::*JobAcls. (Amareshwari + Sriramadasu via cdouglas) + + MAPREDUCE-1219. Remove job level metrics from jobtracker metrics to ease + undue load on jobtracker. (Sreekanth Ramakrishnan via sharad) + + MAPREDUCE-1604. Add Forrest documentation for Job ACLs. + (Amareshwari Sriramadasu via yhemanth) + + MAPREDUCE-1705. Archiving and Purging of HDFS parity files should + handle globbed policies accurately. (Rodrigo Schmidt via dhruba) + + MAPREDUCE-1612. job conf file is not accessible from job history web page. + (Ravi Gummadi and Sreekanth Ramakrishnan via vinodkv) + + MAPREDUCE-1397. NullPointerException observed during task failures. + (Amareshwari Sriramadasu via vinodkv) + + MAPREDUCE-1728. Oracle timezone strings do not match Java. + (Aaron Kimball via tomwhite) + + MAPREDUCE-1609. TaskTracker.localizeJob should not set permissions on + job log directory recursively. (Amareshwari Sriramadasu via vinodkv) + + MAPREDUCE-1657. After task logs directory is deleted, tasklog servlet + displays wrong error message about job ACLs. (Ravi Gummadi via vinodkv) + + MAPREDUCE-1727. TestJobACLs fails after HADOOP-6686. (Ravi Gummadi via vinodkv) + + MAPREDUCE-1611. Refresh nodes and refresh queues doesnt work with service + authorization enabled. (Amar Kamat via vinodkv) + + MAPREDUCE-1276. Correct flaws in the shuffle related to connection setup + and failure attribution. (Amareshwari Sriramadasu via cdouglas) + + MAPREDUCE-1372. ConcurrentModificationException in JobInProgress. + (Dick King and Amareshwari Sriramadasu via tomwhite) + + MAPREDUCE-118. Fix Job.getJobID(). (Amareshwari Sriramadasu via sharad) + + MAPREDUCE-913. TaskRunner crashes with NPE resulting in held up slots, + UNINITIALIZED tasks and hung TaskTracker. (Amareshwari Sriramadasu and + Sreekanth Ramakrishnan via vinodkv) + + MAPREDUCE-1725. Fix MapReduce API incompatibilities between 0.20 and 0.21. + (tomwhite) + + MAPREDUCE-1606. TestJobACLs may timeout as there are no slots for launching + JOB_CLEANUP task. (Ravi Gummadi via vinodkv) + + MAPREDUCE-1765. Correct streaming documentation for StreamXmlRecordReader. + (Corinne Chandel via amareshwari) + + MAPREDUCE-1880. Fix BigDecimal.divide(..) in the pi example. (szetszwo) + + MAPREDUCE-1885. Revert FileSystem create method that takes CreateFlags + (MapReduce part of HADOOP-6826). (Ravi Gummadi via tomwhite) + + MAPREDUCE-1870. Harmonize MapReduce JAR library versions with Common and + HDFS. (tomwhite) + + MAPREDUCE-1791. Remote cluster control functionality needs JavaDocs + improvement (Konstantin Boudnik) + + MAPREDUCE-1942. 'compile-fault-inject' should never be called directly. + (Konstantin Boudnik) + + MAPREDUCE-1876. Fixes TaskAttemptStartedEvent to correctly log event type + for all task types. (Amar Kamat via amareshwari) + + MAPREDUCE-1926. MapReduce distribution is missing build-utils.xml. + (tomwhite) + + MAPREDUCE-2012. Some contrib tests fail in branch 0.21 and trunk. + (Amareshwari Sriramadasu via tomwhite) + + MAPREDUCE-1980. Fixes TaskAttemptUnsuccessfulCompletionEvent and + TaskAttemptFinishedEvent to correctly log event type for all task types. + (Amar Kamat via amareshwari) + + MAPREDUCE-1856. Extract a subset of tests for smoke (DOA) validation (cos) + diff --git a/aarch64/share/doc/hadoop/mapreduce/LICENSE.txt b/aarch64/share/doc/hadoop/mapreduce/LICENSE.txt new file mode 100644 index 0000000..59bcdbc --- /dev/null +++ b/aarch64/share/doc/hadoop/mapreduce/LICENSE.txt @@ -0,0 +1,244 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +APACHE HADOOP SUBCOMPONENTS: + +The Apache Hadoop project contains subcomponents with separate copyright +notices and license terms. Your use of the source code for the these +subcomponents is subject to the terms and conditions of the following +licenses. + +For the org.apache.hadoop.util.bloom.* classes: + +/** + * + * Copyright (c) 2005, European Commission project OneLab under contract + * 034819 (http://www.one-lab.org) + * All rights reserved. + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * - Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * - Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the distribution. + * - Neither the name of the University Catholique de Louvain - UCL + * nor the names of its contributors may be used to endorse or + * promote products derived from this software without specific prior + * written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS + * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE + * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, + * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, + * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN + * ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ diff --git a/aarch64/share/doc/hadoop/mapreduce/NOTICE.txt b/aarch64/share/doc/hadoop/mapreduce/NOTICE.txt new file mode 100644 index 0000000..62fc581 --- /dev/null +++ b/aarch64/share/doc/hadoop/mapreduce/NOTICE.txt @@ -0,0 +1,2 @@ +This product includes software developed by The Apache Software +Foundation (http://www.apache.org/). |