Code

Syncing up code moves from IA#6

Code

Merged

anjackson merged 29 commits intoiipc:masteriipc/webarchive-commons:masterfrom ukwa:masterukwa/webarchive-commons:masterCopy head branch name to clipboardDec 12, 2013

+20,026-331

Commits

Commits on Oct 3, 2013

• ZipNum: fixes: cached locations: don't throw RunTimeException if required (handled later), add a locCacheMaxDuration to cache only if below threshold

  ikreymer

  committed
  28c2995

  Copy full SHA for 28c2995

Commits on Oct 4, 2013

• CDX: Add CloseableCompositeIterator which iterates in sequence, optimization for zipnum clusters to be loaded sequentially when only looking for last line

  ikreymer

  committed
  41d3a58

  Copy full SHA for 41d3a58

Commits on Oct 6, 2013

• ZIP & SLR improvements: store connectedUrl immediately on connection, print connectedUrl on error, skip cached url on fallback to location list

  ikreymer

  committed
  b4f639d

  Copy full SHA for b4f639d

Commits on Oct 14, 2013

• ADD: CloseableIteratorWrapper utility class for wrapping regular iterators

  Show description for 612cdeb
  ikreymer

  committed
  612cdeb

  Copy full SHA for 612cdeb

• ZipNumBlock Loader: * add info on which shard failed to RuntimIOException

  Show description for c5dbc67
  ikreymer

  committed
  c5dbc67

  Copy full SHA for c5dbc67

• ZIPNUMLoader: attempt better error msgs by propagating full error in exception

  ikreymer

  committed
  1fc6019

  Copy full SHA for 1fc6019

Commits on Oct 15, 2013

• FIX: ApacheHttp31SLR save the connected url even on error!

  Show description for 936fb49
  ikreymer

  committed
  936fb49

  Copy full SHA for 936fb49

Commits on Oct 17, 2013

• FIX: ApacheSLR: turn off cookies when using manual cookie

  ikreymer

  committed
  0474441

  Copy full SHA for 0474441

• HttpSLR: add optional error header which can be saved

  Show description for f82aead
  ikreymer

  committed
  f82aead

  Copy full SHA for f82aead

Commits on Oct 18, 2013

• MultiCDXInputSource: Make comparator public

  ikreymer

  committed
  647adec

  Copy full SHA for 647adec

• FIX: MultiCDXInputSource: add optimization for output lazy initing of sequential cluster load

  ikreymer

  committed
  7d4fbf4

  Copy full SHA for 7d4fbf4

Commits on Oct 23, 2013

• FEATURE: add getTotalCount() to cdx input sources

  ikreymer

  committed
  2d2d30e

  Copy full SHA for 2d2d30e

Commits on Oct 24, 2013

• log RuntimeIOException

  ikreymer

  committed
  baf1ad8

  Copy full SHA for baf1ad8

Commits on Nov 1, 2013

• add ThreadLocalHttpConnectionManager (from heritrix-commons)

  Show description for 154386c
  ikreymer

  committed
  154386c

  Copy full SHA for 154386c

• fix path for ThreadLocalHttpConnectionManager

  ikreymer

  committed
  cd43101

  Copy full SHA for cd43101

• fix package

  ikreymer

  committed
  1c58cba

  Copy full SHA for 1c58cba

• move ThreadLocalHttpConnectionManager to original package org.archive.httpclient

  ikreymer

  committed
  da37977

  Copy full SHA for da37977

Commits on Nov 2, 2013

• FEATURE: add support for httpcore 4.3 for SeekableLineReader!

  ikreymer

  committed
  68dab84

  Copy full SHA for 68dab84

• apach43SLR: better reading of entire buffer

  ikreymer

  committed
  54bdd7d

  Copy full SHA for 54bdd7d

Commits on Nov 4, 2013

• canonicalizer: add ExtractRule and RewriteRule

  ikreymer

  committed
  1e8c1af

  Copy full SHA for 1e8c1af

Commits on Nov 16, 2013

• FIX: add cdxlinefactory doesn't require custom format

  Show description for 0ebbad2
  ikreymer

  committed
  0ebbad2

  Copy full SHA for 0ebbad2

Commits on Nov 23, 2013

• Extract outlinks/hopinfo from warc/metadata records

  vinaygoel

  committed
  5077ad8

  Copy full SHA for 5077ad8

Commits on Nov 25, 2013

• reference pom.xml for building with CDH4

  vinaygoel

  committed
  329ff22

  Copy full SHA for 329ff22

Commits on Dec 6, 2013

• moving org.archive.net.PublicSuffixes to ia-web-commons

  nlevitt

  committed
  fc24be8

  Copy full SHA for fc24be8

• moving a bunch of stuff from heritrix-commons to ia-web-commons so that wayback doesn't have to depend on heritrix-commons

  nlevitt

  committed
  a54dd8e

  Copy full SHA for a54dd8e

• Merge branch 'master' of github.com:internetarchive/ia-web-commons

  Show description for 908c7a1
  anjackson

  committed
  908c7a1

  Copy full SHA for 908c7a1

Commits on Dec 11, 2013

• Removed older PublicSuffix code.

  anjackson

  committed
  a732a9e

  Copy full SHA for a732a9e

Commits on Dec 12, 2013

• Merge branch 'master' of github.com:iipc/iipc-web-commons

  anjackson

  committed
  4c195ca

  Copy full SHA for 4c195ca

• Added link to parent project.

  anjackson

  committed
  7653cc0

  Copy full SHA for 7653cc0