Skip to content

Commit

Permalink
HW3 added snappy support
Browse files Browse the repository at this point in the history
it is used to be error during docker image creation, so I abandoned it and used HDP build to get Snappy
see sequenceiq/hadoop-docker#47, second comment for more info and be ware =)
  • Loading branch information
Hokan-Ashir committed Jul 17, 2016
1 parent 0e2b5e1 commit 8dff6a9
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions Homework3/src/main/java/ru/hokan/custom/IpBytesCounter.java
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.compress.CompressionCodec;
import org.apache.hadoop.io.compress.SnappyCodec;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.NLineInputFormat;
Expand Down Expand Up @@ -32,6 +34,8 @@ public static void main(String[] args) throws Exception {
public int run(String[] args) throws Exception {
Configuration config = new Configuration();
config.set("mapred.textoutputformat.separator", ",");
config.setBoolean("mapred.output.compress", true);
config.setClass("mapred.output.compression.codec", SnappyCodec.class, CompressionCodec.class);
config.setInt(NLineInputFormat.LINES_PER_MAP, 1000);

Job job = Job.getInstance(config);
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 8dff6a9

Please sign in to comment.