Release 62 Tropical Parula

Fixes an incompatibility issue between r61's EmrEtlRunner and some older Elastic Beanstalk configurations, plus some other EmrEtlRunner improvements.

Common

• Updated vagrant up to work with latest Peru version (#1475)

Trackers

• Ruby Tracker: bumped git submodule to 0.4.1 (#1488)
• Python Tracker: bumped git submodule to 0.6.0 (#1487)
• PHP Tracker: bumped git submodule to 0.2.1 (#1486)
• JavaScript Tracker: bumped git submodule to 2.3.0 (#1485)
• Java Tracker: bumped git submodule to 0.7.0 (#1484)
• Objective-C Tracker: renamed from iOS Tracker and bump git submodule to 0.3.2 (#1483)

EmrEtlRunner

• Bumped to 0.13.0
• Fixed copy to staging for Tomcat7 logs with hyphen after .txt (#1480)
• Added missing :archive: in BucketHash (#1475)
• Added support for custom bootstrap actions, thanks @danisola! (#1405)
• Removed time_diff as a dependency (#1352)
• Fixed breaking get_assets spec (#1287)
• Now tolerating more exception types in EmrJob's wait_for (#358)
• Bumped Contracts to 0.7 (#1498)
• Moved include Contracts into classes and modules (#1438)

Release 61 Pygmy Parrot

The major additions are:

1. You can now parse Amazon CloudFront access logs using Snowplow
2. The latest Clojure Collector version supports Tomcat 8 and CORS for cross-domain POST from JavaScript and ActionScript
3. EmrEtlRunner's failure handling and Clojure Collector log handling have been improved

• Blog post

Common

• Bumped VERSION file to r61-pygmy-parrot
• Added Gradle to up.playbooks (#1270)
• Added .travis.yml file and Travis button to repo (#1359)
• Added Release button to README (#1428)
• Added License button to README (#1427)

Clojure Collector

• Bumped to 1.0.0
• Updated access-valve to depend on Tomcat 8 classes (#1203)
• Updated .ebextensions to depend on Tomcat 8 (#1202)
• Added ability to disable third-party cookies (#1362)
• Added CORS support (#1146)
• Added CORS-style support for ActionScript3 Tracker (#1330)
• Added support for /:vendor/:version to HEAD (#1166)
• Now using UTF-8 for character encoding throughout (#1354)

Scala Common Enrich

• Bumped to 0.12.0
• Updated SnowplowAdapter to accept "charset=UTF-8" (#1424)
• Base64 decoding does not specify UTF-8 charset (#1403)
• Removed incorrect extra layer of URL decoding from non-Bas64-encoded JSONs (#1396)
• Added support for ti_nm for transaction item name as well as ti_na (#1401)
• Added CloudfrontAccessLogAdapter (#1282)
• Made timestamp field of CollectorPayload an Option (#1417)

Scala Hadoop Enrich

• Bumped to 0.13.0
• Bumped Scala Common Enrich to 0.12.0 (#1395)
• Added test for non-Base64-encoded JSON (#1394)
• Updated tests to include Unicode (#1390)
• Added integration test for CloudfrontAccessLogAdapter (#1423)

Scala Hadoop Bad Rows

• Removed .travis.yml (#1382)

EmrEtlRunner

• Bumped to 0.12.0
• Now appending region name to Clojure Collector log files (#1379)
• Added support for moving and archiving timestamped Clojure Collector log files (#1400)
• Now appending rather than prepending instance names to Clojure Collector log files (#1404)
• Changed Clojure Collector log timestamp format to match CloudFront logs (#1398)
• Added dedicated return code for no files to process (#1397)
• Now allowing tsv// and json// as :etl:collector_format (#1284)
• Now performing S3DistCp from processing for tsv/com.amazon.aws.cloudfront/* (#1431)
• Added output directory empty check prior to staging step (#1151)

StorageLoader

• Updated shell script to only run StorageLoader if EmrEtlRunner found files (#1399)
• Wrote JSON Path file for a com.snowplowanalytics.snowplow/flash_context (#1305)
• Wrote JSON Path file for a com.snowplowanalytics.snowplow/timing event (#1388)
• Wrote JSON Path file for a com.amazon.aws.cloudfront/wd_access_log event (#1285)
• Wrote JSON Path file for a com.google.analytics/cookies context (#1409)
• Wrote JSON Path file for a com.snowplowanalytics.snowplow/desktop_context (#1421)

Redshift

• Added Redshift DDL for a com.snowplowanalytics.snowplow/timing event (#1387)
• Added Redshift DDL for a com.snowplowanalytics.snowplow/flash_context (#1304)
• Added Redshift DDL for a com.amazon.aws.cloudfront/wd_access_log event (#1286)
• Added Redshift DDL for a com.google.analytics/cookies context (#1408)
• Added Redshift DDL for a com.snowplowanalytics.snowplow/desktop_context (#1420)

Release 60 Bee Hummingbird

New functionality in the Kinesis pipeline, and some bug fixes in the Hadoop pipeline.

• Blog post

Common

• Added VERSION file in root to assist vagrant push (#1293)
• Added vagrant push scripting to publish Kinesis apps (#1288)
• Added lzo.yml to up.playbooks (#1325)

Thrift Raw Event

• Bumped Thrift version to 0.9.1 (#1225)
• Added collector-payload-1 and schema-sniffer-1 (#1322)
• Created a subproject for each Thrift class (#1298)
• Updated README and project description to reflect new structure (#1300)
• Renamed to thrift-schemas (#1299)

Scala Stream Collector

• Bumped to 0.3.0
• Started sending CollectorPayloads instead of SnowplowRawEvents (#1226)
• Added support for POST requests (#187)
• Added support for any {api-vendor}/{api-version} for GET and POST (#652)
• Stopped decoding URLs (#1217)
• Changed 1x1 pixel response to use a stable GIF (#1260)
• Renamed default.conf to config.hocon.sample (#1243)
• Started using ThreadLocal to handle Thrift serialization, thanks @denismo and @pkallos! (#1254)
  Scala Stream Collector: added healthcheck for load balancers, thanks @duncan! (#1360)

EmrEtlRunner

• Bumped to 0.11.0
• Added "thrift" collector format (#1301)
• Implemented time_diff manually (#1310)
• Fixed failure reporting when jobflow step(s) created_at is nil (#1351)

Scala Common Enrich

• Bumped to 0.11.0
• Added schema-sniffer-1 and collector-payload-1 dependencies (#1296)
• Bumped user-agent-utils version to 1.14 (#1224)
• Changed EnrichedEvent field name to ip_organization (#1145)
• Changed "thrift" to "thrift-raw" in Loader object (#1302)
• Added tests for getLoader function (#558)

Scala Hadoop Enrich

• Bumped to 0.12.0
• Bumped Scala Common Enrich to 0.11.0 (#1294)
• Added collector-payload-1 and snowplow-thrift-raw-event as test dependencies (#1248)
• Added support for processing Thrift raw events, thanks @pkallos! (#538)
• Added tests to Hadoop Enrich for processing Thrift raw events (#559)

Scala Kinesis Enrich

• Bumped to 0.3.0
• Bumped Scala Common Enrich to 0.11.0 (#1295)
• Renamed default.conf to config.hocon.sample (#1242)

Kinesis Elasticsearch Sink

• Added LICENSE-2.0.txt (#1329)

Kinesis LZO S3 Sink

• Added. Version 0.1.0, thanks @pkallos! (#1016)

Snowplow v0.9.14

Additional webhooks (Mandrill, PagerDuty and Pingdom), bug fixes and a new Vagrant setup.

• Blog post

Common

• Added dedicated Vagrant setup (#1266)
• Added Quickstart section to README (#1268)
• Added script to sync region-specific Snowplow Hosted Assets buckets (#1269)

CloudFront Collector

• Replaced 1x1 pixel with stable GIF (#1259)

Clojure Collector

• Bumped to 0.9.1
• Increased Tomcat's HTTP header tolerance to 64kB (#1249)
• Changed 1x1 pixel response to use a stable GIF (#1258)

EmrEtlRunner

• Bumped to 0.10.0
• Rremoved hyphen from the pattern match for Clojure Collector logs (#1194)
• On job failure, log overall jobflow and individual step statuses (#1153)

Scala Common Enrich

• Bumped to 0.10.0
• Bumped Scala Iglu Client to 0.2.0 (#1222)
• Updated SnowplowAdapter to accept payload_data versions above 1-0-0 (#1220)
• Updated SnowplowAdapter to make charset=utf-8 optional (#1257)
• Added Adapter to pre-process Pingdom events (#1164)
• Added Adapter to pre-process PagerDuty events (#1158)
• Added Adapter to pre-process Mandrill events (#1061)

Scala Hadoop Enrich

• Bumped to 0.11.0
• Bumped Scala Common Enrich to 0.10.0 (#1223)
• Added test job for PingdomAdapter (#1176)
• Added test job for PagerdutyAdapter (#1175)
• Added test job for MandrillAdapter (#1171)
• Added test job for more relaxed payload_data schema matching (#1235)

Scala Hadoop Shred

• Bumped to 0.3.0
• Bumped Scala Common Enrich to 0.10.0 (#1236)
• Bumped Iglu Scala Client to 0.2.0 (#1230)
• Loosened match criteria for unstructured events and contexts (#1231)

StorageLoader

• Wrote JSON Path file for com.pingdom/incident_notify_of_close event (#1182)
• Wrote JSON Path file for com.pingdom/incident_assign event (#1181)
• Wrote JSON Path file for com.pingdom/incident_notify_user event (#1251)
• Wrote JSON Path file for com.pagerduty/incident event (#1177)
• Wrote JSON Path file for com.mandrill/message_sent event (#1059)
• Wrote JSON Path file for com.mandrill/message_bounced event (#1058)
• Wrote JSON Path file for com.mandrill/message_opened event (#1057)
• Wrote JSON Path file for com.mandrill/message_marked_as_spam event (#1056)
• Wrote JSON Path file for com.mandrill/message_delayed event (#1055)
• Wrote JSON Path file for com.mandrill/message_soft_bounced event (#1054)
• Wrote JSON Path file for com.mandrill/message_clicked event (#1053)
• Wrote JSON Path file for com.mandrill/message_rejected event (#1052)
• Wrote JSON Path file for com.mandrill/recipient_unsubscribed event (#1051)

Redshift

• Added Redshift DDL for a com.pingdom/incident_notify_of_close event (#1180)
• Added Redshift DDL for a com.pingdom/incident_assign event (#1179)
• Added Redshift DDL for a com.pingdom/incident_notify_user (#1252)
• Added Redshift DDL for a com.pagerduty/incident event (#1178)
• Added Redshift DDL for a com.mandrill/message_sent event (#1050)
• Added Redshift DDL for a com.mandrill/message_bounced event (#1049)
• Added Redshift DDL for a com.mandrill/message_opened event (#1048)
• Added Redshift DDL for a com.mandrill/message_marked_as_spam event (#1047)
• Added Redshift DDL for a com.mandrill/message_delayed event (#1046)
• Added Redshift DDL for a com.mandrill/message_soft_bounced event (#1045)
• Added Redshift DDL for a com.mandrill/message_clicked event (#1044)
• Added Redshift DDL for a com.mandrill/message_rejected event (#1043)
• Added Redshift DDL for a com.mandrill/recipient_unsubscribed event (#1042)
• Removed trailing commas from com.mailchimp SQL table definitions (#1174)

Snowplow v0.9.13

Important bug fix in URI parsing code.

• Blog post

Scala Common Enrich

• Bumped to 0.9.1
• Added error handling for Netaporter URI parsing (#1216)

Scala Kinesis Enrich

• Bumped to 0.2.1
• Bumped Scala Common Enrich to 0.9.1
• Fixed conflict with Specs2 version, thanks @knservis! (#1213)

Scala Hadoop Enrich

• Bumped to 0.10.1
• Bumped Scala Common Enrich to 0.9.1

General

• Deleted test-file in repository root (#1219)

Snowplow v0.9.12

Improved Kinesis support, including the new Kinesis Elasticsearch Sink.

• Blog post

Scala Stream Collector

• Bumped to 0.2.0
• Changed organization to "com.snowplowanalytics" (#1168)
• Made the --config option mandatory (#1128)
• Added ability to set AWS credentials from environment variables (#1116)
• Now enforcing Java 7 for compilation (#1068)
• Increased request character limit to 32768 (#987)
• Improved performance by using Future, thanks @pkallos! (#580)

Scala Stream Collector, Scala Kinesis Enrich

• Made endpoint configurable, thanks @sambo1972! (#978)
• Added support for IAM roles, thanks @pkallos! (#534)
• Replaced stream list with describe to tighten permissions, thanks @pkallos! (#535)

Scala Kinesis Enrich

• Bumped to 0.2.0
• Bumped Scala Common Enrich to 0.9.0
• Changed organization to "com.snowplowanalytics" (#1167)
• Made the --config option mandatory (#1126)
• Updated instructions in README (#1125)
• Added ability to set AWS credentials from environment variables (#1117)
• Now enforcing Java 7 for compilation (#1067)
• Replaced printlns with Java Logger (#521)
• Started sending bad records to a separate stream (#463)
• Added page_url and page_referrer back into enrichment output (#686)
• Stopped opening a new file for each enriched event, thanks @pkallos! (#714)

Scala Common Enrich

• Bumped to 0.9.0
• Added BadRow from Scala Hadoop Enrich (#1118)
• Added ability to override collector-set nuid with tracker-set tnuid (#1095)
• Made URI parsing more permissive using NetAPorter's URI library, thanks @rupeshmane! (#1172)

Scala Hadoop Enrich

• Bumped to 0.10.0
• Bumped Scala Common Enrich to 0.9.0
• Moved BadRow into Scala Common Enrich (#1119)
• Updated README with new Snowplow capitalization (#1127)

Kinesis Elasticsearch Sink

• Added. Version 0.1.0

Snowplow v0.9.11

Our initial webhook support.

• Blog post

Clojure Collector

• Bumped to 0.9.0
• Added support for /:vendor/:version to GET (#1131)

Scala Common Enrich

• Bumped to 0.8.0
• Bumped json4s to 3.2.11 (#1141)
• Bumped Scala Iglu Client to 0.1.1 (#1140)
• Removed check that POST request has body and content-type (#1132)
• Moved payload API detection into CollectorApi.parse (#1113)
• Fixed bug in CljTomcatLoader expecting request body to be "_" instead of "-" (#1112)
• Added Adapter to pre-process CallRail events (#1108)
• Added Adapter to pre-process MailChimp events (#1086)
• Added Adapter to pre-process Iglu-compatible events (#1060)

Scala Hadoop Enrich

• Bumped to 0.9.0
• Added job test for unrecognized api name/version (#1115)
• Updated DiscardableCfLinesSpec given /not-ice.png is no longer discarded (#1114)
• Added test job for MailchimpAdapter (#1159)
• Added test job for CallrailAdapter (#1160)

Redshift

• Removed not null constraint on change_form's value column (#1162)
• Added Redshift DDL for a com.callrail/call_complete event (#1110)
• Added Redshift DDL for a com.mailchimp/campaign_sending_status event (#1085)
• Added Redshift DDL for a com.mailchimp/cleaned_email event (#1084)
• Added Redshift DDL for a com.mailchimp/email_address_change event (#1083)
• Added Redshift DDL for a com.mailchimp/profile_update event (#1082)
• Added Redshift DDL for a com.mailchimp/unsubscribe event (#1081)
• Added Redshift DDL for a com.mailchimp/subscribe event (#1080)

StorageLoader

• Wrote JSON Path file for com.callrail/call_complete event (#1109)
• Wrote JSON Path file for com.mailchimp/campaign_sending_status event (#1079)
• Wrote JSON Path file for com.mailchimp/cleaned_email event (#1078)
• Wrote JSON Path file for com.mailchimp/email_address_change event (#1077)
• Wrote JSON Path file for com.mailchimp/profile_update event (#1076)
• Wrote JSON Path file for com.mailchimp/unsubscribe event (#1075)
• Wrote JSON Path file for com.mailchimp/subscribe event (#1074)

Snowplow v0.9.10

Minimalistic release designed to support the new Snowplow JavaScript Tracker v2.1.1.

• Blog post

StorageLoader

• Wrote JSON Path file for PerformanceTiming (#1147)
• Wrote JSON Path file for social_interaction (#1029)
• Wrote JSON Path file for site_search (#1027)
• Wrote JSON Path file for change_form (#1025)
• Wrote JSON Path file for submit_form (#1023)
• Wrote JSON Path file for remove_from_cart (#1021)
• Wrote JSON Path file for add_to_cart (#1019)

Redshift

• Converted all Redshift DDLs to use tabs (#1034)
• Added Redshift DDL for PerformanceTiming (#1032)
• Added Redshift DDL for social_interaction (#1030)
• Added Redshift DDL for site_search (#1028)
• Added Redshift DDL for change_form (#1026)
• Added Redshift DDL for submit_form (#1024)
• Added Redshift DDL for remove_from_cart (#1022)
• Added Redshift DDL for add_to_cart (#1020)

Snowplow v0.9.9

Primarily a comprehensive bug fix release, although it also adds the new campaign_attribution enrichment to our enrichment registry.

• Blog post

New trackers

• .NET Tracker: added git submodule. Version 0.1.0 (#1000)
• PHP Tracker: added git submodule. Version 0.1.0 (#1013)

Clojure Collector

• Bumped to 0.8.0
• Fixed regression in log record format caused by #854 (#992)
• Correctly handles multiple IPs in X-Forwarded-For (#970)

StorageLoader

• Bumped to 0.3.3
• Selecting Snowplow's hosted-assets bucket based on region (#1012)

EmrEtlRunner

• Bumped to 0.9.2
• No rows to process now returns 0, not 1 (#1018)
• Fixed bug where --process-enrich doesn't work, thanks @kingo55! (#1089)
• Now checking that output directories are empty before running (#1124)

Scala Common Enrich

• Bumped to 0.7.0
• Bumped scala-maxmind-iplookups to 0.2.0 (#1002)
• Added support for non-GA campaign attribution: phase 1 (#402)
• Rewrote AttributionEnrichments tests as RefererParserEnrichment tests (#974)
• Allow but downcase a-f characters in incoming event_id (#1006)
• Extract useragent from ua parameter (#1011)
• Fixed issue where unset integer fields throw an NPE (#570)
• Fixed issue where unset double fields throw an NPE (#1062)
• Added tests for ConversionUtils.stringToJInteger (#1064)
• Now enforcing Java 7 for compilation (#1065)

Scala Hadoop Enrich

• Bumped to 0.8.0
• Bumped Scala Common Enrich to 0.7.0 (#995)
• Added test for empty integer and double fields to ensure no NPE thrown (#1063)
• Now enforcing Java 7 for compilation (#1066)
• Updated test jobs to reflect updated useragent parsing (#1070)

Snowplow v0.9.8

Our initial release for mobile event analytics

• Blog post

Trackers

• iOS Tracker: added git submodule. Version 0.1.1 (#982)
• Android Tracker: added git submodule. Version 0.1.1 (#983)

Clojure Collector

• Bumped to 0.7.0
• Merged snowplow/tomcat-cf-access-log-valve into Snowplow as clojure-collector/access-valve (#898)
• Bumped access-valve to 0.1.0
• Changed access-valve's package path to com.snowplowanalytics.snowplow.collectors.clojure.accessvalve (#924)
• Changed access-valve to use Gradle (#899)
• Changed access-valve to publish to war-resources/.ebextensions (#900)
• Updated access-valve and added web.xml to log request body and content type (#901)
• Fixed empty querystring in access-valve (#938)
• Fixed IP address forwarding for VPC-based environments (#854)
• Added support for API vendor and version in routing (#925)
• Added support for POST as well as GET (#654)

Scala Stream Collector

• Scala Stream Collector: fixed broken link to thrift-raw-event, thanks @bamos! (#955)

Scala Common Enrich

• Bumped to 0.6.0
• Split out Clojure and CloudFront Collector event processing (#943)
• Added CljTomcatLoaderSpec tests (#963)
• Filtering non-GETs from CloudfrontLoader (#944)
• Replaced all Argonaut code with json4s (#945)
• Renamed CanonicalOutput to EnrichedEvent (#964)
• Replaced CanonicalInput and TrackerPayload with CollectorPayload and RawEvent (#946)
• Updated EnrichmentManager to process RawEvent not CanonicalInput (#903)
• Added Snowplow Tp2 Adapter to convert event JSON to NEL of RawEvents (#904)
• Geo-IP lookup now supports ip parameter on querystring (#961)
• IP address anonymization now works with ip parameter on querystring (#960)

Scala Hadoop Enrich

• Bumped to 0.7.0
• Bumped to Scala Common Enrich 0.6.0 (#940)
• Updated to support generating multiple enriched events from one raw payload (#902)

StorageLoader

• Wrote JSON Path file for mobile_context (#776)
• Wrote JSON Path file for geolocation_context (#962)

Redshift

• Added Redshift DDL for mobile_context (#542)
• Added Redshift DDL for geolocation_context (#950)