CSV line scanner skipping last line? #315
Milestone
Comments
|
Hi @kciesielski When there is no newline at the end of the source it is meant to parse the last line on stream completion. Interesting that it is connected to using a File sink. |
|
Yes, I looked through the sources and I noticed the |
|
Hm, I think it might not be connected to CSV stuff. Adding a |
|
Yeah, maybe it's caused by something in |
|
Just landed after Scaladays.
There is a problem in the CSV stage when data and completion come in a certain way. I know a solution, but haven't succeeded in writing a test yet.
Am 3. Juni 2017 21:48:23 MESZ schrieb Krzysiek Ciesielski <notifications@github.com>:
…Yeah, maybe it's caused by something in `FileIO`. With logging enabled
I could spot that the source finishes earlier in cases when tests fail.
However, I have no idea why adding a newline to the input file would
make the problem go away, logging shows that the whole contet gets read
at once anyway. Indeed a bizarre problem to trace :)
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#315 (comment)
|
ennru
added a commit
to ennru/alpakka
that referenced
this issue
Jun 5, 2017
As @kciesielski described in akka#315 he encountered a bug which sometimes swallowed CSV lines. The issue occurs on stream completion when more than on CSV line is not parsed yet and the last line has no line ending. I haven't been able to re-create the issue in a proper test case as it is connected to stream execution timings.
johanandren
pushed a commit
that referenced
this issue
Jun 5, 2017
s-soroosh
added a commit
to s-soroosh/alpakka
that referenced
this issue
Jun 12, 2017
- Set default settings as default parameter - Improve tests Implement java dsl [FTP] Critical fix for infinite loop of traversing "." and ".." directories Upgrade to aws-java-sdk-dynamodb 1.11.106 Allow to pass a `SSLSocketFactory` to `MqttConnectionSettings` FTP - attribute enrichment of FTPFile akka#153 Add KairosDB connector = akka#135 Limit parallelism for the SqsSource (akka#163) * Create a custom thread pool for the SqsSource and limit concurrency with the buffer size * Provide AWSCredentials for all SqsSource tests * Provide better java api * Use ArrayDeque as FIFO queue Messages for asserts in SqsSourceSettings * Remove AmazonSQSAsyncClient from factory method - Correct the javadoc * Add implicit AmazonSQSAsync to factory method - It now conforms again with the SqsSink - Passing the clients from extern seems the preferred way (awslambda module) * Update documentation to reflect thread pool usage * Improvements from review by ktoso - Replace IntStream with Source.range - Use Sink.head instead of Sink.seq =pro update akka http to 10.0.5 (akka#230) Make Travis fail build on code format differences =sqs fix typo in require in SqsSourceSettings (akka#228) FTP - toPath sink akka#182 Add GCE Pubsub with publish and subscribe. improve naming consistency of private vars. PR feedback. Improve java api, java examples, better json marshalling. use mapAsyncUnordered s3: provide access to the returned response s3: clean up test log s3: add javadsl for request() Time goes by, next try Update connectors.md (akka#237) Make external libs more visible (reactive kafka) (akka#229) * Update TOC depth to 3, to show Reactive Kafka Now people don't notice Kafka is in here since it's "external", expanding the TOC one more level makes it more visible. WDYT? * Update index.md FTP: make lastModified test more robust (fixes akka#236) Add SqsAckSink (akka#129) Add SqsAckSink * update elasticmq version * update dependencies Added possibility configure Sftp connection using private key akka#197. - Added SftpIdentity case class to allow configuring private/public key, - Added option to configure known_hosts file and test to check its usage. - Added spec that should fail password based authentication and revert to private key one, - Added docs paragraph to describe this option. Upgrade to scalafmt 0.6.6 Remove deperecated binPack.callSite scalafmt setting Format with updated scalafmt and fixed settings S3 - add documentation akka#103 Fix alphabetical ordering in the docs Separate out release docs S3 path style access akka#64 Make SqsSourceTest less likely to fail - Reduce amount of sent messages to 1 (multiple batch streaming is tested in the SqsSourceSpec) - Increase timeout Introduced "secure" boolean property for S3 which controls whether HTTPS is used akka#247 README: add scaladex, travis badges And make docs links less scary to click on :) Add CSV data transformation module (akka#213) * Alpakka Issue akka#66: CSV component * Alpakka Issue akka#66: revised CSV parser * Alpakka Issue akka#60: CSV parsing stage * wait for line end before issuing line As the byte string may not contain a whole line the parser needs to read until a line end is reached. * Add Java API and JUnit test; add a bit of documentation * Introduce CsvToMap stage; more documentation * Parse line even without line end at upstream finish * Add Java API for CsvToMap; more documentation * More restricted API, incorporated comments by @johanandren * Format sequence as CSV in ByteString * Add Scala CSV formatting stage * Add Java API for CSV formatting; more docs * Separate enums for Java and Scala DSLs * Use Flow.fromGraph to construct flow * Rename CsvFraming to CsvParsing * Check for Byte Order Mark and ignore it for UTF-8 * Emit Byte Order Marks in formatting; CsvFormatting is just a map * Byte Order Mark for Java API * Add line number to error messages; sample files exported from third party software * Use Charset directly instead of name * csv: autoformatted files * simplified dependency declaration Fixes akka#60. SQS flows + Embedded ElasticMQ akka#255 * Add a flow stage and use ElasticMQ * Use flow-based stage for ACKs * Use AmazonSQSAsync instead of AmazonSQSAsyncClient * Using embedded ElasticMQ for tests add SNS connector with publish sink akka#204 Await futures before performing assertion (fixes akka#235) When an assertion fails after the test has already succeeded it will be ignored, so Await the future before continuing with the check. Document 'docker-compose' for running tests Fail MqttSourceStage mat. value on connection loss And increase the timeout. Might help with akka#189, or otherwise help generate a better error message when it does happen again. Ref akka#2 add IronMq integration Refs akka#2 add at-least-one semantic to IronMq connector Improve documentation and test coverage for IronMq ref akka#2 - Document the IronMq domain classes - Document IronMq client - Test the at-least-once producer/consumer mechanism - Improve the IronMQ connector documentation Ref akka#2 Preserve newline in reference.conf Ref akka#2 Make seure the actor system is fully terminated after each test Ref akka#2 Reformat code Refs akka#2 define a different Committable and CommittableMessage for Java and Scala DSL Refs akka#2 Fix typos in IronMQ documentations Refs akka#2 Remove non needed Environment variables from TravisCI config file Refs akka#2 Add a simple Java test and refactor the Java DSL to looks better in Java FTP: Attempt to fix flaky test on Travis Link to scaladex (akka#266) s3: support for encryption, storage class, custom headers akka#109 s3: Added support for partial file download from S3 akka#264 (akka#265) Add version info and links in index page (akka#273) FTP - append mode for toPath sink + improved upstream failure handling akka#207 Fix broken recovery of EventSource (sse) Replace scala.binaryVersion with scalaBinaryVersion (see akka#278) Fix minor typo in alpakka MQTT Connector doc Add Flow to support RabbitMQ RPC workflow akka#160 Changes Amqp sinks to materialize to Future[Done]. As currently it was very difficult to determine when/if a sink failed due to a amqp error. AMQP: add more options to configuration of the ConnectionFactory, akka#191 Directory sources akka#272 sse: Upgrade to Akka SSE 3 and make test more robust CSV: Fixes ignored second double quote S3: add listBucket method to S3 library (akka#253) * Added recursive listBucket call to get all keys under a specific prefix. * Properly using the request URI method and constructing queries with the Query type * Added tests around query parsing * Fixed formatting and removed recoverWithRetries on listbucket calls as they are already retried on the underlying layer with max-retries * Using signAndGetAs instead of signAndGet as to not duplicate logic. * Implemented quick fixes based on comments. Removed recursive call to get keys and used unfoldAsync to get all keys to run in constant memory. * Added execution context. Fixed broken test * Fixed formatting error. * Cleaned up lisBucket call by added a ListBucketState object instead of the brutal type signature from earlier. * Moved trait for listBucket into the def itself as to remove it from the public namespace. azure-storage-queue connector akka#280 Add attribute parameters to sqs source settings akka#302 Formatting fix for akka#302 Streaming XML parser and utilities. Prepare XML parser to join Alpakka family Remove duplicated region argument in client methods akka#297 Build with Akka 2.5 as well Add Azure Storage Queue documentation to TOC Stub documentation for S3.listBucket S3: fix formatting Run the deployment only against Akka 2.4 PubSub: Add support for emulator host variables Initial commit for apache geode connector CSV: Emit all lines on completion akka#315 XML: make code in tests more consistent Add whitesource plugin Merge branch 'master' into add-kairosdb-connector Add copyright header update docker-compose Make execution context optional in java api Make execution context optional in scala api remove ec from sink spec
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I stumbled upon a weird problem in one stream where I read CSV from a file, transform it and then write results to another file. I managed to isolate this issue in a test here: https://github.com/kciesielski/akka-quickstart-scala/blob/master/src/test/scala/com/lightbend/akka/sample/CsvTest.scala
You can download the whole repo, it's created solely for the purpose of demostrating my issue.
The problem is that sometimes the last line from source CSV does not get emitted by the
CsvParsingstage. It happens only occasionally, so my test above has its assertion wrapped ineventuallyclause.Important notes: Problem doesn't occur if source CSV ends with a newline OR if we read into a different kind of sink than file (for example, Sink.seq).
Maybe (hopefully) I'm using some elements here incorrectly, can someone take a look? A bit of insight from @ennru would be awesome :)
The text was updated successfully, but these errors were encountered: