ReaderError with ASCII-8BIT encoding.

[jcoyne]

This code works:

data = "<info:fedora/scholarsphere:qv33rx50r> <http://purl.org/dc/terms/description> \"\\n\xE2\x80\x99 \" .\n"

repository = RDF::Repository.new
RDF::Reader.for(:ntriples).new(data)  do |reader|
  reader.each_statement do |statement|
    repository << statement
  end
end

repository.dump(:ntriples)

This causes an error:

data = "<info:fedora/scholarsphere:qv33rx50r> <http://purl.org/dc/terms/description> \"\\n\xE2\x80\x99 \" .\n".force_encoding('ASCII-8BIT')

repository = RDF::Repository.new
RDF::Reader.for(:ntriples).new(data)  do |reader|
  reader.each_statement do |statement|
    repository << statement
  end
end

repository.dump(:ntriples)

RDF::ReaderError: expected object in line 1: "\"\n’ \" ."
    from /Users/justin/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/rdf-1.1.0.1/lib/rdf/reader.rb:419:in `fail_object'
    from /Users/justin/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/rdf-1.1.0.1/lib/rdf/ntriples/reader.rb:212:in `block in read_triple'
    from /Users/justin/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/rdf-1.1.0.1/lib/rdf/ntriples/reader.rb:204:in `loop'
    from /Users/justin/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/rdf-1.1.0.1/lib/rdf/ntriples/reader.rb:204:in `read_triple'
    from /Users/justin/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/rdf-1.1.0.1/lib/rdf/reader.rb:382:in `read_statement'
    from /Users/justin/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/rdf-1.1.0.1/lib/rdf/reader.rb:299:in `block in each_statement'
    from /Users/justin/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/rdf-1.1.0.1/lib/rdf/reader.rb:299:in `loop'
    from /Users/justin/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/rdf-1.1.0.1/lib/rdf/reader.rb:299:in `each_statement'
    from (irb):224:in `block in irb_binding'
    from /Users/justin/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/rdf-1.1.0.1/lib/rdf/reader.rb:202:in `call'
    from /Users/justin/.rbenv/versions/2.0.0-p353/lib/ruby/gems/2.0.0/gems/rdf-1.1.0.1/lib/rdf/reader.rb:202:in `initialize'
    from (irb):223:in `new'
    from (irb):223

[gkellogg]

This can be simplified to the following:

data = "<info:fedora/scholarsphere:qv33rx50r> <http://purl.org/dc/terms/description> \"they\\r\\ncapture design\xE2\x80\x99s \" .\n".force_encoding('ASCII-8BIT')
repo = RDF::Repository.new << RDF::NTriples::Reader.new(data)
r.dump(:ntriples)

The problem is in RDF::NTriples::Reader#read_literal, which is failing when it tries to match on LITERAL_PLAIN. Must be some other place where the input is not transformed to UTF-8, which is certainly annoying.

[gkellogg]

The problem seems to be that the object is not valid ASCII-8BIT. The #force_encoding masks this, but if you use #encode instead, you get an undefined conversion error. What seems to be tripping up is the transformation between formats of an illegal format.

As you note in your patch, always giving RDF.rb UTF-8 is probably the right thing. In any case, RDF Literals only cover Unicode (see RDF Concepts), so giving them something else is sort of non-sensical.