• README-JA(日本語)
• What is Sisimai
  • Key features
  • Command line demo
• Setting Up Sisimai
  • System requirements
  • Install
    • From RubyGems
    • From GitHub
• Usage
  • Basic usage
  • Convert to JSON
  • Callback feature
  • One-Liner
  • Output example
• Sisimai Specification
  • Differences between Ruby version and Perl version
  • Other specification of Sisimai
• Contributing
  • Bug report
  • Emails could not be parsed
• Other Information
  • Related sites
  • See also
• Author
• Copyright
• License

What is Sisimai

Sisimai is a Ruby library for analyzing RFC5322 bounce emails and generating structured data from parsed results. The Ruby version of Sisimai is ported from the Perl version of Sisimai at github.com/sisimai/p5-sisimai.

Key Features

• Convert Bounce Mails to Structured Data
  • Supported formats are Ruby(Hash, Array) and JSON(String)
• Easy to Install, Use.
  • gem install
  • git clone & make
• High Precision of Analysis
  • 2 times higher than bounceHammer
  • Support 68 MTAs/MDAs/ESPs
  • Support Feedback Loop Message(ARF)
  • Can detect 29 error reasons

Command line demo

The following screen shows a demonstration of Sisimai at the command line using Ruby(rb-sisimai) and Perl(p5-sisimai) version of Sisimai.

Setting Up Sisimai

System requirements

More details about system requirements are available at Sisimai | Getting Started page.

• Ruby 2.1.0 or later
  • Oj | The fastest JSON parser and object serializer
• Also works on JRuby 9.0.4.0 or later
  • JrJackson | A mostly native JRuby wrapper for the java jackson json processor jar

Install

From RubyGems

$ sudo gem install sisimai
Fetching: sisimai-4.25.5.gem (100%)
Successfully installed sisimai-4.25.5
Parsing documentation for sisimai-4.25.5
Installing ri documentation for sisimai-4.25.5
Done installing documentation for sisimai after 6 seconds
1 gem installed

From GitHub

$ cd /usr/local/src
$ git clone https://github.com/sisimai/rb-sisimai.git
$ cd ./rb-sisimai
$ sudo make depend install-from-local
gem install bundle rake rspec coveralls
...
4 gems installed
bundle exec rake install
sisimai 4.25.5 built to pkg/sisimai-4.25.5.gem.
sisimai (4.25.5) installed.

Usage

Basic usage

make() method provides feature for getting parsed data from bounced email messages like following. Beginning with v4.25.6, new accessor origin which keeps the path to email file as a data source is available.

#! /usr/bin/env ruby
require 'sisimai'
v = Sisimai.make('/path/to/mbox')       # or path to Maildir/

# Beginning with v4.23.0, both make() and dump() method of Sisimai class can
# read bounce messages from variable instead of a path to mailbox
f = File.open('/path/to/mbox', 'r');    # or path to Maildir/
v = Sisimai.make(f.read)

# If you want to get bounce records which reason is "delivered", set "delivered"
# option to make() method like the following:
v = Sisimai.make('/path/to/mbox', delivered: true)

if v.is_a? Array
  v.each do |e|
    puts e.class                # Sisimai::Data
    puts e.recipient.class      # Sisimai::Address
    puts e.timestamp.class      # Sisimai::Time

    puts e.addresser.address    # shironeko@example.org # From
    puts e.recipient.address    # kijitora@example.jp   # To
    puts e.recipient.host       # example.jp
    puts e.deliverystatus       # 5.1.1
    puts e.replycode            # 550
    puts e.reason               # userunknown
    puts e.origin               # /var/spool/bounce/Maildir/new/1740074341.eml

    h = e.damn                  # Convert to HASH
    j = e.dump('json')          # Convert to JSON string
    puts e.dump('json')         # JSON formatted bounce data
  end
end

Convert to JSON

Sisimai.dump() method provides feature for getting parsed data as JSON string from bounced email messages like following.

# Get JSON string from parsed mailbox or Maildir/
puts Sisimai.dump('/path/to/mbox')  # or path to Maildir/

# dump() method also accepts "delivered" option like the following code:
puts Sisimai.dump('/path/to/mbox', delivered: true)

Callback feature

Beginning with Sisimai 4.19.0, make() and dump() methods of Sisimai accept a Lambda (Proc object) in hook argument for setting a callback method and getting the results generated by the method via Sisimai::Data.catch method.

#! /usr/bin/env ruby
require 'sisimai'
callbackto = lambda do |v|
  r = { 'x-mailer' => '', 'queue-id' => '' }

  if cv = v['message'].match(/^X-Postfix-Queue-ID:\s*(.+)$/)
    r['queue-id'] = cv[1]
  end
  r['x-mailer'] = v['headers']['x-mailer'] || ''
  return r
end

data = Sisimai.make('/path/to/mbox', hook: callbackto)
json = Sisimai.dump('/path/to/mbox', hook: callbackto)

puts data[0].catch['x-mailer']      # Apple Mail (2.1283)

More information about the callback feature is available at Sisimai | How To Parse - Callback Page.

One-Liner

% ruby -rsisimai -e 'puts Sisimai.dump($*.shift)' /path/to/mbox

Output example

[{"catch":{"x-mailer":"","return-path":"<shironeko@mx.example.co.jp>"},"token":"cf17945938502bd876603a375f0e9517c921bbab","lhost":"localhost","rhost":"mx-s.neko.example.jp","alias":"","listid":"","reason":"hasmoved","action":"failed","origin":"set-of-emails/maildir/bsd/lhost-sendmail-22.eml","subject":"Nyaaaan","messageid":"0000000011111.fff0000000003@mx.example.co.jp","replycode":"","smtpagent":"Sendmail","softbounce":0,"smtpcommand":"DATA","destination":"example.net","senderdomain":"example.co.jp","feedbacktype":"","diagnosticcode":"450 busy - please try later 551 not our customer 503 need RCPT command [data]","diagnostictype":"SMTP","deliverystatus":"5.1.6","timezoneoffset":"+0900","addresser":"shironeko@example.co.jp","recipient":"kijitora@example.net","timestamp":1397054085}]

Sisimai Specification

Differences between Ruby version and Perl version

The following table show the differences between Ruby version of Sisimai and Perl version of Sisimai. Information about differences between Sisimai and bounceHammer are available at Sisimai | Differences page.

Features	Ruby version	Perl version
System requirements	Ruby 2.1 - 2.6	Perl 5.10 -
	JRuby 9.0.4.0-
Analytical precision ratio(2000 emails)[1]	1.00	1.00
The speed of parsing email(1000 emails)	2.22s[2]	1.35s
How to install	gem install	cpanm, cpm
Dependencies (Except core modules)	1 module	2 modules
LOC:Source lines of code	10600 lines	10800 lines
The number of tests(spec/,t/,xt/) directory	241000 tests	270000 tests
License	BSD 2-Clause	BSD 2-Clause
Support Contract provided by Developer	Available	Available

1. See ./ANALYTICAL-PRECISION
2. Xeon E5-2640 2.5GHz x 2 cores | 5000 bogomips | 1GB RAM | Ruby 2.3.4p301

Other specification of Sisimai

• Parser Engines
• Bounce Reason List
• Data Structure of Sisimai::Data

Contributing

Bug report

Please use the issue tracker to report any bugs.

Emails could not be parsed

Bounce mails which could not be parsed by Sisimai are saved in the repository set-of-emails/to-be-debugged-because/sisimai-cannot-parse-yet. If you have found any bounce email cannot be parsed using Sisimai, please add the email into the directory and send Pull-Request to this repository.

Other Information

Related Sites

• @libsisimai | Sisimai on Twitter (@libsisimai)
• libSISIMAI.ORG | Sisimai | The successor to bounceHammer, Library to parse bounce mails
• Sisimai Blog | blog.libsisimai.org
• Facebook Page | facebook.com/libsisimai
• GitHub | github.com/sisimai/rb-sisimai
• RubyGems.org | rubygems.org/gems/sisimai
• Perl verson | Perl version of Sisimai
• Fixtures | set-of-emails - Sample emails for "make test"

See also

• README-JA.md - README.md in Japanese(日本語)
• RFC3463 - Enhanced Mail System Status Codes
• RFC3464 - An Extensible Message Format for Delivery Status Notifications
• RFC3834 - Recommendations for Automatic Responses to Electronic Mail
• RFC5321 - Simple Mail Transfer Protocol
• RFC5322 - Internet Message Format

Author

@azumakuniyuki

Copyright

Copyright (C) 2015-2023 azumakuniyuki, All Rights Reserved.

License

This software is distributed under The BSD 2-Clause License.