Exception: hashindex_set failed (malloc failed)

[mathbr]

After leaving an initial backup running unattended for a while, I get the following exception:

hashindex: malloc failed
Traceback (most recent call last):
  File "/usr/local/bin/attic", line 3, in <module>
    main()
  File "/usr/local/lib/python3.4/dist-packages/attic/archiver.py", line 728, in main
    exit_code = archiver.run(sys.argv[1:])
  File "/usr/local/lib/python3.4/dist-packages/attic/archiver.py", line 718, in run
    return args.func(args)
  File "/usr/local/lib/python3.4/dist-packages/attic/archiver.py", line 129, in do_create
    self._process(archive, cache, args.excludes, args.exclude_caches, skip_inodes, path, restrict_dev)
  File "/usr/local/lib/python3.4/dist-packages/attic/archiver.py", line 178, in _process
    os.path.join(path, filename), restrict_dev)
  File "/usr/local/lib/python3.4/dist-packages/attic/archiver.py", line 178, in _process
    os.path.join(path, filename), restrict_dev)
  File "/usr/local/lib/python3.4/dist-packages/attic/archiver.py", line 164, in _process
    archive.process_file(path, st, cache)
  File "/usr/local/lib/python3.4/dist-packages/attic/archive.py", line 411, in process_file
    chunks.append(cache.add_chunk(self.key.id_hash(chunk), chunk, self.stats))
  File "/usr/local/lib/python3.4/dist-packages/attic/cache.py", line 227, in add_chunk
    self.chunks[id] = (1, size, csize)
  File "attic/hashindex.pyx", line 162, in attic.hashindex.ChunkIndex.__setitem__ (attic/hashindex.c:3113)
Exception: hashindex_set failed

The backup command was attic create --verbose Foo.attic::2015-04-27 /home/me/Foo and that media currently uses 2.8TB.

It seems like Attic ran out of memory during backup and after checking I saw that ~200MB of my swap space where used which is very unusual given that I have a RAM of 8GB.

Anything wrong with my setup/usage or anything I could try?

My Attic version is 0.15 and was installed via Pip(3)

[mathbr]

Subsequent execution of the same command goes until a specific point and then suddenly memory usage goes up a lot, load average rises and the exception appears. It seems like Attic is terminated by the OOM killer.

[ThomasWaldmann]

I opened a more general issue #302 - this is a known problem.