Memory issues with 3.0 obj loader and large files

[jackd]

Looping over faces for very large obj files creates memory issues that cause my computer to crash. Issues arise in this loop. I wish I could give more details, but the debug cycle is long when I need to hard restart each time.

Example model causing issues:
synset id: 02691156 (plane)
model id: b644db95fe32d115d8d90babf3c5509a

This file is 46mb. My system has 16gb ram.

For anyone with shapenet data already saved/extracted:

import os
import trimesh

def load_model(synset_id, model_id, shapenet_root='/tmp/shapenet_data'):
    return trimesh.load(os.path.join(shapenet_root, synset_id, model_id, 'model.obj'))

if __name__ == '__main__':
    synset_id = '02691156'  # planes
    model_id = 'b644db95fe32d115d8d90babf3c5509a'  # uncompressed size 46596937
    load_model(synset_id, model_id)
    print('successfully loaded model %s / %s' % (synset_id, model_id))

Otherwise, the following downloads the entire planes shapenet archive and attempts to load the above model. That's a lot to download for just one model - sorry about that, but there are licensing issues with redistributing part of it...

import os
import trimesh


class ZipSubdirResolver(trimesh.visual.resolvers.Resolver):
    def __init__(self, archive, subdir):
        assert(hasattr(archive, 'read'))
        self.archive = archive
        self.subdir = subdir

    def get(self, name):
        name = name.lstrip()
        if name.startswith('./'):
            name = name[2:]
        with self.archive.open(os.path.join(self.subdir, name)) as fp:
            return fp.read()


DL_URL = (
        'http://shapenet.cs.stanford.edu/shapenet/obj-zip/ShapeNetCore.v1/'
        '{synset_id}.zip')


def download_zip(synset_id, download_dir='/tmp/shapenet_data'):
    dl_url = DL_URL.format(synset_id=synset_id)
    path = os.path.join(download_dir, os.path.split(dl_url)[1])
    if not os.path.isfile(path):
        import wget
        if not os.path.isdir(download_dir):
            os.makedirs(download_dir)
        wget.download(DL_URL.format(synset_id=synset_id), out=path)
    return path


def load_model(archive, synset_id, model_id):
    subdir = os.path.join(synset_id, model_id)
    obj_path = os.path.join(subdir, 'model.obj')
    resolver = ZipSubdirResolver(archive, subdir)

    with archive.open(obj_path) as fp:
        mesh_or_scene = trimesh.load(fp, file_type='obj', resolver=resolver)
    return mesh_or_scene


if __name__ == '__main__':
    import zipfile
    synset_id = '02691156'  # planes
    model_id = 'b644db95fe32d115d8d90babf3c5509a'  # uncompressed size 46596937
    path = download_zip(synset_id)
    with zipfile.ZipFile(path, 'r') as zf:
        load_model(zf, synset_id, model_id)
    print('successfully loaded model %s / %s' % (synset_id, model_id))

[mikedh]

Ah yeah, can confirm does consume all of my memory and take out my computer haha.

I added some debug stuff to the merge/voxtweak branch that at least makes it possible to debug without a reboot:

trimesh/trimesh/exchange/obj.py

Lines 180 to 199 in e35bfdb

	######## DEBUG
	items = dir()
	sizes = np.zeros(len(items))
	for i, item in enumerate(items):
	try:
	sizes[i] = eval('sys.getsizeof({})'.format(item))
	except:
	pass
	order = sizes.argsort()
	pad = max(len(i) for i in items) + 1
	st = '\n'.join(
	'{}:\t{}MB'.format(i.ljust(pad), s) for i,s in
	zip(np.array(items)[order], sizes[order]/1e6))
	print('\n\n', st)
	# if you don't do this it takes out your computer
	if psutil.virtual_memory().percent > 20:
	from IPython import embed
	embed()
	#### / DEBUG

Otherwise it's watch top and quick fingers with ctl-c 😆. One pretty interesting thing: memory grows during the loop, but once you embed/exit the memory usage falls back (basically) to zero. It seems like it's probably some loop scoping thing that's making a full copy of the whole file every iteration. I didn't notice any objects in scope (from dir) that were obviously giant or growing (this list looks the same on every iteration, even as memory use is going to infinity and beyond):

vn               :	1.6e-05MB
faces_tex        :	1.6e-05MB
uv               :	1.6e-05MB
normal_idx       :	1.6e-05MB
i                :	2.8e-05MB
group_count      :	2.8e-05MB
mtl_position     :	2.8e-05MB
pad              :	2.8e-05MB
search           :	2.8e-05MB
f_start          :	2.8e-05MB
f_idx            :	2.8e-05MB
f_end            :	2.8e-05MB
newline          :	2.8e-05MB
columns          :	2.8e-05MB
per_ref          :	2.8e-05MB
visual           :	5.6e-05MB
resolver         :	5.6e-05MB
mtl_path         :	5.8e-05MB
current_object   :	6.2e-05MB
material         :	6.2e-05MB
current_material :	6.2e-05MB
item             :	7.3e-05MB
name             :	7.6e-05MB
starters         :	0.000104MB
v                :	0.000112MB
array            :	0.000112MB
vt               :	0.000112MB
faces            :	0.000112MB
index            :	0.00012MB
o_split          :	0.00016MB
kwargs           :	0.00024MB
mesh             :	0.00024MB
items            :	0.000448MB
order            :	0.00048MB
sizes            :	0.00048MB
face_lines       :	0.00064MB
materials        :	0.001184MB
material_kwargs  :	0.001184MB
m_chunk          :	0.001258MB
st               :	0.001457MB
joined           :	0.00156MB
o_chunk          :	0.001765MB
file_obj         :	0.004272MB
geometry         :	0.018528MB
face_tuples      :	0.111MB
f_chunk          :	46.596903MB
text             :	46.596987MB

[mikedh]

It's this line:

        else:
            # otherwise just use unmasked vertices                                                   
            uv = None
            mesh.update({'vertices': v.copy(),

[mikedh]

I refactored to only copy referenced vertices in the branch merge/voxtweak, which loads the large mesh with under 3% memory on my machine. That is a heck of a mesh, 12000 components and 1.4m vertices haha. I'm not sure how to square "not copying every vertex" with the weekly "vertices aren't in the original order" issue, but it's definitely more important to not totally explode when someone tries to load a crazy OBJ.

Let me know if that branch works for you, I was getting "ok" load speed, though the large number of components slows it down a lot:

[mikedh]

I added a group_materials option to the OBJ loader, which is True by default and reduces the number of components and load time on that model substantially (from ~29s to 6.5s, pretty competitive with meshlab@3s and blender@29s) while still keeping memory below 3%. For some reason it also fixes some of the texturing on that model. I just merged to master and did a minor bump, the build just finished and released.

Thanks for the report!

[jackd]

Issue resolved with the original problem mesh - next up, 02958343/e23ae6404dae972093c80fb5c792f223: ~100mb, 445k vertices, 882k faces.

Sorry I'm not being very helpful in terms of resolving these issues - I'm down a rabbit hole trying to preprocess shapenet and need to start working my way back up. The easiest (though highly unsatisfying) way of doing that is to blacklist certain models. I figure reporting the blacklist is better than nothing.

[mikedh]

No worries, I wanted to get all of ShapeNet to load at some point!

That mesh was dying because of the numpy string array allocation thing (arrays are allocated so every slot is the size of the largest string), which I replaced with more sensible and slightly faster join/split/fromstring wangling in a branch https://github.com/mikedh/trimesh/tree/fix/objvert. The visuals are kinda screwey, but it does load now (in 14s, not bad).

[jackd]

Confirmed fix of above car mesh in fix/objvert branch.

Attempted to load all shapenet core CAD models on that branch. There seems to be index issues in both fix/objvert and master for the following model ids:

failed_ids = {
    '02828884': ('2f0cd28e2f8cdac16121178eafd002fd',),
    '02933112': (),
    '02958343': (),
    '03001627': ('747d2db60ca6b98f3eec26c23f5bc80b', '1de733a48e5607b22d9c1884c92fce12',),
    '03636649': ('3259e491870a8c49eef5d83b671bb264',),
    '03211117': (),
    '02691156': ('6f0ad1fb7917fd9b50577cf04f3bf74a', 'dbab9feed7e936cfa87372b03d6dc78b', 'f9209166fc259d8885e96081cfe0563b', 'c88275e49bc23ee41af5817af570225e',),
    '04090263': ('661ad35c5c3a908c6d1e04c7ae242f3d',),
    '04256520': ('ecf29f5698798a74104d78b9947ee8', '9f5fd43df32187739f2349486c570dd4',),
    '03691459': (),
    '04379243': ('dc0e0beba650c0b78bc6f322a9608b07', '56ea26c10c478555a31cc7b61ec6561', '1846b3533f41ae82f8c4b4cfc2702232',),
    '04401088': (),
    '04530566': ('18761559208a970188d5590328ce0ddf', '80d381a6760185d8c45977b13fbe7645', '587cc1fc65ac4991ff920fdb73e92549',),
    '02747177': (),
    '02773838': (),
    '02801938': (),
    '02808440': (),
    '02818832': (),
    '02843684': (),
    '02871439': (),
    '02876657': ('8099b9d546231dd27b9c6deef486a7d8', '2d1aa4e124c0dd5bf937e5c6aa3f9597'),
    '02880940': (),
    '02924116': (),
    '02942699': (),
    '02946921': (),
    '02954340': (),
    '02992529': (),
    '03046257': (),
    '03085013': (),
    '03207941': (),
    '03261776': (),
    '03325088': (),
    '03337140': (),
    '03467517': ('3c125ee606b03cd263ae8c3a62777578',),
    '03513137': (),
    '03593526': (),
    '03624134': ('67ada28ebc79cc75a056f196c127ed77',),
    '03642806': (),
    '03710193': (),
    '03759954': (),
    '03761084': (),
    '03790512': (),
    '03797390': (),
    '03928116': (),
    '03938244': (),
    '03948459': (),
    '03991062': (),
    '04004475': (),
    '04074963': (),
    '04099429': (),
    '04225987': (),
    '04330267': ('cfa1ad5b5b023fe81d2e2161e7c7075', '993d687908792787672e57a20276f6e9'),
    '04460130': (),
    '04468005': (),
    '04554684': (),
}

File "[...]/trimesh/exchange/obj.py", line 189, in load_obj
    mask_v[faces] = True
IndexError: index 24495 is out of bounds for axis 0 with size 11302

(failed_ids updated since original post)

[mikedh]

Thanks for the updated ID's! 03001627/747d2db60ca6b98f3eec26c23f5bc80b was seeing the error, and in that case it was coming from mixed data (some faces 1 2 3, others 1/2 3/4 5/5) that we weren't detecting correctly. That example at least is fixed by 1f3b15f

[mikedh]

How is this looking after the changes? I'm going to close this issue for now since we made major changes, though feel free to open a new issue with failing ID's.

Thanks for testing against shapenet!

[jackd]

Sorry for the late response - phd deadline is looming :S.

Things are looking much better. Currently having issues with a single bottle, though I won't be losing any sleep over it myself. I'm running a full sweep now of lesser-used categories too...

02876657/8099b9d546231dd27b9c6deef486a7d8