G-RIPS Research Musing #1

Landed in Berlin on the 19th of June to participate in the G-RIPS program at the Zuse Institut Berlin on the campus of the Freie Universität. Everyone very kindly welcomed me and the other students (most of whom Americans, one of whom from the UK) and introduced us to the two topics we will be working on - three of us, myself included, will be investigating statistical shape models.

Statistical shape models encode information about objects in 3d space and allow continuous deformations between any two shapes of the same kind via interpolation. For our purposes, we are studying human anatomy via SSMs - the company with which we are partnering, 1000Shapes, uses SSMs to extract 3d models from CT scans. The idea is that given a CT scan, which is essentially a stack of images of an anatomical component, one can perform segmentation of the anatomy from the wider image and then project it into 3d space, cleaning up self-intersections and other discontinuities, by constraining it with the SSM.

SSMs are composed of mean shapes and covariances - in this way, sound a lot like an instantion of a Gaussian process on the space of any one anatomical component. One important use case for SSMs and the scanning-modeling work 1000Shapes does is in identifying deviances or parts of a given anatomy which diverge significantly from the population mean. These could be indicators of potential disease.

To use an SSM to model a given person’s anatomy, we start with the scan of that person. We layer a triangle mesh on top of the scan, then landmark it, adding manual labels to easily-identifiable regions of the anatomical component in order that we can teach the optimizer the correspondences between that person’s anatomy and the population mean, which is also landmarked. Then we can do all sorts of stuff - deform via linear interpolation around the shape space, identify deviances, run PCA on it to identify the important modes of the structure.

Even cooler, we can go from very limited real-world data to useful 3d scans via projection and optimization on the database of known anatomies. One very exciting paper took X-rays and found the 3d model closest to that X-ray via projecting the model into the X-ray space. That means you could go from an X-ray to a full 3d model essentially only waiting for the optimization algo to stop running.

Neural network based approaches recently published attempt to get around the landmarking step noted above, which is thought to be the most labor-intensive. FlowSSM, developed by Lüdke and Amiranashvili, is able to classify landmarks on its own. ShapeWorks, another tool, allows user-generated landmarks, but perhaps does not require them?

Today’s Questions

• How do you do PCA on a mesh?
• Does choice of mesh matter?
• Could we do NTK style analysis on the FlowSSM?
• How do you handle 3d image data like this? What and how hard is segmentation?

More to come!