Fragment uses the color/grayscale value of the shape to determine how strong the effect is in that shape. The app starts with whatever the darkest shape is in the SVG. That shape will have a full strength effect in it. Then other shapes that are lighter in color will have a less strong effect than the first. This lets you create layered fragments where the photo moves differently in different sections.
Currently, Fragment does not support gradients. Instead it gets the average color of a gradient and treats the shape as if it was that color. Also different colors don't really matter, it converts any color into grayscale and uses that value to decide on the strength of the effect.
You can see another explanation of the above and an example image in the colors section here: http://fragmentapp.com/help/svg.html