A bird's eye view on the way Fop operates can be found in the document FOP Mechanics
If you want to extend the functionality of FOP by adding new formatting objects, you should do the following:
1. FO Object: Write a class which contains the description of your formatting object and put it into the package fop.fo.flow, fop.fo.pagination (if it is a property it goes to fop.fo.properties. The classes in this package are generated via an xslt stylesheet located in codegen/properties.xml)
2. Element Mapping: Add it to the list in fop.fo.StandardElementMapping (if it is a property you need to add it to fop.fo.PropertyListBuilder)
3. Area: Either your need can be fulfilled within one of the existing classes in fop.layout, then just add the code to handle the new fo/property or you must write a new one.
4. Renderer: Choose the renderer you are interested in. If you worked on an existing layout class you must add code to handle the new features to the already existing area specific method in the renderer class. Otherwise you have to add a new method.