To implement a custom filter or macro with radeox you need to add to XWiki Java code, repackage XWiki and restart the application (eg. in Tomcat or in Jetty). Step by step, you would 

1. Write a new class in the com.xpn.xwiki.render.[filter/macro] package. You can look at the existing classes for an example. Some are simple, like StrongFilter, which rely on the match/print configuration (see item 2). You must define a key for the filter/macro.
2. Add the "match" and "print" regular expressions in the core/src/main/resources/radeox_markup_xwiki.properties file.
3. Register the new class in core/src/main/resources/META-INF/services/* file. Just add the full classname in the corresponding file.

For more details about how a radeox filter/macro works, please consult the radeox documentation http://radeox.org/space/Developer/Radeox_Developer.pdf (around page 16).