What is the exact syntax of a filter expression in the layerdefinition.xml?

Those filters are expressed using the CQL syntax. It is the same as the formulas in the style definitions.
For example, in the online search and filter demo, there is a layer of African countries. A filter, allowing only opec and unesco countries would look like this:
((OPEC == 1) AND (UNESCO == 1))
In this layer, there are defined attributes with the names "OPEC" and "UNESCO" (case-sensitive!), and both attributes are integers.
An example of a filter on a string attribute would look like this: (NAME LIKE '%dunno%')