I'm in the process of applying a new test approach for FIX message validation.
It was decided to create each test case per message which verifies valid, invalid, and malformed data in the field. We agreed to define valid (quite obvious) any supported range of value, invalid as the correct data type but not supported by our system (e.g. wrong precision, too long string value, or unsupported order type), and malformed as invalid data type, a value that violates FIX specification.
I faced the issue of mixing invalid and malformed cases on several behaviors depending on the data type. For example, invalid cases are likely to be handled by Execution report(ExecType=Rejected)/CancelReplaceReject and Business Message Rejects, malformed are supposed to be handled by Session Reject. But, sometimes in some cases that are supposed to be malformed are handled as invalid and vice versa.
I'm thinking of adding Incorrect or Unsupported dimensions of values.
The context of my project is the exchange(matching engine) with a connected FIX engine, for incoming business messages could be rejected by FIX or by matching engine validation rules.
If you share any thoughts or experiences about the testing of error handling I would really appreciate it.
Somehow sounds complicated to me.
A malformed field in FIX tag-value syntax is defined as follows: https://www.fixtrading.org/standards/tagvalue-online/#well-formed-field .
Personally, I'd just put everything that is possible into the data dictionary and let the validation do its job. Other cases (wrong precision, unsupported order type, ...) should be handled by your application logic and rejected with the most specific message, e.g. ExecutionReport as reply to NewOrderSingle, if no specific message is possible then use BusinessMessageReject.
See here: https://www.fixtrading.org/online-specification/business-area-infrastructure/#category-business-rejects