Explainability

Explainability

 Simply stated, explainability is the capacity for a human to explain why a given algorithm provides a certain answer. It is obvious that explainability is a paramount asset for enforcing confidence in AI’s user and increasing acceptance rate of new technologies. It is also a crucial factor for fixing errors which might be detected when the algorithms operate in a real context.

Unfortunately, in recent times we assisted to an exponential growth  of complexity of AI algorithms, which lead to so-called “black box” models: you ingest a lot of data in some neural network architecture, you tweak parameters until a satisfying result is achieved, and you deploy the model. The problem is that the resulting model, albeit providing acceptable results, cannot explain why those results are produced. Neither it can the data scientist that produced it, given the complexity. For this reason in recent years a lot of research was conducted on “explaining models”, i.e. models which try to explain why a black box model produced a given result.

There are many philosophical biases to the concept of “explaining model”, the most important one being that they will produce an explanation which might sound plausible, but which does not necessarily reflect the “reasoning” of the black box model. If it were, the explaining model would be identical to the black box one.

At Irradiantlabs we work hard on the concept of explainability, building on the assumption that complex AI algorithms  should be used for learning rules, not predictions. Predictions are emitted by rules, and as such fully explainable. Notice that with this approach we obtain two important effects:

  • We can put all the complexity we need in the rule learning algorithm: ultimately we do not care how the rules are produced, but only if they are reasonable and effective.
  • We can provide a full explanation of any prediction of the system, even in the form of a natural language statement.

As simple things are often the most difficult to achieve, we devote a strong research effort on explainability, mostly centered on graph learning and rule induction algorithms.