Multi-Modal Approaches

including sonification

Inter-modality Gap

MMKG: multi-modal knowledge graphs

Multimodal Named Entity Recognition with Image Attributes and Image Knowledge

Screen Shot 2021-10-18 at 10.10.37 AM.png

Multimodal Adversarial Networks

"To the best of our knowledge, the proposed MAN could be one of the first works to specifically design for the multimodal representation learning (more than two modalities) with adversarial learning."

CODE by primary author:

Citing Articles (23)

Last edited:

Evaluation Techniques on Multi-Modal Summarization

Last edited:

Haptic Data Visualization

from: "Review of Designs for Haptic Data Visualization"

A combination of aproaches (e.g. auditory and haptic) is best for graph visualization. They assist in both faster and more accurate recognition:

Screen Shot 2021-06-29 at 1.33.55 PM.png

Augmented Cognition Overview

Augmented cognition relates to a number of fields, including my interests in information visualization and automated software engineering. The research article Neurotechnologies for Human Cognitive Augmentation: Current State of the Art and Future Prospects shows that EEG will be very popular in 2040 for various activities such as communication, control, user selection, and training. User selection would be helpful for brushing, highlighting, etc. with visualizations, such as those displayed by smartglasses.

My favorite except from the article shows something that seems even more tangible:

Even when such systems are not directly aimed at augmenting performance, monitoring the mental state of users makes it possible to enhance their performance by adapting the interface they interact with, with so called adaptive interfaces. For instance, Wilson and Russell (2007) described a neuroadaptive system where the users’ task is to detect a target in an environment and where the mental workload is varied according to the feedback given by EEG and other physiological measures.
Neurotechnologies for Human Cognitive Augmentation: Current State of the Art and Future Prospects

Favorite images from the article:


Sonification, etc for Blind Users