visual representation of respect

Top 19 Symbols of Respect and What They Mean

Table of Contents

2. saluting, 4. badge of distinction, 8. certificate of achievement, 9. pin of recognition, 10. rosette, 12. daffodil, 14. monstera plant (chinese culture), 15. emerald, 16. eternal knot in tibetan buddhism, 17. camellia, 18. tipping hat, 19. orchids, wrapping up.

Respect is an essential value that is fundamental to our interactions with others . It’s the foundation of healthy relationships, productive communication, and harmonious communities. We show respect through symbols – gestures, actions, or objects that represent and convey our admiration for others.

In this article, we will explore 19 symbols of respect worldwide. From the traditional bow in Japan to the Namaste in India , we will discover the significance behind each symbol and how it reflects the values of the culture that practices it.

Bowing is a universal symbol of respect found in many cultures worldwide. It’s a gesture of humility and deference towards another person, often used in formal or ceremonial settings.

In Japanese culture, for example, bowing is deeply ingrained and can signify emotions from gratitude to apology. The depth and duration of the bow can indicate the level of respect or social status of the person being addressed.

In other cultures, such as South Korea, bowing is an important symbol of respect. The angle of the bow can differ depending on the occasion and the relationship between the individuals.

Historically, bowing has also been used as a sign of submission or surrender. In medieval Europe, a knight might bow before his lord as a show of loyalty , and in Japan, defeated samurai would perform a deep bow to their victors as a sign of surrender.

Saluting is a gesture of respect used in many cultures and contexts, often associated with military or formal settings.

In the United States, for example, salutes are a standard form of the military protocol used to show respect to higher-ranking officers. The gesture involves raising the right hand to the forehead or temple with the palm facing forward.

The salute takes a different form in other cultures, such as India. Known as the “namaste” gesture, it involves placing the hands at the chest or forehead and bowing the head slightly. This gesture is a sign of respect and greeting and is commonly used in yoga and spiritual practices.

Gift-giving is a common symbol of respect found in many cultures and contexts. It’s a way of showing appreciation, gratitude, or honor to someone else.

In some cultures, such as Japan, gift-giving is deeply ingrained and can carry significant cultural and social meaning. In Japan, for example, gifts are often given as a sign of respect and gratitude, and giving and receiving gifts is steeped in complex social etiquette.

In other cultures, such as the United States, gift-giving is often associated with holidays and special occasions. It can also be used to express sympathy or show support for someone going through a difficult time.

The history of gift-giving can be traced back to ancient times when it was used to establish and maintain social connections and alliances. Today, gift-giving remains an important symbol of respect and gratitude and can vary widely in form and meaning across cultures and contexts.

A badge of distinction is a symbol of respect often used to identify individuals who have achieved a certain status or accomplishment. In many cultures, badges or pins are awarded to military members or individuals who have achieved a high level of academic or professional success.

In some cultures, such as in ancient Rome , badges were used to denote an individual’s rank and social status. In medieval Europe, knights would wear coats of arms or heraldic badges to identify themselves on the battlefield.

In modern times, badges and pins are commonly used in professional and academic settings. For example, in the United States, the Presidential Medal of Freedom is a badge of distinction awarded to individuals who have made significant societal contributions.

A trophy symbolizes respect commonly used to recognize achievement or success in a particular area. Trophies are often associated with sports, where they are awarded to teams or individuals who have won a competition or tournament.

In some cultures, trophies may also be used to recognize achievement or excellence in academic or professional settings.

For example, in the United States, the Pulitzer Prize is a prestigious award often seen as a symbol of respect for exceptional achievement in journalism, literature, or music.

The trophy’s history can be traced back to ancient Greece , where athletes were awarded wreaths or crowns for their victories in the Olympic games. Over time, these symbolic gestures evolved into physical prizes, such as cups or plaques, and eventually into the iconic trophy shapes we know today.

A plaque is a symbol of respect commonly used to recognize achievement or commemorate an event or person of significance. Plaques are often made of metal or wood and can feature inscriptions, engravings, or images that convey a particular message or meaning.

In many cultures, plaques honor individuals who have made significant contributions to society or achieved high success in a particular field. For example, the National Baseball Hall of Fame in the United States features plaques that honor some of the greatest players in the sport’s history.

The history of the plaque can be traced back to ancient Rome , where they were used to commemorate important events or individuals. Over time, plaques have evolved into a powerful symbol of respect that can be used in various contexts.

Thyme is an herb used for centuries to symbolize respect and admiration. In ancient Greece, thyme was associated with bravery and courage, often used to honor soldiers who had fought in battle. In medieval Europe, thyme was believed to have healing properties and was used to treat various ailments.

In some cultures, thyme still symbolizes respect and honor. In Morocco, for example, thyme is an important ingredient in tea, often serving as a sign of hospitality and respect. In Iran, thyme is used in cooking and is believed to have medicinal properties.

A certificate is a document that recognizes the achievement or completion of a particular task or course of study. Certificates are often given as a symbol of respect and honor and can carry significant meaning for the recipient.

In many cultures, certificates are used to recognize achievement or excellence in academic or professional settings. For example, in the United States, certificates of achievement may be given to students who have demonstrated exceptional performance in a particular subject.

In the workplace, certificates may be given to employees who have completed a training program or achieved high success.

The history of certificates can be traced back to ancient Rome, where they were used to verify ownership or citizenship. Over time, certificates evolved into a powerful symbol of respect that can be used in various contexts.

A pin of recognition is a symbol of respect commonly used to acknowledge achievement, service, or contribution in a particular area.

Pins are often small and wearable and can feature designs, logos, or other insignia representing the recognized organization or achievement.

In many cultures, pins recognize individuals who have made significant contributions to society or achieved high success in a particular field. For example, military service members may be awarded pins in the United States to recognize their bravery or service.

Employees may receive pins to recognize their years of service or outstanding performance in the workplace.

A rosette is a decorative ornament made of ribbon or fabric, often featuring a flower-like design. Rosettes symbolize respect and honor and are often given as awards or tokens of appreciation.

In some cultures, rosettes are used in formal or ceremonial settings, such as in military or government contexts. For example, in France, the Legion of Honor is a prestigious award with a red rosette as one of its logos.

In the United Kingdom, rosettes are often worn at horse shows and other equestrian events to indicate the rider’s or horse’s placement.

An average ribbon is a strip of fabric often used as a symbol of respect or recognition. Ribbons can come in various colors and designs, often used to signify a particular cause or achievement.

In many cultures, ribbons are used in formal or ceremonial settings, such as military or government. For example, in the United States, the Purple Heart award features a ribbon worn on the uniform to indicate that the recipient has been wounded in battle.

Ribbons are also commonly used in fundraising or awareness campaigns, where they are used to signify support for a particular cause or issue. For example, the pink ribbon is a well-known symbol of breast cancer awareness.

In ancient Greece , the daffodil was associated with the god Apollo and was seen as a symbol of renewal and new beginnings.

In modern times, the daffodil has become an important symbol of hope, support, and respect for those affected by cancer, particularly in the United States and the United Kingdom.

The daffodil is also an important symbol of spring’s arrival and life’s renewal. In many cultures, the daffodil is associated with Easter and is often used in religious ceremonies and traditions.

In addition to its symbolic meaning, the daffodil is also a popular flower in gardens and floral arrangements.

The color red is often used to symbolize respect and admiration in many cultures worldwide. In Chinese culture , it’s believed that red brings fortune and prosperity and is often used in wedding ceremonies and other festive events.

In many African cultures, red symbolizes courage and bravery, often worn by warriors and other military leaders.

In Western cultures, the color red is often paired with imagery of love and passion and is commonly used in romantic contexts. It’s also used as a symbol of warning or danger, such as in traffic signals or warning signs.

In religious contexts, red is often used to symbolize sacrifice and martyrdom. For example, in Christian traditions , red is associated with the blood of Christ and is often used in artwork and another religious iconography.

The Monstera plant, a.k.a. the Swiss Cheese plant, has become a popular symbol of respect and admiration in recent years, particularly in interior design. Its large, vibrant green leaves often bring a touch of tropical flair to homes and workplaces.

In some cultures, the Monstera plant is associated with prosperity and good fortune. In Feng Shui, a Chinese practice emphasizing energy flow in living spaces, the Monstera plant is believed to bring abundance and positive energy to the home or workplace.

Emerald stones are a powerful and elegant symbol of respect, prized for their beauty and rarity for centuries.

In ancient Egypt , emeralds were associated with fertility and were often used in jewelry and other decorative objects. In Greek mythology , emeralds were believed to have healing properties and were associated with the goddess Aphrodite .

In many cultures, emeralds are also associated with royalty and power. For example, the British Crown Jewels feature several emeralds, including the Delhi Durbar Tiara, which was made for the coronation of King Edward VII in 1903.

Emeralds are also often used in engagement rings and other romantic jewelry, as they are believed to symbolize love, loyalty, and devotion.

The eternal knot also called the Endless Knot , symbolizes respect and enlightenment in Tibetan Buddhism. The knot comprises an intricate interweaving pattern with no beginning or end, representing the interconnectedness of all things in the universe.

In Tibetan Buddhism, the eternal knot is often used to represent the concept of interdependence and the idea that all actions have consequences. It’s also associated with karma and the cycle of rebirth.

Camellias are evergreen shrubs that are a symbol of respect and admiration in many cultures worldwide. They are often associated with love, affection, and gratitude and are frequently used in gift-giving and other expressions of appreciation.

In Japan, camellias are often associated with samurai warriors and are seen as a symbol of courage and strength. In China, camellias are associated with the feminine principle of beauty and grace and are often used in traditional ceremonies and celebrations.

Tipping one’s hat is a traditional symbol of respect used for centuries in many cultures worldwide. It’s often used as a gesture of acknowledgment and gratitude, particularly in formal or ceremonial settings.

In Western culture, tipping one’s hat dates back to medieval times, when knights would raise their helmets to show respect to royalty or other important figures.

Over time, the practice evolved into the more subtle gesture of tipping one’s hat, which is still used today in various contexts.

Orchids are a popular and elegant symbol of respect in many cultures worldwide. They are often associated with beauty, refinement, and luxury and are frequently used in gift-giving and other expressions of appreciation.

Ancient Greece’s orchids were associated with masculinity and were often used in love potions and other medicinal remedies. In Chinese culture, orchids are associated with strength , nobility , and beauty and are often used in traditional ceremonies and celebrations.

Symbols of respect serve as a powerful reminder of our shared values and the importance of treating others with kindness , empathy, and appreciation.

By understanding and incorporating these symbols into our interactions with others, we can foster a culture of respect and inclusivity that transcends borders and promotes harmony among diverse communities.

So, the next time you encounter a symbol of respect, take a moment to reflect on its meaning and consider how you can incorporate it into your own life. Doing so will make a small but meaningful step towards creating a more respectful and harmonious world.

Similar Articles:

Top 10 Symbols of Grace and What They Mean

11 Powerful Symbols of War and Their Meanings

19 Symbols of Nobility and What They Mean

Top 19 Symbols of Leadership from Around the World

Tags: Ancient Symbols

Dani Rhys has worked as a writer and editor for over 15 years. She holds a Masters degree in Linguistics and Education, and has also studied Political Science, Ancient History and Literature. She has a wide range of interests ranging from ancient cultures and mythology to Harry Potter and gardening. She works as the chief editor of Symbol Sage but also takes the time to write on topics that interest her.

15 Meaningful Symbols of Christmas and What They Mean

15 Important Thanksgiving Symbols and What They Mean

9 Halloween Symbols and Why They Represent the Holiday

Traditional Reiki and Its Four Symbols: Power and Controversy

Find Us On Social Media

Quick links.

• Privacy Policy
• Terms of Use

Latest Posts

• The 72 Shem Angels: Guardians of Spiritual Transformation
• Beyond the Surface: The Deep Meanings of Kabbalah Symbols
• Exploring the Book of Abramelin: Secrets of Sacred Magic
• Mercury Rising: Symbolism in Science and Spirituality

© Copyright Symbol Sage, 2024. All Rights Reserved.

Politics and Elections

Ethics and ethical data visualization: a complete guide.

Discover the significance of ethical data visualization in our world. This comprehensive guide covers key ethical principles in data visualization, real-life examples of unethical practices, and strategies to ensure accuracy, fairness, and responsible design in your visual representations of data.

Introduction

We create and use huge amounts of data in all areas of life. It’s important to display and understand this information in an easy-to-grasp and meaningful way. Data visualization plays an important role in bridging the gap between raw data and human understanding. It is enabling us to gain insights, make informed decisions, and communicate complex ideas more effectively. That’s why the question of ethics in data visualization is vital!

From tracking global pandemics to understanding economic trends and social patterns, data visualization has become an essential tool in various fields.

However, with great power comes great responsibility. Data visualization impacts our views and choices, so it’s crucial to make sure visuals are truthful, precise, and ethical.

In this detailed blog post, we’ll explore the importance of ethics in data visualization. We will also cover five ethical principles of data visualization and examine examples of ethical and unethical practices. Finally, we’ll share tips and resources for creating effective, ethical visualizations.

The power of visuals in shaping views and decisions

Visuals greatly impact our perceptions and decision-making, as we naturally process images faster than text. Data visualization effectively grabs attention and communicates complex ideas or patterns due to our visual instincts.

Studies reveal that visuals boost our memory and understanding of information. They help us identify patterns, trends, and anomalies, simplifying conclusions and making informed decisions. Visuals also evoke emotions and empathy, deepening our connection to data and its consequences.

Given this influence, we must recognize the potential outcomes of their work. How data is displayed can affect perception and interpretation, and influence decisions based on that data. By carefully choosing charts, colors, scales, and other elements, we can clarify or confuse the message, guiding or misleading viewers.

This highlights the need for ethical principles in data visualization. The idea here is to ensure visuals are not just eye-catching but also honest, accurate, and unbiased. By embracing ethical practices, we can establish audience trust, foster a well-informed society, and encourage responsible decision-making.

The need for ethical considerations in dataviz efforts

As data visualization gains prominence in our data-driven world, the need for ethical considerations becomes increasingly important. Ethics in data visualization and its considerations are crucial in ensuring that visual representations of data are accurate, fair, and unbiased. As such, they are fostering trust and responsible decision-making.

When data visualizations are created without regard for ethical principles, they can lead to misinterpretations, misinformation, and potentially harmful situations.

There are several reasons why ethical considerations are essential in data visualization.

Why do we need ethical principles in data visualization?

They are summarized in the following five ethical principles in data visualization.

• Accuracy and honesty – Data visualizations should correctly represent the underlying data and not deliberately mislead or deceive the audience. Manipulating visuals to create a false impression or support a specific agenda, erodes trust and leads to poor decision-making.
• Clarity and simplicity – Visualizations should be designed to make the data easier to understand, avoid unnecessary complexity or clutter. Striking a balance between aesthetics and functionality is key to ensuring that the message is clear.
• Fairness and objectivity – Data visualizers should strive to present data objectively, without introducing personal bias or promoting stereotypes. Being transparent about data sources, methodology, and limitations can help establish credibility and promote fair interpretation.
• Privacy and trust – Respecting the privacy of individuals and organizations is critical when visualizing data. We should be mindful of potential privacy concerns and adhere to relevant laws, regulations, and ethical guidelines to protect sensitive information.
• Inclusiveness and accessibility – Ensuring that data visualizations are accessible and inclusive to diverse audiences is an important ethical consideration. This includes using color schemes readable by individuals with color vision deficiencies. It also can mean providing alternative text descriptions (alt text) for visually impaired users. It also means considering cultural sensitivities when designing visuals.

Each of these principles, within the context of dataviz ethics, will be discussed in greater detail later in this article.

Understanding ethics in data visualization

The role of ethics in data viz discipline.

Ethics play a vital role in data visualization. It guides the principles and practices that ensure visual representations of data are truthful, fair, and responsible. Since visuals shape our views and guide decisions, data visualizers need to follow ethical standards for clear and unbiased data communication.

Ethical factors in data visualization cover many issues like data honesty, visual clarity, fair representation, and privacy respect. By sticking to these rules, data visualizers build trust and credibility while effectively sharing complex ideas.

Ethics also matter during the whole data process, from gathering and analyzing data, creating graphics, and the final visual. Ethical aspects should be considered at each step, as biases or errors can harm the visual’s integrity and effectiveness.

Furthermore, ethical obligations extend to visualization consumers, who should remain vigilant against potential manipulation and assess information accuracy judiciously.

Finally, the role of ethics in dataviz is ensuring that we create and use visuals that are honest, accurate, and fair. As such, they will be promoting responsible decision-making and contributing to a more informed and ethical society.

The three basic steps in data visualization and their ethical considerations

Data collection.

Data collection is the first stage in the data visualization process, where relevant and accurate data is gathered from various sources. This essential step forms the foundation for subsequent analysis and visualization.

Data can be collected from primary sources, such as surveys, interviews, or experiments. Also, it can be gathered from secondary sources, like databases, published research, or government reports. In some cases, data may also be obtained through real-time sources, such as sensors or social media platforms.

The two main ethical considerations in this phase are:

• Ensuring data accuracy and completeness – Data collectors must strive to gather high-quality, reliable data to avoid misrepresentations or misleading conclusions. This may involve cross-checking sources, verifying data authenticity, or addressing potential gaps in the dataset.
• Respecting data privacy and consent – When collecting personal or sensitive information, we must privacy regulations and get informed consent from data subjects. This involves being transparent about data usage intentions and safeguarding collected data to prevent unauthorized access or misuse.

Data analysis

Data analysis involves examining and processing collected data to uncover trends, patterns, or insights that will inform the visualization. This step is essential to transforming raw data into meaningful information.

Data visualizers (designers, developers, journalists…) may use various techniques in their work. Those techniques are descriptive statistics, data cleaning, data aggregation, or even more advanced analytics methods like machine learning. They help reveal meaningful patterns or relationships within the data.

Ethical considerations during data analysis include:

• Minimizing biases and errors – Experts should use appropriate methods, tools, and techniques to reduce biases and errors during data processing. This involves critically evaluating data quality, being aware of potential pitfalls, and validating analytical results.
• Transparency in data processing – Analysts need to be transparent about the steps, assumptions, and methodologies used during data analysis. This enables others to verify, replicate, or challenge the results, promoting accountability and trust in the findings.

Data visualization (design)

Data visualization is the final step in this journey. It involves creating visual representations of data that effectively communicate insights, patterns, or trends to an audience. Designers utilize various forms, such as charts, graphs, maps, or interactive visualizations, to make complex data more accessible and understandable.

Good design choices, including colors, scales, and layout, play a significant role in conveying the intended message and influencing viewer interpretation.

The main ethical considerations during data visualization design include:

• Honest data presentation – Designers must avoid manipulating or distorting data to mislead or misinform viewers. This involves choosing appropriate chart types, scales, and data transformations that accurately show the underlying data.
• Accessible and inclusive design – Designers should create visuals suitable to diverse audience needs and preferences. By doing so, it is ensured that the information is accessible to as many people as possible. This may involve considering color blindness, screen reader compatibility, or providing alt descriptions for visual elements

Five ethical principles in data visualization

We mentioned them earlier, and in this section we’ll go deeper into each of the five ethical principles in data visualization.

The five ethical principles in data visualization are:

• Honesty and accuracy
• Clarity and simplicity
• Fairness and objectivity
• Respect for privacy and confidentiality
• Cultural sensitivity and inclusivity

#1 Honesty and accuracy

Being honest and accurate in data visualization means presenting data truthfully, without distortion or manipulation.

This involves using the correct data, avoiding cherry-picking or misrepresenting information to support a specific viewpoint. Honesty and accuracy build trust with the audience, ensuring that the visualizations effectively convey the intended message. To maintain honesty and accuracy, designers should verify data sources, cross-check information, and be transparent about data limitations or uncertainties.

#2 Clarity and simplicity

Clear and simple visualizations make complex data more understandable and accessible. Clarity and simplicity involve choosing appropriate chart types, using consistent and readable fonts, and organizing the layout for easy navigation.

Simplifying visuals helps the audience to quickly grasp key insights, patterns, or trends without confusion. To achieve clarity and simplicity, designers should prioritize essential information, minimize visual clutter, and use colors, labels, and legends effectively.

#3 Fairness and objectivity

Fair and objective data visualizations avoid biases and present data impartially. This principle ensures that visualizations don’t favor a particular viewpoint or mislead the audience with biased interpretations.

Fairness and objectivity involve selecting unbiased data sources, acknowledging data limitations, and presenting alternative perspectives when appropriate.

To achieve fairness and objectivity, we must be aware of biases in data at every step. That includes collection, analysis, and the actual design and presentation work, and take steps to minimize them.

#4 Respect for privacy and confidentiality

Respecting privacy and confidentiality in data visualization means protecting sensitive or personal information.

This principle ensures that visualizations don’t violate privacy rights or expose confidential data. Respecting privacy and confidentiality involves anonymizing data, aggregating information to a safe level, and obtaining consent when necessary.

To maintain privacy and confidentiality, designers should follow relevant data protection regulations, guidelines, and best practices throughout the visualization process.

#5 Cultural sensitivity and inclusivity

Cultural sensitivity and inclusivity in data visualization involve considering diverse audience needs, preferences, and backgrounds.

This principle ensures that visualizations are accessible and respectful to people from various cultural, linguistic, or ability backgrounds. Cultural sensitivity and inclusivity may involve using appropriate colors, symbols, and language to avoid offending or excluding viewers.

To promote cultural sensitivity and inclusivity, we need to research our audience and seek feedback. And not just that – we needt o be open to adjusting our work to accommodate diverse perspectives.

The Hippocratic Oath for dataviz: A commitment to ethical practices

Jason Moore, who’s now a Senior Computer Scientist at the Air Force Research Lab, proposed a version of the Hippocratic Oath for data visualization. It was first introduced at VisWeek in 2011 and then published on Robert Kosara’s blog .

With some modifications, here’s the Hippocratic Oath for data visualization :

“ I shall not use visualization to intentionally hide or confuse the truth which it is intended to portray. I will respect the great power visualization has in garnering wisdom and misleading the uninformed. I accept this responsibility willfully and without reservation, and promise to defend this oath against all enemies, both domestic and foreign. ”

Real-life examples of unethical data visualization practices

Ethical practices are essential to ensure that information is presented honestly, accurately, and without bias.

However, not all visualizations adhere to these principles. That results in misleading or deceptive representations that can distort the audience’s understanding and lead to uninformed decision-making.

In this section, we will explore real-life examples of unethical data visualization practices. We’ll highlighting instances where misleading visuals, data manipulation, and inappropriate visual elements have been used to mislead and confuse.

Learning from these cases, we can raise awareness of the potential pitfalls in data visualization. Moreover, we can emphasize the importance of adhering to ethical principles in creating and consuming visual representations of data.

Misleading or deceptive visualizations

A classic example of misleading visualization is the use of truncated or manipulated y-axes in bar charts or line charts. By adjusting the axis scale, a designer can make small differences in data appear more significant, leading to incorrect conclusions. For example, a company may use a truncated y-axis to exaggerate its growth. Or, to downplay a competitor’s performance, misleading investors or customers.

Gun deaths in Florida

Initially, this chart might make you think that with the introduction of the “Stand Your Ground” law, the number of gun deaths in Florida dropped significantly.

But take another look at the Y-axis. Do you see it? For some reason, the values are ordered in an unusual way – with the highest values at the bottom of the axis and the lowest values at the top.

Most people’s first reaction to this would be that the introduction of this law caused a drop in the number of gun deaths in Florida, not the other way around.

The author, Christine Chan, has tried justifying her decision in a tweet that has since been removed:

@john_self Thanks for the feedback. I prefer to show deaths in negative terms (inverted). It’s a preference really, can be shown either way.

And while it’s likely that she didn’t have an intention to mislead, the impression left made a damage.

People on welfare vs full time jobs

A truncated (or completely missing) Y-axis is the main reason for confusion here. By omitting baseline values and the scale, the difference between the number of people on welfare and those with a full time job seems more drastic than it really is. Bar graph is an appropriate choice for this type of data but it’s missing correct Y-axis.

Adding a meaningful Y-axis with values and a proper scale would help paint a more objective picture here.

CNN’s bar chart shows presidential approval rates

Again, truncated Y-axes give the feeling that the differences between different presidents are even more apparent in this poorly designed piece by CNN. The source is subreddit /dataisugly .

Something doesn’t add up – biggest COVID worries

The sum of all percentages in this case is well over 100%. One doesn’t need to be a data scientist to see that something is off here. Using a pie chart to display percentages is meaningful in many situations, but not every dataset is suitable for it.

In this instance, respondents had the option to choose multiple answers. For example, one might be concerned not only about getting infected themselves but also about their family members becoming infected.

Additionally, it is common practice for the legend to follow the order of pie slices and to have the largest pie slice start at 12 o’clock, proceeding clockwise. In this example, “Getting it” is the first label in the legend section but appears second, regardless of whether we go clockwise or counterclockwise.

Other unethical approaches to data visualization

Manipulation of data to support a specific agenda (cherry picking).

Selectively presenting data points or time periods that support a specific narrative, while ignoring contradictory or less favorable information, can distort the audience’s understanding of an issue.

An example of data manipulation can be seen in the selective presentation of data to support a political or social agenda.

A political group might cherry-pick data points that show a positive trend for their policies. At the same time they would be ignoring negative trends or omitting relevant context. This manipulation can mislead the public and distort their understanding of the issue, promoting a biased perspective.

Inappropriate use of visual elements that trigger bias or stereotype

In 2015, the New York Times published an infographic titled “Murder in America” that aimed to show the relationship between race and homicide victims. However, the visualization faced criticism for perpetuating racial stereotypes. The chart used red dots to represent black victims and blue dots for white victims, with red dots appearing more prominent and alarming. Critics argued that the choice of colors could reinforce negative stereotypes about African Americans and crime. Furthermore, the chart lacked context and socio-economic factors that contribute to crime rates, providing a simplified and potentially misleading view of a complex issue.

It has been removed since.

Ignoring data uncertainty

Ignoring data uncertainty in visualizations can lead to misleading representations of information, causing audiences to develop a false sense of precision and reliability. It’s one of the most overlooked situations and it can happen in a number of cases.

• Election polls and forecasts – During election seasons, various organizations provide polling data and forecast models to predict election outcomes. Failing to include margins of error, confidence intervals, or uncertainty ranges in these visualizations can give the impression that the predictions are more precise and certain than they actually are. This might lead audiences to be overconfident in the forecasted results, which can affect their voting behavior or expectations.
• Scientific research – In fields like medicine or climate science, research findings often include inherent uncertainty due to factors like sample size, measurement errors, or model assumptions. When visualizing these findings, it’s crucial to include error bars, confidence intervals, or other visual cues that represent the uncertainty. Omitting these elements can create a misleading impression of the findings’ certainty, leading to misinformed decisions by policymakers, healthcare professionals, or the public.

Confusing area and radius in scatter plots

An unethical approach in data visualization can involve the inappropriate use of bubble size in scatter plots, causing confusion and misinterpretation of the data.

In scatter plots with bubbles, the size of the bubbles often represents a third variable in addition to the two variables plotted on the x and y axes. However, when the bubble sizes are not scaled properly, it can create misleading impressions about the relationships between variables.

For example, let’s consider a scatter plot that compares a country’s GDP per capita (x-axis) with life expectancy (y-axis), using bubble size to represent the total population. If the bubble size is scaled by radius, rather than area, it can create a distorted perception of the population sizes. Since the area of a circle increases with the square of its radius, a bubble with twice the radius would represent a population four times larger, not twice as large as one might assume. This can lead audiences to overestimate the differences in population between countries.

Strategies for ethical data visualization

Data visualizers must use particular techniques and strategies that encourage ethical decisions throughout the design process to produce visuals that uphold ethical principles.

In this section, we’ll go over four essential strategies for creating ethical dataviz:

• exercising critical thinking and challenging the data source
• selecting the best type of visualization for the intended audience
• achieving a balance between form and function
• ensuring accessibility and inclusivity

Critical thinking and questioning the data source

To ensure the accuracy and honesty of a visualization, it is crucial to critically examine the data source.

Start by assessing the credibility and reliability of the source, and consider potential biases or errors that may be present. Be transparent about the limitations of the data and any assumptions made during the analysis. When working with multiple sources, cross-reference and verify the information to minimize the risk of inaccuracies.

Additionally, be open to feedback and willing to revise the visualization if new or contradictory information emerges.

Choosing the right visualization type for the message

Selecting the appropriate visualization type is essential for effectively communicating the message and avoiding misinterpretation.

Begin by understanding the data and identifying the key insights you wish to convey.

Then, choose a visualization type that best represents these insights.

Do it by considering factors such as:

• the nature of the data (categorical, numerical, or time-based)
• relationships between variables
• the audience’s familiarity with different chart types

Avoid using misleading or overly complex visualizations that may distort the message or confuse the audience.

Balancing aesthetics and functionality

While visually appealing designs can capture attention, it is important not to prioritize aesthetics over functionality.

Strive for a balance between the two by creating visuals that are both engaging and easy to understand. Use color, typography, and layout to enhance the clarity of the message. However, avoid unnecessary embellishments or visual clutter that may distract from the data.

Ensure that the design choices support an accurate interpretation of the information and do not introduce biases or misconceptions.

Ensuring accessibility and inclusivity

An ethical visualization should be accessible and inclusive, taking into account the diverse needs and perspectives of the audience.

To achieve this, consider factors such as color contrast, font size, and alternative text for visually impaired users. Use culturally sensitive and neutral language. Also, be mindful of potential stereotypes or biases that may be reinforced through visual elements.

Additionally, test the visualization with different user groups and gather feedback to identify and address any barriers to accessibility or inclusivity.

By implementing these practices, data visualizers can create visuals that are not only effective but also respectful of their audience’s diverse needs and experiences.

Finally, ethical data visualization is critical in conveying information accurately and responsibly, shaping perceptions, and influencing decision-making.

Data visualizers and consumers can foster trust, promote responsible practices, and contribute to a more informed and fair society. We can do that by following good ethics in data visualization.

This section reiterates the importance of ethical data visualization. It’s a tool helping us foster trust, enable the shared responsibility, and encourage ethical practices for a better society.

Ethics in data visualization and fostering trust

Ethical data visualization helps build trust between data visualizers and their audience. It does that by ensuring that information is presented honestly, accurately, and without bias.

Trustworthy visuals enable audiences to make informed decisions and develop a deeper understanding of complex issues.

By upholding ethical principles, we can establish credibility and foster a trusting relationship with our audiences. And that is vital for effective communication.

The responsibility of data visualizers and consumers

Both data visualizers and consumers have a shared responsibility in promoting ethical practices.

Data designers should follow ethical principles throughout the entire data lifecycle, from collection and analysis to visualization. Consumers should be aware of potential biases and manipulation in the visuals they encounter. They should be able to critically assess the accuracy and reliability of the information presented.

Together, data visualizers and consumers can drive the adoption of ethical practices and create a more responsible data visualization community.

Let’s do it the right way!

Engaging in ethical data visualization not only benefits the individual data visualizer and their audience. It also contributes to a more informed and fair society as a whole.

By committing to ethical practices, we can empower audiences to make better decisions. Also, we can help them develop a deeper understanding of the world around them.

Therefore, with all of this in our minds, I encourage all data visualizers and consumers to actively engage in ethical practices. This will help us in creating a positive impact on society and promoting responsible decision-making.

Additional reading about ethics in data visualization

Here’s some good additional reading that might help you grasp these concepts of ethics in data visualization even more.

• Ethical data viz by Joe Hardin
• The ethics of data visualization by Peter Haferl
• Ethical Dimensions of Visualization Research by Michael Correll
• Practicing good ethics i dataviz by UC Davis
• Ethical infographics In data visualization, journalism meets engineering by Alberto Cairo

You also check out the dataviz fail on a map of Europe done by the European Commission – one of the first posts on this blog.

ABOUT THE AUTHOR

Vibor Cipan

Don't stop reading

Inflation data visualization tutorial and a case study

Inflation rates in Europe 2023 by country (Latest data and maps)

Incredible story of John Nash and his short PhD thesis

2024 Europe gas storage reserves – by country, updated daily

Top 200 most common last names in the USA: A full list and a racial distribution

List and data visualization of the top 20 countries polluting the oceans the most

List of Europe’s most dangerous and worst roads

Seven principles in art and design that’ll improve your data visualizations

Comparison of Slovenia, Croatia, and Western Balkans in 2023 – economy and politics

Interactive map of Pangaea / Pangea with present-day borders and a globe

October in different languages of Europe, maps, and etymology

Unemployment rates in Europe, by country, latest data for 2022

Older articles, september in different languages of europe, maps, and etymology, road quality in europe: the best and worst roads in europe, here’s how heat waves are affecting your mental health and well-being, august in different languages of europe, maps, and etymology, the uk got its new national highest temperature record of 40.3 °c, how many healthy life years do europeans live.

Communication

iResearchNet

Custom Writing Services

Visual representation.

The study and conceptualization of visual representation were primarily associated with art and art history prior to the twentieth century, and drew on the analytical tools of iconology with a focus on the artist’s intention and perception. With the advent of  semiotics, followed by other theories of the visual, the twentieth century marked a broadening in conceptions of visual representation from the realm of art to the realm of the everyday. Studies of visual representation have expanded to include the images that surround people every-day. This includes studies of images in film (Metz 1990), the use of photography (Sontag 1979), advertising (Goffman 1979), scientific imagery (Latour & Woolgar 1986), learning and development (Kress 1996), and the representation of social identities (Hall 1997). This expansion of the domain of the visual has influenced how visual representation is theorized and approached, including a shift in focus from the image to contexts of production and viewers. Today a range of theories is applied to understanding the visual, including theories drawn from anthropology, art history, cognitive psychology, cultural studies, linguistics, psychoanalytical theories, and sociology.

Studies of visual representation have shown that the visual is central to the cultural construction of social life. People’s everyday experience of the world, socially, physically, and psychologically, and their production of meaning are mediated by the visual. Visual representation is not only a matter of how people experience the world; it is also crucial in how the world itself is constructed. The twenty-first century is marked by a plethora of imaging and visual technologies, and in contemporary western society everyday life is saturated with the images that these technologies make available. Studies of late twentieth century culture have noted a turn to the visual (Mitchell 1995; Mirzoeff 1999) in which the modern world has become a visual phenomenon; a world that conflates looking, seeing, and knowing (Jenks 1995) to become a “vision machine” created through new visualizing technologies in which people are all caught (Virilio 1994).

Visual representation is a complex concept that connects with fundamental questions of “reality,” ideology and power, agency, as well as signification and the procedures and potentials for interpreting meaning. There is, however, a general agreement that the meaning of an image is “made” at three sites: (1) the site of the image, (2) the site of production, and (3) the viewer interaction. The meaning of visual representation is realized in the interplay across these three sites or factors.

The concept of visual codes and conventions (socially agreed ways of doing something) is often employed in the analysis of visual representations. Semiotics proposes that there is a wide range of visual, pictorial, material, and symbolic signs that are conventional in the way that they simplify, and yet bear some kind of resemblance to, an object or quality in the “real” world that they signify. This includes abstract signs that become associated through agreement into convention with the object or quality represented. These conventions are extended to a semiotic “code” to form an extended system of semiotic signs. Social semiotics uses the concept of “semiotic resource” rather than code. This serves to foreground the work of the sign-maker in interpreting and assembling semiotic resources.

What is depicted in an image and how it is represented are an obvious starting point for understanding the process of visual representation. Visual codes can, for example, be designed to realize visual narratives or conceptual categories, each of which serves to connect or disconnect depicted elements in different ways. That is, the way these relationships are represented marks (and establishes) what belongs and does not belong together, who acts on what, and so on. These represented relations are realized through a range of codes, including the representation of depicted elements, compositional arrangements, and analytical structures. Visual representations also work by using and representing many of the visual codes that are employed in lived rather than textual forms of communication. In this way visual representations can carry all the sign systems and codes (dress, style, body language, and so on) that can “make” the lived world meaningful. For instance, Goffman’s Gender advertisements (1979) offers a classic exemplar of how depicted elements, compositional structures, posture, and gaze contribute to the production and regulation of gendered identifies.

In addition to their “content,” visual representations position the viewer to look in particular ways. In this way an image can be understood as “telling” viewers who they are and where they are. In doing so the image realizes the ideological design of the viewer’s relationship with the depicted object, person, or event. The viewer can refuse to adopt the viewing position offered by an image, but nonetheless it is present.

In order to get at the meaning of an image, the ways in which the meanings and uses of images are regulated by institutions of production, distribution, and consumption need to be considered. The economics, motives, and intentions of those who produce and disseminate visual representations are aspects of the site of production. That is, visual representations need to be understood in context because these social factors and experiences are not separate from the signifying systems of the visual; they do not exist in an abstract realm – they actually structure it.

The production of an image is therefore integral to its use and meaning. Technologies of production, in particular new technologies, are key to the question of the relationship between visual representation and “reality.” Technologies have changed what it is possible to see and how these entities are seen, through the magnification of detail, slow motion, capturing images that escape “natural” vision, and the changing context of viewing.

Viewer And Gaze

The viewer, looking, and subjectivity are key themes in theorizing and understanding visual representation. Understanding the agency of the viewer demands a shift of analytical emphasis away from the image or text to the social identities and experiences of the viewer. This necessarily connects with the context of viewing as part of the production of meaning. From this perspective, meaning is understood as constituted in the articulation between the viewer and the viewed, between the power of the image to signify and the viewer’s capacity to interpret meaning. Visual representations work (mean) by producing effects each time they are looked at. However, the image also depends for its effects on a certain way of seeing.

Asking who the viewer is raises the question of the alignment and identification of their points of view and perspective with those of the maker of the image. In other words, both the maker and the reader of a visual representation are involved in the process of making sense ( designing meaning ). Seeing the viewer as agentive means that the power of a visual sign remains a potential until the viewer engages with it. The parameters within which interpretations are made are shaped by the viewer’s social position and subjective capacity in conjunction with the conventions and codes of a visual representation.

The mediating effect of contexts of viewing is especially relevant for visual representation in the twenty-first century, when images are mobile across different media and contexts. For example, reading a magazine at home, driving by a billboard on the road, and watching an advert at home on television or with friends in the cinema all call forth different practices of viewing. The context of viewing contributes to realizing different practices of looking and interpreting.

In summary, visual representation can be understood as a cycle of production , circulation, and consumption, a cycle in which visual representations become a site of struggle over what something means.

Visual Representation And Signification

Signification and how to theorize the relationship between referent, signifier, and signified are central to the way visual representation is conceptualized. Current theories of perception, structuralism, poststructuralism, and postmodernism theorize their relationship in different ways. This influences how the power of visual representation is understood to shape people’s experience of the world, what the world is, and what it can be.

Models of perception are concerned with image-making as an internal psychological process, that is, with the perceptions and sensations in the mind of the image-maker and the viewer. Here the goal is the perfect reproduction of reality. This places reality as external to representation, in which the referent “exists” and the signifier reflects it.

From this perspective an image is a transparent window through which the viewer can connect directly (via the signifier) to an original object or presence (the referent). This presents a notion of the image as autonomous and as containing within itself an inherent meaning. This conception of visual signification and image-making as cognition, perception, and a kind of optical truth severs the connection between image and society and fails to account for the relationship between image and power.

Structuralism

In the twentieth century semiotics began to interrogate the elements and structures of systems of representation, language, and the visual. Traditional semiotics (e.g., Saussure 1986) views meaning as consisting of two parts: the signifier and the signified. On the one hand, the signifier denotes a sign’s literal meaning: that is, who or what is depicted. On the other hand, the signified connotes a sign’s associated meanings: that is, the cultural ideas and values associated with the objects depicted. Structuralism asserts that although we depend on the relationship between these two parts to produce meaning, it is an arbitrary relationship established through cultural convention. This presents language and other systems of meaning as self-contained structures, a system of arbitrary and differential signs that do not name or reflect thoughts or concepts that pre-exist it, but rather actively construct them.

Structuralism attempts to understand the social and cultural forces that motivate representation, rather than accepting representation as given or natural. This brings the relationship between referent, signifier, and signified into a social relationship. Viewing is repositioned as a social process that involves the unfolding of meaning through the “activation” of visual codes of recognition, codes learnt through the viewer’s interaction in the social cultural world, rather than an individual physiological and cognitive process.

Locating visual representation in the social domain in this way opens up the necessity of thinking of visual representation as discursive work, work that produces society. This reconnects images with the social and in doing so locates image in relation to power. However, structuralism has been criticized for being ahistorical and for being deterministic in the way that it privileges structure over individual human agency. In this way it is inadequate in theorizing the potential of human agency to resist, challenge, and change systems of representation. Peirce (1991) brought human agency into semiosis by his assertion that sign-making involves the cooperation of a sign, its object, and its interpretant. In other words, for Pierce signs cannot exist independently of a subject to interpret them. This shifts attention from the idea of meaning as embedded in text to a concern for the interactive process of meaning-making .

Poststructuralism

Poststructuralism developed in the 1960s as a critical response to the assumptions of structuralism. Poststructuralism rejects the structuralist notion that there is a consistent structure to a text and argues instead that these structures and systems that underlie a text are themselves a cultural product. Poststructuralism argues that the systems of knowledge that produce it, as well as the text itself, need to be analyzed in order to understand the meaning of a text. In other words, poststructuralism can be seen as the study of the production of knowledge (Worth 1981).

Poststructuralism argues that the author is not the primary source of the meaning of a text (Barthes 1977) and that there is a multiplicity of meanings to any text. Decentering the authority of the author in this way serves to open up the meaning of a text to the work of the reader and other sources for meaning, such as cultural norms and historical context. Placing the reader at the center of meaning-making in this way asserts that no text has one meaning, purpose, or existence. In addition to demanding a rethinking of texts and representations, poststructuralism also destabilizes the notion of the “self.” From this perspective the concept of the self as a single and coherent entity is a fictional construct.

Social semiotics rejects the concept of code as too rigid and static and instead uses the concept of semiotic resource (Hodge & Kress 1988). A focus on semiotic resources serves to recognize the social, historical, and cultural character of visual codes and the ideological work of signs. From a poststructuralist perspective the act of reading a sign is repositioned as the act of sign-making, of giving meaning to a signified. That is, the reader is the signmaker and signs are constantly remade and transformed through people’s engagement with them. In this way, the denotative (the signified) can be seen as always connotative, no matter how literal its relationship to the “real” world.

The relationship of the referent with the signifier and signified is significantly “loosened” in poststructuralism- and the term “signifier” is preferred to “sign” to indicate that no single meaning is secured to a word, and that the possibility of a full and secure meaning is always deferred (Derrida 1978). In other words, meaning is never as fully “fixed” as structuralism appears to suggest. This attempts to move away from the linking of representation with the idea of reconstituting the missing “presence,” the original source “content” of the empirical form of a representation.

Postmodernism

Postmodernism argues that the economic and social conditions of late capitalism, in particular globalization and new technologies, have led to a decentralized, fragmented, and media-dominated society saturated with images. Postmodernism rejects the idea of “representation” in favor of “ simulation ” to emphasize the constructed character of signs. Simulations are understood as simulacra of the “real” that presuppose and precede the “real.” These hyperreal symbols and signs do not have an original, stable referent or a source of meaning. In this way postmoderism breaks the relationship between seeing and knowing. Baudrillard (1983) argues that society has replaced all reality and meaning with symbols and signs, and that what people understand as “reality” is actually a simulation of reality. From this perspective, images are to be fundamentally mistrusted because they claim to signal a referent that they have become detached or severed from.

Postmodernism is a contested term. Whether or not it is a distinct historical period or an extension of modernism remains a matter of debate. There is agreement, however, that the conditions of late capitalism have realized some of the aspects encompassed in postmodernist theory, such as the breakdown of traditional structures (e.g., genre and other stylistic forms) and classificatory categories. This conceptualization of visual representation refutes the possibility of representation as a mimetic reflection of “reality.”

Visual Representation, Identity, And Postcolonialism

The visual produces as well as represents culture, constituting (and constituted by) its relations of power and difference, so that cultures of everyday life are entwined with practices of representation. The continuous circulation and repetition of images presented as the norm or reality in visual media actively define social and cultural norms as fixed and natural. In the ways that people are depicted, vision is complicit with power and discipline through surveillance. Understanding visual representation as embodying and constituting ideologies (the system of thoughts and beliefs that determine the subjects action and behavior) shows how ways of investing meaning in the world are realized in visual representations.

The theorization and study of identities is one example of a significant area that has shown the potential of visual representation in understanding social and cultural phenomena (e.g., Hall 1997). Work has shown how racial, gendered, and other identities are visually represented, recycled, and contested. Visual discourses work to govern and empower particular understandings of a subject through their representation. Building on the work of Foucault (1977), identity theorists have examined visual representation as a part of the regulatory force of culture (sets of practices, cultural norms, meanings, and values) as they apply to the production of identities.

The cultural practices of looking and seeing are (like other social practices) organized around the founding principles of the articulation of difference . Looking at how representations attempt to fix difference offers a way of conceptualizing the complex relationship of power and representation. The practices of representation can be employed, for instance, to mark gender and racial difference. The “visible” signifiers of race and gender on the body can be called on to make these differences seem real and therefore true. The differences we can “see” appear to ground their “truth” beyond the social and historical construction of race and gender into nature and therefore to be unchangeable. The question of how the “other” is produced and reproduced through visual representation is central to postcolonial theories. The idea that discipline and control can be achieved through relations of looking and the knowledge and power that vision allows over what is seen is central to Foucault’s work and postcolonial theories.

Visual representations are, then, a discursive means by which a dominant group works to establish and maintain hegemonic power within a culture in which meaning is constantly reproduced and remade as signs are articulated and rearticulated. Images are thus a site of struggle for meaning , a site of power, and constitutive of society. Cultural production therefore has real political and ideological effects. As a result, visual representation has also been taken up as a potential tool for resistance and the remaking – reimaging – of society.

References:

• Barthes, R. (1977). Image, music, text . London: Fontana.
• Baudrillard, J. (1983). Simulations . New York: Semiotext(e).
• Derrida, J. (1978). Writing and difference (trans. A. Bass). London: Routledge.
• Foucault, M. (1977). Discipline and punish: The birth of the prison . New York: Vintage.
• Goffman, E. (1979). Gender advertisements . New York: HarperCollins.
• Hall, S. (1997). Representation: Cultural representations and signifying practices . London: Sage.
• Hodge, R., & Kress, G. (1988). Social semiotics . Cambridge: Polity.
• Jenks, C. (1995). Visual culture . London: Routledge.
• Kress, G. (1996). Before writing: Rethinking paths to literacy . London: Routledge.
• Kress, G., & van Leeuwen, T. (2006). Reading images: A grammar of visual design . London: Routledge.
• Latour, B., & Woolgar, S. (1986). Laboratory life: The construction of scientific facts . Princeton, NJ: Princeton University Press.
• Metz, C. (1990). Film language: A semiotics of the cinema . Chicago: Chicago University Press.
• Mirzoeff, N. (1999). An introduction to visual culture . London: Routledge.
• Mitchell, W. J. T. (1995). Picture theory: Essays on verbal and visual representation . Chicago: University of Chicago Press.
• Peirce, C. (1991). Peirce on signs: Writings on semiotics by Charles Sanders Peirce . Chapel Hill, NC: University of North Carolina Press.
• Saussure, F. de (1986). Course in general linguistics . Chicago: Open Court.
• Sontag, S. (1979). On photography . London: Penguin.
• Virilio, P. (1994). The vision machine . Indianapolis: Indiana University Press.
• Worth, S. (1981). Studying visual communication (ed. and intro. L. Gross). Philadelphia: University of Pennsylvania Press.

• Open access
• Published: 19 July 2015

The role of visual representations in scientific practices: from conceptual understanding and knowledge generation to ‘seeing’ how science works

• Maria Evagorou 1 ,
• Sibel Erduran 2 &
• Terhi Mäntylä 3  

International Journal of STEM Education volume  2 , Article number:  11 ( 2015 ) Cite this article

75k Accesses

78 Citations

13 Altmetric

Metrics details

The use of visual representations (i.e., photographs, diagrams, models) has been part of science, and their use makes it possible for scientists to interact with and represent complex phenomena, not observable in other ways. Despite a wealth of research in science education on visual representations, the emphasis of such research has mainly been on the conceptual understanding when using visual representations and less on visual representations as epistemic objects. In this paper, we argue that by positioning visual representations as epistemic objects of scientific practices, science education can bring a renewed focus on how visualization contributes to knowledge formation in science from the learners’ perspective.

This is a theoretical paper, and in order to argue about the role of visualization, we first present a case study, that of the discovery of the structure of DNA that highlights the epistemic components of visual information in science. The second case study focuses on Faraday’s use of the lines of magnetic force. Faraday is known of his exploratory, creative, and yet systemic way of experimenting, and the visual reasoning leading to theoretical development was an inherent part of the experimentation. Third, we trace a contemporary account from science focusing on the experimental practices and how reproducibility of experimental procedures can be reinforced through video data.

Conclusions

Our conclusions suggest that in teaching science, the emphasis in visualization should shift from cognitive understanding—using the products of science to understand the content—to engaging in the processes of visualization. Furthermore, we suggest that is it essential to design curriculum materials and learning environments that create a social and epistemic context and invite students to engage in the practice of visualization as evidence, reasoning, experimental procedure, or a means of communication and reflect on these practices. Implications for teacher education include the need for teacher professional development programs to problematize the use of visual representations as epistemic objects that are part of scientific practices.

During the last decades, research and reform documents in science education across the world have been calling for an emphasis not only on the content but also on the processes of science (Bybee 2014 ; Eurydice 2012 ; Duschl and Bybee 2014 ; Osborne 2014 ; Schwartz et al. 2012 ), in order to make science accessible to the students and enable them to understand the epistemic foundation of science. Scientific practices, part of the process of science, are the cognitive and discursive activities that are targeted in science education to develop epistemic understanding and appreciation of the nature of science (Duschl et al. 2008 ) and have been the emphasis of recent reform documents in science education across the world (Achieve 2013 ; Eurydice 2012 ). With the term scientific practices, we refer to the processes that take place during scientific discoveries and include among others: asking questions, developing and using models, engaging in arguments, and constructing and communicating explanations (National Research Council 2012 ). The emphasis on scientific practices aims to move the teaching of science from knowledge to the understanding of the processes and the epistemic aspects of science. Additionally, by placing an emphasis on engaging students in scientific practices, we aim to help students acquire scientific knowledge in meaningful contexts that resemble the reality of scientific discoveries.

Despite a wealth of research in science education on visual representations, the emphasis of such research has mainly been on the conceptual understanding when using visual representations and less on visual representations as epistemic objects. In this paper, we argue that by positioning visual representations as epistemic objects, science education can bring a renewed focus on how visualization contributes to knowledge formation in science from the learners’ perspective. Specifically, the use of visual representations (i.e., photographs, diagrams, tables, charts) has been part of science and over the years has evolved with the new technologies (i.e., from drawings to advanced digital images and three dimensional models). Visualization makes it possible for scientists to interact with complex phenomena (Richards 2003 ), and they might convey important evidence not observable in other ways. Visual representations as a tool to support cognitive understanding in science have been studied extensively (i.e., Gilbert 2010 ; Wu and Shah 2004 ). Studies in science education have explored the use of images in science textbooks (i.e., Dimopoulos et al. 2003 ; Bungum 2008 ), students’ representations or models when doing science (i.e., Gilbert et al. 2008 ; Dori et al. 2003 ; Lehrer and Schauble 2012 ; Schwarz et al. 2009 ), and students’ images of science and scientists (i.e., Chambers 1983 ). Therefore, studies in the field of science education have been using the term visualization as “the formation of an internal representation from an external representation” (Gilbert et al. 2008 , p. 4) or as a tool for conceptual understanding for students.

In this paper, we do not refer to visualization as mental image, model, or presentation only (Gilbert et al. 2008 ; Philips et al. 2010 ) but instead focus on visual representations or visualization as epistemic objects. Specifically, we refer to visualization as a process for knowledge production and growth in science. In this respect, modeling is an aspect of visualization, but what we are focusing on with visualization is not on the use of model as a tool for cognitive understanding (Gilbert 2010 ; Wu and Shah 2004 ) but the on the process of modeling as a scientific practice which includes the construction and use of models, the use of other representations, the communication in the groups with the use of the visual representation, and the appreciation of the difficulties that the science phase in this process. Therefore, the purpose of this paper is to present through the history of science how visualization can be considered not only as a cognitive tool in science education but also as an epistemic object that can potentially support students to understand aspects of the nature of science.

Scientific practices and science education

According to the New Generation Science Standards (Achieve 2013 ), scientific practices refer to: asking questions and defining problems; developing and using models; planning and carrying out investigations; analyzing and interpreting data; using mathematical and computational thinking; constructing explanations and designing solutions; engaging in argument from evidence; and obtaining, evaluating, and communicating information. A significant aspect of scientific practices is that science learning is more than just about learning facts, concepts, theories, and laws. A fuller appreciation of science necessitates the understanding of the science relative to its epistemological grounding and the process that are involved in the production of knowledge (Hogan and Maglienti 2001 ; Wickman 2004 ).

The New Generation Science Standards is, among other changes, shifting away from science inquiry and towards the inclusion of scientific practices (Duschl and Bybee 2014 ; Osborne 2014 ). By comparing the abilities to do scientific inquiry (National Research Council 2000 ) with the set of scientific practices, it is evident that the latter is about engaging in the processes of doing science and experiencing in that way science in a more authentic way. Engaging in scientific practices according to Osborne ( 2014 ) “presents a more authentic picture of the endeavor that is science” (p.183) and also helps the students to develop a deeper understanding of the epistemic aspects of science. Furthermore, as Bybee ( 2014 ) argues, by engaging students in scientific practices, we involve them in an understanding of the nature of science and an understanding on the nature of scientific knowledge.

Science as a practice and scientific practices as a term emerged by the philosopher of science, Kuhn (Osborne 2014 ), refers to the processes in which the scientists engage during knowledge production and communication. The work that is followed by historians, philosophers, and sociologists of science (Latour 2011 ; Longino 2002 ; Nersessian 2008 ) revealed the scientific practices in which the scientists engage in and include among others theory development and specific ways of talking, modeling, and communicating the outcomes of science.

Visualization as an epistemic object

Schematic, pictorial symbols in the design of scientific instruments and analysis of the perceptual and functional information that is being stored in those images have been areas of investigation in philosophy of scientific experimentation (Gooding et al. 1993 ). The nature of visual perception, the relationship between thought and vision, and the role of reproducibility as a norm for experimental research form a central aspect of this domain of research in philosophy of science. For instance, Rothbart ( 1997 ) has argued that visualizations are commonplace in the theoretical sciences even if every scientific theory may not be defined by visualized models.

Visual representations (i.e., photographs, diagrams, tables, charts, models) have been used in science over the years to enable scientists to interact with complex phenomena (Richards 2003 ) and might convey important evidence not observable in other ways (Barber et al. 2006 ). Some authors (e.g., Ruivenkamp and Rip 2010 ) have argued that visualization is as a core activity of some scientific communities of practice (e.g., nanotechnology) while others (e.g., Lynch and Edgerton 1988 ) have differentiated the role of particular visualization techniques (e.g., of digital image processing in astronomy). Visualization in science includes the complex process through which scientists develop or produce imagery, schemes, and graphical representation, and therefore, what is of importance in this process is not only the result but also the methodology employed by the scientists, namely, how this result was produced. Visual representations in science may refer to objects that are believed to have some kind of material or physical existence but equally might refer to purely mental, conceptual, and abstract constructs (Pauwels 2006 ). More specifically, visual representations can be found for: (a) phenomena that are not observable with the eye (i.e., microscopic or macroscopic); (b) phenomena that do not exist as visual representations but can be translated as such (i.e., sound); and (c) in experimental settings to provide visual data representations (i.e., graphs presenting velocity of moving objects). Additionally, since science is not only about replicating reality but also about making it more understandable to people (either to the public or other scientists), visual representations are not only about reproducing the nature but also about: (a) functioning in helping solving a problem, (b) filling gaps in our knowledge, and (c) facilitating knowledge building or transfer (Lynch 2006 ).

Using or developing visual representations in the scientific practice can range from a straightforward to a complicated situation. More specifically, scientists can observe a phenomenon (i.e., mitosis) and represent it visually using a picture or diagram, which is quite straightforward. But they can also use a variety of complicated techniques (i.e., crystallography in the case of DNA studies) that are either available or need to be developed or refined in order to acquire the visual information that can be used in the process of theory development (i.e., Latour and Woolgar 1979 ). Furthermore, some visual representations need decoding, and the scientists need to learn how to read these images (i.e., radiologists); therefore, using visual representations in the process of science requires learning a new language that is specific to the medium/methods that is used (i.e., understanding an X-ray picture is different from understanding an MRI scan) and then communicating that language to other scientists and the public.

There are much intent and purposes of visual representations in scientific practices, as for example to make a diagnosis, compare, describe, and preserve for future study, verify and explore new territory, generate new data (Pauwels 2006 ), or present new methodologies. According to Latour and Woolgar ( 1979 ) and Knorr Cetina ( 1999 ), visual representations can be used either as primary data (i.e., image from a microscope). or can be used to help in concept development (i.e., models of DNA used by Watson and Crick), to uncover relationships and to make the abstract more concrete (graphs of sound waves). Therefore, visual representations and visual practices, in all forms, are an important aspect of the scientific practices in developing, clarifying, and transmitting scientific knowledge (Pauwels 2006 ).

Methods and Results: Merging Visualization and scientific practices in science

In this paper, we present three case studies that embody the working practices of scientists in an effort to present visualization as a scientific practice and present our argument about how visualization is a complex process that could include among others modeling and use of representation but is not only limited to that. The first case study explores the role of visualization in the construction of knowledge about the structure of DNA, using visuals as evidence. The second case study focuses on Faraday’s use of the lines of magnetic force and the visual reasoning leading to the theoretical development that was an inherent part of the experimentation. The third case study focuses on the current practices of scientists in the context of a peer-reviewed journal called the Journal of Visualized Experiments where the methodology is communicated through videotaped procedures. The three case studies represent the research interests of the three authors of this paper and were chosen to present how visualization as a practice can be involved in all stages of doing science, from hypothesizing and evaluating evidence (case study 1) to experimenting and reasoning (case study 2) to communicating the findings and methodology with the research community (case study 3), and represent in this way the three functions of visualization as presented by Lynch ( 2006 ). Furthermore, the last case study showcases how the development of visualization technologies has contributed to the communication of findings and methodologies in science and present in that way an aspect of current scientific practices. In all three cases, our approach is guided by the observation that the visual information is an integral part of scientific practices at the least and furthermore that they are particularly central in the scientific practices of science.

Case study 1: use visual representations as evidence in the discovery of DNA

The focus of the first case study is the discovery of the structure of DNA. The DNA was first isolated in 1869 by Friedrich Miescher, and by the late 1940s, it was known that it contained phosphate, sugar, and four nitrogen-containing chemical bases. However, no one had figured the structure of the DNA until Watson and Crick presented their model of DNA in 1953. Other than the social aspects of the discovery of the DNA, another important aspect was the role of visual evidence that led to knowledge development in the area. More specifically, by studying the personal accounts of Watson ( 1968 ) and Crick ( 1988 ) about the discovery of the structure of the DNA, the following main ideas regarding the role of visual representations in the production of knowledge can be identified: (a) The use of visual representations was an important part of knowledge growth and was often dependent upon the discovery of new technologies (i.e., better microscopes or better techniques in crystallography that would provide better visual representations as evidence of the helical structure of the DNA); and (b) Models (three-dimensional) were used as a way to represent the visual images (X-ray images) and connect them to the evidence provided by other sources to see whether the theory can be supported. Therefore, the model of DNA was built based on the combination of visual evidence and experimental data.

An example showcasing the importance of visual representations in the process of knowledge production in this case is provided by Watson, in his book The Double Helix (1968):

…since the middle of the summer Rosy [Rosalind Franklin] had had evidence for a new three-dimensional form of DNA. It occurred when the DNA 2molecules were surrounded by a large amount of water. When I asked what the pattern was like, Maurice went into the adjacent room to pick up a print of the new form they called the “B” structure. The instant I saw the picture, my mouth fell open and my pulse began to race. The pattern was unbelievably simpler than those previously obtained (A form). Moreover, the black cross of reflections which dominated the picture could arise only from a helical structure. With the A form the argument for the helix was never straightforward, and considerable ambiguity existed as to exactly which type of helical symmetry was present. With the B form however, mere inspection of its X-ray picture gave several of the vital helical parameters. (p. 167-169)

As suggested by Watson’s personal account of the discovery of the DNA, the photo taken by Rosalind Franklin (Fig.  1 ) convinced him that the DNA molecule must consist of two chains arranged in a paired helix, which resembles a spiral staircase or ladder, and on March 7, 1953, Watson and Crick finished and presented their model of the structure of DNA (Watson and Berry 2004 ; Watson 1968 ) which was based on the visual information provided by the X-ray image and their knowledge of chemistry.

X-ray chrystallography of DNA

In analyzing the visualization practice in this case study, we observe the following instances that highlight how the visual information played a role:

Asking questions and defining problems: The real world in the model of science can at some points only be observed through visual representations or representations, i.e., if we are using DNA as an example, the structure of DNA was only observable through the crystallography images produced by Rosalind Franklin in the laboratory. There was no other way to observe the structure of DNA, therefore the real world.

Analyzing and interpreting data: The images that resulted from crystallography as well as their interpretations served as the data for the scientists studying the structure of DNA.

Experimenting: The data in the form of visual information were used to predict the possible structure of the DNA.

Modeling: Based on the prediction, an actual three-dimensional model was prepared by Watson and Crick. The first model did not fit with the real world (refuted by Rosalind Franklin and her research group from King’s College) and Watson and Crick had to go through the same process again to find better visual evidence (better crystallography images) and create an improved visual model.

Example excerpts from Watson’s biography provide further evidence for how visualization practices were applied in the context of the discovery of DNA (Table  1 ).

In summary, by examining the history of the discovery of DNA, we showcased how visual data is used as scientific evidence in science, identifying in that way an aspect of the nature of science that is still unexplored in the history of science and an aspect that has been ignored in the teaching of science. Visual representations are used in many ways: as images, as models, as evidence to support or rebut a model, and as interpretations of reality.

Case study 2: applying visual reasoning in knowledge production, the example of the lines of magnetic force

The focus of this case study is on Faraday’s use of the lines of magnetic force. Faraday is known of his exploratory, creative, and yet systemic way of experimenting, and the visual reasoning leading to theoretical development was an inherent part of this experimentation (Gooding 2006 ). Faraday’s articles or notebooks do not include mathematical formulations; instead, they include images and illustrations from experimental devices and setups to the recapping of his theoretical ideas (Nersessian 2008 ). According to Gooding ( 2006 ), “Faraday’s visual method was designed not to copy apparent features of the world, but to analyse and replicate them” (2006, p. 46).

The lines of force played a central role in Faraday’s research on electricity and magnetism and in the development of his “field theory” (Faraday 1852a ; Nersessian 1984 ). Before Faraday, the experiments with iron filings around magnets were known and the term “magnetic curves” was used for the iron filing patterns and also for the geometrical constructs derived from the mathematical theory of magnetism (Gooding et al. 1993 ). However, Faraday used the lines of force for explaining his experimental observations and in constructing the theory of forces in magnetism and electricity. Examples of Faraday’s different illustrations of lines of magnetic force are given in Fig.  2 . Faraday gave the following experiment-based definition for the lines of magnetic forces:

a Iron filing pattern in case of bar magnet drawn by Faraday (Faraday 1852b , Plate IX, p. 158, Fig. 1), b Faraday’s drawing of lines of magnetic force in case of cylinder magnet, where the experimental procedure, knife blade showing the direction of lines, is combined into drawing (Faraday, 1855, vol. 1, plate 1)

A line of magnetic force may be defined as that line which is described by a very small magnetic needle, when it is so moved in either direction correspondent to its length, that the needle is constantly a tangent to the line of motion; or it is that line along which, if a transverse wire be moved in either direction, there is no tendency to the formation of any current in the wire, whilst if moved in any other direction there is such a tendency; or it is that line which coincides with the direction of the magnecrystallic axis of a crystal of bismuth, which is carried in either direction along it. The direction of these lines about and amongst magnets and electric currents, is easily represented and understood, in a general manner, by the ordinary use of iron filings. (Faraday 1852a , p. 25 (3071))

The definition describes the connection between the experiments and the visual representation of the results. Initially, the lines of force were just geometric representations, but later, Faraday treated them as physical objects (Nersessian 1984 ; Pocovi and Finlay 2002 ):

I have sometimes used the term lines of force so vaguely, as to leave the reader doubtful whether I intended it as a merely representative idea of the forces, or as the description of the path along which the power was continuously exerted. … wherever the expression line of force is taken simply to represent the disposition of forces, it shall have the fullness of that meaning; but that wherever it may seem to represent the idea of the physical mode of transmission of the force, it expresses in that respect the opinion to which I incline at present. The opinion may be erroneous, and yet all that relates or refers to the disposition of the force will remain the same. (Faraday, 1852a , p. 55-56 (3075))

He also felt that the lines of force had greater explanatory power than the dominant theory of action-at-a-distance:

Now it appears to me that these lines may be employed with great advantage to represent nature, condition, direction and comparative amount of the magnetic forces; and that in many cases they have, to the physical reasoned at least, a superiority over that method which represents the forces as concentrated in centres of action… (Faraday, 1852a , p. 26 (3074))

For giving some insight to Faraday’s visual reasoning as an epistemic practice, the following examples of Faraday’s studies of the lines of magnetic force (Faraday 1852a , 1852b ) are presented:

(a) Asking questions and defining problems: The iron filing patterns formed the empirical basis for the visual model: 2D visualization of lines of magnetic force as presented in Fig.  2 . According to Faraday, these iron filing patterns were suitable for illustrating the direction and form of the magnetic lines of force (emphasis added):

It must be well understood that these forms give no indication by their appearance of the relative strength of the magnetic force at different places, inasmuch as the appearance of the lines depends greatly upon the quantity of filings and the amount of tapping; but the direction and forms of these lines are well given, and these indicate, in a considerable degree, the direction in which the forces increase and diminish . (Faraday 1852b , p.158 (3237))

Despite being static and two dimensional on paper, the lines of magnetic force were dynamical (Nersessian 1992 , 2008 ) and three dimensional for Faraday (see Fig.  2 b). For instance, Faraday described the lines of force “expanding”, “bending,” and “being cut” (Nersessian 1992 ). In Fig.  2 b, Faraday has summarized his experiment (bar magnet and knife blade) and its results (lines of force) in one picture.

(b) Analyzing and interpreting data: The model was so powerful for Faraday that he ended up thinking them as physical objects (e.g., Nersessian 1984 ), i.e., making interpretations of the way forces act. Of course, he made a lot of experiments for showing the physical existence of the lines of force, but he did not succeed in it (Nersessian 1984 ). The following quote illuminates Faraday’s use of the lines of force in different situations:

The study of these lines has, at different times, been greatly influential in leading me to various results, which I think prove their utility as well as fertility. Thus, the law of magneto-electric induction; the earth’s inductive action; the relation of magnetism and light; diamagnetic action and its law, and magnetocrystallic action, are the cases of this kind… (Faraday 1852a , p. 55 (3174))

(c) Experimenting: In Faraday's case, he used a lot of exploratory experiments; in case of lines of magnetic force, he used, e.g., iron filings, magnetic needles, or current carrying wires (see the quote above). The magnetic field is not directly observable and the representation of lines of force was a visual model, which includes the direction, form, and magnitude of field.

(d) Modeling: There is no denying that the lines of magnetic force are visual by nature. Faraday’s views of lines of force developed gradually during the years, and he applied and developed them in different contexts such as electromagnetic, electrostatic, and magnetic induction (Nersessian 1984 ). An example of Faraday’s explanation of the effect of the wire b’s position to experiment is given in Fig.  3 . In Fig.  3 , few magnetic lines of force are drawn, and in the quote below, Faraday is explaining the effect using these magnetic lines of force (emphasis added):

Picture of an experiment with different arrangements of wires ( a , b’ , b” ), magnet, and galvanometer. Note the lines of force drawn around the magnet. (Faraday 1852a , p. 34)

It will be evident by inspection of Fig. 3 , that, however the wires are carried away, the general result will, according to the assumed principles of action, be the same; for if a be the axial wire, and b’, b”, b”’ the equatorial wire, represented in three different positions, whatever magnetic lines of force pass across the latter wire in one position, will also pass it in the other, or in any other position which can be given to it. The distance of the wire at the place of intersection with the lines of force, has been shown, by the experiments (3093.), to be unimportant. (Faraday 1852a , p. 34 (3099))

In summary, by examining the history of Faraday’s use of lines of force, we showed how visual imagery and reasoning played an important part in Faraday’s construction and representation of his “field theory”. As Gooding has stated, “many of Faraday’s sketches are far more that depictions of observation, they are tools for reasoning with and about phenomena” (2006, p. 59).

Case study 3: visualizing scientific methods, the case of a journal

The focus of the third case study is the Journal of Visualized Experiments (JoVE) , a peer-reviewed publication indexed in PubMed. The journal devoted to the publication of biological, medical, chemical, and physical research in a video format. The journal describes its history as follows:

JoVE was established as a new tool in life science publication and communication, with participation of scientists from leading research institutions. JoVE takes advantage of video technology to capture and transmit the multiple facets and intricacies of life science research. Visualization greatly facilitates the understanding and efficient reproduction of both basic and complex experimental techniques, thereby addressing two of the biggest challenges faced by today's life science research community: i) low transparency and poor reproducibility of biological experiments and ii) time and labor-intensive nature of learning new experimental techniques. ( http://www.jove.com/ )

By examining the journal content, we generate a set of categories that can be considered as indicators of relevance and significance in terms of epistemic practices of science that have relevance for science education. For example, the quote above illustrates how scientists view some norms of scientific practice including the norms of “transparency” and “reproducibility” of experimental methods and results, and how the visual format of the journal facilitates the implementation of these norms. “Reproducibility” can be considered as an epistemic criterion that sits at the heart of what counts as an experimental procedure in science:

Investigating what should be reproducible and by whom leads to different types of experimental reproducibility, which can be observed to play different roles in experimental practice. A successful application of the strategy of reproducing an experiment is an achievement that may depend on certain isiosyncratic aspects of a local situation. Yet a purely local experiment that cannot be carried out by other experimenters and in other experimental contexts will, in the end be unproductive in science. (Sarkar and Pfeifer 2006 , p.270)

We now turn to an article on “Elevated Plus Maze for Mice” that is available for free on the journal website ( http://www.jove.com/video/1088/elevated-plus-maze-for-mice ). The purpose of this experiment was to investigate anxiety levels in mice through behavioral analysis. The journal article consists of a 9-min video accompanied by text. The video illustrates the handling of the mice in soundproof location with less light, worksheets with characteristics of mice, computer software, apparatus, resources, setting up the computer software, and the video recording of mouse behavior on the computer. The authors describe the apparatus that is used in the experiment and state how procedural differences exist between research groups that lead to difficulties in the interpretation of results:

The apparatus consists of open arms and closed arms, crossed in the middle perpendicularly to each other, and a center area. Mice are given access to all of the arms and are allowed to move freely between them. The number of entries into the open arms and the time spent in the open arms are used as indices of open space-induced anxiety in mice. Unfortunately, the procedural differences that exist between laboratories make it difficult to duplicate and compare results among laboratories.

The authors’ emphasis on the particularity of procedural context echoes in the observations of some philosophers of science:

It is not just the knowledge of experimental objects and phenomena but also their actual existence and occurrence that prove to be dependent on specific, productive interventions by the experimenters” (Sarkar and Pfeifer 2006 , pp. 270-271)

The inclusion of a video of the experimental procedure specifies what the apparatus looks like (Fig.  4 ) and how the behavior of the mice is captured through video recording that feeds into a computer (Fig.  5 ). Subsequently, a computer software which captures different variables such as the distance traveled, the number of entries, and the time spent on each arm of the apparatus. Here, there is visual information at different levels of representation ranging from reconfiguration of raw video data to representations that analyze the data around the variables in question (Fig.  6 ). The practice of levels of visual representations is not particular to the biological sciences. For instance, they are commonplace in nanotechnological practices:

Visual illustration of apparatus

Video processing of experimental set-up

Computer software for video input and variable recording

In the visualization processes, instruments are needed that can register the nanoscale and provide raw data, which needs to be transformed into images. Some Imaging Techniques have software incorporated already where this transformation automatically takes place, providing raw images. Raw data must be translated through the use of Graphic Software and software is also used for the further manipulation of images to highlight what is of interest to capture the (inferred) phenomena -- and to capture the reader. There are two levels of choice: Scientists have to choose which imaging technique and embedded software to use for the job at hand, and they will then have to follow the structure of the software. Within such software, there are explicit choices for the scientists, e.g. about colour coding, and ways of sharpening images. (Ruivenkamp and Rip 2010 , pp.14–15)

On the text that accompanies the video, the authors highlight the role of visualization in their experiment:

Visualization of the protocol will promote better understanding of the details of the entire experimental procedure, allowing for standardization of the protocols used in different laboratories and comparisons of the behavioral phenotypes of various strains of mutant mice assessed using this test.

The software that takes the video data and transforms it into various representations allows the researchers to collect data on mouse behavior more reliably. For instance, the distance traveled across the arms of the apparatus or the time spent on each arm would have been difficult to observe and record precisely. A further aspect to note is how the visualization of the experiment facilitates control of bias. The authors illustrate how the olfactory bias between experimental procedures carried on mice in sequence is avoided by cleaning the equipment.

Our discussion highlights the role of visualization in science, particularly with respect to presenting visualization as part of the scientific practices. We have used case studies from the history of science highlighting a scientist’s account of how visualization played a role in the discovery of DNA and the magnetic field and from a contemporary illustration of a science journal’s practices in incorporating visualization as a way to communicate new findings and methodologies. Our implicit aim in drawing from these case studies was the need to align science education with scientific practices, particularly in terms of how visual representations, stable or dynamic, can engage students in the processes of science and not only to be used as tools for cognitive development in science. Our approach was guided by the notion of “knowledge-as-practice” as advanced by Knorr Cetina ( 1999 ) who studied scientists and characterized their knowledge as practice, a characterization which shifts focus away from ideas inside scientists’ minds to practices that are cultural and deeply contextualized within fields of science. She suggests that people working together can be examined as epistemic cultures whose collective knowledge exists as practice.

It is important to stress, however, that visual representations are not used in isolation, but are supported by other types of evidence as well, or other theories (i.e., in order to understand the helical form of DNA, or the structure, chemistry knowledge was needed). More importantly, this finding can also have implications when teaching science as argument (e.g., Erduran and Jimenez-Aleixandre 2008 ), since the verbal evidence used in the science classroom to maintain an argument could be supported by visual evidence (either a model, representation, image, graph, etc.). For example, in a group of students discussing the outcomes of an introduced species in an ecosystem, pictures of the species and the ecosystem over time, and videos showing the changes in the ecosystem, and the special characteristics of the different species could serve as visual evidence to help the students support their arguments (Evagorou et al. 2012 ). Therefore, an important implication for the teaching of science is the use of visual representations as evidence in the science curriculum as part of knowledge production. Even though studies in the area of science education have focused on the use of models and modeling as a way to support students in the learning of science (Dori et al. 2003 ; Lehrer and Schauble 2012 ; Mendonça and Justi 2013 ; Papaevripidou et al. 2007 ) or on the use of images (i.e., Korfiatis et al. 2003 ), with the term using visuals as evidence, we refer to the collection of all forms of visuals and the processes involved.

Another aspect that was identified through the case studies is that of the visual reasoning (an integral part of Faraday’s investigations). Both the verbalization and visualization were part of the process of generating new knowledge (Gooding 2006 ). Even today, most of the textbooks use the lines of force (or just field lines) as a geometrical representation of field, and the number of field lines is connected to the quantity of flux. Often, the textbooks use the same kind of visual imagery than in what is used by scientists. However, when using images, only certain aspects or features of the phenomena or data are captured or highlighted, and often in tacit ways. Especially in textbooks, the process of producing the image is not presented and instead only the product—image—is left. This could easily lead to an idea of images (i.e., photos, graphs, visual model) being just representations of knowledge and, in the worse case, misinterpreted representations of knowledge as the results of Pocovi and Finlay ( 2002 ) in case of electric field lines show. In order to avoid this, the teachers should be able to explain how the images are produced (what features of phenomena or data the images captures, on what ground the features are chosen to that image, and what features are omitted); in this way, the role of visualization in knowledge production can be made “visible” to students by engaging them in the process of visualization.

The implication of these norms for science teaching and learning is numerous. The classroom contexts can model the generation, sharing and evaluation of evidence, and experimental procedures carried out by students, thereby promoting not only some contemporary cultural norms in scientific practice but also enabling the learning of criteria, standards, and heuristics that scientists use in making decisions on scientific methods. As we have demonstrated with the three case studies, visual representations are part of the process of knowledge growth and communication in science, as demonstrated with two examples from the history of science and an example from current scientific practices. Additionally, visual information, especially with the use of technology is a part of students’ everyday lives. Therefore, we suggest making use of students’ knowledge and technological skills (i.e., how to produce their own videos showing their experimental method or how to identify or provide appropriate visual evidence for a given topic), in order to teach them the aspects of the nature of science that are often neglected both in the history of science and the design of curriculum. Specifically, what we suggest in this paper is that students should actively engage in visualization processes in order to appreciate the diverse nature of doing science and engage in authentic scientific practices.

However, as a word of caution, we need to distinguish the products and processes involved in visualization practices in science:

If one considers scientific representations and the ways in which they can foster or thwart our understanding, it is clear that a mere object approach, which would devote all attention to the representation as a free-standing product of scientific labor, is inadequate. What is needed is a process approach: each visual representation should be linked with its context of production (Pauwels 2006 , p.21).

The aforementioned suggests that the emphasis in visualization should shift from cognitive understanding—using the products of science to understand the content—to engaging in the processes of visualization. Therefore, an implication for the teaching of science includes designing curriculum materials and learning environments that create a social and epistemic context and invite students to engage in the practice of visualization as evidence, reasoning, experimental procedure, or a means of communication (as presented in the three case studies) and reflect on these practices (Ryu et al. 2015 ).

Finally, a question that arises from including visualization in science education, as well as from including scientific practices in science education is whether teachers themselves are prepared to include them as part of their teaching (Bybee 2014 ). Teacher preparation programs and teacher education have been critiqued, studied, and rethought since the time they emerged (Cochran-Smith 2004 ). Despite the years of history in teacher training and teacher education, the debate about initial teacher training and its content still pertains in our community and in policy circles (Cochran-Smith 2004 ; Conway et al. 2009 ). In the last decades, the debate has shifted from a behavioral view of learning and teaching to a learning problem—focusing on that way not only on teachers’ knowledge, skills, and beliefs but also on making the connection of the aforementioned with how and if pupils learn (Cochran-Smith 2004 ). The Science Education in Europe report recommended that “Good quality teachers, with up-to-date knowledge and skills, are the foundation of any system of formal science education” (Osborne and Dillon 2008 , p.9).

However, questions such as what should be the emphasis on pre-service and in-service science teacher training, especially with the new emphasis on scientific practices, still remain unanswered. As Bybee ( 2014 ) argues, starting from the new emphasis on scientific practices in the NGSS, we should consider teacher preparation programs “that would provide undergraduates opportunities to learn the science content and practices in contexts that would be aligned with their future work as teachers” (p.218). Therefore, engaging pre- and in-service teachers in visualization as a scientific practice should be one of the purposes of teacher preparation programs.

Achieve. (2013). The next generation science standards (pp. 1–3). Retrieved from http://www.nextgenscience.org/ .

Google Scholar  

Barber, J, Pearson, D, & Cervetti, G. (2006). Seeds of science/roots of reading . California: The Regents of the University of California.

Bungum, B. (2008). Images of physics: an explorative study of the changing character of visual images in Norwegian physics textbooks. NorDiNa, 4 (2), 132–141.

Bybee, RW. (2014). NGSS and the next generation of science teachers. Journal of Science Teacher Education, 25 (2), 211–221. doi: 10.1007/s10972-014-9381-4 .

Article   Google Scholar  

Chambers, D. (1983). Stereotypic images of the scientist: the draw-a-scientist test. Science Education, 67 (2), 255–265.

Cochran-Smith, M. (2004). The problem of teacher education. Journal of Teacher Education, 55 (4), 295–299. doi: 10.1177/0022487104268057 .

Conway, PF, Murphy, R, & Rath, A. (2009). Learning to teach and its implications for the continuum of teacher education: a nine-country cross-national study .

Crick, F. (1988). What a mad pursuit . USA: Basic Books.

Dimopoulos, K, Koulaidis, V, & Sklaveniti, S. (2003). Towards an analysis of visual images in school science textbooks and press articles about science and technology. Research in Science Education, 33 , 189–216.

Dori, YJ, Tal, RT, & Tsaushu, M. (2003). Teaching biotechnology through case studies—can we improve higher order thinking skills of nonscience majors? Science Education, 87 (6), 767–793. doi: 10.1002/sce.10081 .

Duschl, RA, & Bybee, RW. (2014). Planning and carrying out investigations: an entry to learning and to teacher professional development around NGSS science and engineering practices. International Journal of STEM Education, 1 (1), 12. doi: 10.1186/s40594-014-0012-6 .

Duschl, R., Schweingruber, H. A., & Shouse, A. (2008). Taking science to school . Washington DC: National Academies Press.

Erduran, S, & Jimenez-Aleixandre, MP (Eds.). (2008). Argumentation in science education: perspectives from classroom-based research . Dordrecht: Springer.

Eurydice. (2012). Developing key competencies at school in Europe: challenges and opportunities for policy – 2011/12 (pp. 1–72).

Evagorou, M, Jimenez-Aleixandre, MP, & Osborne, J. (2012). “Should we kill the grey squirrels?” A study exploring students’ justifications and decision-making. International Journal of Science Education, 34 (3), 401–428. doi: 10.1080/09500693.2011.619211 .

Faraday, M. (1852a). Experimental researches in electricity. – Twenty-eighth series. Philosophical Transactions of the Royal Society of London, 142 , 25–56.

Faraday, M. (1852b). Experimental researches in electricity. – Twenty-ninth series. Philosophical Transactions of the Royal Society of London, 142 , 137–159.

Gilbert, JK. (2010). The role of visual representations in the learning and teaching of science: an introduction (pp. 1–19).

Gilbert, J., Reiner, M. & Nakhleh, M. (2008). Visualization: theory and practice in science education . Dordrecht, The Netherlands: Springer.

Gooding, D. (2006). From phenomenology to field theory: Faraday’s visual reasoning. Perspectives on Science, 14 (1), 40–65.

Gooding, D, Pinch, T, & Schaffer, S (Eds.). (1993). The uses of experiment: studies in the natural sciences . Cambridge: Cambridge University Press.

Hogan, K, & Maglienti, M. (2001). Comparing the epistemological underpinnings of students’ and scientists’ reasoning about conclusions. Journal of Research in Science Teaching, 38 (6), 663–687.

Knorr Cetina, K. (1999). Epistemic cultures: how the sciences make knowledge . Cambridge: Harvard University Press.

Korfiatis, KJ, Stamou, AG, & Paraskevopoulos, S. (2003). Images of nature in Greek primary school textbooks. Science Education, 88 (1), 72–89. doi: 10.1002/sce.10133 .

Latour, B. (2011). Visualisation and cognition: drawing things together (pp. 1–32).

Latour, B, & Woolgar, S. (1979). Laboratory life: the construction of scientific facts . Princeton: Princeton University Press.

Lehrer, R, & Schauble, L. (2012). Seeding evolutionary thinking by engaging children in modeling its foundations. Science Education, 96 (4), 701–724. doi: 10.1002/sce.20475 .

Longino, H. E. (2002). The fate of knowledge . Princeton: Princeton University Press.

Lynch, M. (2006). The production of scientific images: vision and re-vision in the history, philosophy, and sociology of science. In L Pauwels (Ed.), Visual cultures of science: rethinking representational practices in knowledge building and science communication (pp. 26–40). Lebanon, NH: Darthmouth College Press.

Lynch, M. & S. Y. Edgerton Jr. (1988). ‘Aesthetic and digital image processing representational craft in contemporary astronomy’, in G. Fyfe & J. Law (eds), Picturing Power; Visual Depictions and Social Relations (London, Routledge): 184 – 220.

Mendonça, PCC, & Justi, R. (2013). An instrument for analyzing arguments produced in modeling-based chemistry lessons. Journal of Research in Science Teaching, 51 (2), 192–218. doi: 10.1002/tea.21133 .

National Research Council (2000). Inquiry and the national science education standards . Washington DC: National Academies Press.

National Research Council (2012). A framework for K-12 science education . Washington DC: National Academies Press.

Nersessian, NJ. (1984). Faraday to Einstein: constructing meaning in scientific theories . Dordrecht: Martinus Nijhoff Publishers.

Book   Google Scholar  

Nersessian, NJ. (1992). How do scientists think? Capturing the dynamics of conceptual change in science. In RN Giere (Ed.), Cognitive Models of Science (pp. 3–45). Minneapolis: University of Minnesota Press.

Nersessian, NJ. (2008). Creating scientific concepts . Cambridge: The MIT Press.

Osborne, J. (2014). Teaching scientific practices: meeting the challenge of change. Journal of Science Teacher Education, 25 (2), 177–196. doi: 10.1007/s10972-014-9384-1 .

Osborne, J. & Dillon, J. (2008). Science education in Europe: critical reflections . London: Nuffield Foundation.

Papaevripidou, M, Constantinou, CP, & Zacharia, ZC. (2007). Modeling complex marine ecosystems: an investigation of two teaching approaches with fifth graders. Journal of Computer Assisted Learning, 23 (2), 145–157. doi: 10.1111/j.1365-2729.2006.00217.x .

Pauwels, L. (2006). A theoretical framework for assessing visual representational practices in knowledge building and science communications. In L Pauwels (Ed.), Visual cultures of science: rethinking representational practices in knowledge building and science communication (pp. 1–25). Lebanon, NH: Darthmouth College Press.

Philips, L., Norris, S. & McNab, J. (2010). Visualization in mathematics, reading and science education . Dordrecht, The Netherlands: Springer.

Pocovi, MC, & Finlay, F. (2002). Lines of force: Faraday’s and students’ views. Science & Education, 11 , 459–474.

Richards, A. (2003). Argument and authority in the visual representations of science. Technical Communication Quarterly, 12 (2), 183–206. doi: 10.1207/s15427625tcq1202_3 .

Rothbart, D. (1997). Explaining the growth of scientific knowledge: metaphors, models and meaning . Lewiston, NY: Mellen Press.

Ruivenkamp, M, & Rip, A. (2010). Visualizing the invisible nanoscale study: visualization practices in nanotechnology community of practice. Science Studies, 23 (1), 3–36.

Ryu, S, Han, Y, & Paik, S-H. (2015). Understanding co-development of conceptual and epistemic understanding through modeling practices with mobile internet. Journal of Science Education and Technology, 24 (2-3), 330–355. doi: 10.1007/s10956-014-9545-1 .

Sarkar, S, & Pfeifer, J. (2006). The philosophy of science, chapter on experimentation (Vol. 1, A-M). New York: Taylor & Francis.

Schwartz, RS, Lederman, NG, & Abd-el-Khalick, F. (2012). A series of misrepresentations: a response to Allchin’s whole approach to assessing nature of science understandings. Science Education, 96 (4), 685–692. doi: 10.1002/sce.21013 .

Schwarz, CV, Reiser, BJ, Davis, EA, Kenyon, L, Achér, A, Fortus, D, et al. (2009). Developing a learning progression for scientific modeling: making scientific modeling accessible and meaningful for learners. Journal of Research in Science Teaching, 46 (6), 632–654. doi: 10.1002/tea.20311 .

Watson, J. (1968). The Double Helix: a personal account of the discovery of the structure of DNA . New York: Scribner.

Watson, J, & Berry, A. (2004). DNA: the secret of life . New York: Alfred A. Knopf.

Wickman, PO. (2004). The practical epistemologies of the classroom: a study of laboratory work. Science Education, 88 , 325–344.

Wu, HK, & Shah, P. (2004). Exploring visuospatial thinking in chemistry learning. Science Education, 88 (3), 465–492. doi: 10.1002/sce.10126 .

Download references

Acknowledgements

The authors would like to acknowledge all reviewers for their valuable comments that have helped us improve the manuscript.

Author information

Authors and affiliations.

University of Nicosia, 46, Makedonitissa Avenue, Egkomi, 1700, Nicosia, Cyprus

Maria Evagorou

University of Limerick, Limerick, Ireland

Sibel Erduran

University of Tampere, Tampere, Finland

Terhi Mäntylä

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Maria Evagorou .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors’ contributions

ME carried out the introductory literature review, the analysis of the first case study, and drafted the manuscript. SE carried out the analysis of the third case study and contributed towards the “Conclusions” section of the manuscript. TM carried out the second case study. All authors read and approved the final manuscript.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( https://creativecommons.org/licenses/by/4.0 ), which permits use, duplication, adaptation, distribution, and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Evagorou, M., Erduran, S. & Mäntylä, T. The role of visual representations in scientific practices: from conceptual understanding and knowledge generation to ‘seeing’ how science works. IJ STEM Ed 2 , 11 (2015). https://doi.org/10.1186/s40594-015-0024-x

Download citation

Received : 29 September 2014

Accepted : 16 May 2015

Published : 19 July 2015

DOI : https://doi.org/10.1186/s40594-015-0024-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Visual representations
• Epistemic practices
• Science learning

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Front Psychol

Differences between Spatial and Visual Mental Representations

Jan frederik sima.

1 SFB/TR 8 Spatial Cognition, Universität Bremen, Bremen, Germany

Holger Schultheis

Thomas barkowsky.

This article investigates the relationship between visual mental representations and spatial mental representations in human visuo-spatial processing. By comparing two common theories of visuo-spatial processing – mental model theory and the theory of mental imagery – we identified two open questions: (1) which representations are modality-specific, and (2) what is the role of the two representations in reasoning. Two experiments examining eye movements and preferences for under-specified problems were conducted to investigate these questions. We found that significant spontaneous eye movements along the processed spatial relations occurred only when a visual mental representation is employed, but not with a spatial mental representation. Furthermore, the preferences for the answers of the under-specified problems differed between the two mental representations. The results challenge assumptions made by mental model theory and the theory of mental imagery.

1. Introduction

Our everyday behavior relies on our ability to process visual and spatial information. Describing the route to work, taking another person’s perspective, or imagining a familiar face or object all depend on our capability to process and reason with visual and spatial information.

Two main theoretic frameworks of visual and spatial knowledge processing have been proposed in cognitive science: mental model theory (Johnson-Laird, 1989 , 1998 ; Tversky, 1993 ) and mental imagery (Finke, 1989 ; Kosslyn, 1994 ; Kosslyn et al., 2006 ). Furthermore, there is also the conception of verbal or propositional mental representations (Rips, 1994 ; Pylyshyn, 2002 ) that employ a sort of logical inference to reason about visual and/or spatial information. However, considerable evidence indicates that analogical mental representations, i.e., mental models or mental images, can better predict and explain the empirical data, in particular, for spatial reasoning (e.g., Byrne and Johnson-Laird, 1989 ; Kosslyn, 1994 ; Johnson-Laird, 2001 ).

In line with behavioral and neuroscientific evidence (e.g., Ungerleider and Mishkin, 1982 ; Levine et al., 1985 ; Newcombe et al., 1987 ; Farah et al., 1988 ; Courtney et al., 1996 ; Smith and Jonides, 1997 ; Mellet et al., 2000 ; Knauff and Johnson-Laird, 2002 ; Klauer and Zhao, 2004 ), mental model theory and the theory of mental imagery both propose a distinction between spatial and visual mental representations. The theory of mental imagery proposes spatial mental images and visual mental images; mental model theory proposes (spatial) mental models and visual mental images. Research based on the theories has, however, mostly focused on one of the two representations: the investigation of the properties of visual mental images in the theory of mental imagery and the investigation of reasoning with (spatial) mental models in mental model theory. Consequently, the relationship and interaction between the two types of mental representations is left largely unspecified in both theories. Although initial attempts exist (e.g., Schultheis and Barkowsky, 2011 ) to explain how visual and spatial mental representations interact and relate to each other, empirical data on the issue is largely missing. Accordingly, the primary aim of this article is to examine the differences and the relationship between visual and spatial mental representations. To achieve this, we first review how mental model theory on the one hand and the theory of mental imagery on the other hand understand spatial and visual mental representations as well as how they interpret the relationship between them. Even though there is much theoretic and empirical work on both theories, the literature lacks a systematic comparison of the theories. In the following, we present such a comparison. From this comparison, it will become clear that the theories actually propose very similar conceptions of spatial and visual mental representations but that their foci of investigation are mostly on different aspects and include phenomena not investigated within the respectively other theory. We examined these different aspects and used them in our experiments to gain new insights into the open issues of the relationship between visual and spatial mental representations. The results can be applied to complement gaps in the two theories.

2. Theories

2.1. mental model theory.

Mental model theory (Johnson-Laird, 1998 ) postulates that there are three representational levels involved in human thinking: propositional representations, mental models, and mental images. The relationships between these three levels are hierarchical in the sense that their construction depends on each other. The following example helps to illustrate this point. Three-term series problems (Johnson-Laird, 1972 ) are common experimental tasks in the study of mental models. They contain two premises and one conclusion that has to be validated or inferred based on the premises. Let the two premises be “A is left of B” and “B is right of C” and let the to-be-drawn conclusion be the relationship between A and C. According to mental model theory the premises are first encoded propositionally. From these propositional premises a mental model of the described configuration is constructed. As it is an essential property of mental models that “the structural relations between the parts of the model are analogous to the structural relations in the world” (Johnson-Laird, 1998 , p. 447), one valid mental model of our example can be depicted in the following way:

We note that a mental model is a special case of the situation defined by the premises, because it only represents one valid situation with respect to the premises. For our example another mental model that satisfies the premises is:

Just like the situation represented by a mental model is a special case of what is described in the premises, mental model theory poses that a mental image is a special case of a given mental model. The mental image that is constructed from a mental model is one specific instance out of many valid instances described by the model, because the image has to specify, for example, the distance between the entities. The underlying mental model is in contrast invariant with respect to the distances. Summarizing the hierarchical structure of mental model theory, we note that a mental image is one out of many projections of the visualizable aspects of a mental model, and a mental model is one out of many analogically structured configurations that are valid given the propositionally represented premises. This suggests a clear hierarchy in which it is necessary to have the more general representations in order to construct a more specific one.

Mental models are described to be analogically structured, amodal, and abstract, e.g., they can represent abstract, non-visualizable relations such as “smarter than.” In contrast, mental images can only represent “visualizable” information and are modality-specific to visual perception (e.g., Johnson-Laird, 1998 ; Knauff and Johnson-Laird, 2002 ). It has been suggested that the analogical nature of mental models might be generally spatial (Knauff et al., 2003 ), i.e., even reasoning with abstract relations like “worse than” or “better than” is handled by a spatio-analogical mental model. This view is supported by the association of mental model reasoning with activation in the parietal lobe (e.g., Goel and Dolan, 2001 ; Knauff et al., 2003 ), which is associated with several processes of spatial cognition (for an overview, see Sack, 2009 ). It was found that the use of “visual” relations, e.g., “dirtier than,” in relational reasoning tasks led to activation in the early visual cortex in contrast to tasks with other (abstract) relations, e.g., “worse than” (Knauff et al., 2003 ). The study also found that “visual” relations led to longer reaction times and it was concluded that tasks using such “visual” relations induce the employment of visual mental images during the mental-model-based reasoning process.

Most of the literature on mental model theory focuses on how mental models explain reasoning. Johnson-Laird and Byrne ( 1991 ) state that reasoning according to the mental model theory consists of three stages: (1) the construction of one mental model (construction phase), (2) the inspection of the mental model (inspection phase), and (3) the variation of the mental model (variation phase). Slightly simplified, the reasoning process works as follows. One first mental model is constructed based on the given premises. This model represents one situation that is valid given the premises. This situation is inspected and can yield a possible conclusion. This conclusion is then verified to be valid in all other possible mental models that can be derived from the premises. If a conclusion is not contradicted in the other valid mental models, the conclusion is confirmed. There is much empirical support for this three stage process in human reasoning (e.g., Johnson-Laird, 2001 ). One interesting phenomenon in reasoning with mental models is the occurrence of preferred mental models when there are multiple valid conclusions. An example for such multiple valid conclusions are the two configurations “CAB” and “ACB” of the above example. It can be observed that there are reliable within-subject and between-subject preferences for which model is constructed first out of several valid mental models. This firstly constructed mental model is termed a preferred mental model. As a consequence, if there are several valid conclusions that can be inferred, there is a preference for one conclusion which corresponds to the preferred mental model. Preferred mental models have been investigated in different domains, but in particular in the domain of spatial reasoning (e.g., Rauh et al., 2005 ; Jahn et al., 2007 ; Schultheis and Barkowsky, 2013 ).

2.2. Theory of mental imagery

The theory of mental imagery (Kosslyn, 1994 ; Kosslyn et al., 2006 ) makes a distinction between spatial mental images and visual mental images. These two mental representations differ in the content they represent and are distinct in their anatomical localization. But they are both assumed to have a (at least partially) spatio-analogical structure. Furthermore, there is also a propositional representation referred to as associative memory, which contains propositional descriptions of the structure of an object or a scene. This information can be used to construct spatial and visual mental images. For the latter, however, one needs to further retrieve encoded shape information from another source, i.e., the object-properties-processing subsystem, which can be thought of as a sort of non-analogical visual memory store located in the temporal lobe.

Spatial mental images (sometimes referred to as object maps) are located in the spatial-properties-processing subsystem in the framework of Kosslyn ( 1994 ). They contain information about the location, size, and orientation of entities. The spatial-properties-processing subsystem is (at least partially) placed in the parietal lobe. Given that areas of the parietal lobe are topographically organized (Sereno et al., 2001 ), it is assumed that spatial mental images are also at least partially spatio-analogical (Kosslyn et al., 2006 ).

Visual mental images are constructed and processed in a structure called the visual buffer. The visual buffer consists of the topographically organized areas of the visual cortex. Visual mental images are thus assumed to be spatio-analogical or “depictive,” i.e., the metrics of what is represented, e.g., a shape, are reflected in the metrics of the representation. Visual mental images represent shape information, as well as, for example, color and depth.

A difference between spatial and visual mental images is that spatial mental images contain more information, in the sense that the current visual mental image in the visual buffer only contains a “visualized” part of what is represented in the spatial mental image (Kosslyn et al., 2006 , p. 138). A visual mental image is a specification of a part of a spatial mental image.

Four types of functions are proposed for visual and spatial mental images: generation, inspection, maintenance, and manipulation. The generation of a mental image can either be just the retrieval of a spatial configuration of entities as a spatial mental image if no visual information is necessary for a given task or it can furthermore include the retrieval of shape information to generate a visual mental image in the visual buffer. Note that the visual buffer does not need to be employed for spatial mental images. Kosslyn et al. ( 2006 ) states that the processing of spatial and visual mental images occurs in parallel, i.e., the image of a shape is generated while a spatial image is generated. They furthermore state that this parallel processing might not always be useful, as the proper construction of a shape requires information about its spatial properties, i.e., location, size, and orientation which are provided by a respective spatial mental image (Kosslyn et al., 2006 , p. 143). For the generation of multi-part visual mental images, a corresponding spatial mental image is necessary to guide the placement of shapes in the visual buffer by specifying the location, orientation, and size. The inspection process can make previously implicit information in a visual or spatial image explicit, i.e., new information is inferred. Visual mental images are inspected by shifting an attention window over the visual buffer. Through this inspection visual information, e.g., properties of a shape, as well as spatial information, e.g., spatial relations, can be inferred. It is also possible that new information is inferred from only a spatial mental image. However, no detailed information on the inspection of/inference in spatial mental images is provided by the theory. The function of image maintenance is used to re-construct parts of mental images as the information fades over time. The function of image manipulation allows the imagination of transformations of mental images. The theory posits that such manipulations are realized by altering the object map, i.e., the spatial mental image, underlying the visual mental image. One would, for example, change the location or size of an entity in the spatial mental image to alter the visual mental image that contains the shape information of that entity.

One interesting phenomenon of mental imagery is the observation of spontaneous eye movements during different visual mental imagery tasks. Brandt and Stark ( 1997 ) had participants imagine a previously memorized grid pattern and found that the eye movements during imagination reflected the content of the original stimuli. Spontaneous eye movements that reflect the processed spatial relations during mental imagery have since been found, for example, during imagination of natural scenes (Holsanova et al., 1998 ), during imagination of detailed paintings and detailed descriptions of scenes while facing a white board as well as in total darkness (Johansson et al., 2006 ), during reasoning with “visual” syllogisms, e.g., “a jar of pickles is below a box of tea bags,” (Demarais and Cohen, 1998 ), and while listening to verbal descriptions of spatial scenes, e.g., “at the bottom there is a doorman in blue” (Spivey and Geng, 2001 ). Johansson et al. ( 2012 ) report a series of experiments, in which participants were selectively forced to not move their eyes during mental imagery. They found that the suppression of eye movements has an impact on the quantity and quality of mental imagery. Their results strongly indicate a functional role of eye movements during mental imagery.

2.3. Open questions

The previous two sections are summarized in Table ​ Table1 1 which provides a comparative overview of the two theories. From the comparison of the two theories, a great overlap in the assumptions made and structures and processes proposed by the two theories is evident. Many aspects of the two theories are revealed to be rather similar, perhaps more similar than one would have expected. In particular, they provide very similar descriptions of a spatial and a visual mental representation with respect to information content, localization, and hierarchical structure between the two representations. There are, however, some diverging predictions with respect to the modality of these representations and their role in reasoning. In the following, we discuss these differences and identify two main questions that arise from the comparison of these two theories.

Comparison of mental model theory and the theory of mental images .

The theory of mental imagery states that spatial mental images are processed in a component called the spatial-properties-processing subsystem. This subsystem is explicitly linked to the dorsal processing stream, which processes spatial information during visual perception (Kosslyn et al., 2006 , p.138). Processing of spatial mental images uses (at least partly) the same processes used during processing of spatial information in visual perception. Mental models on the other hand are commonly assumed to be amodal or multi-modal (e.g., Johnson-Laird and Byrne, 1991 ). Accordingly, mental models are assumed to be used to also reason about abstract, non-spatial, information, e.g., “A is better than B” (Knauff et al., 2003 ), whereas spatial mental images are assumed to process only spatial information. It has, however, been assumed that abstract information, e.g., “better than,” can be translated into spatial information in mental models (Knauff et al., 2003 ). An opinion seemingly shared by Kosslyn ( 1994 ), who states that information like “A is smarter than B” can be represented by dots on a line in a spatial mental image which would then correspond to a mental model in the sense of Johnson-Laird (Kosslyn, 1994 , p. 324). The question that remains is whether the spatial representation, described as a mental model or a spatial mental image, is actually amodal/multi-modal (as claimed by mental model theory) or linked to the modality of visual perception (as seemingly proposed by the theory of mental imagery). Results pointing either way would help refining the theories.

Another open issue is the theories’ seemingly different prediction on the role of the spatial mental representation in reasoning. Unfortunately, both theories remain vague regarding the details of how spatial and visual representations interact during reasoning. In mental model theory it is often explicitly stated that it is mental models and not mental images that underlie human reasoning (Knauff and Johnson-Laird, 2002 ; Knauff et al., 2003 ). The automatic generation of mental images through “visual” relations, e.g., “the dog is dirtier than the cat” is even considered to impede the reasoning process that happens on the level of mental models (Knauff and Johnson-Laird, 2002 ). Of course, mental images can be important for reasoning if certain visual information is necessary, but it is not described how such visual information would be interpreted by nor how it would be transferred into the mental model for further reasoning. In the theory of mental imagery, it is made clear that visual mental images play a major role in reasoning: “[I]magery plays a critical role in many types of reasoning” (Kosslyn, 1994 , p.404). And, contrasting mental model theory, visual mental images are assumed to be much more than just the provider of visual information for spatial mental images, in general, and particularly in reasoning (Kosslyn, 1994 ; Kosslyn et al., 2006 ). The inspection of visual mental images constructed in the visual buffer can lead to new insights and is thus directly involved in the reasoning processes. According to Kosslyn et al. ( 2006 ) a visual mental image is generated using an underlying spatial mental image. However, the concrete role of the spatial mental image in the reasoning process is never elaborated in a way that would suggest that the spatial mental image is of specific importance to reasoning or even that it might be the actual reasoning component (as proposed in mental model theory).

Summarizing, we pointed out two main open issues regarding the differences between spatial mental representations and visual mental representations: (1) whether the spatial mental representation is rather amodal/multi-modal or whether it is also directly linked to visual perception like the visual mental representation; (2) to which extent the two mental representations are involved in reasoning, i.e., whether the spatial mental representation is the primary reasoning component or not.

3. Experiments

The comparison of the two theories, furthermore, showed that there are phenomena which have mostly been investigated only within the framework of one of the two theories. Preferences in under-specified problems have so far only been investigated within the framework of mental model theory while eye movements have been a focus of investigation almost only with mental images. In the presented experiments, we investigated to which extent these two phenomena are transferable to the respectively other type of mental representation. That is, we checked for spontaneous eye movements during reasoning with a spatial mental representation, i.e., a (spatial) mental model, and we checked for possible preferences when employing a visual mental representation, i.e., a visual mental image. In the following, we describe how the investigation of these phenomena informs us about the open questions stated in Section 3.3 .

The tasks used in the experiments are three-term series relational reasoning problems about orientation knowledge. The two experiments differed only in their instructions which were formulated so that they induced the employment of a spatial mental representation in the first experiments and a visual mental representation in the second experiment.

We assume that we will confirm the findings of the literature that systematic eye movements occur during the second experiment (employing a visual mental representation) and that there are significant preferences in the answers of the participants in the first experiment (employing a spatial mental representation). The apparent functional role of eye movements during visual mental imagery provides strong evidence that visual mental representations are linked to processes of visual perception. These spontaneous eye movements reflect the spatial relations of the processed information. Both mental model theory and the theory of mental imagery assume spatial relations to be represented by a spatial mental representation, which supports the construction of a visual mental representation by providing the required spatial information. We tested whether such eye movements along the processed spatial relations would occur during employment of only a spatial mental representation, i.e., without the representation of visual content. The investigation of eye movements in this context can inform us about the question of the modality of spatial mental representations: if systematic eye movements occur during reasoning with spatial mental representations, then this would be a strong indication that mental models are not amodal, but are, in fact, linked to attentional processes of visual perception. A lack of systematic eye movements during reasoning with spatial mental representations, on the other hand, would support the assumption of mental model theory that mental models are amodal. More specifically, this would indicate that the processes of spatial mental representations do not employ the overt attentional processes of visual perception as it is the case for visual mental representations.

Preferred mental models are preferences for certain answers to under-specified reasoning problems that have been found for reasoning with mental models. These preferences are assumed to emerge because participants first construct one, perhaps the most parsimonious, mental model out of several valid models (e.g., Rauh et al., 2005 ). Visual mental images are also assumed to “depict” just one situation at a time; in fact it is hard to imagine how a “depictive” representation could represent more than one situation simultaneously. There are three possible outcomes for our investigation of such preferences for reasoning with visual mental representations: (1) we find no significant preferred answers, (2) we find different preferences for the two mental representations, or (3) we find the same preferences in reasoning with both mental representations. Finding no significant preferences in the answers when a visual mental representation is employed would strongly indicate that the assumption that visual mental representations build upon corresponding spatial mental representations is incorrect. Furthermore, this would indicate that the construction of visual mental representations can be subject to very strong individual differences. Such a finding seems unlikely and would not be predicted by any of the two theories. Should we find the same preferences in both experiments, i.e., for reasoning with both a spatial and a visual mental representation, the assumption of a hierarchal relationship between the two mental representations would be supported. This would strongly suggest that indeed the spatial configuration of a visual mental representation is taken from an underlying spatial mental representation. Should we find different preferences for the two mental representations, refinements of both mental model theory and the theory of mental imagery would be required to explain this disparity. In particular, such a finding would challenge the two theories to elaborate on their assumption that the construction of visual mental representations depends on an underlying spatial mental representation. Additionally, the strong claim made by mental model theory that reasoning is realized by spatial mental representations and not visual mental representations would without additional hypotheses be contradicted by this result.

In the following, the materials and methods employed in both conducted experiments are described.

3.1. Materials and apparatus

The tasks used in the experiments are under-specified three-term series problems about orientation knowledge, specifically cardinal directions. We chose these problems because problems of this type, i.e., three-term series relational spatial reasoning, have been used in several studies of mental model theory before (e.g., Knauff et al., 2003 ; Byrne and Johnson-Laird, 1989 ; Schultheis et al., in revision). We use an eight-sector model of cardinal directions, i.e., the eight directions are north, north-east, east, south-east, south, south-west, west, and north-west. The problems are of the following form:

• Premise 1: A is [direction 1] of B, e.g., A is north of B
• Premise 2: B is [direction 2] of C, e.g., B is east of C
• Conclusion: As seen from A, where is C?

The premises provide two spatial relations between three entities and the third spatial relation has to be inferred. In general, these problems are under-specified, i.e., there can be more than one correct conclusion given the premises. We used two classes of these problems, which we term 45° problems and 90° problems. These problems can be visualized as triangles with one of the three edges missing. This missing edge corresponds to the to-be-inferred spatial relation. We used all possible combinations in which the two given edges form either a 45° or a 90° angle. Figure ​ Figure1 1 depicts an overview of all these problems.

The 16 different types of problems used in the experiments . The upper eight are 45° problems and the lower eight are 90° problems.

We can identify all possible correct solutions for the two problem sets. The 90° problems have three possible solutions and the 45° problems have 4 possible solutions. The different configurations leading to these solutions are depicted in Figure ​ Figure2. 2 . To distinguish the different solutions, we classify the underlying mental representations based on a visualization of the solution as triangles. In this context we use the term “model” to describe the underlying mental configuration, whether it might be a spatial mental representation or a visual mental representation. Models with very different distances for the given spatial relations are termed distorted models (DM) ; models with roughly equal distances for the given relations are termed equal-distance models (EDM) . The remaining valid solution for the 45° problems, the third solution in Figure ​ Figure2, 2 , always leads to one of the four main cardinal directions being inferred and is therefore termed cardinal model (CM) .

Possible valid models for a 45° problem are depicted as 1, 2, 3, and 4 . Possible valid models for a 90° problem are depicted as 5, 6, and 7. The models 1, 4, 5, and 7 are termed distorted models (DM) because the distances between the entities vary a lot from each other. The models 2 and 6 have equal distances and are termed equal-distance models (EDM). The model 3 is termed cardinal model (CM) because the to-be-inferred relation corresponds to one of the main cardinal directions, i.e., north, east, south, or west.

There are 16 different possible problems. We used them all twice with different letters resulting in a total of 32 problems. The 16 different problems consist of eight 45° and eight 90° problems, as depicted in Figure ​ Figure1 1 .

Participants wore a head-mounted SensoMotoric Instruments (SMI) iView X HED eye tracker with a 200 Hz sampling rate to record their eye movements. To prevent expectancy effects, participants were told that the experiment investigates the size of their pupils. A post-experimental questionnaire verified that participants were not aware of the eye tracking.

3.2. Procedure

3.2.1. instructions.

The two experiments used slightly different instructions, so that they conformed with the usual instructions of both studies on mental models as well as studies on visual mental images. At the same time, the minimal change between the experiments helped to keep the tasks as similar as possible and minimize any differences besides the induced mental representation.

The instructions of the first experiment did not contain any suggestions to use visualization or visual information, but simply asked participants to infer the missing relation as fast and as accurately as possible. It is in line with previous experimental studies to assume the employment of mental models, i.e., a spatial mental representation, based on the fact that no visual information is required, given or asked for in the task (e.g., Johnson-Laird, 2001 ; Knauff and Johnson-Laird, 2002 ; Jahn et al., 2007 ).

The instructions of the second experiment only differed slightly from those of the first one. The participants were told that the letters represent cities that are to be imagined as little red squares with the respective letter next to them, which are all placed on a map. This slight variation made the instructions conform with those of several other visual mental imagery studies, i.e., using phrases such as “imagine […]” or “try to mentally see […]” (e.g., Kosslyn et al., 1983 ; Chambers and Reisberg, 1985 ; Borst et al., 2006 ).

In both experiments participants were asked to work as accurately and as fast as possible.

3.2.2. Setup

Participants were seated facing a blank white wall at a distance of approximately 1 m. Their hands were placed on their legs under a table holding a computer mouse in the one hand and a small ball in the other one. This was to prevent participants from using their fingers as an aid to solve the tasks. The eye tracker was mounted on the participant’s head and calibrated. All initial instructions of the experiment were projected on the white wall.

3.2.3. Learning phase

The experiment started with a learning phase to familiarize the participants with the cardinal directions. The learning phase consisted of acoustically presented statements and an answer screen with a question. Each statement was of the form “ K is [direction] of U .” After 4 s the answer screen appeared, which depicted the reference entity U surrounded by the numbers 1 to 8 in a counterclockwise circular order together with the question “ As seen from U, where is K? ” The eight numbers represented the eight cardinal directions (1 = north, 2 = north-west, 3 = west, … 8 = north-east). Participants answered by naming the respective number. In case of an incorrect answer, the correct answer was projected on the wall. The training phase ended as soon as each of the eight cardinal directions was recognized correctly twice in a row.

3.2.4. Problem trials

Participants were presented with a total of 48 trials. Out of those the first four were pre-trials intended to familiarize the participants with the form and procedure of the problems. Out of the remaining 44 trials, 12 were designed as filler trials. These filler trials differed in the order in which the entities were presented: AB, AC, BC, e.g., “ A is north of B; A is west of C; B is? of C ,” in contrast to the order of the remaining 32 problem trials: AB, BC, CA, e.g., “ A is north of B; B is east of C; C is? of A .” The filler trials served a double purpose. First, they were meant to prevent memory effects due to the identical order of all problem trials. Second, filler trials were employed to identify those time intervals in which participants show eye movements along the given directions. We elaborate on this method in Section 4.3 . After the presentation of the four pre-trials, the remaining 44 trials were presented in randomized order.

3.2.5. Presentation

All premises and questions were presented acoustically. There was no projection on the white wall during the premises; after the conclusion phase an answer screen was projected onto the wall. Participants used the mouse to trigger the acoustic presentation of the first premise in each trial. As soon as they understood the statement, they clicked again for the presentation of the second premise. Similarly, they triggered the acoustic presentation of the question after having understood the second premise. Only after participants found an answer, they clicked the mouse again making the answer screen appear. The answer screen was the same as the one used in the learning phase. Participants verbally gave their answer by naming the number associated with the resulting direction. Participants continued to the next trial by clicking the mouse again.

The participants took between 35 and 50 min to complete the experiment.

3.3. Processing of the eye tracking data

We processed the eye tracking data to identify whether eye movements occurred along the spatial relations given in each trial. We employed the same method for both experiments.

The raw eye tracking data collected by the iView X software was first converted using the IDF Event Detector to generate a list of fixations made by the participant. Saccades were calculated automatically from the sequence and coordinates of the participant’s fixations. Using the starting and ending coordinates of each saccade, we classified them into one of eight categories corresponding to the eight cardinal directions used in the trials. All possible angles of a saccade, interpreted as a vector in a Cartesian plane, were uniformly mapped to the set of cardinal directions. Each direction corresponds to a range of angles on the degree circle with each direction taking up (360°/8) = 45°. For example, north corresponded to all angles in the range of 0° ± (45°/2) = 0° ± 22.5° = [337.5°; 22.5°]. Note that the eye movements classified in this way are relative eye movements, i.e., the absolute coordinates do not matter. This is reasonable considering that participants moved their head during trials and that arbitrary eye movements occurred in between. Given this classification, we were able to investigate a possible coupling between the given direction and observed eye movements during a trial. If eye movements are linked to the processing of spatial relations, we expected eye movements to occur not only along the given direction, but also along the opposite one. Assuming a mental representation of, for example, A being north of B, it is plausible to not only expect attention shifts from A to B but also from B to A during inspection as well as construction of the representation. Thus, we always compared the absolute number of observed saccades to the sum of saccades made along the given and the opposite direction. For the first premise, we used the given direction, e.g., for the premise A is north of B we looked for saccades along the north-south axis. For the second premise, we used the direction given in the first premise as well as the new direction given in the second premise, e.g., for B is west of C , we looked for north-south (from premise 1) and for east-west. For the conclusion phase, we used the direction (and its opposite) that was given as the answer by the participant. We applied a binomial test with a probability of 1/4 to test whether the two expected directions were above chance for each participant for the first premise and the conclusion. For the second premise we applied a binomial test with a probability of 1/2 to test whether the four expected directions (two directions from each relation of the two premises) were above chance. For each phase we then applied a binomial test with a probability of 0.05 to check whether the number of participants showing significant eye movements is significantly above chance. The probability of 0.05 corresponds to how often a false positive of the previous binomial test is expected.

No prior information was available on when during the processing of the premises or the conclusion eye movements are to be expected. It is likely that participants spent some time understanding and verbally processing the presented premise or question before they started constructing the mental representation. Similarly, participants required some time preparing the action of clicking the mouse to trigger the next step after they finished the processing of the respective premise or question. We, therefore, used the obtained data during the first premises of the filler trials to gather information on when exactly participants started showing eye movements and whether we could find a temporal pattern. We only looked at eye movements during the first premise, because the filler trials are identical to the problem trials for the first premise. The difference in the order of the presented letters only became evident with the second premise. Therefore, we assumed the same behavior in the first premises of both the problem and the filler trials. We looked at the time interval between the first mention of the direction in the first premise and the time participants click to initiate the second premise. This interval was divided into ten equally long time slots. For each of these ten slots we summed up the eye movements of all participants for each experiment. We checked whether eye movements along the expected directions, i.e., those given in the respective premise (and its opposite), were significantly above chance in each of these intervals. We applied a binomial test using a probability of 1/4 for each of the four pairs of cardinal directions, e.g., north/south compared to east/west, north-east/south-west, and north-west/south-east. We applied this method independently for both experiments and used the identified time slots for the eye movement analysis of the problem trials.

3.4. Ethics statement

The study was conducted within the Collaborative Research Center Spatial Cognition SFB/TR 8 funded by the German Research Foundation (DFG). The DFG’s board of ethics passed the research proposal that underlies the present study. DFG-funded projects do not require additional approval by other ethics committees. The studies are in full agreement with the ethical guidelines of the German Psychological Society (DGPs). Written informed consent was acquired from all participants.

4.1. Experiment 1: Spatial mental representation

4.1.1. participants.

Thirty undergraduate students of the University of Bremen, 12 male and 18 female, volunteered to take part in the experiment for monetary compensation.

Out of the 30 participants, one aborted the experiment and four were discarded due to an error rate of more than 30% incorrectly answered trials. The remaining 25 participants comprised 11 males and 14 females. The 0.05 level of significance was used for all statistical tests in both experiments.

4.1.2. Preferences

For the analysis of the preferences we discarded those trials for which the participants gave no or incorrect answers (12% of all trials). We compared the answers of all participants for all remaining trials to identify possible preferences. We differentiated between 90° and 45° problems and assumed that the given answers indicate the employment of the corresponding model. If no preferences existed, one would expect to observe distorted models and equal-distance models in 66% and 33% of all 90° problem trials, respectively. Likewise, distorted models, equal-distance models, and cardinal models should occur in 50%, 25%, and 25% of all 45° problem trials, respectively. To check for the existence of preferences, we compared the observed model percentages to these hypothetical ones. Figure ​ Figure3 3 shows the resulting preferences for both problem types. There is a clear preference for the equal-distance model in the 90° problems. The answer corresponding to this model was given in 88.34% of all trials ( t (24)  = 17.233; p  < 0.001). The distorted models were employed significantly less than expected by chance with 11.66% ( t (24)  = −17.233; p  < 0.001). We found a significant preference for the equal-distance model in the 45° problems with 62.88% ( t (24)  = 5.352; p  < 0.001), whereas the 23.46% of the cardinal model did not differ significantly from the expected value ( t (24)  = −0.215; p  > 0.8). The distorted models were used significantly less than expected by chance with 13.66% ( t (24)  = −9.995; p  < 0.001).

Preferences in the first experiment . The vertical axis represents the frequency of the given answer. Top: 90° problems; bottom: 45° problems. Error bars show the standard error of the mean. EDM, equal-distance model; CM, cardinal model; DM, distorted models.

4.1.3. Eye movements

Table ​ Table2 2 shows the time slots identified by analyzing the eye movements during the filler trials. We used the last six out of ten time slots for our analysis of the eye movements during the actual problem trials. We decided to use all six slots despite the fact that two out of those did not show significant eye movements in the filler trials, because it is plausible that processing was not interrupted in between, but ran continuously after participants have understood the premise. Table ​ Table3 3 shows that the amount of participants showing eye movements along the given directions is not significant in neither the first nor the second premise (all p  > 0.35), but significant during the conclusion phase ( p  < 0.05).

Analysis of eye tracking data from the first premise of all filler trials .

We applied a binomial test to see whether the eye movements along the given direction (and its opposite) are above the expected value of chance (1/4) within each time slot. Significance is based on an error probability of 0.05 .

The number of participants showing significant eye movements along the given directions .

Significance is based on an error probability of 0.05 .

The left parts of the Figures ​ Figures4 4 and ​ and5 5 show diagrams of the recorded eye movements during all first premises of the form A is west of B and A is north-west of B , respectively. It is evident that the percentage of saccades along the given direction and the opposing direction are not above chance, i.e., 12.5%, for both types of premises.

Distribution of eye movements during first premises of the form “A is west of B.” Amplitude represents the percentage of saccades mapped onto the respective cardinal direction.

Distribution of eye movements during first premises of the form “A is north-west of B.” Amplitude represents the percentage of saccades mapped onto the respective cardinal direction.

4.2. Experiment 2: Visual mental representation

4.2.1. participants.

Thirty one undergraduate students of the University of Bremen, 15 male and 16 female, participated in the study for monetary compensation.

Eight of the 31 participants were discarded due to an error rate of more than 30% incorrectly answered trials. The remaining 23 participants comprised 12 males and 11 females.

4.2.2. Preferences

Preferences were analyzed in the same way as in Experiment 1. We discarded those trials for which the participants gave no or an incorrect answer for the analysis of the preferences (9% of all trials). Figure ​ Figure6 6 shows the preferences for both problem types. For the 90° problems, the equal-distance model was used in 93.2% of all trials, which shows a significant preference ( t (22)  = 29.350; p  < 0.001). Consequently, the distorted models are employed significantly below chance with 6.8% ( t (22)  = −29.350; p  < 0.001). For the 45° problems, we found a significant preference for the equal-distance model with 46.32% ( t (22)  = 2.512; p  < 0.05) as well as for the cardinal model with 47.9% ( t (22)  = 2.683; p  < 0.05). The distorted models were used significantly less compared to their expected value with 5.78% ( t (22)  = −25.360; p  < 0.001).

Preferences in the second experiment . The vertical axis represents the frequency of the given answer. Top: 90° problems; bottom: 45° problems. Error bars show the standard error of the mean. EDM, equal-distance model; CM, cardinal model; DM, distorted models.

4.2.3. Eye movements

Table ​ Table2 2 shows the time slots during which participants showed significant eye movements during the filler trials. Based on this, we used the last seven out of ten time slots for the eye movement analysis for the problem trials. We decided to use all seven slots despite the fact that one out of those did not contain significant eye movements, because we assumed, just as in the first experiment, that processing is not paused in between. Contrasting the first experiment, we found a significant amount of participants showing significant eye movements during all three phases (Prem. 1: p  < 0.001; Prem. 2: p  < 0.05; Concl.: p  < 0.01) as shown in Table ​ Table3 3 .

The right parts of the Figures ​ Figures4 4 and ​ and5 5 show diagrams of the recorded eye movements during all first premises of the form A is west of B and A is north-west of B , respectively. The Figures show that saccades along the given direction as well as the opposing direction are above the frequency of chance (i.e., 12.5%) for both types of premises.

4.2.4. Comparison of eye-movers to non-eye-movers

Based on the literature, we expected to find spontaneous eye movements corresponding to the processed spatial relations when a visual mental representation is employed. In line with this assumption, a majority of participants exhibited systematic eye movements. We compared the participants that showed a significant amount of eye movements along the given directions in any of the phases (both premises or the conclusion) with those that did not show significant eye movements in any of the phases. Given this definition, 13 out of the 23 participants qualified as eye-movers; the 10 remaining participants will be referred to as non-eye-movers.

There was no significant difference between eye-movers and non-eye-movers regarding error rate, reaction times, and sex (all p  > 0.19). The eye-movers and non-eye-movers, however, showed different preferences for the 45° problems as shown in Figure ​ Figure7. 7 . The eye-movers showed a significant preference for the cardinal direction model with 54.84% ( t (12)  = 2.7884; p  < 0.05). The equal-distance model was not significantly preferred with 37.85% ( t (12)  = 1.2527; p  > 0.23) and the distorted models were employed significantly less than expected with 7.31% ( t (12)  = −16.1961; p  < 0.001). The non-eye-movers showed no significant preference for the cardinal direction model with 38.88% ( t (9)  = 0.994; p  > 0.34) but for the equal-distance model with 57.32% ( t (9)  = 2.2926; p  < 0.05). The distorted models were significantly below expectation with 3.8% ( t (9)  = −22.3421; p  < 0.001).

Preferences of the 45° problems in the second experiment . The vertical axis represents the frequency of the given answer. Top: non-eye-movers; bottom: eye-movers. Error bars show the standard error of the mean. EDM, equal-distance model; CM, cardinal model; DM, distorted models.

4.3. Comparison of the experiments

In the first experiment, it is only during the conclusion phase that the number of participants that showed systematic eye movements becomes significant. This finding seems unexpected given that the number of eye-movers of the second experiment is highest during the first premise whereas the number of eye-movers in the first experiment is not significant for neither premise. Furthermore, analysis of the eye movements should be most accurate for the first premise as participants are only aware of one spatial relation at that time and all saccades along the other directions can be assumed not to have any relation to the mental representation constructed. In contrast, during the second premise or the conclusion, all three spatial relations are (at least implicitly) available to the participant and could also result in eye movements, which would, however, not all be counted as “correct” eye movements, because we only checked for the spatial relations of the two premises during the second premise and we only checked for the relation that is given as the answer during the conclusion. Thus, the chance for finding significant eye movements during specifically the conclusion phase should be lower than for the first premise. It can, accordingly, be argued that eye movements during the conclusion phase did not necessarily result from the internal processing of spatial relations but that some participants moved their gaze in anticipation of the answer screen. The answer screen was projected on the wall just after participants clicked to indicate they found an answer. A saccade from the middle of their visual field toward the appropriate number on the answer screen, i.e., the number which represents their given answer, would have been mapped onto the cardinal direction that corresponds to their answer. Thus, there is reason to doubt that the significant number of eye-movers that we find for the conclusion phase in the first experiment is a result of the employed mental representation.

Given the lack of spontaneous eye movements along the processed relations for the non-eye-movers of the second experiment, we conclude that these participants did not employ a visual mental representation. This conclusion is based on the literature (see Section 3.2 ) which shows that employment of visual mental representations is related to the occurrence of such spontaneous eye movements and, furthermore, that these eye movements have a functional role in the employment of visual mental representations (Johansson et al., 2012 ). It may be that the non-eye-movers likely used a spatial mental representation like the participants of the first experiment; this conclusion does, however, not follow from the observation or the literature. We, therefore, remain agnostic regarding the mental representation of the non-eye-movers of the second experiment.

4.3.1. Comparing reasoning with visual and spatial mental representations

As the two experiments consisted of the same task with only slightly different instructions, we compared participants across the experiments 1 . Reaction times that were outside a 2.5*SD range from the mean reaction time of the corresponding phase (first and second premise and the conclusion) were excluded from analysis (3%).

In order to compare the employment of visual mental representations with that of spatial mental representations, we defined two groups: the visual group and the spatial group. The spatial group comprises all participants of the first experiment. The eye-movers of the second experiment constitute the visual group. That is, the spatial group contains those participants which employed a spatial mental representation and the visual group contains those that employed a visual mental representation. There were no significant differences regarding error rate, reaction times, and sex (all p  > 0.35) between the visual and the spatial group. However, the preferences of the two groups differed as indicated by a significant interaction between group (spatial or visual) and type of model (cardinal or equal-distance), F (1,36)  = 5.644 ,p  < 0.05. Figure ​ Figure8 8 shows the preferences of the visual and the spatial group. The spatial group showed a preference for the equal-distance model (EDM) but not for the cardinal model (CM). In contrast, the visual group showed a preference for the cardinal model (CM) but not the equal-distance model (EDM). Table ​ Table4 4 shows an overview of the preferences for the different groups and experiments. Interestingly, the non-eye-movers of the second experiment showed the same preferences as the participants of the first experiment, i.e., a significant preference for the equal-distance model (EDM) and no significant preference for the cardinal model (CM). This may be taken to indicate that the non-eye-movers employed a spatial mental representation despite the fact that the instructions are formulated to induce a visual mental representation.

Preferences of the 45° problems for the spatial group (top) and the visual group (bottom) . The vertical axis represents the frequency of the given answer. Error bars show the standard error of the mean. EDM, equal-distance model; CM, cardinal model; DM, distorted models.

Comparison of preferences for the 45° problems between different groups; S+, frequency significantly above chance; S−, frequency significantly below chance; NS, frequency does not significantly differ from chance; CM, cardinal model; EDM, equal-distance model; DM, distorted models .

5. Discussion

The conducted experiments yielded two main results. First, the employed reasoning task led to no significant systematic eye movements when a spatial mental representation was employed, i.e., for the spatial group. In contrast, we found significant systematic eye movements for a majority of the participants in the second experiment, i.e., the visual group which employed a visual mental representation. Second, there are significant preferences in the answers for the under-specified problems in both the visual and the spatial group. The preferences did, however, differ between the employed mental representations.

These results relate to the two main open issues about the relationship between spatial and visual mental representation (identified in Section 3.3 ): (1) whether spatial mental representations are modality-specific, and (2) whether human visuo-spatial reasoning is realized on the level of spatial mental representations.

Regarding the first issue, we observed systematic eye movements in the second experiment but not for the first experiment. The eye movements observed in second experiment, i.e., the one in which the employment of a visual mental representation was induced, corroborate several studies reporting spontaneous eye movements during visual mental imagery. The fact that we did not find these eye movements for the essentially same reasoning task in the first experiment, i.e., the one in which the employment of a spatial mental representation was induced, suggests that other (attentional) processes are employed when reasoning with spatial mental representations. Since eye movements have been found to play a functional role in processing visual mental representations (Johansson et al., 2012 ) and are therefore not epiphenomenal, we can conclude that reasoning with visual mental representations draws on overt attentional processes of visual perception and reasoning with spatial mental representations does not. This finding lends support to the assumption of mental model theory that spatial mental representations are amodal or multi-modal.

Regarding the second issue – whether reasoning is realized on the level of spatial mental representations – our results show different preferences depending on the employed mental representation. The visual group showed a significant preference for the cardinal model (CM) but not for the equal-distance model (EDM) for the 45° problems. In contrast, the spatial group showed a significant preference for the equal-distance model (EDM) but not for the cardinal model (CM) for the 45° problems. Mental model theory assumes that the hierarchical relationship between visual mental representations and spatial mental representations is such that reasoning happens on the level of the spatial mental representation (Knauff and Johnson-Laird, 2002 ; Knauff et al., 2003 ) specifically when visual information is irrelevant to the task at hand (as it is the case in the presented experiments). This assumption seems in contradiction to the presented results. The fact that we observed different preferences for the two mental representations for essentially the same reasoning task challenges the claim that reasoning is based on spatial mental representations. This similarly affects the theory of mental imagery which also states that visual mental representations require underlying spatial mental representations. In order to construct, inspect and reason with a visual representation, spatial information is necessary to, for example, “know” the location, size, and spatial relations of the shapes that make up a visual mental representation. The results of the experiments are thus hard to reconcile with both the mental model theory and the theory of mental imagery. The assumed hierarchical relationship between spatial and visual mental representations has to be extended with additional explanations about how spatial information is transformed or processed differently in a visual mental representation. In the following, we interpret the results on the preferences with respect to this assumption of the two theories.

The preferred answer given by participants in the spatial group was such that the spatial configuration of the problem has equal distances between the entities. In contrast, the preferred answer of the participants in the visual group was such that the spatial configuration contains distances of different length. This is especially puzzling given the assumption that those spatial relations are supposed to be provided by the spatial mental representation to the visual mental representation. Sticking with the assumption that the spatial information is provided by the underlying spatial representation, one can think of two general explanations: (1) the spatial relations are somehow altered in the context of a visual representation, or (2) the spatial relations are the same but are processed differently in the two mental representations. Regarding the first option, spatial relations might become more specified when represented in a visual mental representation (Schultheis et al., 2007 ). That is, additional properties such as distance are specified. On the level of the spatial mental representation, distance might only be represented with generic default values. This would fit with the preferred mental model of the spatial group, in which all distances are equal. This option is furthermore supported by an assumption of mental model theory: “[w]hen people understand spatial descriptions, they imagine symmetrical arrays in which adjacent objects have roughly equal distances between them […]” (Johnson-Laird and Byrne, 1991 , p. 94). Regarding the second option, the way spatial relations are processed could differ between the two mental representations. This explanation would fit well with the fact that we found spontaneous eye movements that align with the currently processed spatial relations for the visual group, but we did not find such eye movements for the spatial group. An implementation of the second option is proposed by a new model of visuo-spatial mental imagery in which processing of spatial relations is affected by additional visual information and realized by attention shifts such as eye movements (Sima, 2011 ; Sima and Freksa, 2012 ).

6. Conclusion

Our experiments provided two new insights on the so far little investigated relationship between visual and spatial mental representations: (1) visual and spatial mental representations differ in their employment of overt attentional processes of visual perception, (2) there are preferences when employing visual mental representations just as for spatial mental representations, but the preferences can differ for the same reasoning task. These findings are hard to reconcile with current theories on visuo-spatial processing and challenge some of their assumptions. Future work is necessary to shed more light on the exact relationship between visual and spatial mental representations. This will have to include the refinement of the existing theoretical frameworks on the one hand as well as further empirical research on the other hand. Regarding the theories, we have additionally presented a systematic comparison of mental model theory and the theory of mental imagery. This comparison showed that the two theories that are often investigated separately likely investigate the same visual and spatial mental representations. This comparison might serve as the basis of a new unified theory combining the results achieved within both mental model theory and the theory of mental imagery. Regarding the future empirical work, the presented experiments show one way of comparing visual and spatial mental representations while keeping the experimental task essentially the same.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The presented work was done in the project R1-[ImageSpace] of the Transregional Collaborative Research Center SFB/TR 8 Spatial Cognition. Funding by the German Research Foundation (DFG) is gratefully acknowledged. We thank Sven Bertel for fruitful discussions with respect to experimental design and Maren Lindner for her role in designing and conducting the experiments. We thank Thomas Lachmann, David Peebles, and Jelica Nejasmic for their comments and suggestions which helped improve this article.

1 There is evidence that the ( a priori ) differences between the two groups of the two experiments were not more substantial or qualitatively different than if the groups had resulted from random assignments within a single experiment. First, the participants of both experiments were recruited from the same population of students from the University of Bremen. The setup of the two experiments was identical apart from the variation in instructions. This includes specifically the equipment, the room, the experimenter, and the materials. The experiments were conducted within two consecutive semesters. Second, the two groups did not significantly differ with respect to sex ( χ ( 1 ) 2 = 1 . 42 , p > 0 . 2 ) or field of study ( χ ( 2 ) 2 = 1 . 18 , p > 0 . 5 ) . The two groups also did not differ significantly in their age ( t (46)  = –1.084; p  > 0.28, two-tailed) or performance in the paper-folding test ( t (46)  = –0.455; p  > 0.65, two-tailed). Third, using the method described in Masson ( 2011 ), the participants age and performance in the paper-folding test, provided positive evidence for the null hypothesis that the two groups did not differ ( p ( H 0 | D ) = 0.79, p ( H 0 | D ) = 0.86, respectively).

• Borst G., Kosslyn S., Denis M. (2006). Different cognitive processes in two image-scanning paradigms . Mem. Cognit. 34 , 475–490 10.3758/BF03193572 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Brandt S. A., Stark L. W. (1997). Spontaneous eye movements during visual imagery reflect the content of the visual scene . J. Cogn. Neurosci. 9 , 27–38 10.1162/jocn.1997.9.1.27 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Byrne R. M. J., Johnson-Laird P. (1989). Spatial reasoning . J. Mem. Lang. 28 , 564–575 10.1016/0749-596X(89)90013-2 [ CrossRef ] [ Google Scholar ]
• Chambers D., Reisberg D. (1985). Can mental images be ambiguous? J. Exp. Psychol. Hum. Percept. Perform. 11 , 317–328 10.1037/0096-1523.11.3.317 [ CrossRef ] [ Google Scholar ]
• Courtney S. M., Ungerleider L. G., Keil K., Haxby J. V. (1996). Object and spatial visual working memory activate separate neural systems in human cortex . Cereb. Cortex 6 , 39–49 10.1093/cercor/6.1.39 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Demarais A. M., Cohen B. H. (1998). Evidence for image-scanning eye movements during transitive inference . Biol. Psychol. 49 , 229–247 10.1146/annurev.psych.49.1.229 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Farah M. J., Levine D. N., Calvanio R. (1988). A case study of mental imagery deficit . Brain Cogn. 8 , 147–164 10.1016/0278-2626(88)90046-2 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Finke R. A. (1989). Principles of Mental Imagery . Cambridge, MA: MIT Press [ Google Scholar ]
• Goel V., Dolan R. J. (2001). Functional neuroanatomy of three-term relational reasoning . Neuropsychologia 39 , 901–909 10.1016/S0028-3932(01)00024-0 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Holsanova J., Hedberg B., Nilsson N. (1998). “Visual and verbal focus patterns when describing pictures,” in Current Oculomotor Research: Physiological and Psychological Aspects , eds Becker W., Deubel H., Mergner T. (New York: Plenum; ) 303–304 [ Google Scholar ]
• Jahn G., Knauff M., Johnson-Laird P. N. (2007). Preferred mental models in reasoning about spatial relations . Mem. Cognit. 35 , 2075–2087 10.3758/BF03192939 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Johansson R., Holsanova J., Dewhurst R., Holmqvist K. (2012). Eye movements during scene recollection have a functional role, but they are not reinstatements of those produced during encoding . J. Exp. Psychol. Hum. Percept. Perform. 38 , 1289–1314 10.1037/a0026585 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Johansson R., Holsanova J., Holmqvist K. (2006). Pictures and spoken descriptions elicit similar eye movements during mental imagery, both in light and in complete darkness . Cogn. Sci. 30 , 1053–1079 10.1207/s15516709cog0000_86 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Johnson-Laird P. N. (1972). The three-term series problem . Cognition 1 , 57–82 10.1016/0010-0277(72)90045-5 [ CrossRef ] [ Google Scholar ]
• Johnson-Laird P. N. (1989). “Mental models,” in Foundations of Cognitive Science , ed. Posner M. I. (Cambridge, MA: MIT Press; ), 469–499 [ Google Scholar ]
• Johnson-Laird P. N. (1998). “Imagery, visualization, and thinking,” in Perception and Cognition at Century’s End , ed. Hochberg J. (San Diego: Academic Press; ), 441–467 [ Google Scholar ]
• Johnson-Laird P. N. (2001). Mental models and deduction . Trends Cogn. Sci. (Regul. Ed.) 5 , 434–442 10.1016/S1364-6613(00)01751-4 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Johnson-Laird P. N., Byrne R. M. J. (1991). Deduction . Hove: Erlbaum [ Google Scholar ]
• Klauer K., Zhao Z. (2004). Double dissociations in visual and spatial short-term memory [Review] . J. Exp. Psychol. Gen. 133 , 355–381 10.1037/0096-3445.133.3.355 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Knauff M., Fangmeier T., Ruff C. C., Johnson-Laird P. N. (2003). Reasoning, models, and images: behavioral measures and cortical activity . J. Cogn. Neurosci. 15 , 559–573 10.1162/089892903321662949 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Knauff M., Johnson-Laird P. (2002). Visual imagery can impede reasoning . Mem. Cognit. 30 , 363–371 10.3758/BF03194937 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Kosslyn S. (1973). Scanning visual images – some structural implications . Percept. Psychophys. 14 , 90–94 10.3758/BF03198621 [ CrossRef ] [ Google Scholar ]
• Kosslyn S. M. (1980). Image and Mind . Cambridge, MA: Harvard University Press [ Google Scholar ]
• Kosslyn S. M. (1994). Image and Brain: The Resolution of the Imagery Debate . Cambridge, MA: The MIT Press [ Google Scholar ]
• Kosslyn S. M., Reiser B. J., Farah M. J., Fliegel S. L. (1983). Generating visual images: units and relations . J. Exp. Psychol. Gen. 112 , 278–303 10.1037/0096-3445.112.2.278 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Kosslyn S. M., Thompson W. L. (2003). When is early visual cortex activated during visual mental imagery? Psychol. Bull. 129 , 723–746 10.1037/0033-2909.129.5.723 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Kosslyn S. M., Thompson W. L., Ganis G. (2006). The Case for Mental Imagery . New York: Oxford University Press [ Google Scholar ]
• Levine D. N., Warach J., Farah M. (1985). Two visual systems in mental imagery: dissociation of “what” and “where” in imagery disorders due to bilateral posterior cerebral lesions . Neurology 35 , 1010–1018 10.1212/WNL.35.4.562 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Masson M. E. J. (2011). A tutorial on a practical Bayesian alternative to null-hypothesis significance testing . Behav. Res. Methods 43 , 679–690 10.3758/s13428-010-0049-5 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Mellet E., Tzourio-Mazoyer N., Bricogne S., Mazoyer B., Kosslyn S. M., Denis M. (2000). Functional anatomy of high-resolution visual mental imagery . J. Cogn. Neurosci. 12 , 98–109 10.1162/08989290051137620 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Newcombe F., Ratcliff G., Damasio H. (1987). Dissociable visual and spatial impairments following right posterior cerebral lesions: clinical, neuropsychological and anatomical evidence . Neuropsychologia 25 , 149–161 10.1016/0028-3932(87)90127-8 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Pylyshyn Z. W. (2002). Mental imagery: in search of a theory . Behav. Brain Sci. 25 , 157–238 10.1017/S0140525X02000043 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Rauh R., Hagen C., Knauff M., Kuss T., Schlieder C., Strube G. (2005). Preferred and alternative mental models in spatial reasoning . Spat. Cogn. Comput. 5 , 239–269 [ Google Scholar ]
• Rips L. J. (1994). The Psychology of Proof: Deductive Reasoning in Human Thinking . Cambridge, MA: MIT Press [ Google Scholar ]
• Sack A. T. (2009). Parietal cortex and spatial cognition . Behav. Brain Res. 202 , 153–161 10.1016/j.bbr.2009.03.012 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Schultheis H., Barkowsky T. (2011). Casimir: an architecture for mental spatial knowledge processing . Top. Cogn. Sci. 3 , 778–795 10.1111/j.1756-8765.2011.01151.x [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Schultheis H., Barkowsky T. (2013). Variable stability of preferences in spatial reasoning . Cogn. Process. 10.1007/s10339-013-0554-4 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Schultheis H., Bertel S., Barkowsky T., Seifert I. (2007). “The spatial and the visual in mental spatial reasoning: an ill-posed distinction,” in Spatial Cognition V – Reasoning, Action, Interaction , eds Barkowsky T., Knauff M., Ligozat G., Montello D. R. (Berlin: Springer Verlag; ), 191–209 [ Google Scholar ]
• Sereno M. I., Pitzalis S., Martinez A. (2001). Mapping of contralateral space in retinotopic coordinates by a parietal cortical area in humans . Science 294 , 1350–1354 10.1126/science.1063695 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Sima J. F. (2011). “The nature of mental images – an integrative computational theory,” in Proceedings of the 33rd Annual Conference of the Cognitive Science Society, eds Carlson L., Hoelscher C., Shipley T. (Austin, TX: Cognitive Science Society), 2878–2883 [ Google Scholar ]
• Sima J. F., Freksa C. (2012). Towards computational cognitive modeling of mental imagery . Künstliche Intell. 26 , 1–7 10.1007/s13218-011-0163-2 [ CrossRef ] [ Google Scholar ]
• Smith E. E., Jonides J. (1997). Working memory: a view from neuroimaging . Cogn. Psychol. 33 , 5–42 10.1006/cogp.1997.0658 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Spivey M. J., Geng J. J. (2001). Oculomotor mechanisms activated by imagery and memory: eye movements to absent objects . Psychol. Res. 65 , 235–241 10.1007/s004260100059 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Tversky B. (1993). “Cognitive maps, cognitive collages, and spatial mental models,” in Spatial Information Theory: A Theoretical Basis for GIS – Proceedings of COSIT’93, eds Frank A. U., Campari I. (Berlin: Springer), 14–24 [ Google Scholar ]
• Ungerleider L., Mishkin M. (1982). “Two cortical systems,” in Analysis of Visual Behavior , eds Ingle D. J., Goodale M. A., Mansfield R. J. W. (Cambridge: MIT Press; ), 549–586 [ Google Scholar ]

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

Picture Perfect (?): Ethical Considerations in Visual Representation

2004, NEXUS

Related Papers

Sonya de Laat

In this paper, I consider the many ethical dilemmas facing visual anthropologists and those using visual rcpresentation material in their research. The issues are many and are complex. With this paper I scratch only the surface of how to confront and deal with some of them. The main pU&#39;lJose of this paper is not to provide solutions, as ethical questions are always unique to the situations in which they develop. What the paper does do is look at ways in which visual anthropologists. and documentary filmmakers have approached and dealt with a variety of these concerns. An extensive review of historical and contemporary works of visual representations are explored and analysed as examples of the types of ethical issues encountered. In our increasingly postcolonial era. issues of voice. co-authorship and copyright highlight just a few of the current topics covered herein. Techniques such as balanced multivocality. reflexivity. collaboration. and reciprocity are discussed with the a...

Ingrid Rüütel

The following thoughts were inspired by films and discussions seen and heard at the Festival of Visual Anthropology in Pärnu, an annual event initiated by Mark Soosaar in 1987. I am also using my own years of experience in documenting folklore and ethnographic material, and of making video films as a point of dependence. Problems concerning ethics play a significant role in any society but they emerge particularly in the case of cultures and peoples on the verge of extinction. In visual anthropology ethics appears on three different levels: the ethics of the film maker, the ethical principles of the people featured in the film, and the ethics of the TV or cinema audience. The latter may be relatively low (and in serious opposition of the subject matter filmed) and therefore it should be seriously considered what is to be filmed where and to whom it should be shown. The scientific documentation for archives, which is made public only for educational purposes or at academic events for...

Visual Anthropology Review

Jonathan Marion

Blake P Kendall

Visual / Sensorial Anthropology holds a curious position within in the home of the social sciences that it grew out of. And like a rebellious teen breaking free of the parental ties, a fixation of redefining a scope of identity has claimed much of the discipline’s efforts. However this point of distinction cannot be trivialized, for at its core the dilemma of political and ethical consequences of representation, are exposed with a characterized depth. The point of significance here in the question of ethics, pertains to the inherent nature of critically engaging with a medium without a focus on the cultural context within which engaged. And this lay as the missing link in body of literature of the discipline, whereby when we develop an ethical code(s) for visual / sensorial anthropology, we do not merely need to address the overarching points of ownership and representation in context that hinder all film, but so too we need to ask, ethics according to whom?

Marilys Guillemin

This special issue on ethical issues in visual research arose from our collective observation that there is an urgent need for researchers to share and reflect upon stories about the ethical challenges they are facing in their research, including how they have navigated the formal procedural ethics review process and how they have identified and responded to ethical challenges in their research practice. Our approach in this special issue has been to call for tales from the field that raise new questions and highlight concerns within the context of real and ongoing research rather than attempt to derive solutions to ethical problems in an abstract or decontextualized way. The overall collection is therefore one that highlights the importance of good descriptive self-reflexive accounts of ethical and methodological issues, especially in terms of what is useful for other visual researchers and also for members of research ethics boards or committees (REB/REC).

Kim McLeod , Marilys Guillemin

To date, discussions of visual research ethics have largely focussed on the effects for participants, with increasing attention on how researchers are ethically affected by visual research. However, there has been no sustained examination into how visual materials themselves have ethical consequences in visual research. In this paper we argue that visual research presents with particular ethical challenges because of the capacities of visual materials themselves to act in research encounters. The paper draws on a research project where participants generated two different kinds of visual materials: timeline charts and photos. We show how timeline charts and photos elicit different kinds of imaginative, bodily and sensory responses, and activate memory in contrasting ways. We argue these capacities give visual materials agency, or the power to act. The agentic capacities of the visual materials act in specific ways to co-create a network of relations across the research encounters. This network of relations has the capacity to act in particular ethical ways with serious consequences, not just for research participants, but also for researchers. We propose that the action of visual materials needs to be added to discussion about visual research ethics. Drawing on the concept of ethical sustainability, we advocate for extending situated ethics and researcher reflexivity to include consideration of the agentic capacities of visual materials.

International Journal of Modern Anthropology

Dr. Shuchi Srivastava

Visual Anthropology, the study of visual systems, is a specialized study of culture involving photographs and films. The main objective of the study is to present the chronological development and changing trends in visual anthropology. Visual anthropology has a long history started with the photography of various cultural aspects by some professional photographers, and then there was an addition of supplementary use of photographs and films in ethnographic description. Later, reflexive, participatory and dialogic movements were introduced to more depth and objective studies, which finally led to the applied interventions with the use of photo voice technique in community-based participatory research to address social, environmental and health issues and problems. The paper provides an in-depth overview of enhancement in visual techniques and methods used for communication of anthropological observations and insights applicable for human welfare. Keywords: Visual anthropology. Visual system. Anthropological fieldwork. Film. Photography.

Frontiers of Visual Anthropology (Special Issue, Visual Anthropology Vol 20: 2-3)

Glenn Bowman

This special issue of Visual Anthropology is the product of a workshop organized by members of the Visual Anthropology Network of the European Association of Social Anthropologists (EASA). In September 2005, eight speakers came to Oxford University. The papers had been selected om more than 40 proposals for papers dealing with a hugely diverse range of theoretical and practical applications of visual anthropology that we had received in response to our call. What follows is a brief outline of the four themes we proposed to our participants, and the questions we considered most important to discuss. Next follows short introductions to the papers themselves, written by the persons responsible for working with the authors to produce the versions that are being published here. THE AGENDA As we enter the 21st century, visual anthropology has increasingly expanded om the emphasis on ethnographic filmmaking that dominated in the latter part of the 20th century, and has taken on a more diverse identity, one encompassing uses of documentation and elicitation and other visual research methods. Photographic and filmic research methods as well as the analysis of visual texts have been important in visual anthropological practices, om the work of Bateson and Mead [1942] through to Collier [1967, 1973] and beyond, as has been repeatedly emphasized [Ruby and Chalfen 1974; Morphy and Banks 1997]. However, until the end of the 20th century, visual methods of research and analysis were largely undiscussed or taken for granted in a subdiscipline dominated by and identified with ethnographic film production (at the levels of both university training and practice). Currently visual anthropology is engaged in a series of transformations, which to a large degree involve making links with other disciplines, testing how it might be more closely integrated with mainstream anthropology and examining its applied potential outside academia.

Loading Preview

Sorry, preview is currently unavailable. You can download the paper by clicking the button above.

RELATED TOPICS

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

• Search Menu

Sign in through your institution

• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical Numismatics
• Classical Literature
• Classical Reception
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Papyrology
• Greek and Roman Archaeology
• Greek and Roman Epigraphy
• Greek and Roman Law
• Late Antiquity
• Religion in the Ancient World
• Social History
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Emotions
• History of Agriculture
• History of Education
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Evolution
• Language Reference
• Language Variation
• Language Families
• Language Acquisition
• Lexicography
• Linguistic Anthropology
• Linguistic Theories
• Linguistic Typology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Modernism)
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Media
• Music and Culture
• Music and Religion
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Science
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Society
• Law and Politics
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Legal System - Costs and Funding
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Restitution
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Clinical Neuroscience
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Toxicology
• Medical Oncology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Medical Ethics
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Games
• Computer Security
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Psychology
• Cognitive Neuroscience
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Ethics
• Business History
• Business Strategy
• Business and Technology
• Business and Government
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Social Issues in Business and Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic History
• Economic Methodology
• Economic Systems
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Management of Land and Natural Resources (Social Science)
• Natural Disasters (Environment)
• Pollution and Threats to the Environment (Social Science)
• Social Impact of Environmental Issues (Social Science)
• Sustainability
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• Ethnic Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Behaviour
• Political Economy
• Political Institutions
• Political Theory
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Politics and Law
• Politics of Development
• Public Policy
• Public Administration
• Qualitative Political Methodology
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Disability Studies
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

• < Previous chapter
• Next chapter >

5 On the Function of Visual Representation

• Published: April 1996
• Cite Icon Cite
• Permissions Icon Permissions

The advent of the computer age enabled significant developments in the study of visual representation, particularly in the emergence of computational theories which are able to make sense of the large volume of data collected. This background leads into a discussion of the “Literalist View,” which explains the phenomenon of perception as the product of similar logical computations by the brain to reconcile visual stimuli with existing mental “retinotopic structures,” which are assumed to be truthful representations of our world. The chapter then cites several works—namely, that of Churchland, Grimes, and the Nina experiments—that discuss the loopholes in the theory. An alternative, non-Literalist theory is then offered—the “Functional View”—which provides a different insight into how the brain interprets visual stimuli. Specifically, it is posited that there is evidence of selective visual representation, dependent on the importance of the visual stimuli to the particular individual.

Personal account

• Sign in with email/username & password
• Get email alerts
• Save searches
• Purchase content
• Activate your purchase/trial code
• Add your ORCID iD

Institutional access

Sign in with a library card.

• Sign in with username/password
• Recommend to your librarian
• Institutional account management
• Get help with access

Access to content on Oxford Academic is often provided through institutional subscriptions and purchases. If you are a member of an institution with an active account, you may be able to access content in one of the following ways:

IP based access

Typically, access is provided across an institutional network to a range of IP addresses. This authentication occurs automatically, and it is not possible to sign out of an IP authenticated account.

Choose this option to get remote access when outside your institution. Shibboleth/Open Athens technology is used to provide single sign-on between your institution’s website and Oxford Academic.

• Click Sign in through your institution.
• Select your institution from the list provided, which will take you to your institution's website to sign in.
• When on the institution site, please use the credentials provided by your institution. Do not use an Oxford Academic personal account.
• Following successful sign in, you will be returned to Oxford Academic.

If your institution is not listed or you cannot sign in to your institution’s website, please contact your librarian or administrator.

Enter your library card number to sign in. If you cannot sign in, please contact your librarian.

Society Members

Society member access to a journal is achieved in one of the following ways:

Sign in through society site

Many societies offer single sign-on between the society website and Oxford Academic. If you see ‘Sign in through society site’ in the sign in pane within a journal:

• Click Sign in through society site.
• When on the society site, please use the credentials provided by that society. Do not use an Oxford Academic personal account.

If you do not have a society account or have forgotten your username or password, please contact your society.

Sign in using a personal account

Some societies use Oxford Academic personal accounts to provide access to their members. See below.

A personal account can be used to get email alerts, save searches, purchase content, and activate subscriptions.

Some societies use Oxford Academic personal accounts to provide access to their members.

Viewing your signed in accounts

Click the account icon in the top right to:

• View your signed in personal account and access account management features.
• View the institutional accounts that are providing access.

Signed in but can't access content

Oxford Academic is home to a wide variety of products. The institutional subscription may not cover the content that you are trying to access. If you believe you should have access to that content, please contact your librarian.

For librarians and administrators, your personal account also provides access to institutional account management. Here you will find options to view and activate subscriptions, manage institutional settings and access options, access usage statistics, and more.

Our books are available by subscription or purchase to libraries and institutions.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Rights and permissions
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• Reviews / Why join our community?
• For companies
• Frequently asked questions

5. Visual Representation

How can you design computer displays that are as meaningful as possible to human viewers? Answering this question requires understanding of visual representation - the principles by which markings on a surface are made and interpreted. The analysis in this article addresses the most important principles of visual representation for screen design, introduced with examples from the early history of graphical user interfaces . In most cases, these principles have been developed and elaborated within whole fields of study and professional skill - typography , cartography, engineering and architectural draughting, art criticism and semiotics. Improving on the current conventions requires serious skill and understanding. Nevertheless, interaction designers should be able, when necessary, to invent new visual representations.

Introduction to Visual Representation by Alan Blackwell

Alan Blackwell on applying theories of Visual Representation

• 5.1 Typography and text

For many years, computer displays resembled paper documents. This does not mean that they were simplistic or unreasonably constrained. On the contrary, most aspects of modern industrial society have been successfully achieved using the representational conventions of paper, so those conventions seem to be powerful ones. Information on paper can be structured using tabulated columns, alignment, indentation and emphasis, borders and shading. All of those were incorporated into computer text displays. Interaction conventions, however, were restricted to operations of the typewriter rather than the pencil. Each character typed would appear at a specific location. Locations could be constrained, like filling boxes on a paper form. And shortcut command keys could be defined using onscreen labels or paper overlays. It is not text itself, but keyboard interaction with text that is limited and frustrating compared to what we can do with paper (Sellen and Harper 2001).

But despite the constraints on keyboard interaction, most information on computer screens is still represented as text. Conventions of typography and graphic design help us to interpret that text as if it were on a page, and human readers benefit from many centuries of refinement in text document design. Text itself, including many writing systems as well as specialised notations such as algebra, is a visual representation that has its own research and educational literature. Documents that contain a mix of bordered or coloured regions containing pictures, text and diagrammatic elements can be interpreted according to the conventions of magazine design, poster advertising, form design, textbooks and encyclopaedias. Designers of screen representations should take care to properly apply the specialist knowledge of those graphic and typographic professions. Position on the page, use of typographic grids, and genre-specific illustrative conventions should all be taken into account.

Author/Copyright holder: Unknown (pending investigation). Copyright terms and licence: Unknown (pending investigation). See section "Exceptions" in the copyright terms below.

Figure 5.1 : Contemporary example from the grid system website

Figure 5.2 : Example of a symbolic algebra expression (the single particle solution to Schrodinger's equation)

Figure 5.3 : Table layout of funerals from the plague in London in 1665

Figure 5.4 : Tabular layout of the first page of the Gutenberg Bible: Volume 1, Old Testament, Epistle of St. Jerome. The Gutenberg Bible was printed by Johannes Gutenberg, in Mainz, Germany in the 1450s

• 5.1.1 Summary

Most screen-based information is interpreted according to textual and typographic conventions, in which graphical elements are arranged within a visual grid, occasionally divided or contained with ruled and coloured borders. Where to learn more:

thegridsystem.org

Resnick , Elizabeth (2003): Design for Communication: Conceptual Graphic Design Basics. Wiley

• 5.2 Maps and graphs

The computer has, however, also acquired a specialised visual vocabulary and conventions. Before the text-based computer terminal (or 'glass teletype') became ubiquitous, cathode ray tube displays were already used to display oscilloscope waves and radar echoes. Both could be easily interpreted because of their correspondence to existing paper conventions. An oscilloscope uses a horizontal time axis to trace variation of a quantity over time, as pioneered by William Playfair in his 1786 charts of the British economy. A radar screen shows direction and distance of objects from a central reference point, just as the Hereford Mappa Mundi of 1300 organised places according to their approximate direction and distance from Jerusalem. Many visual displays on computers continue to use these ancient but powerful inventions - the map and the graph. In particular, the first truly large software project, the SAGE air defense system, set out to present data in the form of an augmented radar screen - an abstract map, on which symbols and text could be overlaid. The first graphics computer, the Lincoln Laboratory Whirlwind, was created to show maps, not text.

Figure 5.5 : The technique invented by William Playfair, for visual representation of time series data.

Author/Copyright holder: Courtesy of Premek. V. Copyright terms and licence: pd (Public Domain (information that is common property and contains no original authorship)).

Figure 5.6 : Time series data as shown on an oscilloscope screen

Author/Copyright holder: Courtesy of Remi Kaupp. Copyright terms and licence: CC-Att-SA (Creative Commons Attribution-ShareAlike 3.0 Unported)

Figure 5.7 : Early radar screen from HMS Belfast built in 1936

Author/Copyright holder: Courtesy of NOAA's National Weather Service. Copyright terms and licence: pd (Public Domain (information that is common property and contains no original authorship)).

Figure 5.8 : Early weather radar - Hurricane Abby approaching the coast of British Honduras in 1960

Figure 5.9 : The Hereford Mappa Mundi of 1300 organised places according to their approximate direction and distance from Jerusalem

Author/Copyright holder: Courtesy of Wikipedia. Copyright terms and licence: Unknown (pending investigation). See section "Exceptions" in the copyright terms below.

Figure 5.10 : The SAGE system in use. The SAGE system used light guns as interaction devices.

Author/Copyright holder: The MITRE Corporation. Copyright terms and licence: All Rights Reserved. Reproduced with permission. See section "Exceptions" in the copyright terms below.

Figure 5.11 : The Whirlwind computer at the MIT Lincoln Laboratory

• 5.2.1 Summary

Basic diagrammatic conventions rely on quantitative correspondence between a direction on the surface and a continuous quantity such as time or distance. These should follow established conventions of maps and graphs.

Where to learn more:

MacEachren , Alan M. (2004): How Maps Work: Representation, Visualization, and Design. The Guilford Press

• 5.3 Schematic drawings

Ivan Sutherland's groundbreaking PhD research with Whirlwind's successor TX-2 introduced several more sophisticated alternatives (Sutherland 1963). The use of a light pen allowed users to draw arbitrary lines, rather than relying on control keys to select predefined options. An obvious application, in the engineering context of Massachusetts Institute of Technology (MIT) where Sutherland worked, was to make engineering drawings such as the girder bridge in Figure 13. Lines on the screen are scaled versions of the actual girders, and text information can be overlaid to give details of force calculations. Plans of this kind, as a visual representation, are closely related to maps. However, where the plane of a map corresponds to a continuous surface, engineering drawings need not be continuous. Each set of connected components must share the same scale, but white space indicates an interpretive break, so that independent representations can potentially share the same divided surface - a convention introduced in Diderot's encyclopedia of 1772, which showed pictures of multiple objects on a page, but cut them loose from any shared pictorial context.

Author/Copyright holder: Courtesy of Ivan Sutherland. Copyright terms and licence: CC-Att-SA-3 (Creative Commons Attribution-ShareAlike 3.0).

Figure 5.12 : The TX-2 graphics computer, running Ivan Sutherland's Sketchpad software

Figure 5.13 : An example of a force diagram created using Sutherland's Sketchpad

Figure 5.14 : A page from the Encyclopédie of Diderot and d'Alembert, combining pictorial elements with diagrammatic lines and categorical use of white space.

• 5.3.1 Summary

Engineering drawing conventions allow schematic views of connected components to be shown in relative scale, and with text annotations labelling the parts. White space in the representation plane can be used to help the reader distinguish elements from each other rather than directly representing physical space. Where to learn more:

Engineering draughting textbooks

Ferguson , Eugene S. (1994): Engineering and the Mind's Eye. MIT Press

• 5.4 Pictures

The examples so far may seem rather abstract. Isn't the most 'natural' visual representation simply a picture of the thing you are trying to represent? In that case, what is so hard about design? Just point a camera, and take the picture. It seems like pictures are natural and intuitive, and anyone should be able to understand what they mean. Of course, you might want the picture to be more or less artistic, but that isn't a technical concern, is it? Well, Ivan Sutherland also suggested the potential value that computer screens might offer as artistic tools. His Sketchpad system was used to create a simple animated cartoon of a winking girl. We can use this example to ask whether pictures are necessarily 'natural', and what design factors are relevant to the selection or creation of pictures in an interaction design context.

We would not describe Sutherland's girl as 'realistic', but it is an effective representation of a girl. In fact, it is an unusually good representation of a winking girl, because all the other elements of the picture are completely abstract and generic. It uses a conventional graphic vocabulary of lines and shapes that are understood in our culture to represent eyes, mouths and so on - these elements do not draw attention to themselves, and therefore highlight the winking eye. If a realistic picture of an actual person was used instead, other aspects of the image (the particular person) might distract the viewer from this message.

Figure 5.15 : Sutherland's 'Winking Girl' drawing, created with the Sketchpad system

It is important, when considering the design options for pictures, to avoid the 'resemblance fallacy', i.e. that drawings are able to depict real object or scenes because the viewer's perception of the flat image simulates the visual perception of a real scene. In practice, all pictures rely on conventions of visual representation, and are relatively poor simulations of natural engagement with physical objects, scenes and people. We are in the habit of speaking approvingly of some pictures as more 'realistic' than others (photographs, photorealistic ray-traced renderings, 'old master' oil paintings), but this simply means that they follow more rigorously a particular set of conventions. The informed designer is aware of a wide range of pictorial conventions and options.

As an example of different pictorial conventions, consider the ways that scenes can be rendered using different forms of artistic perspective. The invention of linear perspective introduced a particular convention in which the viewer is encouraged to think of the scene as perceived through a lens or frame while holding his head still, so that nearby objects occupy a disproportionate amount of the visual field. Previously, pictorial representations more often varied the relative size of objects according to their importance - a kind of 'semantic' perspective. Modern viewers tend to think of the perspective of a camera lens as being most natural, due to the ubiquity of photography, but we still understand and respect alternative perspectives, such as the isometric perspective of the pixel art group eBoy, which has been highly influential on video game style.

Author/Copyright holder: Courtesy of Masaccio (1401-1428). Copyright terms and licence: pd (Public Domain (information that is common property and contains no original authorship))

Figure 5.16 : Example of an early work by Masaccio, demonstrating a 'perspective' in which relative size shows symbolic importance

Author/Copyright holder: eBoy.com. Copyright terms and licence: All Rights Reserved. Reproduced with permission. See section "Exceptions" in the copyright terms below.

Figure 5.17 : Example of the strict isometric perspective used by the eBoy group

Author/Copyright holder: Courtesy of Masaccio (1401-1428). Copyright terms and licence: pd (Public Domain (information that is common property and contains no original authorship)).

Figure 5.18 : Masaccio's mature work The Tribute Money, demonstrating linear perspective

As with most conventions of pictorial representation, new perspective rendering conventions are invented and esteemed for their accuracy by critical consensus, and only more slowly adopted by untrained readers. The consensus on preferred perspective shifts across cultures and historical periods. It would be naïve to assume that the conventions of today are the final and perfect product of technical evolution. As with text, we become so accustomed to interpreting these representations that we are blind to the artifice. But professional artists are fully aware of the conventions they use, even where they might have mechanical elements - the way that a photograph is framed changes its meaning, and a skilled pencil drawing is completely unlike visual edge-detection thresholds. A good pictorial representation need not simulate visual experience any more than a good painting of a unicorn need resemble an actual unicorn. When designing user interfaces, all of these techniques are available for use, and new styles of pictorial rendering are constantly being introduced.

• 5.4.1 Summary

Pictorial representations, including line drawings, paintings, perspective renderings and photographs rely on shared interpretive conventions for their meaning. It is naïve to treat screen representations as though they were simulations of experience in the physical world. Where to learn more:

Micklewright , Keith (2005): Drawing: Mastering the Language of Visual Expression. Harry N. Abrams

Stroebel , Leslie, Todd , Hollis and Zakia , Richard (1979): Visual Concepts for Photographers. Focal Press

• 5.5 Node-and-link diagrams

The first impulse of a computer scientist, when given a pencil, seems to be to draw boxes and connect them with lines. These node and link diagrams can be analysed in terms of the graph structures that are fundamental to the study of algorithms (but unrelated to the visual representations known as graphs or charts). A predecessor of these connectivity diagrams can be found in electrical circuit schematics, where the exact location of components, and the lengths of the wires, can be arranged anywhere, because they are irrelevant to the circuit function. Another early program created for the TX-2, this time by Ivan Sutherland's brother Bert, allowed users to create circuit diagrams of this kind. The distinctive feature of a node-and-link connectivity diagram is that, since the position of each node is irrelevant to the operation of the circuit, it can be used to carry other information. Marian Petre's research into the work of electronics engineers (Petre 1995) catalogued the ways in which they positioned components in ways that were meaningful to human readers, but not to the computer - like the blank space between Diderot's objects this is a form of 'secondary notation' - use of the plane to assist the reader in ways not related to the technical content.

Circuit connectivity diagrams have been most widely popularised through the London Underground diagram, an invention of electrical engineer Henry Beck. The diagram clarified earlier maps by exploiting the fact that most underground travellers are only interested in order and connectivity, not location, of the stations on the line. (Sadly, the widespread belief that a 'diagram' will be technical and hard to understand means that most people describe this as the London Undergound 'map', despite Beck's insistence on his original term).

Author/Copyright holder: Courtesy of Harry C. Beck and possibly F. H. Stingemore, born 1890, died 1954. Stingmore designed posters for the Underground Group and London Transport 1914-1942. Copyright terms and licence: Unknown (pending investigation). See section "Exceptions" in the copyright terms below.

Figure 5.19 : Henry Beck's London Underground Diagram (1933)

Author/Copyright holder: Computer History Museum, Mountain View, CA, USA. Copyright terms and licence: All Rights Reserved. Reproduced with permission. See section "Exceptions" in the copyright terms below.

Figure 5.20 : Node and link diagram of the kind often drawn by computing professionals

Figure 5.21 : Map of the London Underground network, as it was printed before the design of Beck's diagram (1932)

• 5.5.1 Summary

Node and link diagrams are still widely perceived as being too technical for broad acceptance. Nevertheless, they can present information about ordering and relationships clearly, especially if consideration is given to the value of allowing human users to specify positions. Where to learn more:

Diagrammatic representation books

Lowe , Ric (1992): Successful Instructional Diagram.

• 5.6 Icons and symbols

Maps frequently use symbols to indicate specific kinds of landmark. Sometimes these are recognisably pictorial (the standard symbols for tree and church), but others are fairly arbitrary conventions (the symbol for a railway station). As the resolution of computer displays increased in the 1970s, a greater variety of symbols could be differentiated, by making them more detailed, as in the MIT SDMS (Spatial Data Management System) that mapped a naval battle scenario with symbols for different kinds of ship. However, the dividing line between pictures and symbols is ambiguous. Children's drawings of houses often use conventional symbols (door, four windows, triangle roof and chimney) whether or not their own house has two storeys, or a fireplace. Letters of the Latin alphabet are shapes with completely arbitrary relationship to their phonetic meaning, but the Korean phonetic alphabet is easier to learn because the forms mimic the shape of the mouth when pronouncing those sounds. The field of semiotics offers sophisticated ways of analysing the basis on which marks correspond to meanings. In most cases, the best approach for an interaction designer is simply to adopt familiar conventions. When these do not exist, the design task is more challenging.

It is unclear which of the designers working on the Xerox Star coined the term 'icon' for the small pictures symbolising different kinds of system object. David Canfield Smith winningly described them as being like religious icons, which he said were pictures standing for (abstract) spiritual concepts. But 'icon' is also used as a technical term in semiotics. Unfortunately, few of the Xerox team had a sophisticated understanding of semiotics. It was fine art PhD Susan Kare's design work on the Apple Macintosh that established a visual vocabulary which has informed the genre ever since. Some general advice principles are offered by authors such as Horton (1994), but the successful design of icons is still sporadic. Many software publishers simply opt for a memorable brand logo, while others seriously misjudge the kinds of correspondence that are appropriate (my favourite blooper was a software engineering tool in which a pile of coins was used to access the 'change' command).

It has been suggested that icons, being pictorial, are easier to understand than text, and that pre-literate children, or speakers of different languages, might thereby be able to use computers without being able to read. In practice, most icons simply add decoration to text labels, and those that are intended to be self-explanatory must be supported with textual tooltips. The early Macintosh icons, despite their elegance, were surprisingly open to misinterpretation. One PhD graduate of my acquaintance believed that the Macintosh folder symbol was a briefcase (the folder tag looked like a handle), which allowed her to carry her files from place to place when placed inside it. Although mistaken, this belief never caused her any trouble - any correspondence can work, so long as it is applied consistently.

Copyright terms and licence: pd (Public Domain (information that is common property and contains no original authorship)).

Figure 5.22 : In art, the term Icon (from Greek, eikon, "image") commonly refers to religious paintings in Eastern Orthodox, Oriental Orthodox, and Eastern-rite Catholic jurisdictions. Here a 6th-century encaustic icon from Saint Catherine's Monastery, Mount Sinai

Author/Copyright holder: Apple Computer, Inc. Copyright terms and licence: All Rights Reserved. Reproduced with permission. See section "Exceptions" in the copyright terms below.

Figure 5.23 : In computing, David Canfield Smith described computer icons as being like religious icons, which he said were pictures standing for (abstract) spiritual concepts.

• 5.6.1 Summary

The design of simple and memorable visual symbols is a sophisticated graphic design skill. Following established conventions is the easiest option, but new symbols must be designed with an awareness of what sort of correspondence is intended - pictorial, symbolic, metonymic (e.g. a key to represent locking), bizarrely mnemonic, but probably not monolingual puns. Where to learn more:

Napoles , Veronica (1987): Corporate Identity Design.

• 5.7 Visual metaphor

The ambitious graphic designs of the Xerox Star/Alto and Apple Lisa/Macintosh were the first mass-market visual interfaces. They were marketed to office professionals, making the 'cover story' that they resembled an office desktop a convenient explanatory device. Of course, as was frequently noted at the time, these interfaces behaved nothing like a real desktop. The mnemonic symbol for file deletion (a wastebasket) was ridiculous if interpreted as an object placed on a desk. And nobody could explain why the desk had windows in it (the name was derived from the 'clipping window' of the graphics architecture used to implement them - it was at some later point that they began to be explained as resembling sheets of paper on a desk). There were immediate complaints from luminaries such as Alan Kay and Ted Nelson that strict analogical correspondence to physical objects would become obstructive rather than instructive. Nevertheless, for many years the marketing story behind the desktop metaphor was taken seriously, despite the fact that all attempts to improve the Macintosh design with more elaborate visual analogies , as in General Magic and Microsoft Bob, subsequently failed.

The 'desktop' can be far more profitably analysed (and extended) by understanding the representational conventions that it uses. The size and position of icons and windows on the desktop has no meaning, they are not connected, and there is no visual perspective, so it is neither a map, graph nor picture. The real value is the extent to which it allows secondary notation, with the user creating her own meaning by arranging items as she wishes. Window borders separate areas of the screen into different pictorial, text or symbolic contexts as in the typographic page design of a textbook or magazine. Icons use a large variety of conventions to indicate symbolic correspondence to software operations and/or company brands, but they are only occasionally or incidentally organised into more complex semiotic structures.

Author/Copyright holder:Apple Computer, Inc and Computer History Museum, Mountain View, CA. Copyright terms and licence: All Rights Reserved. Reproduced with permission. See section "Exceptions" in the copyright terms below.

Figure 5.24 : Apple marketed the visual metaphor in 1983 as a key benefit of the Lisa computer. This advertisement said 'You can work with Lisa the same familiar way you work at your desk'. However a controlled study by Carroll and Mazur (1986) found that the claim for immediately familiar operation may have been exaggerated.

Figure 5.25 : The Xerox Alto and Apple Lisa, early products in which bitmapped displays allowed pictorial icons to be used as mnemonic cues within the 'desktop metaphor'

Author/Copyright holder: Courtesy of Mschlindwein. Copyright terms and licence: CC-Att-SA (Creative Commons Attribution-ShareAlike 3.0 Unported).

Figure 5.26 : Apple Lisa

• 5.7.1 Summary

Theories of visual representation, rather than theories of visual metaphor, are the best approach to explaining the conventional Macintosh/Windows 'desktop'. There is huge room for improvement. Where to learn more:

Blackwell , Alan (2006): The reification of metaphor as a design tool . In ACM Transactions on Computer-Human Interaction , 13 (4) pp. 490-530

• 5.8 Unified theories of visual representation

The analysis in this article has addressed the most important principles of visual representation for screen design, introduced with examples from the early history of graphical user interfaces. In most cases, these principles have been developed and elaborated within whole fields of study and professional skill - typography, cartography, engineering and architectural draughting, art criticism and semiotics. Improving on the current conventions requires serious skill and understanding. Nevertheless, interaction designers should be able, when necessary, to invent new visual representations.

One approach is to take a holistic perspective on visual language, information design, notations, or diagrams. Specialist research communities in these fields address many relevant factors from low-level visual perception to critique of visual culture. Across all of them, it can be necessary to ignore (or not be distracted by) technical and marketing claims, and to remember that all visual representations simply comprise marks on a surface that are intended to correspond to things understood by the reader. The two dimensions of the surface can be made to correspond to physical space (in a map), to dimensions of an object, to a pictorial perspective, or to continuous abstract scales (time or quantity). The surface can also be partitioned into regions that should be interpreted differently. Within any region, elements can be aligned, grouped, connected or contained in order to express their relationships. In each case, the correspondence between that arrangement, and the intended interpretation, must be understood by convention, explained, or derived from the structural and perceptual properties of marks on the plane. Finally, any individual element might be assigned meaning according to many different semiotic principles of correspondence.

The following table summarises holistic views, as introduced above, drawing principally on the work of Bertin, Richards, MacEachren, Blackwell & Engelhardt and Engelhardt. Where to learn more:

Engelhardt , Yuri (2002). The Language of Graphics. A framework for the analysis of syntax and meaning in maps, charts and diagrams (PhD Thesis) . University of Amsterdam

Table 5.1 : Summary of the ways in which graphical representations can be applied in design, via different systems of correspondence

Table 5.2 : Screenshot from the site gapminder.org, illustrating a variety of correspondence conventions used in different parts of the page

As an example of how one might analyse (or working backwards, design) a complex visual representation, consider the case of musical scores. These consist of marks on a paper surface, bound into a multi-page book, that is placed on a stand at arms length in front of a performer. Each page is vertically divided into a number of regions, visually separated by white space and grid alignment cues. The regions are ordered, with that at the top of the page coming first. Each region contains two quantitative axes, with the horizontal axis representing time duration, and the vertical axis pitch. The vertical axis is segmented by lines to categorise pitch class . Symbols placed at a given x-y location indicate a specific pitched sound to be initiated at a specific time. A conventional symbol set indicates the duration of the sound. None of the elements use any variation in colour, saturation or texture. A wide variety of text labels and annotation symbols are used to elaborate these basic elements. Music can be, and is, also expressed using many other visual representations (see e.g. Duignan for a survey of representations used in digital music processing).

• 5.9 Where to learn more

The historical examples of early computer representations used in this article are mainly drawn from Sutherland (Ed. Blackwell and Rodden 2003), Garland (1994), and Blackwell (2006). Historical reviews of visual representation in other fields include Ferguson (1992), Pérez-Gómez and Pelletier (1997), McCloud (1993), Tufte (1983). Reviews of human perceptual principles can be found in Gregory (1970), Ittelson (1996), Ware (2004), Blackwell (2002). Advice on principles of interaction with visual representation is distributed throughout the HCI literature, but classics include Norman (1988), Horton (1994), Shneiderman ( Shneiderman and Plaisant 2009, Card et al 1999, Bederson and Shneiderman 2003) and Spence (2001). Green's Cognitive Dimensions of Notations framework has for many years provided a systematic classification of the design parameters in interactive visual representations. A brief introduction is provided in Blackwell and Green (2003).

Research on visual representation topics is regularly presented at the Diagrams conference series (which has a particular emphasis on cognitive science ), the InfoDesign and Vision Plus conferences (which emphasise graphic and typographic information design), the Visual Languages and Human-Centric Computing symposia (emphasising software tools and development), and the InfoVis and Information Visualisation conferences (emphasising quantitative and scientific data visualisation ).

• 5.9.0.1 IV - International Conference on Information Visualization

2008 2007 2006 2005 2004 2003 2002 2001 2000 1999 1998

• 5.9.0.2 DIAGRAMS - International Conference on the Theory and Application of Diagrams

2008 2006 2004 2002 2000

• 5.9.0.3 VL-HCC - Symposium on Visual Languages and Human Centric Computing

2008 2007 2007 2006 2005 2004 2003 2002 2001 2000 1999 1998 1997 1996 1995 1994 1993 1992 1991 1990

• 5.9.0.4 InfoVis - IEEE Symposium on Information Visualization

2005 2004 2003 2002 2001 2000 1999 1998 1997 1995

• 5.10 References

Anderson , Michael, Meyer , Bernd and Olivier , Patrick (2002): Diagrammatic Representation and Reasoning. London, UK,

Bederson , Benjamin B. and Shneiderman , Ben (2003): The Craft of Information Visualization : Readings and Reflections. Morgan Kaufman Publishers

Bertin , Jacques (1967): Semiology of Graphics: Diagrams, Networks, Maps (Sémiologie graphique: Les diagrammes - Les réseaux - Les cartes). English translation by W. J. Berg. Madison, WI, USA, University of Wisconsin Press

Blackwell , Alan (2002): Psychological perspectives on diagrams and their users. In: Anderson , Michael, Meyer , Bernd and Olivier , Patrick (eds.). "Diagrammatic Representation and Reasoning". London, UK: pp. 109-123

Blackwell , Alan and Engelhardt , Yuri (2002): A Meta-Taxonomy for Diagram Research. In: Anderson , Michael, Meyer , Bernd and Olivier , Patrick (eds.). "Diagrammatic Representation and Reasoning". London, UK: pp. 47-64

Blackwell , Alan and Green , T. R. G. (2003): Notational Systems - The Cognitive Dimensions of Notations Framework. In: Carroll , John M. (ed.). "HCI Models, Theories, and Frameworks". San Francisco: Morgan Kaufman Publisherspp. 103-133

Carroll , John M. and Mazur , Sandra A. (1986): LisaLearning . In Computer , 19 (11) pp. 35-49

Garland , Ken (1994): Mr. Beck's Underground Map. Capital Transport Publishing

Goodman , Nelson (1976): Languages of Art. Hackett Publishing Company

Gregory , Richard L. (1970): The Intelligent Eye. London, Weidenfeld and Nicolson

Horton , William (1994): The Icon Book: Visual Symbols for Computer Systems and Documentation. John Wiley and Sons

Ittelson , W. H. (1996): Visual perception of markings . In Psychonomic Bulletin & Review , 3 (2) pp. 171-187

Mccloud , Scott (1994): Understanding Comics: The Invisible Art. Harper Paperbacks

Norman , Donald A. (1988): The Design of Everyday Things. New York, Doubleday

Petre , Marian (1995): Why Looking Isn't Always Seeing: Readership Skills and Graphical Programming . In Communications of the ACM , 38 (6) pp. 33-44

Pérez-Gómez , Alberto and Pelletier , Louise (1997): Architectural Representation and the Perspective Hinge. MIT Press

Richards , Clive (1984). Diagrammatics: an investigation aimed at providing a theoretical framework for studying diagrams and for establishing a taxonomy of their fundamental modes of graphic organization. Unpublished Phd Thesis . Royal College of Art, London, UK

Sellen , Abigail and Harper , Richard H. R. (2001): The Myth of the Paperless Office. MIT Press

Shneiderman , Ben and Plaisant , Catherine (2009): Designing the User Interface : Strategies for Effective Human-Computer Interaction (5th ed.). Addison-Wesley

Spence , Robert (2001): Information Visualization. Addison Wesley

Sutherland , Ivan E. (1963). Sketchpad, A Man-Machine Graphical Communication System. PhD Thesis at Massachusetts Institute of Technology, online version and editors' introduction by Alan Blackwell & K. Rodden. Technical Report 574 . Cambridge University Computer Laboratory

Tufte , Edward R. (1983): The Visual Display of Quantitative Information. Cheshire, CT , Graphics Press

Ware , Colin (2004): Information Visualization: Perception for Design, 2nd Ed. San Francisco, Morgan Kaufman

• 5 Visual Representation

Human-Computer Interaction: The Foundations of UX Design

Get Weekly Design Tips

5.10 commentary by ben shneiderman.

Since computer displays are such powerful visual appliances, careful designers devote extensive effort to getting the visual representation right. They have to balance the demands of many tasks, diverse users, and challenging requirements, such as short learning time, rapid performance, low error rates, and good retention over time. Designing esthetic interfaces that please and even delight users is a further expectation that designers must meet to be successful. For playful and discretionary tasks esthetic concerns may dominate, but for life critical tasks, rapid performance with low error rates are essential. Alan Blackwell's competent description of many visual representation issues is a great start for newcomers with helpful reminders even for experienced designers. The videos make for a pleasant personal accompaniment that bridges visual representation for interface design with thoughtful analyses of representational art. Blackwell's approach might be enriched by more discussion of visual representations in functional product design tied to meaningful tasks. Learning from paintings of Paris is fine, but aren't there other lessons to learn from visual representations in airport kiosks, automobile dashboards, or intensive care units? These devices as well as most graphical user interfaces and mobile devices raise additional questions of changing state visualization and interaction dynamics. Modern designers need to do more than show the right phone icon, they need to show ringing, busy, inactive, no network, conference mode, etc., which may include color changes (highlighted, grayed out), animations, and accompanying sounds. These designers also need to deal with interactive visual representations that happen with a click, double-click, right-click, drag, drag-and-drop, hover, multi-select, region-select, brushing-linking, and more. The world of mobile devices such as phones, cameras, music players, or medical sensors is the new frontier for design, where visual representations are dynamic and tightly integrated with sound, haptics, and novel actions such as shaking, twisting, or body movements. Even more challenging is the expectation that goes beyond the solitary viewer to the collaboration in which multiple users embedded in a changing physical environment produce new visual representations. These changing and interactive demands on designers invite creative expressions that are very different from designs for static signs, printed diagrams, or interpretive art. The adventure for visual representation designers is to create a new language of interaction that engages users, accelerates learning, provides comprehensible feedback, and offers appropriate warnings when dangers emerge. Blackwell touches on some of these issues in the closing Gapminder example, but I was thirsty for more.

5.11 Commentary by Clive Richards

If I may be permitted a graphically inspired metaphor Alan Blackwell provides us with a neat pen sketch of that extensive scene called 'visual representation' (Blackwell 2011).

"Visualisation has a lot more to offer than most people are aware of today" we are told by Robert Kosara at the end of his commentary (Kosara 2010) on Stephen Few's related article on ' Data visualisation for human perception ' (Few 2010). Korsara is right, and Blackwell maps out the broad territory in which many of these visualisation offerings may be located. In this commentary I offer a few observations on some prominent features in that landscape: dynamics, picturing, semiotics and metaphor.

Ben Shneiderman's critique of Blackwell's piece points to a lack of attention to "... additional questions of changing state visualisations and interaction dynamics" (Shneiderman 2010). Indeed the possibilities offered by these additional questions present some exciting challenges for interaction designers - opportunities to create novel and effective combinations of visual with other sensory and motor experiences in dynamic operational contexts. Shneiderman suggests that: "These changing and interactive demands on designers invite creative expressions that are very different from design for static signs, printed diagrams, or interpretive art". This may be so up to a point, but here Shneinderman and I part company a little. The focus of Blackwell's essay is properly on the visual representation side of facilities available to interaction designers, and in that context he is quite right to give prominence to highly successful but static visual representation precedents, and also to point out the various specialist fields of endeavour in which they have been developed. Some of these representational approaches have histories reaching back thousands of years and are deeply embedded within our culture. It would be foolhardy to disregard conventions established in, say, the print domain, and to try to re-invent everything afresh for the screen, even if this were a practical proposition. Others have made arguments to support looking to historical precedents. For example Michael Twyman has pointed out that when considering typographic cueing and "... the problems of the electronic age ... we have much to learn from the manuscript age" (Twyman 1987, p5). He proposes that studying the early scribes' use of colour, spacing and other graphical devices can usefully inform the design of today's screen-based texts. And as Blackwell points out in his opening section on 'Typography and text' "most information on computer screen is still presented as text".

It is also sometimes assumed that the pictorial representation of a dynamic process is best presented dynamically. However it can be argued that the comic book convention of using a sequence of static frames is sometimes superior for focusing the viewer's attention on the critical events in a process, rather than using an animated sequence in which key moments may be missed. This is of course not to deny the immense value of the moving and interactive visual image in the right context. The Gapminder charts are a case in point (http://www.gapminder.org). Blackwell usefully includes one of these, but as a static presentation. These diagrams come to life and really tell their story through the clustering of balloons that inflate or deflate as they move about the screen when driven through simulated periods of time.

While designing a tool for engineers to learn about the operation and maintenance of an oil system for an aircraft jet engine, Detlev Fischer devised a series of interactive animations, called 'Cinegrams' to display in diagrammatic form various operating procedures (Fischer and Richards 1995). He used the cinematic techniques of time compression and expansion in one animated sequence to show how the slow accumulation of debris in an oil filter, over an extended period of time, would eventually create a blockage to the oil flow and trigger the opening of a by-pass device in split seconds. Notwithstanding my earlier comment about the potential superiority of the comic strip genre for displaying some time dependant processes this particular Cinegram proved very instructive for the targeted users. There are many other examples one could cite where dynamic picturing of this sort has been deployed to similarly good effect in interactive environments.

Shneinderman also comments that: "Blackwell's approach might be enriched by more discussion of visual representation in functional product design tied to meaningful tasks". An area I have worked in is the pictorial representation of engineering assemblies to show that which is normally hidden from view. Techniques to do this on the printed page include 'ghosting' (making occluding parts appear as if transparent), 'exploding' (showing components separately, set out in dis-assembly order along an axis) and cutting away (taking a slice out of an outer shell to reveal mechanisms beneath). All these three-dimensional picturing techniques were used by, if not actually invented by, Leonardo Da Vinci (Richards 2006). All could be enhanced by interactive viewer control - an area of further fruitful exploration for picturing purposes in technical documentation contexts.

Blackwell's section on 'Pictures' warns us that when considering picturing options to avoid the "resemblance fallacy" pointing out the role that convention plays, even in so called photo-realistic images. He also points out that viewers can be distracted from the message by incidental information in 'realistic' pictures. From my own work in the field I know that technical illustrators' synoptic black and white outline depictions are regarded as best for drawing the viewer's attention to the key features of a pictorial representation. Research in this area has shown that when using linear perspective type drawings the appropriate deployment of lines of varying 'weight', rather than of a single thickness, can have a significant effect on viewers' levels of understanding about what is depicted (Richards, Bussard and Newman 2007). This work was done specifically to determine an 'easy to read' visual representational style when manipulating on the screen images of CAD objects. The most effective convention was shown to be: thin lines for edges where both planes forming the edge are visible and thicker lines for edges where only one plane is visible - that is where an outline edge forms a kind of horizon to the object.

These line thickness conventions appear on the face of it to have little to do with how we normally perceive the world, and Blackwell tells us that: "A good pictorial representation need not simulate visual experience any more than a good painting of a unicorn need resemble an actual unicorn". And some particular representations of unicorns can aid our understanding of how to use semiotic theory to figure out how pictures may be interpreted and, importantly, sometimes misunderstood - as I shall describe in the following.

Blackwell mentions semiotics, almost in passing, however it can help unravel some of the complexities of visual representation. Evelyn Goldsmith uses a Charles Addams cartoon to explain the relevance of the 'syntactic', 'semantic' and 'pragmatic' levels of semiotic analysis when applied to pictures (Goldsmith 1978). The cartoon in question, like many of those by Charles Addams, has no caption. It shows two unicorns standing on a small island in the pouring rain forlornly watching the Ark sailing away into the distance. Goldsmith suggests that most viewers will have little trouble in interpreting the overlapping elements in the scene, for example that one unicorn is standing behind the other, nor any difficulty understanding that the texture gradient of the sea stands for a receding horizontal plane. These represent the syntactic level of interpretation. Most adults will correctly identify the various components of the picture at the semantic level, however Goldsmith proposes that a young child might mistake the unicorns for horses and be happy with 'boat' for the Ark. But at the pragmatic level of interpretation, unless a viewer of the picture is aware of the story of Noah's Ark, the joke will be lost  - the connection will not be made between the scene depicted in the drawing and the scarcity of unicorns. This reinforces the point that one should not assume that the understanding of pictures is straightforward. There is much more to it than a simple matter or recognition. This is especially the case when metaphor is involved in visual representation.

Blackwell's section on 'Visual metaphor' is essentially a critique of the use of "theories of visual metaphor" as an "approach to explaining the conventional Mackintosh/Windows 'desktop' ". His is a convincing argument but there is much more which may be said about the use of visual metaphor - especially to show that which otherwise cannot be pictured. In fact most diagrams employ a kind of spatial metaphor when not depicting physical arrangements, for example when using the branches of a tree to represent relations within a family (Richards 2002). The capability to represent the invisible is the great strength of the visual metaphor, but there are dangers, and here I refer back to semiotics and particularly the pragmatic level of analysis. One needs to know the story to get the picture.

In our parental home, one of the many books much loved by my two brothers and me, was The Practical Encyclopaedia for Children (Odhams circa 1948). In it a double page spread illustration shows the possible evolutionary phases of the elephant. These are depicted as a procession of animals in a primordial swamp cum jungle setting. Starting with a tiny fish and passing to a small aquatic creature climbing out of the water onto the bank the procession progresses on through eight phases of transformation, including the Moeritherium and the Paleomatodon, finishing up with the land-based giant of today's African Elephant. Recently one of my brothers confessed to me that through studying this graphical diorama he had believed as a child that the elephant had a life cycle akin to that of a frog. He had understood that the procession was a metaphor for time. He had just got the duration wrong - by several orders of magnitude. He also hadn't understood that each separate depiction was of a different animal. He had used the arguably more sophisticated concept that it was the same animal at different times and stages in its individual development.

Please forgive the cliché if I say that this anecdote clearly illustrates that there can be more to looking at a picture than meets the eye? Blackwell's essay provides some useful pointers for exploring the possibilities of this fascinating territory of picturing and visual representation in general.   

• Blackwell A 2011 'Visual representation' Interaction-Design.org
• Few S 2010 ' Data visualisation for human perception ' Interaction-Design.org
• Fischer D and Richards CJ 1995 'The presentation of time in interactive animated systems diagrams' In: Earnshaw RA and Vince JA (eds) Multimedia Systems and Applications London: Academic Press Ltd (pp141 - 159). ISBN 0-12-227740-6
• Goldsmith E 1978 An analysis of the elements affecting comprehensibility of illustrations intended as supportive of text PhD thesis (CNAA) Brighton Polytechnic
• Korsa R 2010 ' Commentary on Stephen Few's article : Data visualisation for human perception' Interaction-Design.org Odhams c. 1949 The practical encyclopaedia for children (pp 194 - 195)
• Richards CJ 2002 'The fundamental design variables of diagramming' In: Oliver P, Anderson M and Meyer B (eds) Diagrammatic representation and reasoning London: Springer Verlag (pp 85 - 102) ISBN 1-85233-242-5
• Richards CJ 2006 'Drawing out information - lines of communication in technical illustration' Information Design Journal 14 (2) 93 - 107
• Richards CJ, Bussard N, Newman R 2007 'Weighing up line weights: the value of differing line thicknesses in technical illustrations' Information Design Journal 15 (2) 171 - 181
• Shneiderman B 2011 'Commentary on Alan Blackwell's article: Visual representation' Interaction-Design.org
• Twyman M 1982 'The graphic representation of language' Information Design Journal 3 (1) 2 - 22

5.12 Commentary by Peter C-H. Cheng

Alan Blackwell has provided us with a fine introduction to the design of visual representations. The article does a great job in motivating the novice designer of visual representations to explore some of the fundamental issues that lurk just beneath the surface of creating effective representations.  Furthermore, he gives us all quite a challenge:

Alan, quite rightly, claims that we must consider the fundamental principles of symbolic correspondence, if we are to design new genres of visual representations beyond the common forms of displays and interfaces.  The report begins to equip the novice visual representation designer with an understanding of the nature of symbolic correspondence between the components of visual representations and the things they represent, whether objects, actions or ideas.  In particular, it gives a useful survey of how correspondence works in a range of representations and provides a systematic framework of how systems of correspondence can be applied to design. The interactive screen shot is an exemplary visual representation that vividly reveals the correspondence techniques used in each part of the example diagram.

However, suppose you really wished to rise to the challenge of creating novel visual representations, how far will a knowledge of the fundamentals of symbolic correspondence take you? Drawing on my studies of the role of diagrams in the history of science, experience of inventing novel visual representations and research on problem solving and learning with diagrams, from the perspective of Cognitive Science, my view is that such knowledge will be necessary but not sufficient for your endeavours.  So, what else should the budding visual representation designer consider? From the perspective of cognitive science there are at least three aspects that we may profitably target.

First, there is the knowledge of how human process information; specifically the nature of the human cognitive architecture. By this, I mean more than visual perception, but an understanding of how we mentally receive, store, retrieve, transform and transmit information. The way the mind deals with each of these basic types of information processing provides relevant constrains for the design of visual representations. For instance, humans often, perhaps even typically, encode concepts in the form of hierarchies of schemas, which are information structures that coordinate attributes that describe and differentiate classes of concepts. These hierarchies of schemas underpin our ability to efficiently generalize or specialize concepts. Hence, we can use this knowledge to consider whether particular forms of symbolic correspondence will assist or hinder the forms of inference that we hope the user of the representation may make. For example, are the main symbolic correspondences in a visual representation consistent with the key attributes of the schemas for the concepts being considered?

Second, it may be useful for the designer to consider the broader nature of the tasks that the user may wish to do with the designed representation.  Resource allocation, optimization, calculating quantities, inferences about of possible outcomes, classification, reasoning about extreme or special cases, and debugging: these are just a few of the many possibilities. These tasks are more generic than the information-oriented options considered in the 'design uses' column of Figure 27 in the article. They are worth addressing, because they provide constraints for the initial stages of representation design, by narrowing the search for what are likely to be effective correspondences to adopt. For example, if taxonomic classification is important, then separation and layering will be important correspondences; whereas magnitude calculations may demand scale mapping, Euclidian and metrical correspondences.

The third aspect concerns situations in which the visual representation must support not just a single task, but many diverse tasks. For example, a visual representation to help students learn about electricity will be used to explain the topology of circuits, make computations with electrical quantities, provide explanations of circuit behaviour (in terms of formal algebraic models and as qualitative causal models), facilitate fault finding or trouble shooting, among other activities. The creation of novel representations in such circumstances is perhaps one of the most challenging for designers. So, what knowledge can help? In this case, I advocate attempting to design representations on the basis of an analysis of the underlying conceptual structure of the knowledge of the target domain. Why? Because the nature of the knowledge is invariant across different classes of task. For example, for problem solving and learning of electricity, all the tasks depend upon the common fundamental conceptual structures of the domain that knit together the laws governing the physical properties of electricity and circuit topology. Hence, a representation that makes these concepts readily available through effective representation designed will probably be effective for a wide range of tasks.

In summary, it is desirable for the aspiring visual representation designer to consider symbolic correspondence, but I recommend they cast their net more widely for inspiration by learning about the human cognitive architecture, focusing on the nature of the task for which they are designing, and most critically thinking about the underlying conceptual structure of the knowledge of the target domain.

5.13 Commentary by Brad A. Myers

I have been teaching human-computer interaction to students with a wide range of backgrounds for many years. One of the most difficult areas for them to learn seems to be visual design. Students seem to quickly pick up rules like Nielsen's Heuristics for interaction (Nielsen & Molich, 1990), whereas the guidelines for visual design are much more subtle. Alan Blackwell's article presents many useful points, but a designer needs to know so much more! Whereas students can achieve competence at achieving Nielsen's "consistency and standards," for example, they struggle with selecting an appropriate representation for their information. And only a trained graphic designer is likely to be able to create an attractive and effective icon. Some people have a much better aesthetic sense, and can create much more beautiful and appropriate representations. A key goal of my introductory course, therefore, is to try to impart to the students how difficult it is to do visual design, and how wide the set of choices is. Studying the examples that Blackwell provides will give the reader a small start towards effective visual representations, but the path requires talent, study, and then iterative design and testing to evaluate and improve a design's success.

• Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interfaces. Paper presented at the Proc. ACM CHI'90 Conf, Seattle, WA, 249-256.
• See also: http://www.useit.com/papers/heuristic/heuristic_list.html

Topics in This Book Chapter

Open access—link to us.

We believe in Open Access and the  democratization of knowledge . Unfortunately, world-class educational materials such as this page are normally hidden behind paywalls or in expensive textbooks.

If you want this to change , cite this book chapter , link to us, or join us to help us democratize design knowledge !

Privacy Settings

Our digital services use necessary tracking technologies, including third-party cookies, for security, functionality, and to uphold user rights. Optional cookies offer enhanced features, and analytics.

Experience the full potential of our site that remembers your preferences and supports secure sign-in.

Governs the storage of data necessary for maintaining website security, user authentication, and fraud prevention mechanisms.

Enhanced Functionality

Saves your settings and preferences, like your location, for a more personalized experience.

Referral Program

We use cookies to enable our referral program, giving you and your friends discounts.

Error Reporting

We share user ID with Bugsnag and NewRelic to help us track errors and fix issues.

Optimize your experience by allowing us to monitor site usage. You’ll enjoy a smoother, more personalized journey without compromising your privacy.

Analytics Storage

Collects anonymous data on how you navigate and interact, helping us make informed improvements.

Differentiates real visitors from automated bots, ensuring accurate usage data and improving your website experience.

Lets us tailor your digital ads to match your interests, making them more relevant and useful to you.

Advertising Storage

Stores information for better-targeted advertising, enhancing your online ad experience.

Personalization Storage

Permits storing data to personalize content and ads across Google services based on user behavior, enhancing overall user experience.

Advertising Personalization

Allows for content and ad personalization across Google services based on user behavior. This consent enhances user experiences.

Enables personalizing ads based on user data and interactions, allowing for more relevant advertising experiences across Google services.

Receive more relevant advertisements by sharing your interests and behavior with our trusted advertising partners.

Enables better ad targeting and measurement on Meta platforms, making ads you see more relevant.

Allows for improved ad effectiveness and measurement through Meta’s Conversions API, ensuring privacy-compliant data sharing.

LinkedIn Insights

Tracks conversions, retargeting, and web analytics for LinkedIn ad campaigns, enhancing ad relevance and performance.

LinkedIn CAPI

Enhances LinkedIn advertising through server-side event tracking, offering more accurate measurement and personalization.

Google Ads Tag

Tracks ad performance and user engagement, helping deliver ads that are most useful to you.

Share Knowledge, Get Respect!

or copy link

Cite according to academic standards

Simply copy and paste the text below into your bibliographic reference list, onto your blog, or anywhere else. You can also just hyperlink to this book chapter.

New to UX Design? We’re giving you a free ebook!

Download our free ebook The Basics of User Experience Design to learn about core concepts of UX design.

In 9 chapters, we’ll cover: conducting user interviews, design thinking, interaction design, mobile UX design, usability, UX research, and many more!

Download Premium UX Design Literature

Enjoy unlimited downloads of our literature. Our online textbooks are written by 100+ leading designers, bestselling authors and Ivy League professors.

New to UX Design? We’re Giving You a Free ebook!

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Published: 07 March 2024

Learning high-level visual representations from a child’s perspective without strong inductive biases

• A. Emin Orhan   ORCID: orcid.org/0000-0002-5486-7385 1 &
• Brenden M. Lake   ORCID: orcid.org/0000-0001-8959-3401 1 , 2  

Nature Machine Intelligence volume  6 ,  pages 271–283 ( 2024 ) Cite this article

3426 Accesses

3 Citations

152 Altmetric

Metrics details

• Computer science
• Human behaviour

This article has been updated

A preprint version of the article is available at arXiv.

Young children develop sophisticated internal models of the world based on their visual experience. Can such models be learned from a child’s visual experience without strong inductive biases? To investigate this, we train state-of-the-art neural networks on a realistic proxy of a child’s visual experience without any explicit supervision or domain-specific inductive biases. Specifically, we train both embedding models and generative models on 200 hours of headcam video from a single child collected over two years and comprehensively evaluate their performance in downstream tasks using various reference models as yardsticks. On average, the best embedding models perform at a respectable 70% of a high-performance ImageNet-trained model, despite substantial differences in training data. They also learn broad semantic categories and object localization capabilities without explicit supervision, but they are less object-centric than models trained on all of ImageNet. Generative models trained with the same data successfully extrapolate simple properties of partially masked objects, like their rough outline, texture, colour or orientation, but struggle with finer object details. We replicate our experiments with two other children and find remarkably consistent results. Broadly useful high-level visual representations are thus robustly learnable from a sample of a child’s visual experience without strong inductive biases.

This is a preview of subscription content, access via your institution

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 12 digital issues and online access to articles

111,21 € per year

only 9,27 € per issue

Buy this article

• Purchase on SpringerLink
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset

Capturing the objects of vision with neural networks

Capturing human categorization of natural images by combining deep networks and cognitive models

Data availability.

Except for SAYCam, all data used in this study are publicly available. Instructions for accessing the public datasets are detailed in Methods . The SAYCam dataset can be accessed by authorized users with an institutional affiliation from the following Databrary repository: https://doi.org/10.17910/b7.564 . The ‘Labeled S’ evaluation dataset, which is a subset of SAYCam, is also available from the same repository under the session name ‘Labeled S’.

Code availability

All of our pretrained models (over 70 different models), as well as a variety of tools to use and analyse them, are available from the following public repository: https://github.com/eminorhan/silicon-menagerie (ref. 63 ). The repository also contains further examples of (1) attention and class activation maps, (2) t -SNE visualizations of embeddings, (3) nearest neighbour retrievals from the embedding models and (4) unconditional and conditional samples from the generative models. The code used for training and evaluating all the models is also publicly available from the same repository.

Change history

11 june 2024.

In the version of the article initially published, the name of Cliona O’Doherty was not included in the peer review information for this article, which has now been amended.

Bomba, P. & Siqueland, E. The nature and structure of infant form categories. J. Exp. Child Psychol. 35 , 294–328 (1983).

Article   Google Scholar  

Murphy, G. The Big Book of Concepts (MIT, 2002).

Kellman, P. & Spelke, E. Perception of partly occluded objects in infancy. Cogn. Psychol. 15 , 483–524 (1983).

Spelke, E., Breinlinger, K., Macomber, J. & Jacobson, K. Origin of knowledge. Psychol. Rev. 99 , 605–632 (1992).

Ayzenberg, V. & Lourenco, S. Young children outperform feed-forward and recurrent neural networks on challenging object recognition tasks. J. Vis. 20 , 310–310 (2020).

Huber, L. S., Geirhos, R. & Wichmann, F. A. The developmental trajectory of object recognition robustness: children are like small adults but unlike big deep neural networks. J. Vis. 23 , 4 (2023).

Locke, J. An Essay Concerning Human Understanding (ed. Fraser, A. C.) (Clarendon Press, 1894).

Leibniz, G. New Essays on Human Understanding 2nd edn (eds Remnant, P. & Bennett, J.) (Cambridge Univ. Press, 1996).

Spelke, E. Initial knowledge: six suggestions. Cognition 50 , 431–445 (1994).

Markman, E. Categorization and Naming in Children (MIT, 1989).

Merriman, W., Bowman, L. & MacWhinney, B. The mutual exclusivity bias in children’s word learning. Monogr. Soc. Res. Child Dev. 54 , 1–132 (1989).

Elman, J., Bates, E. & Johnson, M. Rethinking Innateness: A Connectionist Perspective on Development (MIT, 1996).

Sullivan, J., Mei, M., Perfors, A., Wojcik, E. & Frank, M. SAYCam: a large, longitudinal audiovisual dataset recorded from the infant’s perspective. Open Mind 5 , 20–29 (2022).

Caron, M. et al. Emerging properties in self-supervised vision transformers. In Proc. IEEE/CVF International Conference on Computer Vision 9650–9660 (IEEE, 2021).

Zhou, P. et al. Mugs: a multi-granular self-supervised learning framework. Preprint at https://arxiv.org/abs/2203.14415 (2022).

He, K. et al. Masked autoencoders are scalable vision learners. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition 15979–15988 (IEEE, 2022).

Dosovitskiy, A. et al. An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations (2020).

Xie, S., Girshick, R., Dollár, P., Tu, Z. & He, K. Aggregated residual transformations for deep neural networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 1492–1500 (IEEE, 2017).

Russakovsky, O. et al. ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115 , 211–252 (2015).

Article   MathSciNet   Google Scholar  

Smaira, L. et al. A short note on the Kinetics-700-2020 human action dataset. Preprint at https://arxiv.org/abs/2010.10864 (2020).

Grauman, K. et al. Ego4D: around the world in 3,000 hours of egocentric video. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition . 18995–19012 (IEEE, 2022).

Esser, P., Rombach, R. & Ommer, B. Taming transformers for high-resolution image synthesis. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 12873–12883 (IEEE, 2021).

Radford, A. et al. Language models are unsupervised multitask learners. OpenAI https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf (2019).

Zhou, B., Khosla, A., Lapedriza, A., Oliva, A. & Torralba, A. Learning deep features for discriminative localization. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 2921–2929 (IEEE, 2016).

van der Maaten, L. & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res . 9 , 2579–2605 (2008).

Kuznetsova, A. et al. The Open Images Dataset V4. Int. J. Comput. Vis. 128 , 1956–1981 (2020).

Smith, L. & Slone, L. A developmental approach to machine learning? Front. Psychol. 8 , 2124 (2017).

Bambach, S., Crandall, D., Smith, L. & Yu, C. Toddler-inspired visual object learning. Adv. Neural Inf. Process. Syst. 31 , 1209–1218 (2018).

Zaadnoordijk, L., Besold, T. & Cusack, R. Lessons from infant learning for unsupervised machine learning. Nat. Mach. Intell. 4 , 510–520 (2022).

Orhan, E., Gupta, V. & Lake, B. Self-supervised learning through the eyes of a child. Adv. Neur. In. 33 , 9960–9971 (2020).

Google Scholar  

Lee, D., Gujarathi, P. & Wood, J. Controlled-rearing studies of newborn chicks and deep neural networks. Preprint at https://arxiv.org/abs/2112.06106 (2021).

Zhuang, C. et al. Unsupervised neural network models of the ventral visual stream. Proc. Natl Acad. Sci. USA 118 , e2014196118 (2021).

Zhuang, C. et al. How well do unsupervised learning algorithms model human real-time and life-long learning? In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2022).

Vong, W. K., Wang, W., Orhan, A. E. & Lake, B. M. Grounded language acquisition through the eyes and ears of a single child. Science 383 , 504–511 (2024).

Locatello, F. et al. Object-centric learning with slot attention. Adv. Neur. In. 33 , 11525–11538 (2020).

Lillicrap, T., Santoro, A., Marris, L., Akerman, C. & Hinton, G. Backpropagation and the brain. Nat. Rev. Neurosci. 21 , 335–346 (2020).

Gureckis, T. & Markant, D. Self-directed learning: a cognitive and computational perspective. Perspect. Psychol. Sci. 7 , 464–481 (2012).

Long, B. et al. The BabyView camera: designing a new head-mounted camera to capture children’s early social and visual environments. Behav. Res. Methods https://doi.org/10.3758/s13428-023-02206-1 (2023).

Moore, D., Oakes, L., Romero, V. & McCrink, K. Leveraging developmental psychology to evaluate artificial intelligence. In 2022 IEEE International Conference on Development and Learning (ICDL) 36–41 (IEEE, 2022).

Frank, M. C. Bridging the data gap between children and large language models. Trends Cogn. Sci. 27 , 990–992 (2023).

Object stimuli. Brady Lab https://bradylab.ucsd.edu/stimuli/ObjectCategories.zip

Konkle, T., Brady, T., Alvarez, G. & Oliva, A. Conceptual distinctiveness supports detailed visual long-term memory for real-world objects. J. Exp. Psychol. Gen. 139 , 558 (2010).

Lomonaco, V. & Maltoni, D. CORe50 Dataset. GitHub https://vlomonaco.github.io/core50 (2017).

Lomonaco, V. & Maltoni, D. CORe50: a new dataset and benchmark for continuous object recognition. In Proc. 1st Annual Conference on Robot Learning (eds Levine, S. et al.) 17–26 (PMLR, 2017).

Russakovsky, O. et al. ImageNet Dataset. https://www.image-net.org/download.php (2015).

Geirhos, R. et al. Shortcut learning in deep neural networks. Nat. Mach. Intell. 2 , 665–673 (2020).

Geirhos, R. et al. Partial success in closing the gap between human and machine vision. Adv. Neur. In. 34 , 23885–23899 (2021).

Geirhos, R. et al. ImageNet OOD Dataset. GitHub https://github.com/bethgelab/model-vs-human (2021).

Mehrer, J., Spoerer, C., Jones, E., Kriegeskorte, N. & Kietzmann, T. An ecologically motivated image dataset for deep learning yields better models of human vision. Proc. Natl Acad. Sci. USA 118 , e2011417118 (2021).

Mehrer, J., Spoerer, C., Jones, E., Kriegeskorte, N. & Kietzmann, T. Ecoset Dataset. Hugging Face https://huggingface.co/datasets/kietzmannlab/ecoset (2021).

Zhou, B., Lapedriza, A., Khosla, A., Oliva, A. & Torralba, A. Places: a 10 million image database for scene recognition. IEEE T. Pattern Anal. 40 , 1452–1464 (2017).

Zhou, B. et al. Places365 Dataset. http://places2.csail.mit.edu (2017).

Pont-Tuset, J. et al. The 2017 DAVIS challenge on video object segmentation. Preprint at https://arxiv.org/abs/1704.00675 (2017).

Pont-Tuset, J. et al. DAVIS-2017 evaluation code, dataset and results. https://davischallenge.org/davis2017/code.html (2017).

Lin, T. et al. Microsoft COCO: common objects in context. In Computer Vision – ECCV 2014 (eds Fleet, D. et al.) 740–755 (2014).

COCO Dataset. https://cocodataset.org/#download (2014).

Jabri, A., Owens, A. & Efros, A. Space-time correspondence as a contrastive random walk. Adv. Neur. In. 33 , 19545–19560 (2020).

Kinetics-700-2020 Dataset. https://github.com/cvdfoundation/kinetics-dataset#kinetics-700-2020 (2020).

Ego4D Dataset. https://ego4d-data.org/ (2022).

Kingma, D. & Ba, J. Adam: A method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980 (2014).

VQGAN resources. GitHub https://github.com/CompVis/taming-transformers (2021).

Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B. & Hochreiter, S. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Adv. Neural Inf. Process. Syst. 30 , 6629–6640 (2017).

Orhan, A. E. eminorhan/silicon-menagerie: v1.0.0-alpha. Zenodo https://doi.org/10.5281/zenodo.8322408 (2023).

Download references

Acknowledgements

We thank W. K. Vong, A. Tartaglini and M. Ren for helpful discussions and comments on an earlier version of this paper. This work was supported by the DARPA Machine Common Sense program (B.M.L.) and NSF Award 1922658 NRT-HDR: FUTURE Foundations, Translation and Responsibility for Data Science (B.M.L.).

Author information

Authors and affiliations.

Center for Data Science, New York University, New York, NY, USA

A. Emin Orhan & Brenden M. Lake

Department of Psychology, New York University, New York, NY, USA

Brenden M. Lake

You can also search for this author in PubMed   Google Scholar

Contributions

A.E.O. and B.M.L. conceptualized and designed the study. A.E.O. implemented the experiments. A.E.O. analysed the results with feedback from B.M.L. A.E.O. wrote the first draft. B.M.L. reviewed and edited the paper.

Corresponding author

Correspondence to A. Emin Orhan .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Peer review

Peer review information.

Nature Machine Intelligence thanks Rhodri Cusack, Cliona O’Doherty, Masataka Sawayama and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information.

Supplementary Figs. 1–8 and Tables 1 and 2.

Reporting Summary

Rights and permissions.

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Cite this article.

Orhan, A.E., Lake, B.M. Learning high-level visual representations from a child’s perspective without strong inductive biases. Nat Mach Intell 6 , 271–283 (2024). https://doi.org/10.1038/s42256-024-00802-0

Download citation

Received : 24 May 2023

Accepted : 05 February 2024

Published : 07 March 2024

Issue Date : March 2024

DOI : https://doi.org/10.1038/s42256-024-00802-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Artificial intelligence tackles the nature–nurture debate.

• Justin N. Wood

Nature Machine Intelligence (2024)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Discovering Respects for Visual Similarity

• Conference paper
• First Online: 01 January 2023
• Cite this conference paper

• Olivier Risser-Maroix 11 ,
• Camille Kurtz 11 &
• Nicolas Loménie 11  

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13813))

Included in the following conference series:

• Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR)

526 Accesses

Similarity is a fundamental concept in artificial intelligence and cognitive sciences. Despite all the efforts made to study similarity, defining and measuring it between concepts or images remains challenging. Fortunately, measuring similarity is comparable to answering “ why ”/“ in which respects ” two stimuli are similar. While most related works done in computer sciences try to measure the similarity, we propose to analyze it from a different angle and retrospectively find such respects. In this paper, we provide a pipeline allowing us to find, for a given dataset of image pairs, what are the different concepts generally compared and why . As an indirect evaluation, we propose generating an automatic explanation of clusters of image pairs found using a model pre-trained on texts/images. An experimental study highlights encouraging results toward a better comprehension of visual similarity.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Evaluating Image Similarity Using Contextual Information of Images with Pre-trained Models

QISS: An Open Source Image Similarity Search Engine

Diverse image search with explanations

Rissland, E.: AI and similarity. IEEE Intell. Syst. 21 , 39–49 (2006)

Article   Google Scholar  

Tversky, A., Gati, I.: Studies of similarity. In: Cognition and Categorization (1978)

Google Scholar  

Gentner, D., Markman, A.B.: Structure mapping in analogy and similarity. Am. Psychol. 52 , 45–56 (1997)

Tirilly, P., Mu, X., Huang, C., Xie, I., Jeong, W., Zhang, J.: On the consistency and features of image similarity. In: IIiX, pp. 164–173 (2012)

Liu, T., Cooper, L.A.: The influence of task requirements on priming in object decision and matching. Mem. Cognit. 29 , 874–882 (2001)

Goodman, N.: Seven strictures on similarity. In: Problems and Projects (1972)

Medin, D.L., Goldstone, R.L., Gentner, D.: Respects for similarity. Psychol. Rev. 100 , 254–274 (1993)

Peterson, J.C., Abbott, J.T., Griffiths, T.L.: Evaluating (and improving) the correspondence between deep neural networks and human representations. Cogn. Sci. 42 , 2648–2669 (2018)

Zheng, C.Y., Pereira, F., Baker, C.I., Hebart, M.N.: Revealing interpretable object representations from human behavior. In: ICLR (2019)

Hebart, M.N., Zheng, C.Y., Pereira, F., Baker, C.I.: Revealing the multidimensional mental representations of natural objects underlying human similarity judgements. Nat. Hum. Behav. 4 , 1173–1185 (2020)

Kubilius, J., Bracci, S., Op de Beeck, H.P.: Deep neural networks as a model for human shape sensitivity. PLoS Comput. Biol. 12 , e1004896 (2016)

Pramod, R.T., Arun, S.P.: Do computational models differ systematically from human object perception? In: CVPR, pp. 1601–1609 (2016)

Sadovnik, A., Gharbi, W., Vu, T., Gallagher, A.C.: Finding your lookalike: measuring face similarity rather than face identity. In: CVPR Workshops, pp. 2345–2353 (2018)

Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The effectiveness of deep features as a perceptual metric. In: CVPR, pp. 586–595 (2018)

German, J.S., Jacobs, R.A.: Can machine learning account for human visual object shape similarity judgments? Vis. Res. 167 , 87–99 (2020)

Rosenfeld, A., Solbach, M.D., Tsotsos, J.K.: Totally looks like - how humans compare, compared to machines. In: ACCV, pp. 282–297 (2018)

Hamilton, M., Fu, S., Freeman, W.T., Lu, M.: Conditional image retrieval. CoRR, abs/2007.07177 (2020)

Attarian, M., Roads, B.D., Mozer, M.C.: Transforming neural network visual representations to predict human judgments of similarity. In: SVRHM Workshop (2020)

Chen, C., Li, O., Barnett, A., Su, J., Rudin, C.: This looks like that: deep learning for interpretable image recognition. CoRR, abs/1806.10574 (2018)

Nauta, M., Jutte, A., Provoost, J.C., Seifert, C.: This looks like that, because ... explaining prototypes for interpretable image recognition. CoRR, abs/2011.02863 (2020)

Risser-Maroix, O., Kurtz, C., Loménie, N.: Learning an adaptation function to assess image visual similarities. In: ICIP, pp. 2498–2502 (2021)

Takmaz, A., Probst, T., Pani Paudel, D., Van Gool, L.: Learning feature representations for look-alike images. In: CVPR Workshops (2019)

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., et al.: Learning transferable visual models from natural language supervision. In: ICML, vol. 139, pp. 8748–8763 (2021)

Cichy, R.M., Kaiser, D.: Deep neural networks as scientific models. Trends Cogn. Sci. 23 , 305–317 (2019)

Zhuang, C., et al.: Unsupervised neural network models of the ventral visual stream. Proc. Natl. Acad. Sci. 118 , 237–250 (2021)

Lake, B., Zaremba, W., Fergus, R., Gureckis, T.: Neural networks predict category typicality ratings for images. In: CogSci, pp. 1–15 (2015)

Hariharan, B., Girshick, R.: Low-shot visual recognition by shrinking and hallucinating features. In: ICCV, pp. 3018–3027 (2017)

Lee, D.D., Seung, H.S.: Learning the parts of objects by non-negative matrix factorization. Nature 401 , 788–791 (1999)

Article   MATH   Google Scholar  

Download references

Author information

Authors and affiliations.

Université Paris Cité, LIPADE, Paris, France

Olivier Risser-Maroix, Camille Kurtz & Nicolas Loménie

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Olivier Risser-Maroix .

Editor information

Editors and affiliations.

Concordia University, Montreal, QC, Canada

Adam Krzyzak

Ching Y. Suen

Ca Foscari University of Venice, Venezia, Italy

Andrea Torsello

Nicola Nobile

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Cite this paper.

Risser-Maroix, O., Kurtz, C., Loménie, N. (2022). Discovering Respects for Visual Similarity. In: Krzyzak, A., Suen, C.Y., Torsello, A., Nobile, N. (eds) Structural, Syntactic, and Statistical Pattern Recognition. S+SSPR 2022. Lecture Notes in Computer Science, vol 13813. Springer, Cham. https://doi.org/10.1007/978-3-031-23028-8_14

Download citation

DOI : https://doi.org/10.1007/978-3-031-23028-8_14

Published : 01 January 2023

Publisher Name : Springer, Cham

Print ISBN : 978-3-031-23027-1

Online ISBN : 978-3-031-23028-8

eBook Packages : Computer Science Computer Science (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research