This article is based on the paper Generation of Concept-Representative Symbols.
Website @ CDV Lab.

Nowadays we use computers – in their many forms – not only for work but also as self-improvement tools, be it as a way to always be connected to other people, to assess our body status by constantly measuring it, or to even micromanage our home appliances. Nonetheless, it seems obvious that we are underestimating something that could be highly improved: our mind. More specifically, using computers to foster our creativity.

There is already a number of apps that allow us to turn our vacation pictures into self-named pieces of art; but isn’t there a more useful way of using our computers for creative purposes?

On one hand, creativity support tools help users in activities ranging from drawing or sketching _[1,2] to typeface design _[3]. These tools, however, are limited in terms of creative contributions to the process.

On the other hand, some researchers are already focused on trying to make computers be creative on their own – for example, some artificial artists are AARON _[4], The Painting Fool _[5] or DARCI _[6]. These generative systems normally do not consider the feedback from the user.

Nonetheless, we believe that for creative stimulation the sweet spot is somewhere between these two ends.

Computer as a Colleague

Inspired by the way humans naturally collaborate in creative domains, as such as musical improvisation, we believe we should have a similar approach towards computers. When considering computers as an extension of our body, co-creativity seems the way to go.

In co-creativity, participants are normally considered equal, building on previous contributions and mutually influencing each other. Each brings its own knowledge and experience to the process, leading to different interpretations and unexpected solutions _[7,8].

In this sense, we would change our view: not seeing computers as something that blindly follows our orders, but rather as our partners in creation. Such systems already exist and are hybrids between creativity support tools and generative systems, including:

• music: a percussion robot that mimics musicians and is able to generate synchronized improvisations _[9];
• movement: an interactive art installation in which humans and artificial intelligent agents collaborate in movement improvisation _[10];
• visual arts: a drawing system capable of contributing in a creative way by suggesting new elements _[11].

Computers have the potential to not only turn our bad ideas into good ones, but also to possibly lead us to a “Eureka!” screaming frenzy.

Visualizing it  

As far as brainstorming for a concrete idea is concerned, a visual approach seems to be the most logical one – after all, sketching is often used in the ideation process _[12]. However, in order to generate ideas, we first need to represent them.

Humans have been visually representing ideas since more than two hundred thousand years ago. Take for example cave paintings. These representations vary from being completely pictorial, e.g. pictograms, to more abstract, e.g. ideographs.

Fig. 1 Spectrum of abstraction. The more simple it is, the more people it can represent. Adapted from [13]. 

This shifting along the spectrum of abstraction (see Fig. 1) may bring interpretation issues when aiming for clarity but does not pose a threat when dealing with unlocking creativity and fostering ideas.

In addition, breaking down complexity into simplicity has already been used to retrieve the essence of concepts. (One such process can be seen in Pablo Picasso’s "Bull", a set of eleven lithographs produced in 1945.)

Representing concepts and their connections does not have to be done in an obvious way – it can be done by only giving visual clues. In this sense, semiotics and hidden meaning seem to be great options.

There are three main sources of semiotic properties that can change the meaning of a representation: shape, color and position.

Shape

Firstly, shape and color seem to have the most immediate impact. Wheeler _[14] even points out that in terms of sequence of visual perception and cognition, the brain firstly acknowledges shapes and only after considers color.

There are unlimited ways of visually representing a concept. However, when just considering the shape itself, its visual qualities may cause us to automatically perceive it in a certain way, even unrelated to the concept being represented.

Consider these two simple shapes: one smooth and curved; another rough and jagged (see Fig. 2). 

Fig. 2 Nonsense shapes firstly presented by Wolfgang Köhler _[15]. Shape A is described as smooth and curved; shape B as rough and jagged. This mapping between sound and visual shapes was later known as the bouba/kiki effect.

According to studies done by Bernard Lyman  _[15], there are concepts consistently attributed to these meaningless shapes. Some concepts were even agreed upon by 100% of participants, such as “calm” for shape A and “angry” and “resentful” for shape B. A tendency can also be observed in some abstract non-emotional concepts – e.g. 87% attribute shape A to “eternity”; 80% attribute shape B to “consciousness” _[15].

This attribution bias goes one step further when considering names without any apparent meaning. According to psychologist Wolfgang Köhler _[16], most people attribute “maluma” to shape A and “takete” to shape B, without any hesitation.

Humans tend to perform mappings among domains, namely between image and sound. Sharp shapes tend to be associated with harsh-sounding names and organic shapes with smooth ones _[17], which can explain these name-shape attributions.

Color

Color has also been proven to facilitate interpretation of visual representations of concepts. Using colors that are related to what is represented improves our reading speed _[18]. However, the opposite also happens: using an unsuitable or incompatible color may lead to unwanted consequences, such as slower reading comprehension and speed, (see the Stroop effect.) 

Choosing a color is no ordinary task as it is directly connected to the meaning of the representation. By simply attributing a different color to the same symbol, its perceptual meaning also changes. 

Consider how the meaning of a banana can change with its color:One example of this can be seen in Fig. 3 which shows how the meaning of a banana can change with its color.

• A green banana is most likely unripe and thus not ready to be eaten;
• A yellow banana is already mature;
• A red banana, despite existing, will probably seem weird to anyone not familiar with them.

Nevertheless, this aspect has more to it than simply picking a color based on a single meaning. The same color changes in interpretation according to what is being represented.

Fig. 3 Two examples that show how color can affect meaning. On the left is the banana example and on the right the traffic light example. In both examples, changing color changes the meaning. When comparing both, one realises that the same color can have different meaning.

Imagine you’re at a traffic intersection. If you always understood banana-green as a sign to “patiently wait,” then driving will most likely result in a loud honking symphony (see Fig. 3). In the same way, upon facing a red traffic light, there is often not much time to contemplate its meaning (as you might have with a red banana), because that confusion and hesitation could cause a life-threatening situation.

Position

Like the ones before, the semiotic factor of position can entirely change the meaning of a representation. This is observed in Fig. 4, in which the same elements are positioned differently, leading to three different interpretations. Using position in symbol combination to obtain different meanings has been applied by several authors, one of which is Charles Bliss _[19].

Fig.4 Difference in meaning according to position. The left pictogram means "give," the middle means "receive" and the one on the right means "take." Adapted from [20].

The Ultimate Task

Since we form ideas when we are being stimulated by either visual, auditory or tactile stimulus, we believe that the future of creativity is somehow connected to computational sensorial stimulation.

By focusing on the visual domain, the biggest problem lies in aligning the user interpretation with the computer's. Finding a way for computers to share our semiotic knowledge will improve their ability to successfully cooperate with us in a creative task.

However, this is related to a complex problem in computation which deals with how computers use common sense information. Such a problem should be seen as a priority if we are to fully shift away from using computers as tools and see them as partners as well as make computational co-creativity applications available to the general public.

This article is based on the paper: 
Cunha, J. M., Martins, P., Cardoso, A., & Machado, P. (2015). Generation of concept-representative symbols. In ICCBR (Workshops), 156–160. 

Website @ CDV Lab.

Acknowledgement:
a special thanks to Filipe Carneiro for helping with the featured image. See some of his concept art and design related works.

[1] Dixon, D., Prasad, M., & Hammond, T. (2010, April). iCanDraw: using sketch recognition and corrective feedback to assist a user in drawing human faces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 897-906). ACM.
[2] Lee, Y. J., Zitnick, C. L., & Cohen, M. F. (2011, August). Shadowdraw: real-time user guidance for freehand drawing. In ACM Transactions on Graphics (TOG) (Vol. 30, No. 4, p. 27). ACM.
[3] Cunha, J. M., Martins, T., Martins, P., Bicker, J., & Machado, P. (2016). TypeAdviser: a type design aiding-tool. In C3GI@ ESSLLI.
[4] McCorduck, P. (1991). Aaron's code: meta-art, artificial intelligence, and the work of Harold Cohen. Macmillan.
[5] Colton, S. (2011). The painting fool in new dimensions. In Proceedings of the 2nd International Conference on Computational Creativity (Vol. 112).
[6] Norton, D., Heath, D., & Ventura, D. (2013). Finding creativity in an artificial artist. The Journal of Creative Behavior, 47(2), 106-124.
[7] Mamykina, L., Candy, L., & Edmonds, E. (2002). Collaborative creativity. Communications of the ACM, 45(10).
[8] Jordanous, A. (2017). Co-creativity and perceptions of computational agents in co-creativity. In Proceedings of the Eighth International Conference on Computa- tional Creativity, Atlanta, US. ACC. 
[9] Hoffman, G., & Weinberg, G. (2010, April). Shimon: an interactive improvisational robotic marimba player. In CHI'10 Extended Abstracts on Human Factors in Computing Systems (pp. 3097-3102). ACM.
[10] Long, D., Jacob, M., Davis, N., & Magerko, B. (2017, June). Designing for Socially Interactive Systems. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition (pp. 39-50). ACM.
[11] Davis, N., Hsiao, C. P., Singh, K. Y., & Magerko, B. (2016, September). Co-creative drawing agent with object recognition. In Twelfth Artificial Intelligence and Interactive Digital Entertainment Conference.
[12] Knapp. J. (2016, February  24). Google Ventures On How Sketching Can Unlock Big Ideas.
[13] McCloud, S. (1993). Understanding comics: The invisible art. Northhampton, MA: Kitchen Sink Press.
[14] Wheeler, A. (2009). Designing brand identity. Jakarta: John Wiley and Sons Inc.
[15] Lyman, B. (1979). Representation of complex emotional and abstract meanings by simple forms. Perceptual and Motor Skills, 49(3), 839-842.
[16] Köhler, W. (1970). Gestalt psychology: An introduction to new concepts in modern psychology. WW Norton & Company.
[17] Ramachandran, V. S., & Hubbard, E. M. (2003). Hearing colors, tasting shapes. SCIENTIFIC AMERICAN-AMERICAN EDITION-, 288(5), 52-59.
[18] Lin, S., et al.: Selecting semantically-resonant colors for data visualization. In: Computer Graphics Forum. vol. 32, pp. 401–410. Wiley Online Library (2013)
[19] Bliss, C. K. (1965). Semantography (Blissymbolics): A Logical Writing for an illogical World. Semantography Blissymbolics Publ.
[20] Krampen, M., Götte, M., & Kneidl, M. (2007). Die Welt der Zeichen: Kommunikation mit Piktogramme n/ The World of Signs: Communication by Pictogram. avedition GmbH, Ludwigsburg.

GuardarGuardar