Methods for two dimensional stroke based painterly rendering. Effects and applications

Thesis Booklet
painterly rendering. Effects and applications
Levente Kovacs
Supervisor: Prof. Dr. Tamas Sziranyi
Veszprem, 2006.
decade ago. Non-photorealistic is just what the name suggests - images
created not with the goal to reproduce original imagery as close as possible,
but to produce different representations of a model, something that might
even be considered artistic. The first goals of these techniques were to some-
how imitate real life painting effects artificially. Since the first trials many
different methods appeared, both in 2 and 3D, for many different purposes.
Yet the first goal has always remained: produce visually pleasant represen-
tations of model images or create such images artificially with dedicated
tools.
Stroke based rendering (SBR) methods form a special subset of non-
photorealistic rendering techniques. In SBR the painterly effects are pro-
duced by simulating real life brush stroke to an extent, and the painting
process itself, to produce images by a so called artificial painting process.
An overview of some important methods and techniques of stroke-based
2
non-photorealistic rendering can be found in the books of Gooch & Gooch.
(Non-Photorealistic Rendering. AK Peters, Ltd. 2001.) and Strothotte et
al. (Non-Photorealistic Computer Graphics, Morgan Kaufman Publishers,
2002.).
During the work, the author’s goals were to create new painterly
effects on images using artificial stroke template based rendering approaches,
investigating stochastic and non-stochastic painting approaches, developing
alternative solutions, application of different image processing and image
structure extraction methods in stroke based painting, develop new stroke
based painting methods. At the same time, investigating other connected
areas like optimizing template based painting, possible easy to handle and
portable representations and encoding of rendered images, and applications.
The main results of the work are in developing new stroke based
painting methods, in optimizations and coding of generated paintings, in
connecting the painting methods with scale-space feature extraction and
also in creating focus based level of detail control during the painting pro-
cess. Both the theoretical and application results are presented.
3
Research methodology
The goal and focus of artificial painterly rendering techniques is usually on
producing different kinds of effects, the kinds and styles of these effects. Sev-
eral methods have been developed which can generate many painting-like
effects, automatically, semi-automatically, or in a user controlled environ-
ment. In this work, that was one of the goals of the author, creating new
painterly rendering methods, but there has been also a more important
guideline: how can the painting process be controlled, on the highest pos-
sible level, by image structure information, and thus developing automatic
effect generator methods which only need an input model image to run. An-
other important guideline was that the developed painting methods should
draw off from stochastic methods in their nature. The development of the
different painting methods was based on these guidelines. A next point al-
ways kept in consideration was that how the painted images produced by
the different developed methods could be handled and stored efficiently in
an easy to handle form, which in itself was a new point of view in the field
4
of painterly rendering.
The painterly rendering methods are the central point of this thesis.
During the research work, the author has used both existing and new tools
and resources. During the painterly rendering work, the tests and trials were
performed on standard test images well known from the image processing
field, and also on images collected by the author for these purposes. In the
case of the relative focus map estimation method, the tests and evaluations
were performed on standard texture data sets and test images, and in the
case of focus map based indexing tests an image database collected by the
author was also used. The base of all evaluations and comparisons were clas-
sical image and signal processing error measures. For development and test
applications the author has used several mathematical and programming
environments.
The research work has been done at the Image Processing Labora-
tory of the Department of Image Processing and Neurocomputing, Pannon
University (formerly University if Veszprem).
Possible Applications 5
Possible Applications
Application areas of the methods presented in this work are those of painterly,
cartoon-like image and video transformation. E.g. nowadays video games
and also cinema movies are using more and more non-photorealistically
rendered elements, thus research in this field is still a demanding and con-
tinuously developing field. The author’s results give solutions mostly in the
image structure-driven automatic effect generation area. Possible applica-
tions, as ao presented in the thesis, are the following:
• Creation and coding of stroke-based animations - producing animation
effects by painterly rendering methods, and also compactly storing and
coding them,
• Stroke-rendered paintings - generating different style representations
of real painting with various stroke templates, into portable and scal-
able form,
Possible Applications 6
ing painterly rendered images in different resolutions and detail, and
viewing/transmitting them realtime,
• Painting into vector graphics with level of detail control - generat-
ing level of detail controlled painterly rendered images into scalable,
portable, application-agnostic form,
• Using focus maps for feature extraction - image database indexing by
important regions based on the presented relative focus map estima-
tion method.
The author has also worked on movie restoration problems, namely in
the restoration development works of the 1949 Ludas Matyi film (DIMORF
project). The author’s [15] work on painterly image and video trasnforma-
tion has 3 independent citations.
Thesis Groups 7
1. Thesis group: basic methods
I introduced new methods into the family of 2D automatic, stochastic,
non-photorealistic painterly rendering techniques for producing new
effects, for more effective and efficient realization, and I have shown
that these can also produce relevant improvement in quality, speed
and also in the encoding and compression of the generated images.
(a) Thesis: The use of binary and grayscale stroke templates make
possible to produce different types of representations and styles
of the models. Thus artificial paintings with individual styles can
be produced.
(b) Thesis: I introduced the so called interpolative layer filling color
morphology into the multilayer painting process and showed that
it can speed up the steps of the painterly rendering process and
also produces more compact encodings of these images.
(c) Thesis: In multilayer multiscale painterly rendering the key for
Thesis Groups 8
finding the optimum in quality/size/time is in the choice of the
right scales of the used strokes, which - as experiments showed -
usually means no more than three scales.
(d) Thesis: I proved that the painterly generated images of stroke
based rendering processes can be more efficiently encoded and
stored by coding the stroke stream representation that they have
been built of, than coding the painted raster images.
2. Thesis group: combining scale-space image features and painterly ren-
dering
I applied the results of scale-space approaches used for image struc-
ture analysis and feature extraction for improving automatic and cre-
ating new stroke based rendering methods, by introducing such new
techniques which make the generation and representation of painted
images more natural. I showed that automatic painterly rendering by
using scale-space structure information can be efficient alternatives of
classical methods.
(a) Thesis: I showed that the edge/ridge information extracted from
model images by scale-space structure analysis can be used to
create non-stochastic one layer painting and different painting
variations.
(b) Thesis: I showed that the combined edge and ridge information
extracted from the model images provides such an effective con-
trol for the automatic painterly rendering that is a more natural
Thesis Groups 9
image description and representation.
3. Thesis group: use of relative focus maps for region based control of
painting quality
I introduced an automatic method based on blind deconvolution, for
extracting relative focus maps from ordinary images and I proved its
practical applicability. I introduced an automatic non-photorealistic
painterly rendering method, which uses the model image’s relative
focus map to control the painterly detail on image areas.
(a) Thesis: I introduced and showed the usability of localized blind
deconvolution for extraction of relative focus maps from ordinary
images, for automatically extracting the relevant image areas.
(b) Thesis: I experimentally proved that deconvolution-based rel-
ative focus map estimation is well usable also on images with
various textures for extracting important foreground areas.
(c) Thesis: I proved the applicability of relative focus maps though
using it for focus based image indexing. Thus, I showed the possi-
bility of indexing image databases by relative focus information.
(d) Thesis: I showed that, by using the relative focus maps of the
model images, such an automatic painterly rendering method can
be constructed which can control the local level of detail of the
painting by the focus map information.
4. Thesis group: applications and extension to videos
Thesis Groups 10
I showed that storing and encoding the painterly rendered images as
a stroke series makes possible to display these images in different res-
olutions in real time without recoding. I applied automatic stroke
based painterly rendering to produce painterly renderings of videos
and showed that by combination with the appropriate motion de-
tection methods it is usable for generating painterly effects in image
sequences.
(a) Thesis: Stroke series based representation of painted images is
not just an effective storage method, but is also usable for display-
ing different resolution versions of the same rendered image with-
out re-transforming the painting in another resolution. I showed
the possibility of rendering the painted images into the portable,
application-agnostic SVG (scalable vector graphics) format. A
test application also shows that real time resolution scaling of
painted images is available.
(b) Thesis: I applied painterly rendering to consecutive video frames
to create animation-like effects, by extracting motion fields from
image sequences, using the motion fields to extract the areas that
are different from the previous frame and determine the strokes’
position and orientation information.
Publication List
[1] Kovacs, L., Sziranyi, T. “2D Multilayer Painterly Rendering with Automatic Focus Extraction,” WSCG 2006 Full Papers Proceedings, ISBN 80-86943- 03-8, pp. 141-145, Plzen, 2006.
[2] Kovacs, L., Sziranyi, T. “Painterly Rendering by Automatic Feature Extrac- tion,” HACIPPR 2005, pp. 287-295, Veszprem, Hungary, 2005.
[3] Kovacs, L., Sziranyi, T. “Image Indexing By Focus Map,” ACIVS 2005, Lecture Notes in Computer Science vol. 3708, pp. 300-307, Antwerp, 2005. IF:0.513
[4] Kovacs, L., Sziranyi, T. “Relative Focus Map Estimation Using Blind De- convolution,” Optics Letters, vol. 30, pp. 3021-3023, 2005. IF: 3.882
[5] Kovacs, L., and Sziranyi, T. “Painterly Rendering of Images and Real Paint- ings with SVG Support.” Third Hungarian Conference on Computer Graph- ics and Geometry, pp. 2-8, nov. 17-18, Budapest, 2005.
[6] Kovacs, L., Sziranyi, T. “Coding of Stroke-Based Animations,” WSCG 2004, Plzen, Checz Republic, ISBN:80-903100-6-0, pp. 81-84, 2004.
[7] Kovacs, L., Sziranyi, T. “Painterly Rendering Controlled by Multiscale Im- age Features,” SCCG 2004 (Spring Conference on Computer Graphics), Pub- lished by ACM, ISBN: 1-58113-914-4, Budmerice, Slovakia, pp.183-190, 2004.
[8] Kovacs, L., Sziranyi, T. “Efficient Coding of Stroke-Rendered Paintings,” 17th ICPR, Cambridge, UK, august 23-26, vol. 2, pp. 835-839, 2004.
Publication List 12
[9] Kovacs, L., Sziranyi, T. “Rendering and Coding of Still and Motion Picture by Stochastic Painting,” IV. KepAf konferencia, Miskolc-Tapolca, pp. 171- 178, 2004.
[10] Czuni, L., Hanis, A., Kovacs, L., Kranicz, B., Licsar, A., Sziranyi, T., Kas, I., Kovacs, Gy., Manno, S. “Digital Motion Picture Restoration System for Film Archives (DIMORF),” SMPTE Motion Imaging Journal, May/June 2004, pp. 170-178, 2004.
[11] Czuni, L., Csaszar, G., Hanis, A., Kovacs, L., Licsar, A., Sziranyi, T. “Semi Automatic Digital Motion Picture Restoration System with Learning Capa- bilities,” 17th ICPR, Workshop: Learning for Adaptable Visual Systems, CD Proceedings. 2004.
[12] Kovacs, L., Sziranyi, T. “Stroke-Based Painting and Coding of Image Se- quences,” 2. Magyar Grafika es Geometria Konferencia, pp. 142-150, BME, Budapest, 2003.
[13] Bolecz, M., Czuni, L., Gal, B., Hanis, A., Kovacs, L., Kranicz, B., Licsar, A., Sziranyi, T., Kas, I., Kovacs, Gy., Manno, S. “Digital Motion Picture Restoration System for Film Archives (DIMORF),” IBC2003, Amsterdam, 2003.
[14] Kovacs, L., Sziranyi, T. “Creating Video Animations Combining Stochastic Paintbrush Transformation and Motion Detection,”, III. KepAf konferencia, Domaszek, pp. 210-217, 2002.
[15] Kovacs, L., Sziranyi, T. “Creating Animations Combining Stochastic Paint- brush Transformation and Motion Detection,” 16th ICPR, Quebec City, Canada pp. 1090-1093, 2002.
[16] Kovacs, L., and Sziranyi, T. “Blind Deconvolution Based Automatic Fo- cus Map Extraction Method,” Hungarian Patent Office, submitted under P050164/2, november 2005.
[17] Kovacs, L., Sziranyi, T., “Focus Area Extraction by Blind Deconvolution for Defining Regions of Interest,” IEEE Transactions on Pattern Analysis and Machine Intelligence, revised version under review, 2006. IF: 4.352