We introduce an intuitive gesture-based interaction technique for creating and manipulating simple three-dimensional (3D) shapes. Specifically, the developed interface utilizes low-cost depth camera to capture user's hand gesture as the input, maps different gestures to system commands and generates 3D models from midair 3D sketches (as opposed to traditional two-dimensional (2D) sketches). Our primary contribution is in the development of an intuitive gesture-based interface that enables novice users to rapidly construct conceptual 3D models. Our development extends current works by proposing both design and technical solutions to the challenges of the gestural modeling interface for conceptual 3D shapes. The preliminary user study results suggest that the developed framework is intuitive to use and able to create a variety of 3D conceptual models.
Issue Section:
Research Papers
References
1.
Robertson
, B. F.
, and Radcliffe
, D. F.
, 2009
, “Impact of Cad Tools on Creative Problem Solving in Engineering Design
,” Comput.-Aided Des.
, 41
(3
), pp. 136
–146
.2.
Sharma
, A.
, Madhvanath
, S.
, Shekhawat
, A.
, and Billinghurst
, M.
, 2011
, “Mozart: A Multimodal Interface for Conceptual 3D Modeling
,” 13th International Conference on Multimodal Interfaces
(ICMI
), Alicante, Spain, Nov. 14–18, pp. 307
–310
.https://www.researchgate.net/publication/221052364_MozArt_A_Multimodal_Interface_for_Conceptual_3D_Modeling3.
Rubine
, D.
, 1991
, “Specifying Gestures by Example
,” 18th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '91)
, Providence, RI, Apr. 27–30, pp. 329
–337
.4.
Takahashi
, T.
, and Kishino
, F.
, 1992
, “A Hand Gesture Recognition Method and Its Application
,” Syst. Comput. Jpn.
, 23
(3
), pp. 38
–48
.5.
Radhakrishnan
, S.
, Lin
, Y.
, Zeid
, I.
, and Kamarthi
, S.
, 2013
, “Finger-Based Multitouch Interface for Performing 3D CAD Operations
,” Int. J. Human-Comput. Stud.
, 71
(3
), pp. 261
–275
.6.
Seth
, A.
, Su
, H.-J.
, and Vance
, J. M.
, 2006
, “Sharp: A System for Haptic Assembly and Realistic Prototyping
,” ASME
Paper No. DETC2006-99476
. 7.
Edit
, V.
, 2008
, “Using Hand Motions in Conceptual Shape Design
,” Product Engineering
, Springer, Berlin, pp. 367–382.8.
Segen
, J.
, and Kumar
, S.
, 1998
, “Gesture Vr: Vision-Based 3D Hand Interace for Spatial Interaction
,” Sixth ACM International Conference on Multimedia
, Bristol, UK, Sept. 13–16, pp. 455
–464
.9.
Wang
, R.
, Paris
, S.
, and Popović
, J.
, 2011
, “6D Hands: Markerless Hand-Tracking for Computer Aided Design
,” 24th Annual ACM Symposium on User Interface Software and Technology
, Santa Barbara, CA, Oct. 16–19, pp. 549
–558
.https://homes.cs.washington.edu/~jovan/papers/wang-2011-6dh.pdf10.
Bai
, H.
, Gao
, L.
, El-Sana
, J.
, and Billinghurst
, M.
, “Free-Hand Interaction for Handheld Augmented Reality Using an RGB-Depth Camera
,” SIGGRAPH Asia 2013 Symposium on Mobile Graphics and Interactive Applications
, Hong Kong, Nov. 19–22, Paper No. 22.11.
Stannus
, S.
, Fu
, W.-T.
, and Lucieer
, A.
, 2014
, “Natural 7DOF Input for 3D Navigation
,” 26th Australian Computer-Human Interaction Conference on Designing Futures: The Future of Design
, Sydney, Australia, Dec. 2–5, pp. 216
–219
.12.
Song
, J.
, Cho
, S.
, Baek
, S.-Y.
, Lee
, K.
, and Bang
, H.
, 2014
, “Gafinc: Gaze and Finger Control Interface for 3D Model Manipulation in Cad Application
,” Comput.-Aided Des.
, 46
, pp. 239
–245
.13.
Vinayak
,Murugappan
, S.
, Piya
,C.
, and Ramani
,K.
, 2013
, “Handy-Potter: Rapid Exploration of Rotationally Symmetric Shapes Through Natural Hand Motions
,” ASME J. Comput. Inf. Sci. Eng.
, 13
(2
), p. 021008
.14.
Han
, Y. C.
, and Han
, B-J.
, 2014
, “Virtual Pottery: A Virtual 3D Audiovisual Interface Using Natural Hand Motions
,” Multimedia Tools Appl.
, 73
(2
), pp. 917
–933
.15.
Vinayak
, K.
, and Ramani
, 2015
, “A Gesture-Free Geometric Approach for Mid-Air Expression of Design Intent in 3D Virtual Pottery
,” Comput.-Aided Des.
, 69
, pp. 11
–24
.16.
Colombo
, G.
, Facoetti
, G.
, Rizzi
, C.
, and Vitali
, A.
, 2016
, “Mixed Reality to Design Lower Limb Prosthesis
,” Comput.-Aided Des. Appl.
, 13
(6
), pp. 799
–807
.17.
Fuge
, M.
, Yumer
, M. E.
, Orbay
, G.
, and Kara
, L. B.
, 2012
, “Conceptual Design and Modification of Freeform Surfaces Using Dual Shape Representations in Augmented Reality Environments
,” Comput.-Aided Des.
, 44
(10
), pp. 1020
–1032
.18.
Jinmiao
, H.
, Prakhar
, J.
, and Rahul
, R.
, 2018
, “Gesture-Based System for Next Generation Natural and Intuitive Interfaces
,” Ai Edam
, 15
, p. 1
.19.
Olsen
, L.
, Samavati
, F. F.
, Sousa
, M. C.
, and Jorge
, J. A.
, 2009
, “Sketch-Based Modeling: A Survey
,” Comput. Graph.
, 33
(1
), pp. 85
–103
.20.
Takeo
, I.
, Satoshi
, M.
, and Hidehiko
, T.
, 2007
, “Teddy: A Sketching Interface for 3D Freeform Design
,” ACM SIGGRAPH 2007 Courses
, ACM, New York, p. 21
21.
Nealen
, A.
, Igarashi
, T.
, Sorkine
, O.
, and Alexa
, M.
, 2007
, “Fibermesh: Designing Freeform Surfaces With 3D Curves
,” ACM Trans. Graph. (TOG)
, 26
(3
), p. 41
.22.
Karpenko
, O.
, Hughes
, J. F.
, and Raskar
, R.
, 2002
, “Free-Form Sketching With Variational Implicit Surfaces
,” Comput. Graph. Forum
, 21
(3
), pp. 585
–594
.23.
Tai
, C.-L.
, Zhang
, H.
, and Fong
, J. C.-K.
, 2004
, “Prototype Modeling From Sketched Silhouettes Based on Convolution Surfaces
,” Comput. Graph. Forum
, 23
(1
), pp. 71
–83
.24.
Alexe
, A.
, Gaildrat
, V.
, and Barthe
, L.
, 2004
, “Interactive Modelling From Sketches Using Spherical Implicit Functions
,” Third International Conference on Computer Graphics, Virtual Reality, Visualisation and Interaction in Africa
, Stellenbosch, South Africa, Nov. 3–5, pp. 25
–34
.25.
Schmidt
, R.
, Wyvill
, B.
, Sousa
, M. C.
, and Jorge
, J. A.
, 2006
, “Shapeshop: Sketch-Based Solid Modeling With Blobtrees
,” ACM SIGGRAPH 2006 Courses, ACM, New York, p. 14
26.
Gharib
, I.
, and Qin
, S.
, 2013
, “Integration of Sketch-Based Conceptual Design and Commercial Cad Systems for Manufacturing
,” Int. J. Adv. Manuf. Technol.
, 68
(9–12
), pp. 2669
–2681
.27.
Zeleznik
, R. C.
, Herndon
, K. P.
, and Hughes
, J. F.
, 2007
, “ Sketch: An Interface for Sketching 3D Scenes
,” ACM SIGGRAPH 2007 Courses, ACM, New York, p. 19
.28.
Jatupoj
,P.
, 2005, “Sketchboard: The Simple 3D Modeling From Architectural Sketch Recognition
,” Proceedings of the 10th International Conference on Computer Aided Architectural Design Research in Asia
, CAADRIA, New Delhi, India, pp. 3–22.29.
Qin
, S. F.
, Wright
, D. K.
, and Jordanov
, I. N.
, 2000
, “From On-Line Sketching to 2D and 3D Geometry: A System Based on Fuzzy Knowledge
,” Comput.-Aided Des.
, 32
(14
), pp. 851
–866
.30.
Chaudhuri
, S.
, Kalogerakis
, E.
, Guibas
, L.
, and Koltun
, V.
, 2011
, “Probabilistic Reasoning for Assembly-Based 3D Modeling
,” ACM Trans. Graph. (TOG)
, 30
(4
), p. 35
.31.
Kalogerakis
, E.
, Chaudhuri
, S.
, Koller
, D.
, and Koltun
, V.
, 2012
, “A Probabilistic Model for Component-Based Shape Synthesis
,” ACM Trans. Graph.
, 31
(4
), p. 55.32.
Chaudhuri
, S.
, Kalogerakis
, E.
, Giguere
, S.
, and Funkhouser
, T.
, 2013
, “Attribit: Content Creation With Semantic Attributes
,” 26th Annual ACM Symposium on User Interface Software and Technology
, St. Andrews, UK, Oct. 8–11, pp. 193
–202
.33.
Jaiswal
, P.
, Huang
, J.
, and Rai
, R.
, 2016
, “Assembly-Based Conceptual 3D Modeling With Unlabeled Components Using Probabilistic Factor Graph
,” Comput.-Aided Des.
, 74
, pp. 45
–54
.34.
Sung
, M.
, Su
, H.
, Kim
, V. G.
, Chaudhuri
, S.
, and Guibas
, L.
, 2017
, “Complement Me: Weakly-Supervised Component Suggestions for 3D Modeling
,” ACM Trans. Graph. (TOG)
, 36
(6
), p. 226
.35.
Krupka
, E.
, Karmon
, K.
, Bloom
, N.
, Freedman
, D.
, Gurvich
, I.
, Hurvitz
, A.
, Leichter
, I.
, Smolin
, Y.
, Tzairi
, Y.
, Vinnikov
, A.
, and Bar-Hillel, A., 2017
, “Toward Realistic Hands Gesture Interface: Keeping It Simple for Developers and Machines
,” CHI Conference on Human Factors in Computing Systems
, Denver, CO, May 6–11, pp. 1887
–1898
.36.
Vitali
, A.
, and Rizzi
, C.
, 2017
, “A Virtual Environment to Emulate Tailor's Work
,” Comput.-Aided Des. Appl.
, 14
(5
), pp. 671
–679
.37.
Microsoft
, 2018
, “Gestures Mixed Reality
,” Microsoft, Redmond, WA, accessed Aug. 8, 2018, https://docs.microsoft.com/en-us/windows/mixed-reality/gestures38.
The CGAL Project,
2018
, CGAL User and Reference Manual
, 4.12 ed., CGAL Editorial Board
.39.
Chew
, L. P.
, 1989
, “Constrained Delaunay Triangulations
,” Algorithmica
, 4
(1–4
), pp. 97
–108
.40.
Mapari
, R. B.
, and Kharat
, G.
, 2016
, “American Static Signs Recognition Using Leap Motion Sensor
,” Second International Conference on Information and Communication Technology for Competitive Strategies
, Udaipur, India, Mar. 4–5, p. 67
.41.
Marin
, G.
, Dominio
, F.
, and Zanuttigh
, P.
, 2014
, “Hand Gesture Recognition With Leap Motion and Kinect Devices
,” IEEE International Conference on Image Processing
(ICIP
), Paris, France, Oct. 27–30, pp. 1565
–1569
.42.
Zeiler
, M. D.
, and Fergus
, R.
, 2014
, “Visualizing and Understanding Convolutional Networks
,” European Conference on Computer Vision
, Zurich, Switzerland, Sept. 6–12, pp. 818
–833
.https://cs.nyu.edu/~fergus/papers/zeilerECCV2014.pdf43.
Simonyan
, K.
, and Zisserman
, A.
, 2014
, “Very Deep Convolutional Networks for Large-Scale Image Recognition
,” arXiv Preprint: 1409:1556
.https://arxiv.org/abs/1409.155644.
Russakovsky
, O.
, Deng
, J.
, Su
, H.
, Krause
, J.
, Satheesh
, S.
, Ma
, S.
, Huang
, Z.
, Karpathy
, A.
, Khosla
, A.
, Bernstein
, M.
, Berg
, A. C.
, and Fei-Fei
, L.
, 2015
, “ImageNet Large Scale Visual Recognition Challenge
,” Int. J. Comput. Vision (IJCV)
, 115
(3
), pp. 211
–252
.45.
46.
Dalal
, N.
, and Triggs
, B.
, 2005
, “Histograms of Oriented Gradients for Human Detection
,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition
(CVPR
), San Diego, CA, June 20–25, pp. 886
–893
.47.
Hammond
, T.
, and Davis
, R.
, 2005
, “Ladder, a Sketching Language for User Interface Developers
,” Comput. Graph.
, 29
(4
), pp. 518
–532
.48.
Kara
, L. B.
, and Stahovich
, T. F.
, 2005
, “An Image-Based, Trainable Symbol Recognizer for Hand-Drawn Sketches
,” Comput. Graph.
, 29
(4
), pp. 501
–517
.49.
Rabiner
, L. R.
, 1990
, “Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition
,” Readings in Speech Recognition
, Morgan Kaufmann Publishers Inc
., San Francisco, CA
, pp. 267
–296
50.
Viterbi
, A.
, 1967
, “Error Bounds for Convolutional Codes and an Asymptotically Optimum Decoding Algorithm
,” IEEE Transactions Inf. Theory
, 13
(2
), pp. 260
–269
.51.
Tsang
, S.
, Balakrishnan
, R.
, Singh
, K.
, and Ranjan
, A.
, 2004
, “A Suggestive Interface for Image Guided 3D Sketching
,” SIGCHI Conference on Human Factors in Computing Systems
, Vienna, Austria, Apr. 24–29, pp. 591
–598
.52.
Wigdor
, D.
, and Wixon
, D.
, 2011
, Brave Nui World: Designing Natural User Interfaces for Touch and Gesture
, Elsevier
, Burlington, MA.53.
Nielsen
, M.
, Störring
, M.
, Moeslund
, T. B.
, and Granum
, E.
, 2003
, “A Procedure for Developing Intuitive and Ergonomic Gesture Interfaces for HCI
,” International Gesture Workshop, Genoa, Italy, Apr. 15–17.54.
Thakur
, A.
, and Rai
, R.
, 2015
, “User Study of Hand Gestures for Gesture Based 3D CAD Modeling
,” ASME
Paper No. DETC2015-46086
. 55.
Mansutti
, A.
, Rodriguez
, M. C.
, Bordegoni
, M.
, and Cugini
, U.
, 2017
, Tactile Display for Virtual 3D Shape Rendering
, Springer
, Cham, Switzerland.Copyright © 2018 by ASME
You do not currently have access to this content.
