The Track

The field of 3D shape retrieval has achieved an important level of maturity, which can be evidenced in the number of techniques and derived applications so far. Also, many efforts have been done for the evaluation of these techniques, with SHREC competitions gaining more and more interest from the community in past editions. Still, in our opinion, there is one aspect that deserves careful attention: scalability. Currently, there are large-scale 3D repositories such as Google 3D Warehouse (more than 1,000,000 models) or TurboSquid (more than 300,000 models), whose presence suggests that in the near future 3D searching capabilities will require to pay attention on efficiency and robustness. Obviously, scalability is associated to large repositories and it is exactly the aspect this track intends to cover.

Our track presents a large-scale scenario for 3D shape retrieval. This task is built upon a non-rigid shape retrieval task which has been shown to be succesful in recent years in previous SHREC editions [1][2]. Indeed, our decision of building a large-scale dataset upon a non-rigid retrieval taks relies on the observation that there is a good number of methods and the achieved effectiveness is high. We believe that this new track will allow us to go one step further in the evaluation of existing algorithms and their robustness and efficiency

The dataset

Our base dataset is the publicly available collection from Sumner[3]. It contains 229 non-rigid shapes categorized in 9 classes. Subsequently, we populate our dataset including unclassified objects from several publicly available shape collections such as Princeton Shape Benchmark [4], Konstanz database[5], SHREC' 09 Generic dataset [6], and SHREC' 14 Large Scale Comprehensive Retrieval [7]. After including models from the previous datasets, we generate a large set of shapes using the method proposed in [8] to generate simulated range images using the SHREC' 14 Large Scale dataset as target models. After obtaining a first large-scale set of shapes, we applied a careful post-processing step in order to repair non-manifold objects and merge objects with more than 1 connected component. Our final dataset only contains manifold objects with one connected component. This pre-processing step will guarantee that most of the current approaches work with our dataset.

In total, our dataset contains 96,487 models; which, compared to the largest dataset known (SHREC'14 Large Scale Comprehensive Retrieval contains 8,987 models), is one order of magnitude larger. The names of the objects have the format LX.off, where X is a number in the range 0 to 96486. For this track, participants must use the first 229 objects (from Sumner dataset) as queries; that is, objects S0.off to S228.off.

Downloads

The complete dataset has been splitted for easy download. The evaluation code and groundtruth can be obtained from this package. Please read the README file for further information. If the information or evaluation code is used, the track paper must be cited as
  • Sipiran, I.; Bustos, B.; Schreck, T.; Bronstein, A. M.; Bronstein, M.; Castellani, U.; Choi, S.; Lai, L.; Li, H.; Litman, R.; Sun, L.: Scalability of Non-Rigid 3D Shape Retrieval. Proc. Eurographics Workshop on 3D Object Retrieval (3DOR). Pages 121-128. 2015.[online article]

Results submission

Participants should submit a distance matrix for each run. Up to 5 matrices may be submitted corresponding to different algorithms or a different parameter setting. The matrix distance must be stored in a file (with a white space as separator) and its size must be of 229 x 96,487. The i-th row of the matrix corresponds with the distances from the i-th query (L{i}.off) to every model in the dataset. We will consider the same order imposed by the number in the name file for both target and query set. So entry A(i,j) corresponds to the distance from L{i}.off to L{j}.off.

In addition, participants must report the following information:

  • System specification: CPU (model, speed in GHz, number of CPU's, RAM per CPU in GB). In case participants use GPU, the required information is model, speed in MHz, memory in GB and number of GPU's.
  • Processing time in seconds: In order to properly evaluate the several stages of a 3D retrieval system, participants should make the difference between offline processing (for example, dictionary computation in BoF approaches or other preprocessing steps in the dataset) and online processing (the time to compute distances between a query object and the entire dataset). Please, provide the average query time for the 229 query objects.

Effectiveness evaluation

We will use four measures to compute the effectiveness of algorithms:

  • Mean Average Precision (MAP): Given a query, its average precision is the average of all precision values computed in each relevant object in the retrieved list. Given several queries, the mean average precision is the mean of average precision of each query.
  • Nearest Neighbor (NN): Given a query, it is the precision at the first object of the retrieved list.
  • First Tier (FT): Given a query, it is the precision when C objects have been retrieved, where C is the number of relevant objects to the query.
  • Second Tier (ST): Given a query, it is the precision when 2*C objects have been retrieved, where C is the number of relevant objects to the query.

To Participants

People interested in participating in this track must register by sending an email to sipiran@dbvis.inf.uni-konstanz.de. The registration will help to keep track of the contest.

Timeline

  • December 29th, 2014: Track kick-off
  • January 19th, 2015: Deadline for track registration
  • February 14th, 2015: Deadline for submission of results
  • February 28th, 2015: Submission of track paper

Contact

For additional information, please do not hesitate to contact Ivan Sipiran (sipiran@dbvis.inf.uni-konstanz.de).

Organizers

  • Ivan Sipiran, Department of Computer and Information Sciences, University of Konstanz.
  • Benjamin Bustos, Department of Computer Science, University of Chile.
  • Tobias Schreck, Department of Computer and Information Sciences, Univertsity of Konstanz.

References

  1. Z. Lian, A. Godil, B. Bustos, M. Daoudi, J. Hermans, S. Kawamura, Y. Kurita, G. Lavoue, H.V. Nguyen, R. Ohbuchi, Y. Ohkita, Y. Ohishi, F. Porikli, M. Reuter, I. Sipiran, D. Smeets, P. Suetens, H. Tabia, and D. Vandermeulen: SHREC'11 Track: Shape Retrieval on Non-rigid 3D Watertight Meshes , In: H. Laga and T. Schreck, A. Ferreira, A. Godil, I. Pratikakis, R. Veltkamp (eds.), Proceedings of the Eurographics/ACM SIGGRAPH Symposium on 3D Object Retrieval, 2011.
  2. Lian, Z., Godil, A., Bustos, B., Daoudi, M., Hermans, J., Kawamura, S., Kurita, Y., Lavoue, G., Van Nguyen, H., Ohbuchi, R., Ohkita, Y., Ohishi, Y., Porikli, F., Reuter, M., Sipiran, I., Smeets, D., Suetens, P., Tabia, H., Vandermeulen, D.: A comparison of methods for non-rigid 3D shape retrieval. Pattern Recognition, 46(1):449-461.
  3. Robert W. Sumner, Jovan Popovic. Deformation Transfer for Triangle Meshes. ACM Transactions on Graphics. 23, 3. August 2004.
  4. Philip Shilane, Patrick Min, Michael Kazhdan, and Thomas Funkhouser The Princeton Shape Benchmark. Shape Modeling International, Genova, Italy, June 2004.
  5. Vranic Dejan. 3D model retrieval. PhD thesis, University of Leipzig, 2004.
  6. Afzal Godil, Helin Dutagaci, Ceyhun Akgul, Apostolos Axenopoulos, Benjamin Bustos, Mohamed Chaouch, Petros Daras, Takahiko Furuya, Sebastian Kreft, Zhouhui Lian, Thibault Napoleon, Athanasios Mademlis, Ryutarou Ohbuchi, Paul Rosin, Bulent Sankur,Tobias Schreck, Xianfang Sun, Masaki Tezuka, Anne Verroust-Blondet, Michael Walter, and Yucel Yemez. SHREC'09 track: Generic shape retrieval. In Proc. Eurographics 2009 Workshop on 3D Object Retrieval (3DOR'09), pages 61-68. Eurographics Association.
  7. Bo Li, Yijuan Lu, Chunyuan Li, Afzal Godil, Tobias Schreck, Masaki Aono, Qiang Chen, Nihad Karim Chowdhury, Bin Fang, Takahiko Furuya, Henry Johan, Ryuichi Kosaka, Hitoshi Koyanagi, Ryutarou Ohbuchi, Atsushi Tatsuma. SHREC' 14 Track: Large Scale Comprehensive 3D Shape Retrieval. Eurographics Workshop on 3D Object Retrieval 2014 (3DOR 2014): 131-140, 2014.
  8. Sipiran, I., Meruane, R., Bustos, B., Schreck, T., Li, B., Lu, Y., Johan, H.: SHREC'13 Track: Large-Scale Partial Shape Retrieval Using Simulated Range Images. Proc. Eurographics Workshop on 3D Object Retrieval (3DOR'13), pages 81-88. Eurographics Association. 2013.