Machine Learning Research

Submit a Manuscript

Publishing with us to make your research visible to the widest possible audience.

Propose a Special Issue

Building a community of authors and readers to discuss the latest research and develop new ideas.

Automatic Indexing of Digital Objects Through Learning from User Data

Digital data objects increasingly take the form of a non-textual nature, and the effective retrieval of these objects using their intrinsic contents largely depends on the underlying indexing mechanism. Since current multimedia objects are created with ever-increasing speed and ease, they often form the bulk of the data contents in large data repositories. In this study, we provide an effective automatic indexing mechanism based on learning reinforcement by systematically exploiting the big data obtained from different user interactions. Such human interaction with the search system is able to encode the human intelligence in assessing the relevance of a data object against user retrieval intentions and expectations. By methodically exploiting the big data and learning from such interactions, we establish an automatic indexing mechanism that allows multimedia data objects to be gradually indexed in the normal course of their usage. The proposed method is especially efficient for the search of multimedia data objects such as music, photographs and movies, where the use of straightforward string-matching algorithms are not applicable. The method also permits the index to respond to change in relation to user feedback, which at the same time avoids the system landing in a local optimum. Through the use of the proposed method, the accuracy of searching and retrieval of multimedia objects and documents may be significantly enhanced.

Autonomous Agent, Digital Data Objects, Index Generation, Multimedia Information Search, Probability Generating Function, Reinforcement Learning, Stochastic Modelling

Clement Leung, Yuanxi Li. (2023). Automatic Indexing of Digital Objects Through Learning from User Data. Machine Learning Research, 7(2), 18-23.

Copyright © 2022 Authors retain the copyright of this article.
This article is an open access article distributed under the Creative Commons Attribution License ( which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

1. B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey, “Maximum Entropy Inverse Reinforcement Learning,” Proc. Twenty-Third AAAI Conference on Artificial Intelligence (AAAI 08), vol. 8, pp. 1433-1438, 2008.
2. L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement learning: A survey” Journal of artificial intelligence research, vol. 4, pp. 237-285, 1996.
3. M. Kearns, and S. Singh, “Near-optimal reinforcement learning in polynomial time,” In Int. Conf. on Machine Learning, 1998.
4. H. Santana, G. Ramalho, V. Corruble, and B. Ratitch, “Multi-agent patrolling with reinforcement learning,” Proc. Third International Joint Conference on Autonomous Agents and Multiagent Systems, vol. 3, pp. 1122-1129, IEEE Computer Society, 2004.
5. R. I. Brafman, and M. Tennenholtz, “R-max-a general polynomial time algorithm for near-optimal reinforcement learning,” Journal of Machine Learning Research, vol. 3, pp. 213-231, 2002.
6. L. Panait, and S. Luke, “Cooperative multi-agent learning: The state of the art,” Autonomous agents and multi-agent systems, vol. 11, no. 3, pp. 387-434, 2005.
7. E. Ipek, O. Mutlu, J. F. Martínez, and R. Caruana, “Self-optimizing memory controllers: A reinforcement learning approach,” ACM SIGARCH Computer Architecture News, vol. 36, no. 3, IEEE Computer Society, 2008.
8. L. Busoniu, R. Babuska, and B. De Schutter, “A comprehensive survey of multiagent reinforcement learning,” IEEE Transactions on Systems, Man, And Cybernetics-Part C: Applications and Reviews, vol. 38, no. 2 2, 2008.
9. S. V. Albrecht, and P. Stone, “Autonomous agents modelling other agents: A comprehensive survey and open problems,” Artificial Intelligence 258, pp. 66-95, 2018.
10. A. Tampuu, T. Matiisen, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, and R. Vicente, “Multiagent cooperation and competition with deep reinforcement learning,” PloS one, vol. 12, no. 4: e0172395, 2017.
11. A. W. Moore, and C. G. Atkeson, “Prioritized sweeping: Reinforcement learning with less data and less time,” Machine learning, vol. 13, no. 1, pp. 103-130, 1993.
12. E. Brochu, V. M. Cora, and N. De Freitas, “A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning,”, unpublished.
13. Q. Wei, F. L. Lewis, Q. Sun, P. Yan, and R. Song, “Discrete-time deterministic Q-learning: A novel convergence analysis,” IEEE transactions on cybernetics, vol. 47, no. 5, pp. 1224-1237, 2017.
14. C. J. Watkins, and P. Dayan, “Q-learning,” Machine learning 8.3-4 pp. 279-292, 1992.
15. H. Van Hasselt, and M. A. Wiering, “Using continuous action spaces to solve discrete problems,” Proc. International Joint Conference on Neural Networks (IJCNN 09), pp. 1149-1156. IEEE, 2009.
16. N. Hansen, S. D. Müller, and P. Koumoutsakos, “Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES),” Evolutionary computation, vol. 11, no. 1 pp. 1-18, 2003.
17. W. Feller, An Introduction to Probability Theory and its Applications, vol. 1, 3rd Edition, Wiley & Sons, 1968.
18. S. Ross, A First Course in Probability, 9th Edition, Pearson, 2014.
19. R. Gupta, M. Khomami Abadi, J. A. Cárdenes Cabré, F. Morreale, T. H. Falk, and N. Sebe, “A quality adaptive multimodal affect recognition system for user-centric multimedia indexing”. In: Proceedings of the 2016 ACM on international conference on multimedia retrieval. ACM, p. 317-320, 2016.
20. Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A.,“Deep reinforcement learning: A brief survey,” IEEE Signal Processing Magazine, 34 (6), 26-38, 2017.
21. X. Yao, J. Du, N. Zhou, and C. Chen, “Microblog Search Based on Deep Reinforcement Learning,” In Proceedings of 2018 Chinese Intelligent Systems Conference (pp. 23-32). Springer, Singapore, 2019.
22. Y. C. Wu, T. H. Lin, Y. D. Chen, H. Y Lee, and L. S. Lee, “Interactive spoken content retrieval by deep reinforcement learning”. arXiv preprint arXiv: 1609.05234, 2016.
23. S. Lan, R. Panda, Q. Zhu, and A. K. Roy-Chowdhury, “FFNet: Video fast-forwarding via reinforcement learning”, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6771-6780, 2018.
24. R. Hong, Y. Yang, M. Wang, and X. S. Hua, “Learning visual semantic relationships for efficient visual retrieval”, IEEE Transactions on Big Data, 1 (4), 152-161, 2015.
25. R. Yan, A. Hauptmann, and R. Jin, “Multimedia search with pseudo-relevance feedback. In International Conference on Image and Video Retrieval” (pp. 238-247). Springer, Berlin, Heidelberg, 2003.
26. J. Deng, C. H. C. Leung: Dynamic Time Warping for Music Retrieval Using Time Series Modeling of Musical Emotions. IEEE Transactions on Affective Computing, Vol. 6, No. 2, pp. 137-151 (2015).
27. H. L. Zhang, C. H. C. Leung, G. K. Raikundalia: Topological analysis of AOCD-based agent networks and experimental results. Journal of Computer and System Sciences, pp. 255–278, (2008).
28. Azzam, I., Leung, C. H. C., Horwood, J.: Implicit concept-based image indexing and retrieval. In Proceedings of the IEEE International Conference on Multi-media Modeling, pp. 354-359, Brisbane, Australia (2004).
29. H. Zhang, C. H. C. Leung and G. K. Raikundalia: Classification of intelligent agent network topologies and a new topological description language for agent networks. In Proceedings of the 4th International Conference on Intelligent Information Processing, Adelaide, Australia, pp. 21-31 (2006).
30. N. L. J. Kuang, C. H. C. Leung, and V. Sung: Stochastic Reinforcement Learning. In Proc. IEEE International Conference on Artificial Intelligence and Knowledge Engineering, pp. 244-248, California, USA (2018).
31. N. L. J. Kuang, and C. H. C. Leung: Performance Dynamics and Termination Errors in Reinforcement Learning – A Unifying Perspective. In Proc. IEEE International Conference on Artificial Intelligence and Knowledge Engineering, pp. 129-133, California, USA (2018).
32. N. L. J. Kuang, C. H. C. Leung, Analysis of Evolutionary Behavior in Self-Learning Media Search Engines, in Proceedings of the IEEE International Conference on Big Data, Los Angeles, USA, (2019).
33. N. L. J. Kuang, C. H. C. Leung, Performance Effectiveness of Multimedia Information Search Using the Epsilon-Greedy Algorithm, in Proceedings of the 2019 IEEE International Conference on Machine Learning and Applications, pp. 929-936, Florida, USA (2019).
34. E. Thorpe, Portfolio Choice and the Kelly Criterion, Academic Press, 1975.