International Journal of Control, Automation, and Systems

Institute of Control, Robotics and Systems (제어로봇시스템학회)
권호 표지

A Motivation-Based Action-Selection-Mechanism Involving Reinforcement Learning

  • Lee, Sang-Hoon (College of Information and Communications, Hanyang University) ;
  • Suh, Il-Hong (College of Information and Communications, Hanyang University) ;
  • Kwon, Woo-Young (College of Information and Communications, Hanyang University)
  • Published : 2008.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

An action-selection-mechanism(ASM) has been proposed to work as a fully connected finite state machine to deal with sequential behaviors as well as to allow a state in the task program to migrate to any state in the task, in which a primitive node in association with a state and its transitional conditions can be easily inserted/deleted. Also, such a primitive node can be learned by a shortest path-finding-based reinforcement learning technique. Specifically, we define a behavioral motivation as having state-dependent value as a primitive node for action selection, and then sequentially construct a network of behavioral motivations in such a way that the value of a parent node is allowed to flow into a child node by a releasing mechanism. A vertical path in a network represents a behavioral sequence. Here, such a tree for our proposed ASM can be newly generated and/or updated whenever a new behavior sequence is learned. To show the validity of our proposed ASM, experimental results of a mobile robot performing the task of pushing- a- box-in to- a-goal(PBIG) will be illustrated.

Keywords

References

  1. A. Guillot and J.-A. Meyer, "Computer simulations of adaptive behavior in animals," Proc. Computer Animation '94, pp. 122-131, May 25-28, 1994
  2. S. W. Wilson, "The animat path to AI," Proc. of the First International Conference on Simulation of Adaptive Behavior on From Animals To Animats, Cambridge, MA, USA, MIT Press, 1990
  3. P. Maes, "Modeling adaptive autonomous agents," Artificial Life, vol. 1, no. 1-2, pp. 135-162, 1994 https://doi.org/10.1162/artl.1993.1.1_2.135
  4. T. Tyrrell, Computational Mechanisms for Action Selection, Ph.D. Dissertation, University of Edinburg, 1993
  5. M. J. Huber, S. Kumar, and D. McGee, "Toward a suite of performatives based upon joint intention theory," Lecture Notes in Computer Science, vol. 3396, pp. 226-241, 2005
  6. J. Rosenblatt and D. Payton, "A fine-grained alternative to the subsumption architecture for mobile robot control," Proc. of International Joint Conference on Neural Networks IJCNN, pp. 317-323, 18-22 June 1989
  7. E. Spier and D. McFarl, "A finer-grained motivational model of behavior sequencing," Proc. of the Fourth International Conference on Simulation of Adaptive Behavior, From Animals to Animats 4, 1997
  8. P. Maes, "A bottom-up mechanism for behavior selection in an artificial creature," Proc. of the First International Conference on Simulation of Adaptive Behavior on From Animals To Animats, Cambridge, MA, USA, MIT Press, pp. 238-246, 1990
  9. B. M. Blumberg, P. M. Todd, and P. Maes, "No bad dogs: Ethological lessons for learning in hamsterdam," Proc. of the Fourth International Conference on the Simulation of Adaptive Behavior, From Animals to Animats, 1996
  10. Y. Hoshino, T. Takagi, U. Di Profio, and M. Fujita, "Behavior description and control using behavior module for personal robot," Proc. of IEEE International Conference on Robotics and Automation, vol. 4, pp. 4165-4171, Apr 26-May 1, 2004
  11. T. Sawada, T. Takagi, and M. Fujita, "Behavior selection and motion modulation in emotionally grounded architecture for qrio sdr-4xii," Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems, vol. 3, pp. 2514-2519, Sept. 28-Oct. 2, 2004
  12. R. Brooks, "A robust layered control system for a mobile robot," IEEE Trans. on Robotics and Automation, vol. 2, no. 1, pp. 14-23, March 1986 https://doi.org/10.1109/JRA.1986.1087032
  13. E. de Sevin and D. Thalmann, "A motivational model of action selection for virtual humans," Proc. of the Computer Graphics International, Washington, DC, USA, IEEE Computer Society, pp. 213-220, 2005
  14. M. Scheutz and V. Andronache, "Architectural mechanisms for dynamic changes of behavior selection strategies in behavior-based systems," IEEE Trans. on Systems, Man and Cybernetics, Part B, vol. 34, no. 6, pp. 2377-2395, Dec. 2004 https://doi.org/10.1109/TSMCB.2004.837309
  15. C. M. Witkowski, Schemes for Learning and Behavior: A New Expectancy Model, Ph.D. Dissertation, University of London, 1997
  16. M. Humphrys, Action Selection Methods Using Reinforcement Learning, Ph.D. Dissertation, University of Cambridge, 1996
  17. S. Lee and I. H. Suh, "A programming framework supporting an ethology-based behavior control architecture," Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4138-4144, Oct. 2006
  18. E. Hogewoning, J. Broekens, J. Eggermont, and E. G. P. Bovenkamp, "Strategies for affectcontrolled action-selection in Soar-RL," Lecture Notes in Computer Science, vol. 4528, pp. 501-510, 2007
  19. R. Ameur and J.-C. Heudin, "Interactive intelligent agent architecture," Proc. WI-IAT Workshops Web Intelligence and International Agent Technology Workshops IEEE/WIC/ACM International Conference on, pp. 331-334, Dec. 2006
  20. M. Azar, M. Ahmadabadi, A. Farahmand, and B. Araabi, "Learning to coordinate behaviors in soft behavior-based systems using reinforcement learning," Proc. of International Joint Conference on Neural Networks, pp. 241-248, 2006
  21. N. Goerke and T. Henne, "Learning hierarchical action selection for an autonomous robot," Proc. of International Joint Conference on Neural Network, pp. 4958-4965, 16-21 July 2006
  22. J. Qiao, Z. Hou, and X. Ruan, "Q-learning based on neural network in learning action selection of mobile robot," Proc. of IEEE International Conference on Automation and Logistics, pp. 263-267, 18-21 Aug. 2007
  23. Z. Shen, C. Miao, Y. Miao, X. Tao, and R. Gay, "A goal-oriented approach to goal selection and action selection," Proc. of IEEE International Conference on Fuzzy Systems, pp. 114-121, 2006
  24. J. Jaafar, E. McKenzie, and A. Smaill, "A fuzzy action selection method for virtual agent navigation in unknown virtual environments," Proc. of IEEE International Fuzzy Systems Conference, pp. 1-6, 23-26 July 2007
  25. P. Pirjanian, "Behavior coordination mechanisms-state-of-the-art," Tech-report IRIS-99-375, Institute for Robotics and Intelligent Systems, University of Southern California, Tech. Rep., 1999
  26. J. H. Connell, "A behavior-based arm controller," IEEE Trans. on Robotics and Automation, vol. 5, no. 6, pp. 784-791, Dec. 1989 https://doi.org/10.1109/70.88099
  27. I. H. Suh, S. Lee, W. Y. Kwon, and Y.-J. Cho, "Learning of action patterns and reactive behavior plans via a novel two-layered ethology-based action selection mechanism," Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1799-1805, 2-6 Aug. 2005
  28. H. S. Ahn and J. Y. Choi, "Emotional behavior decision model based on linear dynamic systems for intelligent service robots," Proc. of the 16th IEEE International Symposium on Robot and Human interactive Communication, pp. 786-791, 26-29 Aug. 2007
  29. J. Bryson, "Hierarchy and sequence vs. full parallelism in action selection," Proc. of the Sixth International Conference From Animals to Animats 6, R. P. Jean-Arcady Meyer, Ed. MIT Press, 2000
  30. C. L. Hull, Principle of Behavior, Appleton-Century-Crofts, New York, 1943
  31. W. Kwon, I. H. Suh, S. Lee, and Y.-J. Cho, "Fast reinforcement learning using stochastic shortest paths for a mobile robot," Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 82-87, Oct. 29-Nov. 2 2007
  32. M. Kim and H. Kim, "Behavior coordination mechanism for intelligent home," Proc. of the 5th IEEE/ACIS International Conference on Computer and Information Science, pp. 452-457, 10-12 July 2006
  33. R. Arkin, M. Fujita, T. Takagi, and R. Hasegawa, "Ethological modeling and architecture for an entertainment robot," Proc. of IEEE International Conference on Robotics and Automation, vol. 1, pp. 453-458, 2001
  34. M. N. Nicolescu and M. J. Matari'c, "A hierarchical architecture for behavior-based robots," Proc. of the First International Joint Conference on Autonomous Agents and Multiagent Systems, ACM, New York, NY, USA, pp. 227-233, 2002
  35. A. Doucet, N. de Freitas and N. Gordon, Sequential Monte Carlo Methods in Practice, N. Gordon, Ed. Springer-Verlag, 2001