phd thesis on reinforcement learning

• DSpace@MIT Home
• MIT Libraries
• Doctoral Theses

Show simple item record

Data Efficient Reinforcement Learning

Files in this item

This item appears in the following Collection(s)

Show Statistical Information

• My Playlists
• Media Upload

• Best of Illinois
• Big Ten Network
• College Features
• News Bureau
• Video Services
• Diversity Equity & Inclusion
• Alumni Newsmakers
• Campus Conversations
• Campus Tours
• Commencements
• Opening Symposium
• iSEE Congress 2014
• Illinois Athletics
• Illinois Online
• Archive: Compass2g
• Video Creation Examples
• News@Illinois
• Research & Innovation
• Teaching Excellence
• Computing History
• ACES Technology Services
• Agricultural and Biological Engineering
• Agricultural and Consumer Economics
• Agricultural Communications
• Internships
• Animal Sciences
• Career Services
• Crop Sciences
• Division of Nutritional Sciences
• Food Science and Human Nutrition
• Human Development and Family Studies
• Special Events
• Kinesiology and Community Health
• Speech and Hearing Science
• Carle Illinois Virtual Open House
• College of Media
• I-TEACH I-LEAD
• User Services
• FAA Online Courses
• School of Architecture
• School of Art and Design
• School of Music
• General Studies
• Gies College of Business
• Doctoral Hooding
• Research Live
• For Students
• For Faculty and Staff
• Aerospace Engineering
• Bioengineering
• Civil and Environmental Engineering
• Computer Science
• Electrical & Computer Engineering
• Cell - Matrix Mechanobiology Meeting Presentations
• Engineer Guy
• Everitt Award for Teaching Excellence
• Hall of Fame - Engineering
• Labor & Employment Relations
• Center for East Asian and Pacific Studies
• Center for Latin American and Caribbean Studies
• French and Italian
• Jewish Studies
• LAS International Programs
• LAS Teaching Academy
• MCB Advising Program
• Program in Jewish Culture and Society
• Talent Show
• Sociology Alumni
• Sociology Advising
• School of Information Sciences
• Social Work
• Veterinary Medicine
• Beckman Institute
• Cancer Center at Illinois
• Center for Advanced Study
• Coordinated Science Lab
• CyberGIS Center
• Institute for Genomic Biology
• Illinois Sustainable Technology Center
• Research at Illinois
• Illinois Abroad and Global Exchange
• Illinois Leadership Center
• Illinois Student Government
• The Career Center
• Center for Global Studies
• Teaching Tips & Tutorials
• Faculty Stories
• Workshop and Event Recordings
• CITL Promos
• European Union Center
• Campus Wellbeing Services
• Faculty/Staff Assistance Services
• Know Your U
• Orientation
• The Illinois Professional
• Campus Email
• Classroom Technology
• Digital Media
• Digital Signage
• IT Power Plant
• Mediaspace - Kaltura Help and Tutorials
• Training Services
• Training Videos
• History Philosophy and Newspaper Library
• Illinois Open Publishing Network
• International and Area Studies Library
• Media Commons
• Mortenson Center
• Music & Performing Arts Library
• Rare Book and Manuscript Library
• Social Sciences Health and Education Library
• University of Illinois Archives
• University Senate
• Web Con Campus Workshops

2022 PhD Thesis Award: Kaiqing Zhang, "Reinforcement Learning for Multi-Agent and Robust Control Systems: Towards Large-scale and Reliable Autonomy"

Related media.

Sample-Efficient Deep Reinforcement Learning for Continuous Control

Repository uri, repository doi.

Reinforcement learning (RL) is a powerful, generic approach to discovering optimal policies in complex sequential decision-making problems. Recently, with flexible function approximators such as neural networks, RL has greatly expanded its realm of applications, from playing computer games with pixel inputs, to mastering the game of Go, to learning parkour movements by simulated humanoids. However, the common RL approaches are known to be sample intensive, making them difficult to be applied to real-world problems such as robotics. This thesis makes several contributions toward developing RL algorithms for learning in the wild, where sample-efficiency and stability are critical. The key contributions include Normalized Advantage Functions (NAF), extending Q-learning for continuous action problems; Interpolated Policy Gradient (IPG), unifying prior policy gradient algorithm variants through theoretical analyses on bias and variance; and Temporal Difference Models (TDM), interpreting a parameterized Q-function as a generalized dynamics model for novel temporally abstracted model-based planning. Importantly, this thesis highlights that these algorithms can be seen as bridging gaps between branches of RL – model-based with modelfree, and on-policy with off-policy. The proposed algorithms not only achieve substantial improvements over the prior approaches, but also provide novel perspectives on how to mix different branches of RL effectively to gain the best of both worlds. NAF has subsequently been shown to be able to train two 7-DoF robot arms to open doors using only 2.5 hours of real-world experience, making it one of the first demonstrations of deep RL approaches on real robots.

Description

Qualification, awarding institution, sponsorship, collections.

• AI Collection
• Oxford Thesis Collection
• CC0 version of this metadata

Deep multi-agent reinforcement learning

A plethora of real world problems, such as the control of autonomous vehicles and drones, packet delivery, and many others consists of a number of agents that need to take actions based on local observations and can thus be formulated in the multi-agent reinforcement learning (MARL) setting. Furthermore, as more machine learning systems are deployed in the real world, they will start having impact on each other, effectively turning most decision making problems into multiagent proble...

Email this record

Please enter the email address that the record information will be sent to.

Please add any additional information to be included within the email.

Cite this record

Chicago style, access document.

• DeepMARL.pdf ( Preview , pdf, 4.8MB, Terms of use )

Why is the content I wish to access not available via ORA?

Content may be unavailable for the following four reasons.

• Version unsuitable We have not obtained a suitable full-text for a given research output. See the versions advice for more information.
• Recently completed Sometimes content is held in ORA but is unavailable for a fixed period of time to comply with the policies and wishes of rights holders.
• Permissions All content made available in ORA should comply with relevant rights, such as copyright. See the copyright guide for more information.
• Clearance Some thesis volumes scanned as part of the digitisation scheme funded by Dr Leonard Polonsky are currently unavailable due to sensitive material or uncleared third-party copyright content. We are attempting to contact authors whose theses are affected.

Alternative access to the full-text

Request a copy.

We require your email address in order to let you know the outcome of your request.

Provide a statement outlining the basis of your request for the information of the author.

Please note any files released to you as part of your request are subject to the terms and conditions of use for the Oxford University Research Archive unless explicitly stated otherwise by the author.

Contributors

Bibliographic details, item description, related items, terms of use, views and downloads.

If you are the owner of this record, you can report an update to it here: Report update to this record

Report an update

We require your email address in order to let you know the outcome of your enquiry.

• Show simple item record
• Show full item record
• Export item record

Files in This Item:

Page view(s) 1

Download(s) 5.

Google Scholar TM

Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.

Meta Reinforcement Learning through Memory

Modern deep reinforcement learning (RL) algorithms, despite being at the forefront of artificial intelligence capabilities, typically require a prohibitive amount of training samples to reach a human-equivalent level of performance. This severe data inefficiency is the major obstruction to deep RL’s practical application: it is often near impossible to apply deep RL to any domain without at least a simulator available. Motivated to address this critical data inefficiency, in this thesis we work towards the design of meta-learning agents that are capable of rapidly adapting to new environments. In contrast to standard reinforcement learning, meta-learning learns over distributions of environments, from which specific tasks are sampled and with which the meta-learner is directly optimized to improve the speed of policy improvement on. By exploiting a distribution of tasks which share common substructure with the tasks of interest, the meta-learner can adjust its own inductive biases to enable rapid adaptation at test time.

This thesis focuses on the design of meta-learning algorithms which exploit memory as the main mechanism driving rapid adaptation in novel environments. Meta learning with inter-episodic memories are a class of meta-learning methods that leverage a memory architecture conditioned on the entire interaction history of a particular environment to produce a policy. The learning dynamics driving policy improvement in a particular task are thus subsumed by the computational process of the sequence model, essentially offloading the design of the learning algorithm to the architecture. While conceptually straightforward, meta-learning using inter-episodic memory is highly effective and remains a state-of-the-art method. 

We present and discuss several techniques for meta-learning through memory. The first part of the thesis focuses on the “embodied” class of environments, where an agent has a physical manifestation in an environment resembling the natural world. We exploit this highly structured set of environments to work towards the design of a monolithic embodied agent architecture that has the capabilities of rapid memorization, planning and state inference. In the second part of the thesis, we move to focus on methods that apply in general environments without strong common substructure. First, we re-examine the modes of interaction a meta-learning agent has with the environment: proposing to replace the typically sequential processing of interaction history with a concurrent execution framework where multiple agents act in the environment in parallel. Next, we discuss the use of a general and powerful sequence model for inter-episodic memory, the gated transformer, demonstrating large improvements in performance and data efficiency. Finally, we develop a method that significantly reduces the training cost and acting latency of transformer models in (meta-)reinforcement learning settings, with the aim to both (1) make their use more widespread within the research community, and, (2) unlock their use in real-time and latency-constrained applications, such as in robotics.

Degree Type

• Dissertation
• Machine Learning

Degree Name

• Doctor of Philosophy (PhD)

Usage metrics

• Artificial Intelligence and Image Processing

Reinforcement Learning Using Neural Networks, with Applications to Motor Control

PhD thesis, by Rémi Coulom

This thesis is a study of practical methods to estimate value functions with feedforward neural networks in model-based reinforcement learning. Focus is placed on problems in continuous time and space, such as motor-control tasks. In this work, the continuous TD(lambda) algorithm is refined to handle situations with discontinuous states and controls, and the vario-eta algorithm is proposed as a simple but efficient method to perform gradient descent. The main contributions of this thesis are experimental successes that clearly indicate the potential of feedforward neural networks to estimate high-dimensional value functions. Linear function approximators have been often preferred in reinforcement learning, but their success is restricted to relatively simple mechanical systems, or require a lot of prior knowledge. The method presented in this thesis was tested successfully on an original task of learning to swim by a simulated articulated robot, with 4 control variables and 12 independent state variables.

(only the first pages are in French, the rest is in English):

For those who cannot run the win32 demos below, some avi movies demonstrating the movements of swimmers (DivX codec required):

A few interactive (win32) swimmer demos (click in the window to change swimming direction):

• swimmer3.exe
• swimmer4-Slow.exe
• swimmer4-Fast.exe
• sw5-l0.exe (beginner)
• sw5-l5.exe (expert)
• sw5-l6.exe (performance drop)

Source code of the swimmer simulator:

• swimmer.tar.bz2
• RARSRLDemo.exe

Disclaimer/Complaints regulations

If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Ask the Library , or send a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You will be contacted as soon as possible.

• Corpus ID: 60875658

Reinforcement learning for robots using neural networks

• Longxin Lin
• Published 1992
• Computer Science, Engineering

Figures and Tables from this paper

1,012 Citations

Scaling up reinforcement learning for robot control, safer reinforcement learning for robotics, texplore: real-time sample-efficient reinforcement learning for robots, q-learning for robots, towards a common implementation of reinforcement learning for multiple robotic tasks, guided deep reinforcement learning for swarm systems, q-learning for robots 1 q-learning for robots, guided deep reinforcement learning for robot swarms guided deep reinforcement learning, texplore: temporal difference reinforcement learning for robots and time-constrained domains, learning robot control - using control policies as abstract actions, 71 references, input generalization in delayed reinforcement learning: an algorithm and performance comparisons, learning in embedded systems, programming robots using reinforcement learning and teaching, memory approaches to reinforcement learning in non-markovian domains, explanation-based neural network learning for robot control, efficient memory-based learning for robot control, self-improving reactive agents: case studies of reinforcement learning frameworks, reinforcement learning with a hierarchy of abstract models, learning a cost-sensitive internal representation for reinforcement learning, scaling reinforcement learning to robotics by exploiting the subsumption architecture, related papers.

Showing 1 through 3 of 0 Related Papers

This week: the arXiv Accessibility Forum

Help | Advanced Search

Quantitative Finance > Portfolio Management

Title: reinforcement learning for portfolio management.

Abstract: In this thesis, we develop a comprehensive account of the expressive power, modelling efficiency, and performance advantages of so-called trading agents (i.e., Deep Soft Recurrent Q-Network (DSRQN) and Mixture of Score Machines (MSM)), based on both traditional system identification (model-based approach) as well as on context-independent agents (model-free approach). The analysis provides conclusive support for the ability of model-free reinforcement learning methods to act as universal trading agents, which are not only capable of reducing the computational and memory complexity (owing to their linear scaling with the size of the universe), but also serve as generalizing strategies across assets and markets, regardless of the trading universe on which they have been trained. The relatively low volume of daily returns in financial market data is addressed via data augmentation (a generative approach) and a choice of pre-training strategies, both of which are validated against current state-of-the-art models. For rigour, a risk-sensitive framework which includes transaction costs is considered, and its performance advantages are demonstrated in a variety of scenarios, from synthetic time-series (sinusoidal, sawtooth and chirp waves), simulated market series (surrogate data based), through to real market data (S\&P 500 and EURO STOXX 50). The analysis and simulations confirm the superiority of universal model-free reinforcement learning agents over current portfolio management model in asset allocation strategies, with the achieved performance advantage of as much as 9.2\% in annualized cumulative returns and 13.4\% in annualized Sharpe Ratio.

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .