Two-Sided Deep Reinforcement Learning for Dynamic Mobility-on-Demand Management with Mixed Autonomy

References

• Afeche P, Liu Z, Maglaras C (2018) Ride-hailing networks with strategic drivers: The impact of platform control capabilities on performance. Preprint, submitted February 12, https://dx.doi.org/10.2139/ssrn.3120544.Google Scholar
• Ahmed S, Muhammod R, Khan ZH, Adilina S, Sharma A, Shatabda S, Dehzangi A (2021) Acp-mhcnn: An accurate multi-headed deep-convolutional neural network to predict anticancer peptides. Sci. Rep. 11(1):1–15.Crossref, Google Scholar
• Alonso-Mora J, Samaranayake S, Wallar A, Frazzoli E, Rus D (2017) On-demand high-capacity ride-sharing via dynamic trip-vehicle assignment. Proc. Natl. Acad. Sci. USA 114(3):462–467.Crossref, Google Scholar
• Arik SÖ, Jun H, Diamos G (2018) Fast spectrogram inversion using multi-head convolutional neural networks. IEEE Signal Processing Lett. 26(1):94–98.Crossref, Google Scholar
• Banerjee S, Johari R, Riquelme C (2015) Pricing in ride-sharing platforms: A queueing-theoretic approach. Roughgarden T, Feldman M, Schwarz M, eds. Proc. Sixteenth ACM Conf. Econom. Comput. (ACM, New York), 639–639.Google Scholar
• Buşoniu L, Babuška R, De Schutter B (2010) Multi-agent reinforcement learning: An overview. Srinivasan D, Jain LC, eds. Innovations in Multi-Agent Systems and Applications–1: Studies in Computational Intelligence (Springer, Berlin, Heidelberg), 183–221.Crossref, Google Scholar
• Cachon GP, Daniels KM, Lobel R (2017) The role of surge pricing on a service platform with self-scheduling capacity. Manufacturing Service Oper. Management 19(3):368–384.Link, Google Scholar
• Castillo JC, Knoepfle D, Weyl G (2017) Surge pricing solves the wild goose chase. Daskalakis C, Babaioff M, Moulin H, eds. Proc. 2017 ACM Conf. Econom. Comput. (ACM, New York), 241–242.Google Scholar
• Chen X, Di X (2021) Ridesharing user equilibrium with nodal matching cost and its implications for congestion tolling and platform pricing. Transportation Res. Part C: Emerging Tech. 129:103233.Crossref, Google Scholar
• Chen TD, Kockelman KM (2016) Management of a shared autonomous electric vehicle fleet: Implications of pricing schemes. Transportation Res. Record 2572(1):37–46.Crossref, Google Scholar
• Chen Y, Liu Y (2022) Integrated optimization of planning and operations for shared autonomous electric vehicle systems. Transportation Sci., ePub ahead of print August 24, https://doi.org/10.1287/trsc.2022.1156.Link, Google Scholar
• Chen L, Mislove A, Wilson C (2015) Peeking beneath the hood of Uber. Cho K, Fukuda K, eds., Pai V, Spring N, Program Chairs, Proc. 2015 Internet Measurement Conf. (ACM, New York), 495–508.Google Scholar
• Chen XM, Zheng H, Ke J, Yang H (2020a) Dynamic optimization strategies for on-demand ride services platform: Surge pricing, commission rate, and incentives. Transportation Res. Part B: Methodological 138:23–45.Crossref, Google Scholar
• Chen C, Wei H, Xu N, Zheng G, Yang M, Xiong Y, Xu K, Li Z (2020b) Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic signal control. Proc. Conf. AAAI Artificial Intelligence 34(4):3414–3421.Crossref, Google Scholar
• Chow Y, Yu JY, Pavone M (2015) Two phase Q-learning for bidding-based vehicle sharing. Preprint, submitted September 29, https://arxiv.org/abs/1509.08932.Google Scholar
• Coppola P, Silvestri F (2019) Autonomous vehicles and future mobility solutions. Coppola P, Esztergar-Kiss D, eds. Autonomous Vehicles and Future Mobility (Elsevier, Amsterdam), 1–15.Crossref, Google Scholar
• Di X, Ban XJ (2019) A unified equilibrium framework of new shared mobility systems. Transportation Res. Part B: Methodological 129:50–78.Crossref, Google Scholar
• Di X, Liu HX (2016) Boundedly rational route choice behavior: A review of models and methodologies. Transportation Res. Part B: Methodological 85:142–179.Crossref, Google Scholar
• Di X, Liu HX, Ban XJ (2016) Second best toll pricing within the framework of bounded rationality. Transportation Res. Part B: Methodological 83:74–90.Crossref, Google Scholar
• Di X, Liu HX, Ban X, Yang H (2017) Ridesharing user equilibrium and its implications for high-occupancy toll lane pricing. Transportation Res. Record 2667(1):39–50.Crossref, Google Scholar
• Di X, Liu HX, Zhu S, Levinson DM (2017) Indifference bands for boundedly rational route switching. Transportation 44(5):1169–1194.Crossref, Google Scholar
• Duan L, Wei Y, Zhang J, Xia Y (2020) Centralized and decentralized autonomous dispatching strategy for dynamic autonomous taxi operation in hybrid request mode. Transportation Res. Part C: Emerging Tech. 111:397–420.Crossref, Google Scholar
• Flet-Berliac Y, Preux P (2019) MERL: Multi-head reinforcement learning. Preprint, submitted September 26, https://arxiv.org/abs/1909.11939.Google Scholar
• Furuhata M, Dessouky M, Ordóñez F, Brunet M-E, Wang X, Koenig S (2013) Ridesharing: The state-of-the-art and future directions. Transportation Res. Part B: Methodological 57:28–46.Crossref, Google Scholar
• Gao Y, Jiang D, Xu Y (2018) Optimize taxi driving strategies based on reinforcement learning. Internat. J. Geographical Inform. Sci. 32(8):1677–1696.Crossref, Google Scholar
• Godfrey GA, Powell WB (2002a) An adaptive dynamic programming algorithm for dynamic fleet management, i: Single period travel times. Transportation Sci. 36(1):21–39.Link, Google Scholar
• Godfrey GA, Powell WB (2002b) An adaptive dynamic programming algorithm for dynamic fleet management, ii: Multiperiod travel times. Transportation Sci. 36(1):40–54.Link, Google Scholar
• Guériau M, Dusparic I (2018) SAMoD: Shared autonomous mobility-on-demand using decentralized reinforcement learning. 21st Internat. Conf. Intelligent Transportation Systems (IEEE, New York), 1558–1563.Google Scholar
• Haliem M, Mani G, Aggarwal V, Bhargava B (2020) A distributed model-free ride-sharing approach for joint matching, pricing, and dispatching using deep reinforcement learning. Preprint, submitted October 5, https://arxiv.org/abs/2010.01755.Google Scholar
• Haydari A, Yilmaz Y (2020) Deep reinforcement learning for intelligent transportation systems: A survey. IEEE Trans. Intelligent Transportation Systems 23(1):11–32.Crossref, Google Scholar
• He F, Wang X, Lin X, Tang X (2018) Pricing and penalty/compensation strategies of a taxi-hailing platform. Transportation Res. Part C: Emerging Tech. 86:263–279.Crossref, Google Scholar
• Hu M, Zhou Y (2020) Price, wage, and fixed commission in on-demand matching. Preprint, submitted September 1, https://dx.doi.org/10.2139/ssrn.2949513.Google Scholar
• Hu S, Dessouky MM, Uhan NA, Vayanos P (2021) Cost-sharing mechanism design for ride-sharing. Transportation Res. Part B: Methodological 150:410–434.Crossref, Google Scholar
• Karamanis R, Angeloudis P, Sivakumar A, Stettler M (2018) Dynamic pricing in one-sided autonomous ride-sourcing markets. 21st IEEE Internat. Conf. Intelligent Transportation Systems (IEEE, New York), 3645–3650.Google Scholar
• Ke J, Xiao F, Yang H, Ye J (2019) Optimizing online matching for ride-sourcing services with multi-agent deep reinforcement learning. Preprint, submitted February 17, https://arxiv.org/abs/1902.06228.Google Scholar
• Ke J, Yang H, Li X, Wang H, Ye J (2020) Pricing and equilibrium in on-demand ride-pooling markets. Transportation Res. Part B: Methodological 139:411–431.Crossref, Google Scholar
• Kim B, Kim J, Huh S, You S, Yang I (2020) Multi-objective predictive taxi dispatch via network flow optimization. IEEE Access 8:21437–21452.Crossref, Google Scholar
• Konda VR, Tsitsiklis JN (2000) Actor-critic algorithms. Solla S, Leen T, Muller K, eds. Advances in Neural Information Processing Systems (NIPS, Denver), 1008–1014.Google Scholar
• Lei C, Jiang Z, Ouyang Y (2020) Path-based dynamic pricing for vehicle allocation in ridesharing systems with fully compliant drivers. Transportation Res. Part B: Methodological 132:60–75.Crossref, Google Scholar
• Li Y (2017) Deep reinforcement learning: An overview. Preprint, submitted January 25, https://arxiv.org/abs/1701.07274v1.Google Scholar
• Li Q, Liao F (2020) Incorporating vehicle self-relocations and traveler activity chains in a bi-level model of optimal deployment of shared autonomous vehicles. Transportation Res. Part B: Methodological 140:151–175.Crossref, Google Scholar
• Li Y, Liu Y (2021) Optimizing flexible one-to-two matching in ride-hailing systems with boundedly rational users. Transp. Res. Part E: Logist. Transportation Rev. 150:102329.Crossref, Google Scholar
• Li Y, Yuan Y (2017) Convergence analysis of two-layer neural networks with relu activation. Preprint, submitted May 28, https://arxiv.org/abs/1705.09886v1.Google Scholar
• Li Y, Liu Y, Xie J (2020) A path-based equilibrium model for ridesharing matching. Transportation Res. Part B: Methodological 138:373–405.Crossref, Google Scholar
• Li M, Di X, Liu HX, Huang H-J (2020) A restricted path-based ridesharing user equilibrium. J. Intelligent Transportation Systems 24(4):383–403.Crossref, Google Scholar
• Li M, Qin Z, Jiao Y, Yang Y, Wang J, Wang C, Wu G, Ye J (2019) Efficient ridesharing order dispatching with mean field multi-agent reinforcement learning, World Wide Web Conf. (Association for Computing Machinery, New York), 983–994.Google Scholar
• Lin K, Zhao R, Xu Z, Zhou J (2018a) Efficient collaborative multi-agent deep reinforcement learning for large-scale fleet management. Preprint, submitted February 18, https://arxiv.org/abs/1802.06444v1.Google Scholar
• Lin K, Zhao R, Xu Z, Zhou J (2018b) Efficient large-scale fleet management via multi-agent deep reinforcement learning. Guo Y, Farooq F, eds. Proc. 24th ACM SIGKDD Internat. Conf. Knowledge Discovery & Data Mining (Association for Computing Machinery, New York), 1774–1783.Google Scholar
• Litman T (2017) Autonomous Vehicle Implementation Predictions (Victoria Transport Policy Institute, Victoria, BC, Canada).Google Scholar
• Liu Y, Li Y (2017) Pricing scheme design of ridesharing program in morning commute problem. Transportation Res. Part C: Emerging Tech. 79:156–177.Crossref, Google Scholar
• Liu Y, Xie J, Chen N (2022) Stochastic one-way carsharing systems with dynamic relocation incentives through preference learning. Transportation Res. Part E: Logist. Transportation Rev. 166:102884.Crossref, Google Scholar
• Lokhandwala M, Cai H (2018) Dynamic ride sharing using traditional taxis and shared autonomous taxis: A case study of NYC. Transportation Res. Part C: Emerging Tech. 97:45–60.Crossref, Google Scholar
• Luo Q, Saigal R (2017) Dynamic pricing for on-demand ride-sharing: A continuous approach. Preprint, submitted October 23, https://dx.doi.org/10.2139/ssrn.3056498.Google Scholar
• Ma J, Xu M, Meng Q, Cheng L (2020) Ridesharing user equilibrium problem under OD-based surge pricing strategy. Transportation Res. Part B: Methodological 134:1–24.Crossref, Google Scholar
• Mao C, Liu Y, Shen Z-JM (2020) Dispatch of autonomous vehicles for taxi services: A deep reinforcement learning approach. Transportation Res. Part C: Emerging Tech. 115:102626.Crossref, Google Scholar
• Miao F, Han S, Lin S, Stankovic JA, Zhang D, Munir S, Huang H, He T, Pappas GJ (2016) Taxi dispatch with real-time sensing data in metropolitan areas: A receding horizon control approach. IEEE Trans. Automation Sci. Engrg. 13(2):463–478.Crossref, Google Scholar
• Mnih V, Badia AP, Mirza M, Graves A, Lillicrap T, Harley T, Silver D, Kavukcuoglu K (2016) Asynchronous methods for deep reinforcement learning. Internat. Conf. Machine Learning (PMLR, New York), 1928–1937.Google Scholar
• Mo D, Chen XM, Zhang J (2022) Modeling and managing mixed on-demand ride services of human-driven vehicles and autonomous vehicles. Transportation Res. Part B: Methodological 157:80–119.Crossref, Google Scholar
• Nazari M, Oroojlooy A, Snyder LV, Takáč M (2018) Reinforcement learning for solving the vehicle routing problem. Preprint, submitted February 12, https://arxiv.org/abs/1802.04240v1.Google Scholar
• Noruzoliaee M, Zou B, Liu Y (2018) Roads in transition: Integrated modeling of a manufacturer-traveler-infrastructure system in a mixed autonomous/human driving environment. Transportation Res. Part C: Emerging Tech. 90:307–333.Crossref, Google Scholar
• Nourinejad M, Roorda MJ (2016) Agent based model for dynamic ridesharing. Transportation Res. Part C: Emerging Tech. 64:117–132.Crossref, Google Scholar
• Ordóñez F, Dessouky MM (2017) Dynamic ridesharing. Batta R, Peng J, eds. Leading Developments from INFORMS Communities, INFORMS Tutorials in Operations Research (INFORMS, Catonsville, MD), 212–236.Abstract, Google Scholar
• O’Keeffe K, Anklesaria S, Santi P, Ratti C (2021) Using reinforcement learning to minimize taxi idle times. J. Intelligent Transportation Systems 26(4):1–16.Google Scholar
• Pakusch C, Meurer J, Tolmie P, Stevens G (2020) Traditional taxis vs automated taxis–Does the driver matter for millennials? Travel Behav. Soc. 21:214–225.Crossref, Google Scholar
• Pang J-S, Zhang M, Dessouky MM, Gu W, Center MT, Region PS (2020) Modeling e-hailing and car-pooling services in a coupled morning-evening commute framework, Technical report, California Department of Transportation, Division of Research and Innovation, Sacramento, CA.Google Scholar
• Powell WB (2007) Approximate Dynamic Programming: Solving the Curses of Dimensionality, vol. 703 (John Wiley & Sons, New York).Crossref, Google Scholar
• Powell WB (2009) What you should know about approximate dynamic programming. Naval Res. Logist. 56(3):239–249.Crossref, Google Scholar
• Qin Z, Tang J, Ye J (2019) Deep reinforcement learning with applications in transportation. Teredesai A, Kumar V, Li Y, Rosales R, Terzi E, Karypis G eds. Proc. 25th ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining (ACM, New York), 3201–3202.Google Scholar
• Qin Z, Tang X, Jiao Y, Zhang F, Xu Z, Zhu H, Ye J (2020) Ride-hailing order dispatching at didi via reinforcement learning. INFORMS J. Appl. Analytics 50(5):272–286.Link, Google Scholar
• Ramezani M, Nourinejad M (2018) Dynamic modeling and control of taxi services in large-scale urban networks: A macroscopic approach. Transportation Res. Part C: Emerging Tech. 94:203–219.Crossref, Google Scholar
• Sayarshad HR, Chow JYJ (2017) Non-myopic relocation of idle mobility-on-demand vehicles as a dynamic location-allocation-queueing problem. Transportation Res. Part E: Logist. Transportation Rev. 106:60–77.Crossref, Google Scholar
• Shetty S (2020) Uber’s self-driving cars are a key to its path to profitability. Accessed November 1, 2022, https://www.cnbc.com/2020/01/28/ubers-self-driving-cars-are-a-key-to-its-path-to-profitability.htmlGoogle Scholar
• Shojaeighadikolaei A, Ghasemi A, Jones KR, Bardas AG, Hashemi M, Ahmadi R (2020) Demand responsive dynamic pricing framework for prosumer dominated microgrids using multiagent reinforcement learning. Preprint, submitted September 23, https://arxiv.org/abs/2009.10890.Google Scholar
• Shou Z, Di X (2020) Reward design for driver repositioning using multi-agent reinforcement learning. Transportation Res. Part C: Emerging Tech. 119:102738.Crossref, Google Scholar
• Shou Z, Di X, Ye J, Zhu H, Hampshire R (2019) Where to find next passengers on e-hailing platforms? A Markov decision process approach. Preprint, submitted May 23, https://arxiv.org/abs/1905.09906v1.Google Scholar
• Simao HP, Day J, George AP, Gifford T, Nienow J, Powell WB (2009) An approximate dynamic programming algorithm for large-scale fleet management: A case application. Transportation Sci. 43(2):178–197.Link, Google Scholar
• Sutton RS, Barto AG (2018) Reinforcement Learning: An Introduction (MIT Press, Cambridge, MA).Google Scholar
• Tang X, Li M, Lin X, He F (2020) Online operations of automated electric taxi fleets: An advisor-student reinforcement learning framework. Transportation Res. Part C: Emerging Tech. 121:102844.Crossref, Google Scholar
• Torreño A, Onaindia E, Komenda A, Štolba M (2017) Cooperative multi-agent planning: A survey. ACM Comput. Surveys 50(6)84:1–84:32.Google Scholar
• Turan B, Pedarsani R, Alizadeh M (2020) Dynamic pricing and fleet management for electric autonomous mobility on demand systems. Preprint, submitted October 1, https://arxiv.org/abs/1909.06962.Google Scholar
• Ulmer MW, Goodson JC, Mattfeld DC, Hennig M (2019) Offline–online approximate dynamic programming for dynamic vehicle routing with stochastic requests. Transportation Sci. 53(1):185–202.Link, Google Scholar
• Van Seijen H, Fatemi M, Romoff J, Laroche R, Barnes T, Tsang J (2017) Hybrid reward architecture for reinforcement learning. Submitted June 13, https://arxiv.org/abs/1706.04208v1.Google Scholar
• Vinsensius A, Wang Y, Chew EP, Lee LH (2020) Dynamic incentive mechanism for delivery slot management in e-commerce attended home delivery. Transportation Sci. 54(3):567–587.Link, Google Scholar
• Vosooghi R, Puchinger J, Jankovic M, Vouillon A (2019) Shared autonomous vehicle simulation and service design. Transportation Res. Part C: Emerging Tech. 107:15–33.Crossref, Google Scholar
• Wang X, He F, Yang H, Gao HO (2016) Pricing strategies for a taxi-hailing platform. Transportation Res. Part E: Logist. Transportation Rev. 93:212–231.Crossref, Google Scholar
• Wang X, Liu W, Yang H, Wang D, Ye J (2020a) Customer behavioural modelling of order cancellation in coupled ride-sourcing and taxi markets. Transportation Res. Part B: Methodological 132:358–378.Crossref, Google Scholar
• Wang C, Zhang J, Xu L, Li L, Ran B (2019) A new solution for freeway congestion: Cooperative speed limit control using distributed reinforcement learning. IEEE Access 7:41947–41957.Crossref, Google Scholar
• Wang E, Ding R, Yang Z, Jin H, Miao C, Su L, Zhang F, Qiao C, Wang X (2020b) Joint charging and relocation recommendation for e-taxi drivers via multi-agent mean field hierarchical reinforcement learning. IEEE Trans. Mobile Comput. 21(4):1274–1290Crossref, Google Scholar
• Wei Q, Rodriguez JA, Pedarsani R, Coogan S (2019) Ride-sharing networks with mixed autonomy. 2019 Amer. Control Conf. (IEEE, New York), 3303–3308.Google Scholar
• Wollenstein-Betech S, Paschalidis IC, Cassandras CG (2020) Joint pricing and rebalancing of autonomous mobility-on-demand systems. 59th IEEE Conf. Decision Control (CDC) (IEEE, New York), 2573–2578.Google Scholar
• Wong R, Szeto W, Wong S (2014) A cell-based logit-opportunity taxi customer-search model. Transportation Res. Part C: Emerging Tech. 48:84–96.Crossref, Google Scholar
• Xie J, Yang Z, Lai X, Liu Y, Yang XB, Teng T-H, Tham C-K (2022) Deep reinforcement learning for dynamic incident-responsive traffic information dissemination. Transportation Res., Part E: Logist. Transportation Rev. 166:102871.Crossref, Google Scholar
• Xu H, Pang J-S, Ordóñez F, Dessouky M (2015) Complementarity models for traffic equilibrium with ridesharing. Transportation Res. Part B: Methodological 81:161–182.Crossref, Google Scholar
• Xu Z, Li Z, Guan Q, Zhang D, Li Q, Nan J, Liu C, Bian W, Ye J (2018) Large-scale order dispatch in on-demand ride-hailing platforms: A learning and planning approach. Guo Y, Farooq F, eds. Proc. 24th ACM SIGKDD Internat. Conf. Knowledge Discovery & Data Mining (ACM, New York), 905–913.Google Scholar
• Yang H, Leung CW, Wong SC, Bell MG (2010) Equilibria of bilateral taxi–customer searching and meeting on networks. Transportation Res. Part B: Methodological 44(8–9):1067–1083.Crossref, Google Scholar
• Yang Z, Merrick KE, Abbass HA, Jin L (2017) Multi-task deep reinforcement learning for continuous action control. IJCAI (U. S.) 17:3301–3307.Google Scholar
• Yang H, Shao C, Wang H, Ye J (2020a) Integrated reward scheme and surge pricing in a ride sourcing market. Transportation Res. Part B: Methodological 134:126–142.Crossref, Google Scholar
• Yang K, Tsao MW, Xu X, Pavone M (2021) Real-time control of mixed fleets in mobility-on-demand systems, 2021 IEEE Internat. Intelligent Transportation Systems Conf. (ITSC) (IEEE, New York), 3570–3577.Google Scholar
• Yang Y, Wang X, Yuanbo X, Huang Q (2020b) Multiagent reinforcement earning-based taxi predispatching model to balance taxi supply and demand. J. Adv. Transportation 2020:1–12.Google Scholar
• Yang Y, Luo R, Li M, Zhou M, Zhang W, Wang J (2018) Mean field multi-agent reinforcement learning. Preprint, February 15, https://arxiv.org/abs/1802.05438v1.Google Scholar
• Zha L, Yin Y, Du Y (2017) Surge pricing and labor supply in the ride-sourcing market. Transportation Res. Procedia 23:2–21.Crossref, Google Scholar
• Zha L, Yin Y, Du Y (2018) Surge pricing and labor supply in the ride-sourcing market. Transportation Res. Part B: Methodological 117:708–722.Crossref, Google Scholar
• Zha L, Yin Y, Xu Z (2018) Geometric matching and spatial pricing in ride-sourcing markets. Transportation Res. Part C: Emerging Tech. 92:58–75.Crossref, Google Scholar
• Zha L, Yin Y, Yang H (2016) Economic analysis of ride-sourcing markets. Transportation Res. Part C: Emerging Tech. 71:249–266.Crossref, Google Scholar
• Zhang R, Pavone M (2016) Control of robotic mobility-on-demand systems: A queueing-theoretical perspective. Internat. J. Robotics Res. 35(1–3):186–203.Crossref, Google Scholar
• Zhang D, Liu Y, He S (2019) Vehicle assignment and relays for one-way electric car-sharing systems. Transportation Res. Part B: Methodological 120:125–146.Crossref, Google Scholar
• Zhang C, Odonkor P, Zheng S, Khorasgani H, Serita S, Gupta C (2020) Dynamic dispatching for large-scale heterogeneous fleet via multi-agent deep reinforcement learning. Preprint, submitted August 24, https://arxiv.org/abs/2008.10713.Google Scholar
• Zhu Z, Ke J, Wang H (2021) A mean-field Markov decision process model for spatial-temporal subsidies in ride-sourcing markets. Transportation Res. Part B: Methodological 150:540–565.Crossref, Google Scholar