In this paper, we consider a multi-cell downlink mmWave communication network, where the base stations (BS) are assumed to be incapable of synchronously accommodating service requests from all users. The objective is to develop the joint user scheduling and beam selection strategy that minimizes the long-term average delay cost while satisfying the instantaneous quality of service constraint of each user. To achieve the long-term performance, we propose a distributed algorithm to develop the joint strategy based on multi-agent reinforcement learning. Simulation results show that the proposed intelligent distributed algorithm can learn from the dynamic environment and enhance the long-term network performance.