It looks to me like you drew the diodes the wrong way around in your schematic? You need the TX pin to be able to pull the shared line down...
Are you relying on the internal pull-up for the RX pin on the ESP32? I would hook up something like a 2.2K-4.7K external pull up with so many diodes connected.
Also, why the 10K in the 5V supply path? Isn't that asking for trouble when there are current spikes?