Tesla FSD with Teleops (Teslops)

The bull case for the Tesla stock centres around autonomy/FSD: selling it to customers and using it to enable the Tesla ride-hailing network — but there is a consensus that the current state of Tesla’s FSD, the slow pace of improvement, and the continual missing of targets suggest that Tesla is nowhere near Level 4 and a very long way from creating a ride-hailing service that doesn’t require human drivers.

The main problem is edge cases. Even once the ‘feature complete’ version of their FSD is working well, the experience of their competitors in this space have shown that that the edge cases are extremely hard to address.

In deciding whether to buy or sell TSLA I would suggest that this is the most important question: what is their plan to get from L2/L3 to L4/L5.

A Solution — Tesla FSD plus Teleops

This problem can be resolved by complementing Tesla FSD with “teleops” (remote control) fallback, where the teleops is used for edge cases with FSD in an enhanced shadow mode.

Actually, I had rather assumed that this was Tesla’s plan: that it is one of the reasons they stuck to their simpler (cheaper) sensor stack while exhibiting such great confidence in the near future potential of their FSD to achieve autonomy. The recent tweet by Musk about them excluding even radar from the latest incarnation of FSD makes little sense to me otherwise. And if they aren’t planning to utilize teleops (and have no other magic in their back pocket) then I consider Musk’s claims about L5 last year to be truly jumping the shark.

But I’m baffled that I rarely see any reference to this approach. Instead I keep seeing discussion of them starting with a human driver based service, even on earning calls. Recently I also read/heard Cathie Wood of ARK Investment call on Tesla to launch their ride-hailing service with human drivers, detailed here. This would have fewer technical and regulatory dependencies than teleops, but it still appears to me to be a clearly inferior interim solution — it does not help in providing FSD to Tesla owners — and would be an enormous undertaking and commitment with a much higher cost structure.

Time for me to try to make the case for Teslops: Tesla FSD augmented with Teleops.

Some Details

The FSD + teleops solution would be most useful for location based edge cases: driving segments that involve difficult situations or uncertainty. Where possible, routing will avoid these (a good idea from a safety perspective anyway) but where that is impossible a remote driver who is experienced in the scenario at hand can be brought online seconds ahead of time. If there is a communications issue then the FSD takes control and behaves as conservatively as possible — maybe even just pulling over to the side of the road.

For other types of edge cases, eg. sudden and unexpected events, teleops could still be used but would be less effective so Tesla would have to focus early efforts on training the initial reaction of their FSD system. I believe that this type of edge case will be much less common then the others (that’s why they are unexpected) and that the FSD can be trained to handle the initial moments of these — before teleops can come online — but this is just my speculation so I consider this one of the weaker aspects of the case I’m making.

Note that FSD will always be on and available — it will just be in something like shadow mode (the mode currently used when the human driver is in control of the vehicle) when teleops is active but doing additional processing geared mainly towards minimizing the amount of data that needs to be sent. So FSD as fallback to teleops which is fallback to FSD — let’s just say they complement each other very well, and I don’t think teleops makes much sense without some on-board autonomy for driving systems.

Another weak point in my case is the availability of the sensors — it doesn’t seem to me that Tesla has made adequate provision for keeping the cameras clean in adverse conditions. However, I anticipate there being Tesla employees at their charging stations to clean the car while the car is charging, and who would also clean the cameras. I also expect that Tesla will start their network in locations with consistent good weather — perhaps full (unattended) FSD and the ride-hailing network will simply not be available to customers in bad weather? Not ideal for a ride-hailing network but I believe Waymo has had to accept this limitation as well.

However, if an inability to keep the sensors working properly is a deal breaker for teleops then it is a serious problem for FSD and for Tesla in general, and then why would it not be the current focus of attention and of incremental hardware improvements?

Regulatory issues are certainly an important potential impediment, but it is very significant that a fully autonomous commercial service (Waymo) is already operational in the US, and that teleops is not new. As well, since Tesla already has FSD and the hardware that would be used for teleops on most of their vehicles, they already have tons of relevant data and could quickly generate more in support of this new proposition: that their FSD plus teleops should be allowed to operate in full autonomous mode in specific locations. Tesla’s oft declared vision is of FSD everywhere but there is no way of getting around location dependence when we are talking about regulation. And going live everywhere (even just in the US) at launch would be a bad idea for all sorts of practical reasons anway. In the end, if Tesla can demonstrate that this solution is safe then they should be able to achieve regulatory approval in enough states (and the province of Ontario, I hope) to get started, and if they can’t then I am wrong and it shouldn’t be used.

On a semantic and rather pedantic note, Waymo has made it clear that while their ‘driver’ (their autonomous driving system) may have to rely on human intervention for strategic planning, they are not using teleops to drive the vehicle so their system should be considered fully autonomous. I can well understand that they want to downplay any potential crutches that would allow competitors to close the gap that they have gained with such enormous investment, but people won’t notice or care about this crutch if it works well. Others have focussed on the same distinction, but in the end all that matters is whether the solution is effective in terms of safety and cost.

I think the primary issue that people will bring up to argue against the use of teleops will be the reliability and latency of the cellular communications. I’m not going to get into a detailed, technical discussion of this mainly because it is a quickly moving target:

* Cellular keeps getting better, in particular, 5G can provide lower latency than 4G and it can allow for reserving slices of spectrum with defined characteristics, at difficult locations.

* Tesla can continue to reduce the latency of its software stack both in the car and for the operators.

* The human operators will get better with experience, and potentially more specialized with scale, eg. an operator could specialize in handling roundabouts, flat tires or snowy conditions.

So that makes the feasibility of this solution a matter of when and where, rather then if, and the best evidence that ‘when’ is soon is that remote controlled robots (as drones, industrial robots, and vehicles) over cellular links are already here — even with on-board autonomy that is inferior to Tesla’s and without 5G network slicing.

The cost is another moving target with the number of edge cases that require teleops and the cost of cellular data continually improving. Every use of teleops would be valuable training data that would improve the fallback performance of FSD and eventually eliminate the edge case. Most importantly, my assumption that the cost of this is not a problem is based on the assumption that it will be used for only a very small portion of each trip and on the amount of additional revenue it enables.

I will throw in one final, enticing consideration … once Tesla vehicles are uploading data and getting back driving instructions to augment their FSD, all in real-time, this opens up new possibilities: the server could resolve some the situations without resorting to human operators, eg. using what it knows from other Tesla vehicles in the area and by applying additional computational resources. This capability would be improved with time and with increasing density of Tesla vehicles, and one would expect that the areas where they offer their ride-hailing service would have lots of Tesla’s, and the places where FSD needed to resort to “outside help” would usually be dense urban areas with high vehicle density. I would imagine that this is a long-term end goal for most of the autonomous driving systems anyway.

Prediction

This post is more about having a discussion than making hard assertions, but I believe in making concrete predictions where possible, so here is mine: I predict that Tesla will begin employing teleops in their vehicles as a fallback to their FSD before the end of 2022. This will be available to customers who purchased their FSD package and it will be used to enable their ride-hailing service. The locations in which they launch will be determined by weather, cellular, and regulatory considerations.

I consider that an aggressive timeline for Tesla FSD to improve enough to match the vision I have outlined above, but it is worth considering that, in locations with excellent cellular service, this solution could be launched with a fairly heavy dependence on the teleops as long as there was certainty that this could be improved upon quickly — the extra costs would be manageable since the scale of the service would be quite limited at this point. Waymo is a rather extreme example of starting in a small geographic area!

I acknowledge, though, that, far from this being the consensus of analysts and pundits etc., the idea of Teslops barely gets mentioned. So I have to ask you what you think about this and what I might be missing?

Disclaimer

Nella is a Tesla customer and a (very small time) stockholder, but we have no other connection with Tesla. Neither of us have any association with teleops companies.

This article expands on a previous one that we wrote on the same topic. We must admit that we got quite a negative reaction to the idea back then, but now with the advancing plans of some of the teleops startups lending credence to the viability of our proposal, we felt it was time to press the case again.