As soon as the 2019 “strength” report of autonomous driving came out, I had to take a quick look and couldn’t help but boil.
The industry has made rapid progress – from total mileage to average takeover, unmanned vehicles seem to be in a “drag racing” mode compared to last year. More importantly, Chinese companies have performed very well. According to the average takeover index, 4 of the top 10 companies in the world are from China.
Baidu’s North American autopilot team, with a separate rudder, defeated Waymo at the location of Google’s base camp and reached the top for the first time.
That’s right, this is the “2019 Autonomous Driving Takeover Report” just released by the California Department of Vehicles (DMV), and the above highlights are not all.
After the report was published, Waymo, which lost the Iron Throne for the first time, quit.
Waymo publicly stated that it opposed the data requirements and ability judgment criteria of this official report.
So what is going on, let’s start with the report.
Baidu overtakes Google’s Waymo, Chinese company making rapid progress
California, the United States, is the most enlightened area in the world for autonomous driving road testing. Currently, 64 unmanned vehicle companies from around the world have been assembled and approved for road testing.
As a result, the California Department of Vehicles (DMV)’s annual “2019 Autonomous Driving Takeover Report” has received special attention-because no other report can better show a horizontal comparison.
It’s like everyone is competing in the same venue and scene, and finally the organizing committee releases an annual report, which can be used as a reference for strength.
It’s just that this report is also criticized. It is submitted by various testing companies independently, and the core indicators are difficult to quantify. It can be understood that this is a report that each company is a judge by itself, and each company uses its own conscience.
So what about this year’s report? First of all, at the player level, a total of 36 unmanned vehicle companies submitted reports.
Secondly, on the whole, there has been great development and great progress. On average, it takes 508 kilometers of unmanned driving to take over once, an increase of 21 times compared with 2018.
In addition, there are new conclusions that are different from the past: Baidu’s North American team has pulled Waymo from the first throne for the first time, and Chinese companies have performed very well.
The specific measurement indicator is naturally the number of disengagements.
What is “Out of Takeover”? That is to say, in the process of automatic driving, the human safety officer grabbed the control right, let the automatic driving system and the vehicle disengage, and the human took over the driving. It often occurs when dealing with various situations that the automatic driving system cannot handle.
To expand, it is how many cars in total, how far the cars have run, and how many times people have taken over.
Then you can figure out how far you need to take over on average.
Focus on what you need to test. Later, we will discuss how far you need to take over this average run.
From this indicator alone, the data results are as follows:
The abscissa is too long, and since it would be inconvenient to view on a mobile phone, let’s show the top 10 companies:
And the companies after the top ten:
Note that the blue histogram is for 2019 (that is, the latest data), and the orange histogram is for 2018 (a year ago).
As can be seen from the above charts, Baidu USA is the first, with an average of 29,060 kilometers to be taken over once – that means running around the equator for two laps, it only needs to take over three times.
And Google sibling Waymo, which ranks second this year, needs to be taken over once on average for 21,283 kilometers.
Cruise, AutoX, and PonyAI were then ranked third, fourth, and fifth globally.
Among the top five, Chinese companies such as Baidu, AutoX and PonyAI are not yet fully displayed in California, but the results are already astonishing.
Other Chinese companies are also doing very well. Didi’s self-driving car appeared in the report for the first time, ranking eighth, WeRide Weride ranked 11th, and SAIC, the third-to-last last year, ranked 17th.
Compared to the 2018 data, the progress is really great:
Baidu climbed to No. 1 from No. 7. AutoX, progressed from 9th to 4th. There is also Pony AI. Although the ranking has not changed this year, its performance has also increased by more than 5 times.
Last year’s champions and runners-up, Waymo and Cruise, although made progress, did not have the 87-fold increase of Baidu, ranking second and third.
Of course, there are people who quit with this result.
Executives and corporate groups speak out: DMV statistics are inappropriate
Before, the statistical methods and standards of the DMV report have been controversial.
And just recently, there are several core players who have formed a group to protest with their real names. Their main point of questioning is whether it is reasonable to use the “number of takeovers” as the core assessment indicator.
Literally, the more frequent takeovers, the less time the autopilot system works, and it seems that the autopilot company is less powerful.
△ Hou Xiaodi
Last year, Hou Xiaodi, co-founder & CTO of TuSimple, expressed many times: the standard of separation is very subjective, and it mainly evaluates the company’s moral standard rather than technical ability. “Zhang San is strict and takes over if he violates the traffic rules. A car accident takes over.”
Because standards are subjective, every company can play around with words in their reports.
Hou Xiaodi believes that driving mileage on complex urban roads is completely different from driving miles in empty suburbs, which is not included in the report.
“If you don’t need to take over this test version for thousands of miles, it means that the Feature has not encountered problems, then it is time to develop the next version. The meaning of testing is to find problems, and constantly find problems to solve them before entering the stable version iteration period. ,”He says.
Therefore, the frequency of takeovers is not meaningful for autonomous driving in the test phase, and the experience of test rides should be compared. So Tucson stopped submitting reports this year.
Before this year’s DMV report came out, the helmsmen from Cruise and Aurora fired their guns early.
△Dan Ammann
Dan Ammann, CEO of GM’s Cruise subsidiary, bluntly said at an investor conference Wednesday: “There’s no really good way to show the progress of the company other than its own internal data.”
Cruise co-founder Kyle Vogt also mentioned the problem of “swiping mileage” in his own blog post. He believes that disengagement often occurs in several different situations:
1. Emergency situations, such as someone or a car suddenly rushed out in front of the car;
2. The situation is not really urgent, but the security officer is too cautious to take over, just like your mother always sees you open fire and cook and does not worry about taking over;
3. Be polite to other cars on the road. After all, automatic driving is different from human driving. If other car owners feel awkward or confused, the safety officer will intervene;
4. The automatic driving system makes an error, and if it does not take over, an accident will occur.
Vogt believes that only the fourth situation is the one where the self-driving system is really problematic, so it would be unfair to include the first three.
Therefore, the disengagement frequency cannot measure whether the automatic driving is ready for commercial landing. It should compare the difference between the automatic driving car and the human driving to see if the automatic driving is easier to use than the human driving.
After all, human beings are now often involved in crashes while driving.
△Chris Urmson
Aurora co-founder Chris Urmson was once the founding CTO of Google’s self-driving car.
He thinks the problem is two-fold. On the one hand, there is no clear definition of what constitutes a disengagement, so the numbers are meaningless.
On the other hand, companies may be developing new features or testing them in more complex road conditions, which will inevitably lead to higher disengagement frequency.
“The number of disengagements driving 100 million miles in a flat, dry area with no cars, people, and intersections is not comparable to the number of disengagements driving 100 miles in a city as complex as Pittsburgh,” Urmson said.
As such, Aurora says it measures technological progress internally with engineering velocity, not disengagement.
After the above-mentioned doubts and diss, the DMV report came out this morning, and Waymo also joined the bombardment camp.
Waymo tweeted this morning that the disengagement metrics do not provide relevant insights into the capabilities of Waymo drivers and do not believe the (California) disengagement data should be used to compare performance.
What do you think of this year’s report?
So so much diss, how should you view this year’s report?
Perhaps there are two dimensions that can be used as reference, one is the whole, and the other is the “progress” of each family.
Speaking of one thousand and ten thousand, DMV is still the only organization that can continuously track the status of the global unmanned vehicle industry. From this data, we can also see some of the development of the industry.
There were only 28 self-driving car companies reporting in 2018, and 36 this year.
From the road test data, there is a rapid growth. In 2018, a total of 3,258,074 kilometers were tested in California, and this year it reached 4,626,627 kilometers, an increase of 42%.
The road test mileage has increased, but the overall number of takeovers has declined.
The overall number of takeovers in 2018 was 143,720, and this year it was only 9,107, a decrease of 93.7%.
Calculated based on the data on how long an average takeover can drive a driverless car. In 2018, it was 22.7 kilometers. After running a circle around the Fifth Ring Road (98 kilometers) in Beijing, manual takeovers were required 4-5 times. This year, it reached 508 kilometers—running five laps in the Fifth Ring Road in Beijing without taking over.
The above overall data can be regarded as boosting confidence in the unmanned vehicle industry. Of course, there is a premise: the standards for each company to submit reports are not significantly different from last year.
Secondly, the throne has changed hands and the new stars are prominent, and it can be seen that the progress of each company’s performance curve.
According to the situation of each company, we have vertically sorted out the growth rate from 2018 to 2019:
Among them, there are a total of 7 companies with a growth rate of more than 5 times, as shown in the following figure:
Except for Apple and Qualcomm, all are from China.
There are also several companies “bucking the trend”, including Zoox, WeRide, Weride, Aurora, Drive.ai, Nissan, Nullmax, and Nvidia.
Of course, some of these companies have encountered difficulties. For example, Drive.ai, which is owned by Mr. Wu Enda, was sold to Apple.
In addition, in view of the fact that many industry experts believe that how far the DMV runs on average, it is unreasonable to manually take over the test center. It may be that these companies have increased the difficulty of the test for themselves, and went to places with more people, more traffic jams, and more dangerous roads to test.
That is to say, several “regressed” companies actually have very high requirements on themselves. The test is to meet the problems and challenges, so under the indicator of separation, the data has regressed.
Apple and Uber, which counted down last year, have also changed this year. Apple has increased from 1.8 to 189.8, an increase of 104 times, which may also be the real force.
Uber’s self-driving business, which has been keeping a low profile in 2019 because of human accidents, did not appear this time.
Finally, in this report disclosure, there are two other Chinese forces that will be disqualified from the California road test because they have not submitted the takeover report:
One is Roadstar, after all the company has been wound up.
The other is Xmotors.ai – Xiaopeng Motors. Xiaopeng’s specific reason was not disclosed further.
In short, this is this year’s DMV autonomous driving “strength” list. I wonder what your evaluation is after reading it?
Report link:
https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/disengagement_report_2019
As soon as the 2019 “strength” report of autonomous driving came out, I had to take a quick look and couldn’t help but boil.
The industry has made rapid progress – from total mileage to average takeover, unmanned vehicles seem to be in a “drag racing” mode compared to last year. More importantly, Chinese companies have performed very well. According to the average takeover index, 4 of the top 10 companies in the world are from China.
Baidu’s North American autopilot team, with a separate rudder, defeated Waymo at the location of Google’s base camp and reached the top for the first time.
That’s right, this is the “2019 Autonomous Driving Takeover Report” just released by the California Department of Vehicles (DMV), and the above highlights are not all.
After the report was published, Waymo, which lost the Iron Throne for the first time, quit.
Waymo publicly stated that it opposed the data requirements and ability judgment criteria of this official report.
So what is going on, let’s start with the report.
Baidu overtakes Google’s Waymo, Chinese company making rapid progress
California, the United States, is the most enlightened area in the world for autonomous driving road testing. Currently, 64 unmanned vehicle companies from around the world have been assembled and approved for road testing.
As a result, the California Department of Vehicles (DMV)’s annual “2019 Autonomous Driving Takeover Report” has received special attention-because no other report can better show a horizontal comparison.
It’s like everyone is competing in the same venue and scene, and finally the organizing committee releases an annual report, which can be used as a reference for strength.
It’s just that this report is also criticized. It is submitted by various testing companies independently, and the core indicators are difficult to quantify. It can be understood that this is a report that each company is a judge by itself, and each company uses its own conscience.
So what about this year’s report? First of all, at the player level, a total of 36 unmanned vehicle companies submitted reports.
Secondly, on the whole, there has been great development and great progress. On average, it takes 508 kilometers of unmanned driving to take over once, an increase of 21 times compared with 2018.
In addition, there are new conclusions that are different from the past: Baidu’s North American team has pulled Waymo from the first throne for the first time, and Chinese companies have performed very well.
The specific measurement indicator is naturally the number of disengagements.
What is “Out of Takeover”? That is to say, in the process of automatic driving, the human safety officer grabbed the control right, let the automatic driving system and the vehicle disengage, and the human took over the driving. It often occurs when dealing with various situations that the automatic driving system cannot handle.
To expand, it is how many cars in total, how far the cars have run, and how many times people have taken over.
Then you can figure out how far you need to take over on average.
Focus on what you need to test. Later, we will discuss how far you need to take over this average run.
From this indicator alone, the data results are as follows:
The abscissa is too long, and since it would be inconvenient to view on a mobile phone, let’s show the top 10 companies:
And the companies after the top ten:
Note that the blue histogram is for 2019 (that is, the latest data), and the orange histogram is for 2018 (a year ago).
As can be seen from the above charts, Baidu USA is the first, with an average of 29,060 kilometers to be taken over once – that means running around the equator for two laps, it only needs to take over three times.
And Google sibling Waymo, which ranks second this year, needs to be taken over once on average for 21,283 kilometers.
Cruise, AutoX, and PonyAI were then ranked third, fourth, and fifth globally.
Among the top five, Chinese companies such as Baidu, AutoX and PonyAI are not yet fully displayed in California, but the results are already astonishing.
Other Chinese companies are also doing very well. Didi’s self-driving car appeared in the report for the first time, ranking eighth, WeRide Weride ranked 11th, and SAIC, the third-to-last last year, ranked 17th.
Compared to the 2018 data, the progress is really great:
Baidu climbed to No. 1 from No. 7. AutoX, progressed from 9th to 4th. There is also Pony AI. Although the ranking has not changed this year, its performance has also increased by more than 5 times.
Last year’s champions and runners-up, Waymo and Cruise, although made progress, did not have the 87-fold increase of Baidu, ranking second and third.
Of course, there are people who quit with this result.
Executives and corporate groups speak out: DMV statistics are inappropriate
Before, the statistical methods and standards of the DMV report have been controversial.
And just recently, there are several core players who have formed a group to protest with their real names. Their main point of questioning is whether it is reasonable to use the “number of takeovers” as the core assessment indicator.
Literally, the more frequent takeovers, the less time the autopilot system works, and it seems that the autopilot company is less powerful.
△ Hou Xiaodi
Last year, Hou Xiaodi, co-founder & CTO of TuSimple, expressed many times: the standard of separation is very subjective, and it mainly evaluates the company’s moral standard rather than technical ability. “Zhang San is strict and takes over if he violates the traffic rules. A car accident takes over.”
Because standards are subjective, every company can play around with words in their reports.
Hou Xiaodi believes that driving mileage on complex urban roads is completely different from driving miles in empty suburbs, which is not included in the report.
“If you don’t need to take over this test version for thousands of miles, it means that the Feature has not encountered problems, then it is time to develop the next version. The meaning of testing is to find problems, and constantly find problems to solve them before entering the stable version iteration period. ,”He says.
Therefore, the frequency of takeovers is not meaningful for autonomous driving in the test phase, and the experience of test rides should be compared. So Tucson stopped submitting reports this year.
Before this year’s DMV report came out, the helmsmen from Cruise and Aurora fired their guns early.
△Dan Ammann
Dan Ammann, CEO of GM’s Cruise subsidiary, bluntly said at an investor conference Wednesday: “There’s no really good way to show the progress of the company other than its own internal data.”
Cruise co-founder Kyle Vogt also mentioned the problem of “swiping mileage” in his own blog post. He believes that disengagement often occurs in several different situations:
1. Emergency situations, such as someone or a car suddenly rushed out in front of the car;
2. The situation is not really urgent, but the security officer is too cautious to take over, just like your mother always sees you open fire and cook and does not worry about taking over;
3. Be polite to other cars on the road. After all, automatic driving is different from human driving. If other car owners feel awkward or confused, the safety officer will intervene;
4. The automatic driving system makes an error, and if it does not take over, an accident will occur.
Vogt believes that only the fourth situation is the one where the self-driving system is really problematic, so it would be unfair to include the first three.
Therefore, the disengagement frequency cannot measure whether the automatic driving is ready for commercial landing. It should compare the difference between the automatic driving car and the human driving to see if the automatic driving is easier to use than the human driving.
After all, human beings are now often involved in crashes while driving.
△Chris Urmson
Aurora co-founder Chris Urmson was once the founding CTO of Google’s self-driving car.
He thinks the problem is two-fold. On the one hand, there is no clear definition of what constitutes a disengagement, so the numbers are meaningless.
On the other hand, companies may be developing new features or testing them in more complex road conditions, which will inevitably lead to higher disengagement frequency.
“The number of disengagements driving 100 million miles in a flat, dry area with no cars, people, and intersections is not comparable to the number of disengagements driving 100 miles in a city as complex as Pittsburgh,” Urmson said.
As such, Aurora says it measures technological progress internally with engineering velocity, not disengagement.
After the above-mentioned doubts and diss, the DMV report came out this morning, and Waymo also joined the bombardment camp.
Waymo tweeted this morning that the disengagement metrics do not provide relevant insights into the capabilities of Waymo drivers and do not believe the (California) disengagement data should be used to compare performance.
What do you think of this year’s report?
So so much diss, how should you view this year’s report?
Perhaps there are two dimensions that can be used as reference, one is the whole, and the other is the “progress” of each family.
Speaking of one thousand and ten thousand, DMV is still the only organization that can continuously track the status of the global unmanned vehicle industry. From this data, we can also see some of the development of the industry.
There were only 28 self-driving car companies reporting in 2018, and 36 this year.
From the road test data, there is a rapid growth. In 2018, a total of 3,258,074 kilometers were tested in California, and this year it reached 4,626,627 kilometers, an increase of 42%.
The road test mileage has increased, but the overall number of takeovers has declined.
The overall number of takeovers in 2018 was 143,720, and this year it was only 9,107, a decrease of 93.7%.
Calculated based on the data on how long an average takeover can drive a driverless car. In 2018, it was 22.7 kilometers. After running a circle around the Fifth Ring Road (98 kilometers) in Beijing, manual takeovers were required 4-5 times. This year, it reached 508 kilometers—running five laps in the Fifth Ring Road in Beijing without taking over.
The above overall data can be regarded as boosting confidence in the unmanned vehicle industry. Of course, there is a premise: the standards for each company to submit reports are not significantly different from last year.
Secondly, the throne has changed hands and the new stars are prominent, and it can be seen that the progress of each company’s performance curve.
According to the situation of each company, we have vertically sorted out the growth rate from 2018 to 2019:
Among them, there are a total of 7 companies with a growth rate of more than 5 times, as shown in the following figure:
Except for Apple and Qualcomm, all are from China.
There are also several companies “bucking the trend”, including Zoox, WeRide, Weride, Aurora, Drive.ai, Nissan, Nullmax, and Nvidia.
Of course, some of these companies have encountered difficulties. For example, Drive.ai, which is owned by Mr. Wu Enda, was sold to Apple.
In addition, in view of the fact that many industry experts believe that how far the DMV runs on average, it is unreasonable to manually take over the test center. It may be that these companies have increased the difficulty of the test for themselves, and went to places with more people, more traffic jams, and more dangerous roads to test.
That is to say, several “regressed” companies actually have very high requirements on themselves. The test is to meet the problems and challenges, so under the indicator of separation, the data has regressed.
Apple and Uber, which counted down last year, have also changed this year. Apple has increased from 1.8 to 189.8, an increase of 104 times, which may also be the real force.
Uber’s self-driving business, which has been keeping a low profile in 2019 because of human accidents, did not appear this time.
Finally, in this report disclosure, there are two other Chinese forces that will be disqualified from the California road test because they have not submitted the takeover report:
One is Roadstar, after all the company has been wound up.
The other is Xmotors.ai – Xiaopeng Motors. Xiaopeng’s specific reason was not disclosed further.
In short, this is this year’s DMV autonomous driving “strength” list. I wonder what your evaluation is after reading it?
Report link:
https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/disengagement_report_2019
The Links: SKD110-16 DMF-50773NF-FW