Hi,

We are in the process of deploying BIOS updates to all of our Lenovo laptops. In total around 300.

The updates are deployed through Config Manager aka SCCM / MECM.

With this deployment, the system is being prepped with a BIOS update, which will flash on the next reboot.

Before executing, the script will evaluate model and BIOS version and date, to ensure we only flash models not already flashed to latest version and to ensure we flash the correct BIOS file corresponding with the hardware model. In close to 100% of cases there's a success on this task when reading return codes and output from the script

So far, so good.

Following this, we've observed the following:

Around 80% of cases the BIOS is flashed correctly on the next reboot. Everything is as it should be.

Around 10% of cases the BIOS is not being updated on the next reboot. Nothing happens. This can fixed by attempted to flash the BIOS again. (Mostly)

Around 10% of cases the BIOS starts flashing, but gets stuck in "Flashing embedded controller". This is stuck for hours and will not clear until you do a reset of the computer with a pin through the whole on the bottom side of the laptop.

As far as I can tell, the last issue is very common, and has been occurring for many months, through other entries I've read on this forum and others. Some of these has been responded to by Lenovo representatives, but the "solution" is always to put a pin through the reset hole after waiting several hours.

Obviously, in an enterprise setup, the above solution as absolutely unacceptable. On top of this, a failure rate of 20% is simply too high.

Models we've observed this with ranges from T14s gen 1 and 2, X1 carbon gen 9, Yoga gen 6. I don't consider this to be limited to these models, it's simply where we've seen it issue occur, but since the issue seems to be very random in nature, it could be down to simple luck we haven't observed it on other models.

My questions are: Have anyone been able to find a stable solution to the above scenario? Are Lenovo acknowledging this issue and working on a solution?

Hi Rasmus.

We have experienced an almost identical scenario with our T14 Gen 2 using Lenovo Patch and SCCM to apply 1.52. ThinkPad-T14-Gen2-BIOS-Update-v1-52-failing - English Community - LENOVO COMMUNITY. 

As yet Lenovo has not been able to assist us with narrowing down on what the issue could be. Currently my working theory is that users are carrying out a "hybrid shutdown" prior to the deployment deadline reboot.  This is an excerpt from the readme for the 1.52 update "If hybrid shutdown is made after the BIOS update on Windows 10, the update will not be done correctly." - https://download.lenovo.com/pccbbs/mobiles/n34uj18w.txt

We have since removed all BIOS updates from Software Update Groups scheduled for the other models in our fleet. I am not sure what the solution is for keeping our fleet BIOS versions up to date without causing major disruptions to users. Forcing a reboot directly after a BIOS update during the working day is not currently an option. Leaving the reboot to after working hours means that the necessary reboot prior to a "hybrid shutdown" cannot be guaranteed.

 

 

 

Hi,

Thank you for your response.

It seems indeed the issues you are experiencing are similar to ours. Hybrid shutdown could be the culprit for sure. I will have to run some tests before concluding anything on that though. But it is not something we've considered in our deployment. The notion was simply that if they prep flash had executed, the BIOS would flash on the next proper reboot, which would occur within a month due to software updates from Microsoft.

I will consider adding a forced reboot, but not entirely sold on the idea, as it will highly inconvenience the end user.

However, from my observations, we've had better results with BIOS flashes during PXE, where it runs as one of the last steps in the task sequence during OSD. This step has a forced reboot, and I don't recall having any problems in this scenario.

Still not ideal at all, and would prefer that Lenovo increased the robustness of BIOS updates.

@Rasmus W @march1977

We are investigating this issue on systems.  We will report back when we have more information.

TLawson

 

I have deployed the update to my T14 Gen 2i devices.  Another engineer has deployed the update to their environment.  Neither of us have been able to reproduce the issue.

We are still looking into this issue.

TLawson

Upon further review, the update is being applied using normal installation procedures.  It appears that the content within the update is failing to complete the Embedded Controller correctly.  Please create a case with our support team for this issue.

 

TLawson

I agree, it is not always possible to reproduce the error, because it is very random.

Adding to my previous post, I will say that the observations I've had with updating BIOS on an endpoint in use differ from BIOS updating an endpoint during OSD.

I don't think I recall having a single error during OSD. However, in the case of updating an endpoint in use is a whole different thing. The failure rate is massive. This could easily have to do with user behavior after applying the BIOS update before restarting. If the endpoint is not booted immediately after applying the update, the users actions might unintentionally prevent the update from being applied properly on the next boot. This is why I ask if you could add more robustness to the BIOS application. The task sequence used to update the BIOS during OSD and an endpoint in use is the exact same, meaning it's the same WMI logic determining if the endpoint is applicable.

Of course, you could argue that we should just boot the endpoints immediately after applying the update, but that would be very disruptive for the user.

For some statistics, I deployed the update to our entire company. Of 303 Lenovo devices, 48 are marked as compliant with latest BIOS. I pulled this data from the SCCM hardware inventory today. It is now three weeks ago I deployed it, and almost all of the devices has been restarted since applying the update. Of models in our inventory, T14s Gen 1 by far the majority. Not that I've noticed any correlation between model and failure rate.

Are there any updates on this issue? We are facing the same while updating BIOS on several hundred machines - there is a huge error rate. Same picture as Rasmus W is describing.

After speaking to Lenovo we have decided not to push out BIOS updates via Lenovo Patch as we were not enforcing an immediate reboot. This meant that users were using Windows 10 hybrid (fast boot) shoutdown which causes this failure upon boot. No matter which meathod you decide to use for updating BIOS ensure an immediate hard reboot is carried out