Welcome to our peer-to-peer forums, where owners help owners. Need help now? Visit eSupport here.

English Community

Linux Operating SystemsOther Linux Discussions
All Forum Topics
Options

584 Posts

07-22-2019

Canada

439 Signins

4801 Page Views

  • Posts: 584
  • Registered: ‎07-22-2019
  • Location: Canada
  • Views: 4801
  • Message 21 of 35

Re:Unusable TSC on P14s and X13 with the latest LTS kernel

2021-05-19, 23:46 PM

Thanks for all the details and workaround Gabriel.

 

I've raised a ticket to the firmware team. (For my reference  LO-1237).

I'll update when I hear back from them. 

 

Mark

Reply
Options

584 Posts

07-22-2019

Canada

439 Signins

4801 Page Views

  • Posts: 584
  • Registered: ‎07-22-2019
  • Location: Canada
  • Views: 4801
  • Message 22 of 35

Re:Unusable TSC on P14s and X13 with the latest LTS kernel

2021-05-20, 13:59 PM

Hi @osnix 

 

With the 'shutdown -h now' step is it 100% reproducible for you, or does it take a bunch of attempts?

The FW team were unable to reproduce in a dozen attempts. I updated my BIOS this morning (I had been waiting on the LVFS update due out very soon, but went the USB stick route) and so far I've not seen it - I think I'm on attempt 9. Note - I'm on a T14 AMD but that's essentially the same as the P14s.

I'll keep trying throughout the day but wanted to get an idea of how persistent I needed to be.

 

@xkrx - For the L15 AMD, that's not a Linux certified platform so I don't have access to one and more limited levers to pull on with the firmware team I'm afraid. Looks like they're rolling out the BIOS Linux fixes there anyway which is *awesome* but just want to set expectations.

 

Mark

Reply
Options

343 Posts

03-06-2021

Germany

187 Signins

2065 Page Views

  • Posts: 343
  • Registered: ‎03-06-2021
  • Location: Germany
  • Views: 2065
  • Message 23 of 35

Re:Unusable TSC on P14s and X13 with the latest LTS kernel

2021-05-20, 14:31 PM

@ MarkRHPearson wrote:

Hi @osnix 

 

With the ' -h now' step is it 100% reproducible for you, or does it take a bunch of attempts?

The FW team were unable to reproduce in a dozen attempts. I updated my BIOS this morning (I had been waiting on the LVFS update due out very soon, but went the USB stick route) and so far I' not seen it - I think I' on attempt 9. Note - I' on a T14 AMD but that' essentially the same as the P14s.

I' keep trying throughout the day but wanted to get an idea of how persistent I needed to be.

 

@xkrx - For the L15 AMD, that' not a Linux certified platform so I don' have access to one and more limited levers to pull on with the firmware team I' afraid. Looks like they' rolling out the BIOS Linux fixes there anyway which is *awesome* but just want to set expectations.

 

Mark

 

For me is always reproducible. Any shutdown / poweroff ( or even using systemctl < command > ) 

will trigger that.

 

IOW, shutdown, boot , bug is there, use reboot bug is gone :)

 

I updated using the iso file with a USB stick too. Maybe is a BIOS setting we have different?

Kernel config maybe? You can have mine but is a hell of a config since is a distro kernel.

 

https://gitweb.frugalware.org/frugalware-current/raw/master/source/base/kernel/config.x86_64

 

I just updated to the latest 5.12.5 today and tested with this one too, the same thing.

 

What makes me wonder is the fact a 'reboot' command will fix it.

From my memory, there was some change in the shutdown code in Linux

sometimes around 5.10 ( I think ), but that should not break a BIOS.

 

Like you saw is not only me hitting it. ( even the user doesn't have a Linux certified platform

the bug is the same ). 

 

Just let me know if I can do something else to help to find the cause of this.

 

I don't know about your internal rules, but I'm willing to test whatever you wish

inclusive beta BIOS(es), run tools or build kernels the way you do that internally.

 

Gabriel

 

Reply
Options

343 Posts

03-06-2021

Germany

187 Signins

2065 Page Views

  • Posts: 343
  • Registered: ‎03-06-2021
  • Location: Germany
  • Views: 2065
  • Message 24 of 35

Re:Unusable TSC on P14s and X13 with the latest LTS kernel

2021-05-20, 14:40 PM

This is from 5.12.5. ( shutdown -h , and boot up again ):

 

[    1.648775] clocksource: Switched to clocksource tsc
[    4.436868] clocksource: timekeeping watchdog on CPU7: Marking clocksource 'tsc' as unstable because the skew is too large:
[    4.436896] clocksource:                       'tsc' cs_now: 7ce25f6da cs_last: 778c6dddb mask: ffffffffffffffff
[    4.436906] tsc: Marking TSC unstable due to clocksource watchdog
[    4.436936] TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'.

 

It is not always the same CPU is out of sync so there has to be something else going on.

 

Here a small sample of one of my testing partitions:

 

crazy@ThinkPad-P14s:~$ sudo journalctl | grep timekeeping
[sudo] password for [crazy@ThinkPad-P14s]: 
Mai 19 00:03:36 frugalware kernel: clocksource: timekeeping watchdog on CPU1: Marking clocksource 'tsc-early' as unstable because the skew is too large:
Mai 19 17:43:20 ThinkPad-P14s kernel: clocksource: timekeeping watchdog on CPU7: Marking clocksource 'tsc' as unstable because the skew is too large:
Mai 19 17:44:21 ThinkPad-P14s kernel: clocksource: timekeeping watchdog on CPU6: Marking clocksource 'tsc' as unstable because the skew is too large:
Mai 19 17:45:40 ThinkPad-P14s kernel: clocksource: timekeeping watchdog on CPU7: Marking clocksource 'tsc' as unstable because the skew is too large:
Mai 19 17:49:17 ThinkPad-P14s kernel: clocksource: timekeeping watchdog on CPU7: Marking clocksource 'tsc' as unstable because the skew is too large:
Mai 19 17:52:27 ThinkPad-P14s kernel: clocksource: timekeeping watchdog on CPU3: Marking clocksource 'tsc' as unstable because the skew is too large:
Mai 19 17:53:44 ThinkPad-P14s kernel: clocksource: timekeeping watchdog on CPU7: Marking clocksource 'tsc' as unstable because the skew is too large:
Mai 19 17:55:35 ThinkPad-P14s kernel: clocksource: timekeeping watchdog on CPU7: Marking clocksource 'tsc' as unstable because the skew is too large:
Mai 19 18:23:20 ThinkPad-P14s kernel: clocksource: timekeeping watchdog on CPU7: Marking clocksource 'tsc' as unstable because the skew is too large:
Mai 19 18:27:14 ThinkPad-P14s kernel: clocksource: timekeeping watchdog on CPU7: Marking clocksource 'tsc' as unstable because the skew is too large:
Mai 19 18:31:46 ThinkPad-P14s kernel: clocksource: timekeeping watchdog on CPU7: Marking clocksource 'tsc' as unstable because the skew is too large:
Mai 19 18:44:49 ThinkPad-P14s kernel: clocksource: timekeeping watchdog on CPU6: Marking clocksource 'tsc' as unstable because the skew is too large:
Mai 19 18:50:24 ThinkPad-P14s kernel: clocksource: timekeeping watchdog on CPU6: Marking clocksource 'tsc' as unstable because the skew is too large:
Mai 19 19:02:29 ThinkPad-P14s kernel: clocksource: timekeeping watchdog on CPU6: Marking clocksource 'tsc' as unstable because the skew is too large:
Mai 19 19:04:55 ThinkPad-P14s kernel: clocksource: timekeeping watchdog on CPU7: Marking clocksource 'tsc' as unstable because the skew is too large:
Mai 20 16:12:00 ThinkPad-P14s kernel: clocksource: timekeeping watchdog on CPU7: Marking clocksource 'tsc' as unstable because the skew is too large:

Reply
Options

83 Posts

12-26-2020

Austria

47 Signins

655 Page Views

  • Posts: 83
  • Registered: ‎12-26-2020
  • Location: Austria
  • Views: 655
  • Message 25 of 35

Re:Unusable TSC on P14s and X13 with the latest LTS kernel

2021-05-20, 14:49 PM

@All,

 

Let's try to collate the available data and see if there is a relation between a certain configuration and reproducibility of a bug.

Spreadsheet: https://docs.google.com/spreadsheets/d/1alrg1lvoLqdZxqYGin5Uzqhc0pQxf1_PLDeSTMjQxNw/edit?usp=sharing

 

Please feel free to PM me your GMail address and I will add you as a collaborator.

Reply
Options

584 Posts

07-22-2019

Canada

439 Signins

4801 Page Views

  • Posts: 584
  • Registered: ‎07-22-2019
  • Location: Canada
  • Views: 4801
  • Message 26 of 35

Re:Unusable TSC on P14s and X13 with the latest LTS kernel

2021-05-20, 15:07 PM

Kernel version makes a difference...

 

I was testing with our Ubuntu 20.04 OEM image updated to latest (so 5.8 kernel, but all sorts of patches that Canonical add) and wasn't seeing the problem.

By pure happenstance I was working with AMD on the docking issues we have yesterday and had built a vanilla 5.12.4 for running some tests....and I see the problem right away there.

Explains why the firmware team won't have seen it too - they won't be building their own kernels.

 

I'll do some digging....maybe a bisect (bleurgh...). I don't know this area of the kernel at all I'm afraid.

 

Mark

Reply
Options

343 Posts

03-06-2021

Germany

187 Signins

2065 Page Views

  • Posts: 343
  • Registered: ‎03-06-2021
  • Location: Germany
  • Views: 2065
  • Message 27 of 35

Re:Unusable TSC on P14s and X13 with the latest LTS kernel

2021-05-20, 15:40 PM

@ MarkRHPearson wrote:

Kernel version makes a difference...

 

I was testing with our Ubuntu 20.04 OEM image updated to latest (so 5.8 kernel, but all sorts of patches that Canonical add) and wasn' seeing the problem.

By pure happenstance I was working with AMD on the docking issues we have yesterday and had built a vanilla 5.12.4 for running some tests....and I see the problem right away there.

Explains why the firmware team won' have seen it too - they won' be building their own kernels.

 

I' do some digging....maybe a bisect (bleurgh...). I don' know this area of the kernel at all I' afraid.

 

Mark

 

Distro kernels .. ( and the Topic I still didn't start about =) )

 

While I use the distro config I won't use patches other than

things to unlock a CONFIG_* option when testing something like this.

 

Mark, to be honest, Lenovo needs to test on vanilla kernel.org kernels too.

Distributions ( old, new, Frankenstein ) kernels are irrelevant to kernel.org ones.

 

Ubuntu, RH, Debian, etc are adding not only weird backports but their own stuff ( sometimes broken ),

in which case you won't see a bug, or trigger it differently, or hit one no one else sees.

 

A simple bash/C/C++/rust/go script/program to build the latest vanilla kernel is easy to implement,

and then even the FW team(s) could run a: ./tool --build latest, drink a tea, reboot to the latest kernel

and run their tests.

 

OFC just my 2c but I think such bugs will bring us always back to Distro vs Linux support ;)

 

Gabriel

Reply
Options

6 Posts

03-25-2021

Croatia

6 Signins

15 Page Views

  • Posts: 6
  • Registered: ‎03-25-2021
  • Location: Croatia
  • Views: 15
  • Message 28 of 35

Re:Unusable TSC on P14s and X13 with the latest LTS kernel

2021-05-21, 6:59 AM

@ osnix wrote:

Thx, could you test something maybe ?

 

1) open a terminal and use reboot command to reboot the box

2) open a terminal and verify the bug is not there use: dmesg | grep tsc

3) shut down the box using shutdown -h now

4) wait at least 10sec then power on the box

5) verify you hit the bug: dmesg | grep tsc

 

Tried it 3-4 times, a shutdown seems to produce the problem on next boot as you described.

 

If you could reproduce that way then I may have a workaround for now.

 

Append the following to your kernel command line ( better grub & rebuild your grub.cfg & initrd ):

 

tsc=nowatchdog

 

tsc=nowatchdog or reliable options should stop the kernel from detecting and logging it, but it won't fix the problem itself ?

 

Reply
Options

6 Posts

03-25-2021

Croatia

6 Signins

15 Page Views

  • Posts: 6
  • Registered: ‎03-25-2021
  • Location: Croatia
  • Views: 15
  • Message 29 of 35

Re:Unusable TSC on P14s and X13 with the latest LTS kernel

2021-05-21, 7:16 AM

@ MarkRHPearson wrote:

@xkrx - For the L15 AMD, that' not a Linux certified platform so I don' have access to one and more limited levers to pull on with the firmware team I' afraid. Looks like they' rolling out the BIOS Linux fixes there anyway which is *awesome* but just want to set expectations.

 

I understand and I would not ask for driver support.

However a reliable TSC is a hardware feature, and this seems to be a BIOS problem which should cause problems on Windows too.

Hopefully it will get fixed along with all the others.

 

Reply
Options

343 Posts

03-06-2021

Germany

187 Signins

2065 Page Views

  • Posts: 343
  • Registered: ‎03-06-2021
  • Location: Germany
  • Views: 2065
  • Message 30 of 35

Re:Unusable TSC on P14s and X13 with the latest LTS kernel

2021-05-21, 7:51 AM

@ xkrx wrote:

@ osnix wrote:

Thx, could you test something maybe ?

 

1) open a terminal and use reboot command to reboot the box

2) open a terminal and verify the bug is not there use: dmesg | grep tsc

3) shut down the box using shutdown -h now

4) wait at least 10sec then power on the box

5) verify you hit the bug: dmesg | grep tsc

 

Tried it 3-4 times, a shutdown seems to produce the problem on next boot as you described.

 

If you could reproduce that way then I may have a workaround for now.

 

Append the following to your kernel command line ( better grub & rebuild your grub.cfg & initrd ):

 

tsc=nowatchdog

 

tsc=nowatchdog or reliable options should stop the kernel from detecting and logging it, but it won' fix the problem itself ?

 

 

No, it won't fix the problem, it just disables the watchdog clock on tsc... 

Like I've said is just a workaround for now.

Reply
Forum Home

Community Guidelines

Please review our Guidelines before posting.

Learn More

Check out current deals!

Go Shop
X

Save

X

Delete

X

No, I don’t want to share ideas Yes, I agree to these terms