cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
compudj
Fanfold Paper
Posts: 8
Location: Montreal, Canada
Views: 8,838
Message 1 of 65

x230: SATA errors with 180GB Intel 520 SSD under heavy write load

Hi,

We spent a couple of days cornering what appears to be an issue with the Intel 520 SSD drives in Lenovo x230 laptops. It was first showing up on a clean Debian installation, while installing a guest operating system into a VM. Looking around on forums, there appears to be some people having issues with database workloads too. So I decided to create a small user-space program to repoduce the problem. IMPORTANT: Before you try it, be ready for a system crash requiring a cold reboot.. It's available at:

git://git.efficios.com/test-ssd.git

direct link to .c file:
https://git.efficios.com/?p=test-ssd.git;a=blob;f=test-ssd-write.c;hb=refs/heads/master

This program simply performs random-access-writes of 4Kb into a single file.

Executive summary of our findings (the details are in the test-ssd-write.c header in the git repo):

- We reproduced this issue on 4 x230 machines (all our x230 have 180GB Intel drives, and they are all affected),
- We took a SSD from one of the machines, moved it into an x200, and the problem still occurs,
- The problem seems to occur independently of the filesystem (reproduced on ext3 and ext4),
- Problem reproduced by test-ssd-write.c (git tree above): After less than 5 minutes of the heavy write workload, we get SATA errors and we need to cold reboot the machine to access the drive again. Example usage (don't forget to prepare for a computer freeze):

  ./test-ssd-write somefileondisk 209715200 1234 -z

  (see options by just running ./test-ssd-write)

The problem occurs with drive model SSDSC2BW180A3L, with both firmwares LE1i and LF1i (those are Lenovo firmwares). We could reproduce the issue on 3.2 (Debian), 3.5 (Debian), 3.7.9 (Arch) distribution kernels. We could reproduce it with x230 BIOS G2ET90WW (2.50) 2012-20-12 and G2ET86WW (2.06) 2012-11-13, but since it can be reproduced on a x200 too, it does not appear to be a BIOS issue.

We tried the program on a range of other SSD drives, one of those including the same SandForce 2281 controller (details within test-ssd-write.c header). So our current guess is that the Lenovo firmware on the SSD might be part of the problem, but it might be good if we could to confirm that Intel's firmwares work fine.

Thoughts, ideas, hints about who to contact on this issue would be very much welcome,

Thanks,

Mathieu

jde
What's DOS?
Posts: 1
Location: montreal
Views: 8,743
Message 2 of 65

Re: x230: SATA errors with 180GB Intel 520 SSD under heavy write load

I can confirm the exact same problem with Debian and Ubuntu.

I can reproduce it easily on Debian by installing a VM.

On Ubuntu it is harder to reproduce but I already hit this problem when installing updates (dpkg does a lot of sync/fdatasync just like your test program).


I think the difference between the 2 distros that explain why it is easier to reproduce on one is the default I/O scheduler (cfq for Debian and deadline for Ubuntu).

 

Lenovo already completely replaced my laptop (X230 with SSDSC2BW180A3L) and I still have this problem.

 

We need a firmware fix.

 

Thanks !

jgalar
Paper Tape
Posts: 1
Location: Canada
Views: 8,737
Message 3 of 65

Re: x230: SATA errors with 180GB Intel 520 SSD under heavy write load

I can also confirm I have the same issue using Arch Linux (which uses the CFQ scheduler).

syrious
Punch Card
Posts: 45
Location: US
Views: 8,678
Message 4 of 65

Re: x230: SATA errors with 180GB Intel 520 SSD under heavy write load

is there maybe a firmware update available for this SSD via intel? I don't expect Lenovo would update the firmware of the drive before installing.

compudj
Fanfold Paper
Posts: 8
Location: Montreal, Canada
Views: 8,667
Message 5 of 65

Re: x230: SATA errors with 180GB Intel 520 SSD under heavy write load

The firmware on the x230 Intel SSD drive is made by Lenovo. The last version available (which I tried) is LF1i, published within their SSD firmware update utility FWSS28 on February 14, 2013.

Indeed, trying the test-case with an Intel firmware might help pinpointing the issue, but I did not deploy non-Lenovo firmware on my SSD drive, fearing it might void my warranty.

Thanks,

Mathieu

syrious
Punch Card
Posts: 45
Location: US
Views: 8,659
Message 6 of 65

Re: x230: SATA errors with 180GB Intel 520 SSD under heavy write load

I was thinking the same thing after I wrote that.

 

I would say as a last result you could probably try the intel firmware, but I believe you are correct in regards to the warranty.  Worst case, unless you brick the SSD, I don't see why you couldn't swap intel and Lenovo firmware, it might run faster! lol

 

Sorry I can't be of more help Mathieu.

stgraber
What's DOS?
Posts: 1
Location: Canada
Views: 8,610
Message 7 of 65

Re: x230: SATA errors with 180GB Intel 520 SSD under heavy write load

FWIW, I've been using my X230 with Intel OEM SSDs and haven't seen a problem with them.

I also have a bunch of SandForce based SSDs from Intel and OCZ that I use in various machine and that don't reproduce the behaviour that Mathieu and Julien showed me, it's therefore quite possible that it's the Lenovo-specific firmware that's the cause of those problems.

mtrudel
What's DOS?
Posts: 1
Location: Canada
Views: 8,565
Message 8 of 65

Re: x230: SATA errors with 180GB Intel 520 SSD under heavy write load

I got the same errors on my own Lenovo X230 (2306-CTO), that I received with an Intel Series 520 180G SSD, also with firmware LE1i. This forced me to replace the drive -- the errors cause users to lose trust in the safety of their data.

 

mwalls
Fanfold Paper
Posts: 16
Location: Norman, Oklahoma
Views: 8,558
Message 9 of 65

Re: x230: SATA errors with 180GB Intel 520 SSD under heavy write load

You might want to look at this thread, sounds like the same problem:

http://forums.lenovo.com/t5/T400-T500-and-newer-T-series/T430s-Intel-SSD-520-180GB-issue/td-p/888083...

-Monty.
compudj
Fanfold Paper
Posts: 8
Location: Montreal, Canada
Views: 8,509
Message 10 of 65

Re: x230: SATA errors with 180GB Intel 520 SSD under heavy write load

Thanks Monthy. I've just posted on their thread so it is known that those two issues are linked.

Top Kudoed Authors