drewp

Corrupt ZFS pool recovery

Posted in zfs by Drew Perttula at 2009-09-25T19:06:43+00:00

Here's a ZFS repair success story:

I'm using zfs-fuse on linux. I had an already-weird setup where a replace operation wouldn't finish:

	  pool: stor3                                                  
	 state: DEGRADED
	config:

		stor3                      DEGRADED
		  mirror                   DEGRADED
		    sdd                    ONLINE
		    replacing              DEGRADED
		      3323475527286891755  UNAVAIL  cannot open
		      sdb                  ONLINE

The disk called sdb was making ATA resets in my dmesg log, and one night when I ran 'fdisk -l', the machine rebooted entirely. I didn't even know linux would do that.

On reboot, the volume wouldn't mount. (These commands are running at the same time in two shells.)

    slash(pts/1):/tmp/zfs/src% sudo /usr/local/sbin/zfs-fuse --no-daemon 
    lib/libzpool/dmu.c:610: dmu_write: Assertion `0 == dmu_buf_hold_array(os, object, offset, 
size, FALSE, FTAG, &numbufs, &dbp)` failed.
    zsh: abort      sudo /usr/local/sbin/zfs-fuse --no-daemon

    slash(pts/3):/tmp/zfs/src# zfs mount stor3
    zfsfuse_ioctl_read_loop(): file descriptor closed
    cannot mount 'stor3': Input/output error

I got a new disk and dd'ed the data from sdd to a 500GB file. It broke around 400GB. I think dd has some don't-stop-on-errors options, but what I really wanted was ddrescue, which logs and retries bad blocks. In the end, I had 77kB of errors in 152 places. This doesn't seem like a high number to me at all. Zfs would have used its mirror to notice those kinds of errors, but for this catastrophic failure I don't (think I) care about that few errors.

I tried a lot of zpool import combinations on zfs-fuse and an opensolaris live cd. Usually the result was that the zpool command would crash zfs-fuse or just hang on some lock. On solaris, I couldn't even kill the zpool command with a ctrl-c.

Here is some search bait showing the kinds of crashes and lockups I could get:

#2  0x00000000004e4020 in vpanic (fmt=0x510648 "zfs: allocating allocated segment(offset=%llu size=%llu)
", adx=0x7f1bd272b6c0)
    at lib/libsolkerncompat/cmn_err.c:53
#3  0x00000000004e414e in vcmn_err (ce=3, fmt=0x510648 "zfs: allocating allocated segment(offset=%llu size=%llu)
", adx=0x7f1bd272b6c0)
    at lib/libsolkerncompat/cmn_err.c:73
#4  0x0000000000488548 in zfs_panic_recover (fmt=0x510648 "zfs: allocating allocated segment(offset=%llu size=%llu)
")
    at lib/libzpool/spa_misc.c:1123
#5  0x00000000004c5c9f in space_map_add (sm=0x7f1bd283f870, start=0, size=1) at lib/libzpool/space_map.c:102
#6  0x00000000004c6d73 in space_map_load (sm=0x7f1bd283f870, ops=0x0, maptype=0 '\0', 
smo=0x7f1bd283fd00, os=0x7f1bd0fb2c80)
    at lib/libzpool/space_map.c:327
#7  0x000000000048eccb in vdev_dtl_load (vd=0x7f1bd283f800) at lib/libzpool/vdev.c:1544
#8  0x000000000048f5eb in vdev_load (vd=0x7f1bd283f800) at lib/libzpool/vdev.c:1708
#9  0x000000000048f56a in vdev_load (vd=0x7f1bd2841000) at lib/libzpool/vdev.c:1694
#10 0x000000000048f56a in vdev_load (vd=0x7f1bd2842000) at lib/libzpool/vdev.c:1694
#11 0x000000000048f56a in vdev_load (vd=0x7f1bd2842800) at lib/libzpool/vdev.c:1694
#12 0x000000000047c7c3 in spa_load (spa=0x7f1bd284f000, config=0x7f1bd2850f20, state=SPA_LOAD_TRYIMPORT, mosconfig=1)
    at lib/libzpool/spa.c:1548
#13 0x000000000047ed9a in spa_tryimport (tryconfig=0x7f1bd2850f20) at lib/libzpool/spa.c:2721
#14 0x000000000040f95b in zfs_ioc_pool_tryimport (zc=0x7f1bd0fdb000) at zfs-fuse/zfs_ioctl.c:1105
#15 0x0000000000413e10 in zfsdev_ioctl (dev=0, cmd=23046, arg=140734269694352, flag=0, cr=0x0, rvalp=0x0) 
at zfs-fuse/zfs_ioctl.c:3670
#16 0x00000000004051cb in cmd_ioctl_req (sock=11, cmd=0x7f1bd272c0a0) at zfs-fuse/cmd_listener.c:49

I found this article with another ZFS recovery story. In his solution, Nathan used dd to carefully write zeros over some of the uberblock structures on the disk. This makes ZFS pick slightly older uberblocks that hopefully aren't corrupt.

Since I already had the zfs-fuse source up, I just made the driver reject the newest uberblocks. This meant I didn't have to edit my 500GB file, which sounded like a pain (especially if I wanted to revert).

int
uberblock_verify(uberblock_t *ub)
{
	if (ub-<ub_magic == BSWAP_64((uint64_t)UBERBLOCK_MAGIC))
		byteswap_uint64_array(ub, sizeof (uberblock_t));

	printf("uberblock_verify block txg %lu time %lu ", ub-<ub_txg, ub-<ub_timestamp);

	if (ub-<ub_txg < 1058110) { 
	  printf("txg too new; skipping
");
	  return EINVAL;
	}

	if (ub-<ub_magic != UBERBLOCK_MAGIC) {
	  printf(" failed magic
");
		return (EINVAL);
	}
	printf(" ok
");
	return (0);
}

The first printf showed me the hundreds of historical (and duplicated) uberblocks on the disk. I picked a TransaXtion Group a few in from the end, and made uberblock_verify fail on the later ones.

This helped, but then spa_history_log failed to write its message to a log on the disk:

#1  0x00000000004e3f89 in cv_wait (cv=0x7f3a8ce775b8, mp=0x7f3a8ce774b0) at lib/libsolkerncompat/condvar.c:59
#2  0x0000000000489e6c in txg_wait_synced (dp=0x7f3a8ce772c0, txg=1058116) at lib/libzpool/txg.c:449
#3  0x0000000000476fad in dsl_sync_task_group_wait (dstg=0x7f3ad2d56af0) at lib/libzpool/dsl_synctask.c:119
#4  0x0000000000477412 in dsl_sync_task_do (dp=0x7f3a8ce772c0, checkfunc=0, 
    syncfunc=0x485099 <spa_history_log_sync>, arg1=0x7f3ad2d5a000, arg2=0x7f3ab0549c50, blocks_modified=0)
    at lib/libzpool/dsl_synctask.c:209
#5  0x0000000000485857 in spa_history_log (spa=0x7f3ad2d5a000, 
    history_str=0x7f3a8d3e5000 "zpool import -d /1t stor3", what=LOG_CMD_NORMAL) at lib/libzpool/spa_history.c:300
#6  0x000000000040da78 in zfs_log_history (zc=0x7f3ad2cc4000) at zfs-fuse/zfs_ioctl.c:252
#7  0x00000000004138fe in zfsdev_ioctl (dev=0, cmd=23042, arg=140734786238112, flag=0, cr=0x0, rvalp=0x0)
    at zfs-fuse/zfs_ioctl.c:3662
#8  0x0000000000404cd7 in cmd_ioctl_req (sock=9, cmd=0x7f3ab054b0a0) at zfs-fuse/cmd_listener.c:49

I disabled spa_history_log by having it return 0 at the top.

Finally, zpool started hanging here, and I didn't know what to do:

#0  0x00007f62696b4746 in pthread_rwlock_rdlock () from /lib/libpthread.so.0
#1  0x00000000004e74e3 in rw_enter (rwlp=0x7f6223cfb7c8, rw=0) at lib/libsolkerncompat/rwlock.c:62
#2  0x000000000046c07e in dsl_dir_open_spa (spa=0x7f6269cb9000, name=0x7f6269c23000 "stor3", tag=0x4fcf00, 
    ddp=0x7f62474a9c08, tailp=0x7f62474a9bf8) at lib/libzpool/dsl_dir.c:314
#3  0x000000000045f44e in dsl_dataset_hold (name=0x7f6269c23000 "stor3", tag=0x7f6269caaf70, dsp=0x7f62474a9c58)
    at lib/libzpool/dsl_dataset.c:566
#4  0x00000000004479c0 in dmu_objset_open (name=0x7f6269c23000 "stor3", type=DMU_OST_ANY, mode=9, 
    osp=0x7f62474a9ca0) at lib/libzpool/dmu_objset.c:378
#5  0x000000000040fd99 in zfs_ioc_objset_stats (zc=0x7f6269c23000) at zfs-fuse/zfs_ioctl.c:1391
#6  0x00000000004138b7 in zfsdev_ioctl (dev=0, cmd=23058, arg=140737203102944, flag=0, cr=0x0, rvalp=0x0)
    at zfs-fuse/zfs_ioctl.c:3656
#7  0x0000000000404cd7 in cmd_ioctl_req (sock=9, cmd=0x7f62474aa0a0) at zfs-fuse/cmd_listener.c:49

Some faraway code had the lock, and I don't have the right kind of debugger-fu.

Fortunately, Mark Musante from Sun visited the #zfs channel around this time. Upon seeing the last trace and reviewing some of the other context, he suggested disabling spa_sync entirely. This made the import succeed. 'zpool list' made the zfs-fuse process crash again, but as long as my only command is the 'zpool import', I am able to keep the filesystem mounted.

Mark was concerned about any writes happening on this hacked mount, including atime updates. I'm pretty sure I disabled atime long ago, and I'm not going to run any further zpool or zfs commands on here, just a big rsync read.

The following parties get donations or presents:

Mark Musante from the ZFS team at Sun
etherpad, for providing the whiteboard/live-pastebin I used with Mark
Antonio Diaz Diaz, for ddrescue
freenode, for hosting the #zfs IRC channel
zfs-fuse project

Other things I learned: opensolaris has an 'ath' driver for my laptop's wifi card which is much more reliable and quick to connect than the linux driver.

When you try to make an ssh tunnel like '-R 999:localhost:22' (for ssh'ing back the other way), ssh will silently fail when the port can't be opened (999 < 1024, so only root can use it).

Sun is still doing features for ZFS. (Mark: "We are working on an automatic uberblock rollback feature, which will be really handy.")

Sandeep - September 27, 2009
how did u disable spa_sync.
Could you elaborate just a bit about your hacking of the zfs code - whether it was on linux or sol and what files did u change, etc. ?
thanks!
Drew Perttula - September 28, 2009
To disable spa_sync, I just put a return at the top of the function in lib/libzpool/spa.c:
void
spa_sync(spa_t *spa, uint64_t txg)
{
return;
...
I did this on linux, where it was easy for me to rebuild and restart the filesystem. To rebuild, just run 'scons', and to restart, just rerun the zfs-fuse process and then try commands like cmd/zpool/zpool.
soloapelty - April 16, 2016
http://google.com mg.solo-solo@mail.ru
AngeloBic - October 1
No matter how you do it, losing weight is an individual thing. Friends can help you but they can't lose the weight for you. You've got to take charge of your weight loss from the beginning if you want to get serious about losing weight. The following tips will help.
Terryvow - October 2
Doctor Who is now considered a British Institute and has come a long way since it first aired on November 23rd 1963. The very first show saw the Doctor travel 100,00 years into the past to help some dim cavemen discover light. After 26 seasons and seven Doctors later the series came off our screens in 1989 much to the disappointment of the huge devoted fanbase. In 1996 an attempt was made to revive Doctor Who but it wasnt until June 2005 when it came back with a vengeance with Christopher Eccleston as the ninth Doctor that put the series back on the map as it were. It then went on for 5 years with David Tenant portraying the Doctor until 2010 when Matt Smith took over the role. Today it is still a great family show and has attracted many new fans. If youre a new or old fan of the show there are Tours and museums you can go and see some of the locations and memorabilia of this classic show. The Doctor Who Tour of London will take you on over 15 locations from the show, some from the new series and some from old sites like the location of The Invasion and Resurrection of the Darleks. The tour also takes you to the TV museum in London where you will get to see some of the cosumes worn in the show and props used. Also you can buy gifts and memorabilia from the shop. You will learn all about how the shows were made so the tour is also educational. If you want to take pictures of the locations thats not a problem. Remember the front door of 10 Downing Street in Aliens of London? Well you can get up and close to this and get your picture taken in front of the door. Rose Tyler fans will love the tour as you get to drop by her home in the show. Why not go that extra mile and actually meet a Doctor Who star. Well this is possible with private or group tours. You will get the general tour but included will be a pre-arranged meeting or lunch with a celebrity from the show. This will obviously depend on availability of the celebrity and the cost will reflect the popularity of that celebrity. There are tours in London and also Wales. The Wales tours take you to Cardiff where you will see lots of location which were featured in shows since 2005. You can leave from London or at Leigh Delamere services station on the M4. There is a Doctor Who exhibition in Cardiff which you get to see. At the end of the Doctor Who tour you get a souvenir group picture sent you by email which is a nice touch. For seriously devoted Doctor Who fans there is a 3 day tour which takes you to all the locations in both London and Cardiff. You will see locations from the past 45 years as well as recent sites from the lasted Doctor Who series. Day one is based in London where you get to see 15 sites. Day two takes you to Cardiff where you get to mean the real life owner of the to see we have an Gothic property used as the location of the school in Human Nature. The final day is partly spent in Cardiff with a walking tour at Cardiff Bay, then you head back to London but a stop at Stonehenge to see the site of the Pandoica. Then its dinner at The Cloven Hoof pub in Devils End b efore you taken back to central London.
CaseyIodig - October 7
Some people, especially those running on busy daily schedules tend to use the pills to help maintain weight since they can not afford to follow all the diet programs. This is not advised. It is recommended that one seek advice from a professional in this field before using the pills. This can save one from many dangers associated with the misuse. The diet pills should always be taken whole. Some people tend to divide the pills to serve a longer period of time. This is not advised and can lead to ineffectiveness. If it is required that one takes a complete tablet, it means that a certain amount of the ingredients are required to achieve the desired goal. It is also recommended that one does not crush the pill and dissolve it in beverages. Chemicals found in beverages have the potential of neutralizing the desired nutrients in the pill thereby leading to ineffectiveness. The best way to take the tablets is swallowing them whole with a glass of water. The diet pills speed up the metabolic processes. This is the key factor that leads to the burning of all the fats in the body. This means that one passes out lots of urine, which subsequently leads to dehydration. It is imperative that the user take lots of water round the clock. This will help curb dehydration, which can lead to health problems. In addition to that, water offers the required medium for the function of the nutrients and elimination of the fats. When buying the review of diet pills, it is imperative that one gets the most recommended dose. People tend to compromise the quality and effectiveness of the tablets due to the variation in cost. The low priced pills depict poor quality, which means their effectiveness is not reliable. Some have also been found to cause health problems. The dose should also be taken as recommended. Over dose will not speed up the process but rather lead to complication. This will increase risk of side effects. If the taking of the pill is forgotten, do not take more to compensate for the lost time. The diet plan enclosed with the diet pills has also to be followed. According to the requirements, the termination of the diet must be done even with no results. This means your body is irresponsive.
FelixBep - October 9
Writing a medical thesis or dissertation is a task done by almost all postgraduate and master's medical students. Dissertation is derived from the Latin word disserto which means discuss. It is essential to write successful medical papers such as medicine essays and medical thesis papers. There are several reasons as to why students write medicine essays. One of the reasons is to promote enhancement of critical judgment, research skills as well as analytical skills. Moreover, medicine essay writing produce students with the ability to 4evaluate and analyze data critically. The initial step for writing medicine essays is to choose a topic. A writer should have at least three topics to choose from. The topic has to be interesting, feasible and relevant. It is essential to write quality medicine essay. Hence, students need to have analytical skills and perfect writing skills. The writing skills will enable them write outstanding essay papers that can be highly regarded by instructors and professors. Teachers often require a lot and expect a lot from their students in terms of medicine essay writing. for this reason, students find essay writing to be an extremely difficult task and hence resort to buying custom medicine essays. A custom medicine essay has to be written by professional writers who are qualified in the field of nursing. Moreover, the custom medicine essay has to be original and plagiarism free. This means that it has to be written from scratch by experts with many years experience. The many years experience should enable a writer to write any form of medical paper including medical thesis, medicine essay and even medicine research paper. Moreover, experience will enable a writer to write a medicine essay that can guarantee academic success. Students get custom medicine essays from custom writing company. It is essential to choose the best company so that one can get the best custom medicine essay. The best and the most reliable medicine essay writing company should have some unique characteristics such as affordability and the ability to provide original and superior quality medicine essays. The other quality is that the company has to hire expert writers who can write quality medicine essays and other types of medical papers. The essays should not only be quality but also plagiarism free and free of grammatical and spelling mistakes. A custom medicine essay has a similar structure to any other academic essay assignment. It has an introduction that introduces the topic and tells the reader what the essay is all about. The second section is the body that has many paragraphs supporting the main topic. Finally there is the conclusion that briefly summarizes what has been discussed in the body section of the essay. Students should choose reliable writing companies so that they can get quality custom papers on several fields such as technology, sociology and law in addition to medicine field. Our custom writing company is the best company that all clients should rely on when in need of any given type of medicine paper. We provide quality papers that not only plagiarism free but also original. Moreover, our custom papers are affordable and able to guarantee academic excellence at all times. All our medical papers are reliable and sure of satisfying clients at all times.
New comment

Your name:

Your email address (won't be redisplayed):

Comment: </div> <div> <input type="submit" class="postButton" value="Post comment"/></div> </form> </li> </ol>  </div>  (prev/next buttons not implemented; use search) </div>    <hr/> </div> </div> </div> </div> </div> <div id="footer"> <p>The Green Marinée template by <a href="http://e-lusion.com" title="Ian Main - e-lusion.com">Ian Main</a> - Built for Wordpress 1.5, ported to this custom site by Drew</p> <div class="extras"> <ul>  <li><a href="/atom" title="Subscribe to Atom feed">Atom</a></li> </ul> </div> </div> </div> </body> </html>

drewp

Recent Posts

Search

Categories

Corrupt ZFS pool recovery