Bug #2586

Optimize 'onedb' command (upgrade & fsck)

Added by OpenNebula Systems Support Team over 7 years ago. Updated about 7 years ago.

Status:ClosedStart date:12/20/2013
Priority:NormalDue date:
Assignee:Carlos Martín% Done:

0%

Category:Core & System
Target version:Release 4.6
Resolution:fixed Pull request:
Affected Versions:Development

Description

Need to reproduce scenario where upgrade takes significant amount time (more 1 hr)

Approach : use nokogiri & code optimizations

patch-fsck.rb Magnifier (1.24 KB) Rolandas Naujikas, 04/17/2014 03:16 PM


Related issues

Related to Backlog #2845: Improve fsck performance Pending 04/22/2014

Associated revisions

Revision 57099e26
Added by Carlos Martín over 7 years ago

Feature #2586: Optimize onedb upgrade

Revision 49231076
Added by Carlos Martín over 7 years ago

Feature #2586: Optimize onedb fsck

Revision 358a59d3
Added by Carlos Martín over 7 years ago

Feature #2586: Optimize onedb upgrade
(cherry picked from commit 57099e26426ae63756f7c8cae91de7fe29ad1608)

Revision 8067c0db
Added by Carlos Martín over 7 years ago

Feature #2586: Optimize onedb fsck
(cherry picked from commit 492310768a15f90eb4428fcc43d3f190d53bd91f)

Revision 6b097141
Added by Carlos Martín over 7 years ago

Feature #2586: Better onedb upgrade time log

Revision d57b69d0
Added by Carlos Martín over 7 years ago

Feature #2586: Better onedb upgrade time log
(cherry picked from commit 6b0971415b802b388cddc8db8d781b597ed79a8f)

Revision 0820cf86
Added by Carlos Martín about 7 years ago

Bug #2586: Remove formatting in fsck for Nokogiri docs. Contributed by Rolandas Naujikas

Revision 8277a09b
Added by Carlos Martín about 7 years ago

Bug #2586: Do not remove attibutes with space values

Revision 85c4b7bd
Added by Carlos Martín about 7 years ago

Bug #2586: Better nokogiri formatting options

Revision ee013ae4
Added by Carlos Martín about 7 years ago

Revert "Bug #2586: Do not remove attibutes with space values"

This reverts commit 8277a09b1e0aef92db714e30ba6211738d9b6499.

Revision 49b897fc
Added by Carlos Martín about 7 years ago

Revert "Bug #2586: Remove formatting in fsck for Nokogiri docs. Contributed by Rolandas Naujikas"

This reverts commit 0820cf86d77965453f7683e74d81277fd32f4838.

Revision 45d60eb8
Added by Carlos Martín about 7 years ago

Bug #2586: Better nokogiri formatting options

Revision 2baaea53
Added by Carlos Martín about 7 years ago

Bug #2586: Print fsck total time

Revision e82842b8
Added by Carlos Martín about 7 years ago

Bug #2586: Fix extra verbose time log for fsck

Revision a53fd0d4
Added by Carlos Martín about 7 years ago

Bug #2586: A few more improvements to fsck performance

Revision 072d7085
Added by Carlos Martín about 7 years ago

Bug #2586: fsck time formatting

History

#1 Updated by Ruben S. Montero over 7 years ago

  • Tracker changed from Request to Feature
  • Category set to Core & System
  • Status changed from Pending to New
  • Target version set to Release 4.6

#2 Updated by Jaime Melis over 7 years ago

  • Assignee set to Carlos Martín

#3 Updated by Ruben S. Montero over 7 years ago

  • Status changed from New to Closed
  • Resolution set to fixed

#4 Updated by Rolandas Naujikas about 7 years ago

I found those changes in one-4.4 branch breaking stable release of OpenNebula 4.4.1.
By using Nokogiry::XML all user_pool and group_pool body columns are rewrote to indented presentation of XML and breaks other parts in opennebula core and user cli utilities. For e.g. usually body in user_pool is represented as one long line (except if data inside have line breaks):

<USER><ID>34</ID><GID>0</GID><GROUPS><ID>0</ID><ID>1</ID></GROUPS><GNAME>oneadmin</GNAME>...</USER>

but now they look like:
<USER>
  <ID>34</ID>
  <GID>0</GID>
  <GROUPS>
    <ID>0</ID>
    <ID>1</ID>
  </GROUPS>
  <GNAME>oneadmin</GNAME>
  ...
</USER>

#5 Updated by Rolandas Naujikas about 7 years ago

I found a solution for fsck.rb. We can use

doc.root.xpath('//text()[not(normalize-space())]').remove

to remove all formatting space (I'm not sure about space in values) see http://stackoverflow.com/questions/1155293/removing-anything-between-xml-tags-and-their-content
After we can put xml in one line with
:body       => doc.root.serialize(:save_with => 0),

see http://stackoverflow.com/questions/4611403/need-to-remove-newlines-from-object-embed-tags-only-using-nokogiri

Regards

#6 Updated by Ruben S. Montero about 7 years ago

  • Tracker changed from Feature to Bug
  • Status changed from Closed to New
  • Resolution deleted (fixed)
  • Affected Versions Development added

Hi Rolands, Thanks for your detailed feedback!. I've moved this to bug tracker and assign to 4.6, so we can include this before the final release of 4.6

#7 Updated by Carlos Martín about 7 years ago

Hi Rolandas, could you please tell us where was OpenNebula crashing because of the xml formatting?
thanks!

#8 Updated by Carlos Martín about 7 years ago

I guess the problem was that this fix c026120a wasn't cherry-picked to one-4.4. This problem should not happen in the 4.6 branch.

Thanks a lot for your patch. I removed the xpath, because if I understood it right it only removes ATTRIBUTE = " ", which should be valid in OpenNebula. Did I miss any other reason why it was needed?

The documentation is also updated with instructions to update the fsck file from the version in the repo.

Best regards.

#9 Updated by Rolandas Naujikas about 7 years ago

Carlos Martín wrote:

Hi Rolandas, could you please tell us where was OpenNebula crashing because of the xml formatting?
thanks!

When user_pool/group_pool body contains pretty XML formatting (after bug in onedb fsck), then there were many strange errors related to ACLs and permission denied. oneuser show crashing on secondary groups. Strange permission problems. After all spaces and newlines were removed - problems disappeared.

Rolandas

#10 Updated by Rolandas Naujikas about 7 years ago

Carlos Martín wrote:

I guess the problem was that this fix c026120a wasn't cherry-picked to one-4.4. This problem should not happen in the 4.6 branch.

Thanks a lot for your patch. I removed the xpath, because if I understood it right it only removes ATTRIBUTE = " ", which should be valid in OpenNebula. Did I miss any other reason why it was needed?

That was quick way to correct XML formating, but not good in general and not needed if you don't hit bug in onedb fsck. Nokogiri is preserving original XML formatting with right .serialize() call.

The documentation is also updated with instructions to update the fsck file from the version in the repo.

Best regards.

#11 Updated by Carlos Martín about 7 years ago

#12 Updated by Carlos Martín about 7 years ago

  • Status changed from New to Closed
  • Resolution set to fixed

I think this ticket is ready for the 4.6 release.

Rolandas, I decided to fix the empty lines in Nokogiri using this commit 45d60eb8. Although it wasn't strictly necessary, since master and release-4.4.1 were working fine, the problem was only in the one-4.4 head. Anyway, thank you for your feedback.

Also available in: Atom PDF