Bug #2586
Optimize 'onedb' command (upgrade & fsck)
Status: | Closed | Start date: | 12/20/2013 | |
---|---|---|---|---|
Priority: | Normal | Due date: | ||
Assignee: | Carlos Martín | % Done: | 0% | |
Category: | Core & System | |||
Target version: | Release 4.6 | |||
Resolution: | fixed | Pull request: | ||
Affected Versions: | Development |
Description
Need to reproduce scenario where upgrade takes significant amount time (more 1 hr)
Approach : use nokogiri & code optimizations
Related issues
Associated revisions
Feature #2586: Optimize onedb upgrade
Feature #2586: Optimize onedb fsck
Feature #2586: Optimize onedb upgrade
(cherry picked from commit 57099e26426ae63756f7c8cae91de7fe29ad1608)
Feature #2586: Optimize onedb fsck
(cherry picked from commit 492310768a15f90eb4428fcc43d3f190d53bd91f)
Feature #2586: Better onedb upgrade time log
Feature #2586: Better onedb upgrade time log
(cherry picked from commit 6b0971415b802b388cddc8db8d781b597ed79a8f)
Bug #2586: Remove formatting in fsck for Nokogiri docs. Contributed by Rolandas Naujikas
Bug #2586: Do not remove attibutes with space values
Bug #2586: Better nokogiri formatting options
Revert "Bug #2586: Do not remove attibutes with space values"
This reverts commit 8277a09b1e0aef92db714e30ba6211738d9b6499.
Revert "Bug #2586: Remove formatting in fsck for Nokogiri docs. Contributed by Rolandas Naujikas"
This reverts commit 0820cf86d77965453f7683e74d81277fd32f4838.
Bug #2586: Better nokogiri formatting options
Bug #2586: Print fsck total time
Bug #2586: Fix extra verbose time log for fsck
Bug #2586: A few more improvements to fsck performance
Bug #2586: fsck time formatting
History
#1 Updated by Ruben S. Montero over 7 years ago
- Tracker changed from Request to Feature
- Category set to Core & System
- Status changed from Pending to New
- Target version set to Release 4.6
#2 Updated by Jaime Melis over 7 years ago
- Assignee set to Carlos Martín
#3 Updated by Ruben S. Montero over 7 years ago
- Status changed from New to Closed
- Resolution set to fixed
#4 Updated by Rolandas Naujikas about 7 years ago
I found those changes in one-4.4 branch breaking stable release of OpenNebula 4.4.1.
By using Nokogiry::XML all user_pool and group_pool body columns are rewrote to indented presentation of XML and breaks other parts in opennebula core and user cli utilities. For e.g. usually body in user_pool is represented as one long line (except if data inside have line breaks):
<USER><ID>34</ID><GID>0</GID><GROUPS><ID>0</ID><ID>1</ID></GROUPS><GNAME>oneadmin</GNAME>...</USER>
but now they look like:
<USER> <ID>34</ID> <GID>0</GID> <GROUPS> <ID>0</ID> <ID>1</ID> </GROUPS> <GNAME>oneadmin</GNAME> ... </USER>
#5 Updated by Rolandas Naujikas about 7 years ago
- File patch-fsck.rb added
I found a solution for fsck.rb. We can use
doc.root.xpath('//text()[not(normalize-space())]').remove
to remove all formatting space (I'm not sure about space in values) see http://stackoverflow.com/questions/1155293/removing-anything-between-xml-tags-and-their-content
After we can put xml in one line with
:body => doc.root.serialize(:save_with => 0),
see http://stackoverflow.com/questions/4611403/need-to-remove-newlines-from-object-embed-tags-only-using-nokogiri
Regards
#6 Updated by Ruben S. Montero about 7 years ago
- Tracker changed from Feature to Bug
- Status changed from Closed to New
- Resolution deleted (
fixed) - Affected Versions Development added
Hi Rolands, Thanks for your detailed feedback!. I've moved this to bug tracker and assign to 4.6, so we can include this before the final release of 4.6
#7 Updated by Carlos Martín about 7 years ago
Hi Rolandas, could you please tell us where was OpenNebula crashing because of the xml formatting?
thanks!
#8 Updated by Carlos Martín about 7 years ago
I guess the problem was that this fix c026120a wasn't cherry-picked to one-4.4. This problem should not happen in the 4.6 branch.
Thanks a lot for your patch. I removed the xpath, because if I understood it right it only removes ATTRIBUTE = " ", which should be valid in OpenNebula. Did I miss any other reason why it was needed?
The documentation is also updated with instructions to update the fsck file from the version in the repo.
Best regards.
#9 Updated by Rolandas Naujikas about 7 years ago
Carlos Martín wrote:
Hi Rolandas, could you please tell us where was OpenNebula crashing because of the xml formatting?
thanks!
When user_pool/group_pool body contains pretty XML formatting (after bug in onedb fsck), then there were many strange errors related to ACLs and permission denied. oneuser show crashing on secondary groups. Strange permission problems. After all spaces and newlines were removed - problems disappeared.
Rolandas
#10 Updated by Rolandas Naujikas about 7 years ago
Carlos Martín wrote:
I guess the problem was that this fix c026120a wasn't cherry-picked to one-4.4. This problem should not happen in the 4.6 branch.
Thanks a lot for your patch. I removed the xpath, because if I understood it right it only removes ATTRIBUTE = " ", which should be valid in OpenNebula. Did I miss any other reason why it was needed?
That was quick way to correct XML formating, but not good in general and not needed if you don't hit bug in onedb fsck. Nokogiri is preserving original XML formatting with right .serialize() call.
The documentation is also updated with instructions to update the fsck file from the version in the repo.
Best regards.
#11 Updated by Carlos Martín about 7 years ago
- Related to Backlog #2845: Improve fsck performance added
#12 Updated by Carlos Martín about 7 years ago
- Status changed from New to Closed
- Resolution set to fixed
I think this ticket is ready for the 4.6 release.
Rolandas, I decided to fix the empty lines in Nokogiri using this commit 45d60eb8. Although it wasn't strictly necessary, since master and release-4.4.1 were working fine, the problem was only in the one-4.4 head. Anyway, thank you for your feedback.