GSoC 2011 Tables Weekly Reports

From Scribus Wiki
Revision as of 11:46, 5 July 2011 by Estan (talk | contribs) (Fixed link.)
Jump to navigation Jump to search

This is a collection of weekly reports from my work with implementing tables in Scribus as part of Google Summer of Code 2011. They are kept here for future reference and can possibly be used for future design documentation. The reports are in chronological order.

Report #1

This report was sent to the scribus-dev mailing list on Sunday June 5, 2011. Here is a link to the mail.

Work Report

  • Added a skeleton of a table style. This was way back in April, just to see what work is involved in creating a new type of style. The style is the most basic one imaginable. At the moment I think it has a single property for background color and styles can be managed in the SM. Also, it doesn't do inheritance yet, though that should be relatively easy to fix.
  • Added class PageItem_Table, a new page item for tables. The class represents a grid of table cells by keeping a few lists with row / column geometries.
  • Added ability to insert a new PageItem_Table from toolbar or Insert menu followed by mouse dragging. Just like the current "tables".
  • Added API to PageItem_Table for the basic table operations
    • insert and remove rows and columns,
    • resize rows and column.
  • Added support for merging cells to PageItem_Table as well as an accompanying helper class CellArea used when keeping track of areas of merged cells. The CellArea class also comes with what I think is the very first unit tests in Scribus. The tests are in tests/ and can be runned using `make test'. In the future, feel free to add more unit tests here if you think there are classes in Scribus that that are easy to unit test and could benefit from it.
  • Added some rudimentary painting to PageItem_Table, just to be able to see the cells. Still a long way to go for proper table / cell painting, which is kind of non-trivial.
  • As scriptability allows me to experiment with tables before any UI is done, I've added the following new scripting methods. These have been added to the old scripter, but should be easy enough to port to ScripterNG later on.
    • createTable(x, y, width, height, numRows, numColumns, ["name"])
    • getTableRows(["name"])
    • getTableColumns(["name"])
    • insertTableRows(index, numRows, ["name"])
    • removeTableRows(index, numRows, ["name"])
    • getTableRowHeight(row, ["name"])
    • setTableRowHeight(row, height, ["name"])
    • insertTableColumns(index, numColumns, ["name"])
    • removeTableColumns(index, numColumns, ["name"])
    • getTableColumnWidth(column, ["name"])
    • setTableColumnWidth(column, width, ["name"])
    • mergeTableCells(row, column, numRows, numColumns, ["name"])

Project Status

The goal in the schedule for this first week was simply "Data structures for tables". Intentionally a quite moderate goal, as I didn't know how long it would take to get up to speed. So unsurprisingly, I think I'm a little ahead of schedule. Worth noting though is that the tables are currently only skeletal in their nature; they hold no content and are simply a grid of empty cells. This is all according to plan though, and integration with text frames is scheduled for later.

Problems / Questions

Only have one quite specific question at the moment (though more will come for sure):

  • How do I set a clipping rectangle on the painter when painting an item in PageItem::DrawObj_Item(...) ? Some method call to set the clipping path followed by a call to setClipPath on the painter? I looked a bit at the other items but haven't quite figured it out yet. So if someone knows off-hand how to do it that would be great.

Next Week

The goals for next week according to the schedule is

  • Basic table layout with fixed column widths and mock content in cells.
  • Insertion/removal of rows/columns.
  • Basic drawing of table.

but as at least some of this has already been done, I think I'll start looking at other things as well. Ideas include

  • Cell borders / padding.
  • Add splitCells(int, int, int, int), symmetric to the current mergeCells(int, int, int, int).
  • Painting of table / cells.
  • Table and cell styles.
  • Start looking at UI:
    • Editing through Property Palette.
    • Canvas mode for editing and selecting cells / rows / columns.
  • Start looking at saving / loading (probably a bit premature).

Lots of things to pick and choose from, and if you have thoughts on which direction to go in first, I'm idle ears. Otherwise I'll just pick what feels most natural and go with it.

I don't think I want to start looking at integration of text frames just yet as I have a feeling that is going to be one of the toughest parts of the project. Besides, there are plenty of things to do before that can be done orthogonally to the integration of text frames. I'd like to have quite robust support for the other things before starting to crack that nut. Keep my back clear and don't spread myself too thin so to speak. So I think I'll stick to my schedule when it comes to this and not stick my head in that beehive until week 4 ;)

Report #2

This report was sent to the scribus-dev mailing list on Sunday June 12, 2011. Here is a link to the mail.

Work Report

  • Fixed 4 small bugs in the scripting methods.
  • Added a poor mans unit testing "framework" in the form a script (tests/script/test_tables.py) that automatically runs a set of test methods that exercises the table functionality and reports any regressions. The script gives output of the form
Running table tests...
	1/8: test_construction()............. Passed
	2/8: test_insert_columns()........... Passed
	3/8: test_insert_rows().............. Passed
	4/8: test_merge_cells().............. Passed
	5/8: test_remove_columns()........... Passed
	6/8: test_remove_rows().............. Passed
	7/8: test_set_column_width()......... Passed
	8/8: test_set_row_height()........... Passed
100% passed, 0 tests failed out of 8
  • Added CellStyle, a new type of style for table cells. Currently it's as simple as the table style. New styles of this type can be managed in the SM. Also made sure that table/cell styles are shown below the other styles in the SM.
  • Fixed the drawing problem in DrawObj_Post we were talking about.
  • Added a new class TableCell for representing table cells.
  • Moved some adjustment code for areas of merged cells from PageItem_Table into CellArea, thereby simplifying the code in PageItem_Table a little. This made it possible to unit test the code properly and made me find a couple of bugs that I've now fixed. The code will probably also come in handy when doing selections of cell areas later on.
  • Fixed a (silly) bug when inserting a row/column before the first row/column of a table.
  • (Somewhere around here made my 100:th commit, yay :)
  • Changed TableCell into an explicitly shared class using QExplicitlySharedDataPointer for its data. This makes it possible for the table to easily invalidate cells that have been returned by cellAt(int, int) when rows are removed or cells merged. It also frees me from the burden of working with bare pointers to TableCells, minimizing the risk of dangling pointers.
  • Various small cleanups to the table code.

In addition to the work listed above, this week I've also done quite a bit of thinking about the painting of the tables and about their internal representation. I guess like most programmers, I find that if I make a long and hard thinking about the code before writing it, it will almost write itself, whereas if I just go ahead and start coding, it's easy to code yourself into some nasty corners. As I will have limited time after the summer, I want the code to be hackable/maintainable and clear.

Project Status

The past week has seen a little bit less work from me than the last I think. This is mostly because I had that dreaded trailing exam to take care of on the Thursday. This is now taken care of and GSoC can have my undivided attention. On the schedule for the past week was as I mentioned in my last report

  • Basic table layout with fixed column widths and mock content in cells.
  • Insertion/removal of rows/columns.
  • Basic drawing of table.

All three items are done, except I've actually removed some of the first painting code I wrote as I'm currently working on proper painting (more about this in Next Week below).

Problems / Questions

No specific questions this time. You promptly replied to me regarding that painting issue I had with DrawObj_Post(...). Thanks for that. I've added a TODO at the fix in the code about this needing some further investigation.

I do however have one non-code-related concern which turned up while writing this report; I noticed that after malex did the updating of my gsoc11tables Git repo from your internal SVN, some of my old commits started showing up twice in `git log'. As an example, my very first commit "Add a (very) rough first prototype of a table style." is now visible as both

0b17201cf816a883b80daae139f24c6a7c093939 and
0a8327820da7a88952744ac4a97314f075018869

in the Git history or gsoc11tables. The second one is the one that turned up after the merging and it has malex as commiter and me as author (Git makes a difference between the two). I'm no Git expert, but think this may be due to the pushing of SVN changes into the Git repo is done using git-svn. Ideally we would have no such duplicates, but I'm not sure how to achieve it. So; malex, do you think you could look into this? If it's not possible then it's not possible, but it's kind of annoying when trying to work out the history of things when old commits show up as if they were new. Maybe ask someone really git-wizard person about it.

Next Week

On the menu for next week according to my initial schedule is

  • More advanced layout.
  • Variable column widths.

But as variable column width is already done, and I already have some of the more advanced table layout features such as row/column spanning working (which was actually not in my schedule until week 7), it seems I'm still a little bit ahead. So far, it seems that I've underestimated how much work I'd be able to get done during these first two/three weeks. That said, the work ahead is plenty and I may well have underestimated the work involved in some of the later items on the schedule, so this is a good thing.

My plan of action for next week is instead to start looking at one of the items for week 4 in the schedule: "Cell formatting properties such as border and padding". In fact, this is what I'm working on right now, making some changes to the internals of the table in preparation for this. I've had a look at the quite powerful styles for lines that Scribus has, and I think they will be useful for my work. There are some tricky but interesting problems when it comes to painting borders on a table, especially regarding the order/priority of which border to paint, but also when it comes to joining of borders that meet at a corner or crossing. During my GSoC for KOffice in 2009, me and Casper Boemann were working in tandem on the tables, and he was doing a lot of the painting work, so I'm very much looking forward to taking this on as it's new territory for me.

Report #3

This report was sent to the scribus-dev mailing list on Sunday June 19, 2011. Here is a link to the mail.

Work Report

  • Added a QList<QList<TableCell>> structure of cells to PageItem_Table. And made sure that cells removed from this structure are invalidated properly.
  • Added the cellAt(int, int) API for retrieving cells from the table. Retrieved cells can be queried for their validity with isValid().
  • Added QDebug operator<<(...) to TableCell for convenient printing of cells.
  • Added correct updating of the row/column span of cells when rows/columns are removed/inserted, or when several areas of merged cells are united.
  • Added a thorough internal assertValid() method to PageItem_Table which checks the consistency of the table. Also added an accompanying ASSERT_VALID macro, which compiles to nothing unless WANT_DEBUG is set.
  • Added scripting methods getCellRowSpan(...) and getCellColumnSpan(...).
  • Added unit test for mergeTableCells(...) to test script.
  • Added scripting API for setting/getting cell border colors and widths.
  • Moved cell related command out of cmdtable.h/.cpp and into their own cmdcell.h/.cpp.
  • Added API for border widths/colors to PageItem_Table.
  • Simplified the cellRect(int, int) and isCovered(int, int) methods in PageItem_Table by using the new cellAt(int, int) API.
  • Moved styles/overview.txt into the wiki and expanded it with a little section on style property lookup.
  • Added getNamedResources(...) and replaceNamedResources to CellStyle and TableStyle.
  • Added two new document-wide style contexts to ScribusDoc, docTableStyles and docCellStyles, along with API that uses them:
    • const TableStyle& tableStyle(QString name)
    • const StyleSet<TableStyle>& tableStyles()
    • bool isDefaultStyle(const TableStyle& style) const
    • void redefineTableStyles(const StyleSet<TableStyle>& newStyles, bool removeUnused = false)
    • void replaceTableStyles(const QMap<QString, QString>& newNameForOld)
  • Created default styles for tables/cells in ScribusDoc::init().
  • Added API for handling table/cell style names to ResourceCollection.
  • Added apply() and updateStylesCache() to SMTableStyle and SMCellStyle, finally allowing the SM to redefine these new styles on the document.
  • Added getCellStyle()/setCellStyle(...) to scripting API.
  • Added properties for border width/color to TableStyle and CellStyle.
  • Changed PageItem_Table and TableCell to use an internal style for storing local properties. The style has the new style context added to ScribusDoc as context, and setting a named style on a cell or table will set the parent style of the internal style.
  • Added API and scripting API for background color to tables and cells.
  • Added missing update(...) functions to TableStyle and CellStyle.
  • Added getTableStyle()/setTableStyle(...) to scripting API.
  • Added some initial painting of cell / table backgrounds. Color is taken from style and direct formatting also works.
  • Various documentation fixes, bug fixes and cleanups.

Project Status

On the schedule for this past week was

  • More advanced layout.
  • Variable column widths.

But as those were already done, I decided last week that this week I'd instead start working on proper painting of cell borders. However, despite getting a lot of things done, it seems my schedule has finally caught up with me. You could say that I'm now on schedule instead of ahead of it.

The main reason why I'm no longer staying ahead is that it's been more work than anticipated in bringing the styles up to snuff. A lot of legwork to get them integrated into the the document and updates SM<-->ScribusDoc working.

Things are pretty much ready now though. New properties can now easily be added and the tables and cells will pick them up. Work on cell/table border painting can now begin.

Problems / Questions

No specific questions this time either. I had some during the week, but those were resolved over e-mail/IRC. (Thanks to Craig/avox!).

Next Week

In the timeline for next week it reads

  • Cell formatting properties such as border and padding.
  • Cell content layout; use Scribus' text frames to populate cells.

And for the first time since my first week, I'll actually be doing what the schedule says :) I'll work on the painting of tables/cells. Solid backgrounds are actually already painted, as of a commit some hour ago. Hopefully I'll also be able to also start working on bringing Scribus' text frames into the game. I'm sure that will be a lot of work, which is why it's in my schedule for the week after next as well.

Report #4

This report was sent to the scribus-dev mailing list on Monday June 27, 2011. Here is a link to the mail.

Work Report

  • Some small cleanups to insertRows()/-Columns() and removeRows()/-Columns().
  • Added a new empty page "Table" to the PP.
  • Added properties for table border model and border drawing options to the table style / table.
  • Drafted a design of how borders could be collected by "border collectors" inheriting from a common base class and started working on it.
  • Added a drawLine(...) taking QPointF to ScPainter.
  • Realized my initial design mentioned above was probably unnecessarily complex / over-designed and abandoned it.
  • Added a PageItem_Table::collapseBorders(...) which does primitive collapsing of two borders.
  • Added four functions to PageItem_Table for getting the correctly collapsed borders of a cell.
  • Added PageItem_Table::drawBorders(...) for painting a list of borders.
  • Added PageItem_Table::drawTableCollapsed which paints the entire table with collapsed borders. It works by making one iteration across the cells of the table, painting cell backgrounds and collecting borders as it goes. It then paints the collected horizontal and vertical borders in the correct order in a second pass.
  • Switched to using QRectF / QPointF instead of FRect / FPoint.
  • Added PageItem_Table::adjustBorderJoins(...) which adjusts the endpoints of the four borders surrounding a cell according to the border drawing order (horizontal first or vertical first).
  • Added setTableBorderDrawingOptions(...) to scripter API.

Project Status

On the schedule for this past week was

  • Cell formatting properties such as border and padding.
  • Cell content layout; use Scribus' text frames to populate cells.

And quite frankly (and unfortunately) I haven't managed to do half of the things I wanted to. The reasons for this are several and mostly has to do with bad planning on my part, so I take full responsibility for all of them.

Firstly, we had visitors from Singapore coming in on Tue-Wed, and the plan was that Hanna would take care of showing them around town et.c., as she was supposed to have those days off. But they called her in for work in the last minute, leaving me to take care of the guests. Secondly, for Midsummer celebrations we were as usual going up to some friend's summer house up in the northwest. Before I left I made sure that tethering using my Android phone would work in Linux, and looked up the 3G coverage up there, as it's quite a remote location. My hope was to get some hacking done while up there. Tethering worked out of the box and the coverage map I looked said it would be OK. Unfortunately it was only good for certain providers, and coverage was horrible (even regular calls were a bit shaky for me). Finally, as the tickets for the train we were going to take up there on Friday rapidly grew outrageously expensive, we had to book tickets for early Thursday instead. This all meant that when I got home late Saturday night, I hadn't gotten much work done at all. Hence the meagre list in the Work Report this time around. It's a combination of unrealistic planning on my part and some bad circumstances.

But enough about that. The project status right now is that basic painting of solid collapsed borders is pretty much done. The approach I finally settled on seems to work out OK, though some things could of course be improved.

Problems / Questions

Nothing at the moment. One thing that gave me a little surprise was that I realized e.g. FRect::right() will not return left() + width() but left() + width() - 1 (just like that infamous "bug" QRect has for historical reasons). I've since switched to QRectF.

Next Week

In the timeline for next week is

  • More integration of text frames. Fixing of issues/road bumps.
  • UI: Row/column selection/insertion/removal/resizing.

How appropriate that I managed to get the words "road bumps" in there for this week :)

On a more serious note, what I'll be working hard on during next week is of course getting back on schedule. First I'll fix any remaining issues with the basic border drawing I have now. This means specifically

  • Add a vertical and horizontal offset to the table layout. The basic table grid should actually be offset by (max(left borders of first column)/2, max(top borders of first row)/2)
  • Add cell padding (this is easy, it's just a few more properties on the cells).

After that I should have content bounding rectangles for the cells, and can finally turn my head to integrating Scribus' text frames. As the schedule also mentions UI for next week, I'll try to at least start looking at canvas modes and cell selections.

Report #5

This report was sent to the scribus-dev mailing list on Monday June 27, 2011. Here is a link to the mail.

Work Report

  • Fixed the API docs of Scribus to work with Doxygen 1.7+.
    • Some HTML elements had changed, so the CSS was updated to match.
    • Updated Doxyfile with `doxygen -u'.
    • Added *.js and *.repository to `clean' target as those are generated nowadays. No *.ttf seems to be generated anymore, but I kept them in to stay compatible with older Doxygen.
    • Enabled JAVADOC_AUTOBRIEF (OK:ed by avox on IRC).

The above is not directly related to my project, but as I try to document all code I'm writing I wanted to see what it looks like by generating it locally. Anyway, it was 15-20 min of work. Moving on, I also

  • Added class TableBorderLine which represents a single line in a table/cell border. Perhaps it can be scrapped in favor of the SingleLine class in scribusstructs.h as they are very similar, but I'm keeping it for now.
  • Added class TableBorder which represents a border along one side of a table/cell. It is essentially a list of TableBorderLine with a convenience function bool joinsWith(const TableBorder& other) const; to check if it can be joined with another border. A TableBorder with no TableBorderLines in it is considered a null border. These are used for convenience during painting to represent the absence of a border.
  • Updated the scripting API and table/cell styles to use the new border system instead of individually set border color/width/et.c. properties.
  • Added abstract base class TablePainter representing a painter that can paint a PageItem_Table. The class has a single pure virtual function void paintTable(ScPainter*) in which subclasses should do their work.
  • Added class CollapsedTablePainter, an implementation of TablePainter that can paint a table with borders collapsed and joined at intersections.
  • Modified PageItem_Table to hold a pointer to a TablePainter and paint itself using this painter in the DrawObj_Item(...) function.
  • Added three utility functions, kept in a new TableUtils namespace for easy unit testing. The functions are used by CollapsedTablePainter when collapsing and joining borders. They are
    • collapseBorders(...) which given two borders returns them collapsed,
    • joinHorizontal(...) which given a horizontal table border and its adjoining borders will return the adjustments necessary for joining the border,
    • joinVertical(...) which given a vertical table border and its adjoining borders will return the adjustments necessary for joining the border,
  • Added columnPosition(int) and rowPosition(int) API to PageItem_Table.
  • Added an QString asString() const; to CellStyle for easy printing.

In addition to just listing the changes like this, I feel I need to elaborate a bit on what has taken such time. I did explain it a little bit in an earlier mid-week report. The problem has been a hard one for mainly two reasons:

  • Each cell may have N neighboring cells along one of its edges. The objective of the painting algorithm is to iterate over all cell edges in the table, and for each cell edge, iterate over all shared border sections. It must then figure out all (possibly six) cells surrounding the border section to be painted, along with the collapsed borders between them (or between them and the table). When that is done, the border section (or rather, the coordinates at which to paint the border section) needs to be adjusted for joining. This is delegated to the joining algorithm.
  • There are 41 one possible join cases (that I've identified at least). They can be seen in the attached picture, which I finally had to draw to sort it out in my head. The objective of the joining algorithm is then to, given a border section along with the other border sections coming in to meet it at its start and end point, return adjusted start and end points as well as offsets for the individual border lines in order to make a proper join.

On top of this is the question of painting order. The joining algorithm must be necessity make an assumption about painting order; either horizontal borders should be painted on top for the algorithm to work, or vertical should be painted on top.

The joining algorithm I've written assumes that all horizontal borders in the table are painted on top of vertical ones, which means two iterations across the cells. I've been unable to find an algorithm which makes no assumption about the painting order. I don't think there is any actually, and iteration across the table cells should be fast enough. Perhaps in the future optimizations could be made, but then I think they should rather be done to the internal structures of the table to make cell accesses fast, rather than convoluting the joining algorithm.

All in all, a tough nut to crack, and there are currently the following two small problems with the painting/joining left:

  1. The joining algorithm does not handle some cases properly.
  2. The painting algoritm sometimes mistakenly identifies a neighboring border as a null border.

For 1), I feel that I need a break from this problem to not go insane, and will let it go for now. The code is completely isolated in TableUtils::joinHorizontal(...) and TableUtils::joinVertical(...) and can be improved on later. I welcome anyone who likes puzzles to look at the picture with the 41 join cases and see if they can find an optimal algorithm that covers all cases with a minimal amount of code.

For 2) I know the reason and how to fix it, it's just that I haven't gotten around to it. In short, what happens is that instead of collapsing with the table border, it's requesting a cell outside the table, which will return an invalid cell with a null border, and collapses with that.

As I've intentionally kept the collapsing and joining algorithms as separate functions in the TableUtils namespace, what remains is to write unit tests that cover all the 41 cases.

Project Status

I'm still behind schedule by at least one week. The reason I decided to put all my efforts into getting this painting working correctly is I didn't want to leave it unfinished, or leave a design that would later come back and bite me when trying to paint complex borders.

I had no idea it would be such much hard work. I think I've used up all my sketchbooks on this problem, and lost some sleep over it. I feel now that, to not lose my mind, I really need to get on with some of the other problems, such as UI and integration with text frames.

I've looked ahead a bit on my schedule, and found that

Week 6:

  • Wrapping up for mid-term evaluation.
  • Fix bugs/finish up what I have so far.

Week 7:

  • Row and column spanning.

Week 8:

  • Table/row/column/cell styles.

Since row and column spanning is done, and the styles are in a pretty good shape (they are there, and new properties can be added removed easily), I feel that there's still a good opportunity for me to catch up.

Problems / Questions

My biggest question right now is:

  • What should be the general approach for integration of text frames? The table will hold a set of text frames for its cells, but then? The user should be prohibited from doing certain things to the text frames, such as selecting and resizing them. This should be handled by the table. In a way they will be similar to locked text frames, until the user decides he wants to edit their content. Should I go for an approach where I add a bool isTableCellContent() or similar to PageItem_TextFrame? Or should PageItem_TextFrame even be inherited, creating a new item for cell content? I'm open to all ideas, so if you happen to have given this problem some thought, please share, because frankly I haven't (yet).

Next Week

I'll yet again be working on catching up with my schedule, which means working on integration of text frames and basic UI. Since mid-term evaluation is next week, perhaps I should also write some kind of mid-term report. What do you think Craig, do you think its worthwhile or just a waste of coding time? Perhaps you feel up to date enough with my status. I also really should get around to making another blog post showing the new features, as I've only made one post so far.