GSoC 2011 Tables Weekly Reports

This is a collection of weekly reports from my work with implementing tables in Scribus as part of Google Summer of Code 2011. They are kept here for future reference and can possibly be used for future design documentation. The reports are in chronological order.

= Report #1 = This report was sent to the scribus-dev mailing list on Sunday June 5, 2011. Here is a link to the mail.

Work Report

 * Added a skeleton of a table style. This was way back in April, just to see what work is involved in creating a new type of style. The style is the most basic one imaginable. At the moment I think it has a single property for background color and styles can be managed in the SM. Also, it doesn't do inheritance yet, though that should be relatively easy to fix.
 * Added class PageItem_Table, a new page item for tables. The class represents a grid of table cells by keeping a few lists with row / column geometries.
 * Added ability to insert a new PageItem_Table from toolbar or Insert menu followed by mouse dragging. Just like the current "tables".
 * Added API to PageItem_Table for the basic table operations
 * insert and remove rows and columns,
 * resize rows and column.
 * Added support for merging cells to PageItem_Table as well as an accompanying helper class CellArea used when keeping track of areas of merged cells. The CellArea class also comes with what I think is the very first unit tests in Scribus. The tests are in tests/ and can be runned using `make test'. In the future, feel free to add more unit tests here if you think there are classes in Scribus that that are easy to unit test and could benefit from it.
 * Added some rudimentary painting to PageItem_Table, just to be able to see the cells. Still a long way to go for proper table / cell painting, which is kind of non-trivial.
 * As scriptability allows me to experiment with tables before any UI is done, I've added the following new scripting methods. These have been added to the old scripter, but should be easy enough to port to ScripterNG later on.
 * createTable(x, y, width, height, numRows, numColumns, ["name"])
 * getTableRows(["name"])
 * getTableColumns(["name"])
 * insertTableRows(index, numRows, ["name"])
 * removeTableRows(index, numRows, ["name"])
 * getTableRowHeight(row, ["name"])
 * setTableRowHeight(row, height, ["name"])
 * insertTableColumns(index, numColumns, ["name"])
 * removeTableColumns(index, numColumns, ["name"])
 * getTableColumnWidth(column, ["name"])
 * setTableColumnWidth(column, width, ["name"])
 * mergeTableCells(row, column, numRows, numColumns, ["name"])

Project Status
The goal in the schedule for this first week was simply "Data structures for tables". Intentionally a quite moderate goal, as I didn't know how long it would take to get up to speed. So unsurprisingly, I think I'm a little ahead of schedule. Worth noting though is that the tables are currently only skeletal in their nature; they hold no content and are simply a grid of empty cells. This is all according to plan though, and integration with text frames is scheduled for later.

Problems / Questions
Only have one quite specific question at the moment (though more will come for sure):


 * How do I set a clipping rectangle on the painter when painting an item in PageItem::DrawObj_Item(...) ? Some method call to set the clipping path followed by a call to setClipPath on the painter? I looked a bit at the other items but haven't quite figured it out yet. So if someone knows off-hand how to do it that would be great.

Next Week
The goals for next week according to the schedule is


 * Basic table layout with fixed column widths and mock content in cells.
 * Insertion/removal of rows/columns.
 * Basic drawing of table.

but as at least some of this has already been done, I think I'll start looking at other things as well. Ideas include


 * Cell borders / padding.
 * Add splitCells(int, int, int, int), symmetric to the current mergeCells(int, int, int, int).
 * Painting of table / cells.
 * Table and cell styles.
 * Start looking at UI:
 * Editing through Property Palette.
 * Canvas mode for editing and selecting cells / rows / columns.
 * Start looking at saving / loading (probably a bit premature).

Lots of things to pick and choose from, and if you have thoughts on which direction to go in first, I'm idle ears. Otherwise I'll just pick what feels most natural and go with it.

I don't think I want to start looking at integration of text frames just yet as I have a feeling that is going to be one of the toughest parts of the project. Besides, there are plenty of things to do before that can be done orthogonally to the integration of text frames. I'd like to have quite robust support for the other things before starting to crack that nut. Keep my back clear and don't spread myself too thin so to speak. So I think I'll stick to my schedule when it comes to this and not stick my head in that beehive until week 4 ;)

= Report #2 = This report was sent to the scribus-dev mailing list on Sunday June 12, 2011. Here is a link to the mail.

Work Report

 * Fixed 4 small bugs in the scripting methods.
 * Added a poor mans unit testing "framework" in the form a script (tests/script/test_tables.py) that automatically runs a set of test methods that exercises the table functionality and reports any regressions. The script gives output of the form

Running table tests... 1/8: test_construction............. Passed 2/8: test_insert_columns........... Passed 3/8: test_insert_rows.............. Passed 4/8: test_merge_cells.............. Passed 5/8: test_remove_columns........... Passed 6/8: test_remove_rows.............. Passed 7/8: test_set_column_width......... Passed 8/8: test_set_row_height........... Passed 100% passed, 0 tests failed out of 8


 * Added CellStyle, a new type of style for table cells. Currently it's as simple as the table style. New styles of this type can be managed in the SM. Also made sure that table/cell styles are shown below the other styles in the SM.
 * Fixed the drawing problem in DrawObj_Post we were talking about.
 * Added a new class TableCell for representing table cells.
 * Moved some adjustment code for areas of merged cells from PageItem_Table into CellArea, thereby simplifying the code in PageItem_Table a little. This made it possible to unit test the code properly and made me find a couple of bugs that I've now fixed. The code will probably also come in handy when doing selections of cell areas later on.
 * Fixed a (silly) bug when inserting a row/column before the first row/column of a table.
 * (Somewhere around here made my 100:th commit, yay :)
 * Changed TableCell into an explicitly shared class using QExplicitlySharedDataPointer for its data. This makes it possible for the table to easily invalidate cells that have been returned by cellAt(int, int) when rows are removed or cells merged. It also frees me from the burden of working with bare pointers to TableCells, minimizing the risk of dangling pointers.
 * Various small cleanups to the table code.

In addition to the work listed above, this week I've also done quite a bit of thinking about the painting of the tables and about their internal representation. I guess like most programmers, I find that if I make a long and hard thinking about the code before writing it, it will almost write itself, whereas if I just go ahead and start coding, it's easy to code yourself into some nasty corners. As I will have limited time after the summer, I want the code to be hackable/maintainable and clear.

Project Status
The past week has seen a little bit less work from me than the last I think. This is mostly because I had that dreaded trailing exam to take care of on the Thursday. This is now taken care of and GSoC can have my undivided attention. On the schedule for the past week was as I mentioned in my last report


 * Basic table layout with fixed column widths and mock content in cells.
 * Insertion/removal of rows/columns.
 * Basic drawing of table.

All three items are done, except I've actually removed some of the first painting code I wrote as I'm currently working on proper painting (more about this in Next Week below).

Problems / Questions
No specific questions this time. You promptly replied to me regarding that painting issue I had with DrawObj_Post(...). Thanks for that. I've added a TODO at the fix in the code about this needing some further investigation.

I do however have one non-code-related concern which turned up while writing this report; I noticed that after malex did the updating of my gsoc11tables Git repo from your internal SVN, some of my old commits started showing up twice in `git log'. As an example, my very first commit "Add a (very) rough first prototype of a table style." is now visible as both

and

in the Git history or gsoc11tables. The second one is the one that turned up after the merging and it has malex as commiter and me as author (Git makes a difference between the two). I'm no Git expert, but think this may be due to the pushing of SVN changes into the Git repo is done using git-svn. Ideally we would have no such duplicates, but I'm not sure how to achieve it. So; malex, do you think you could look into this? If it's not possible then it's not possible, but it's kind of annoying when trying to work out the history of things when old commits show up as if they were new. Maybe ask someone really git-wizard person about it.

Next Week
On the menu for next week according to my initial schedule is


 * More advanced layout.
 * Variable column widths.

But as variable column width is already done, and I already have some of the more advanced table layout features such as row/column spanning working (which was actually not in my schedule until week 7), it seems I'm still a little bit ahead. So far, it seems that I've underestimated how much work I'd be able to get done during these first two/three weeks. That said, the work ahead is plenty and I may well have underestimated the work involved in some of the later items on the schedule, so this is a good thing.

My plan of action for next week is instead to start looking at one of the items for week 4 in the schedule: "Cell formatting properties such as border and padding". In fact, this is what I'm working on right now, making some changes to the internals of the table in preparation for this. I've had a look at the quite powerful styles for lines that Scribus has, and I think they will be useful for my work. There are some tricky but interesting problems when it comes to painting borders on a table, especially regarding the order/priority of which border to paint, but also when it comes to joining of borders that meet at a corner or crossing. During my GSoC for KOffice in 2009, me and Casper Boemann were working in tandem on the tables, and he was doing a lot of the painting work, so I'm very much looking forward to taking this on as it's new territory for me.

= Report #3 = This report was sent to the scribus-dev mailing list on Sunday June 19, 2011. Here is a link to the mail.

Work Report

 * Added a QList> structure of cells to PageItem_Table. And made sure that cells removed from this structure are invalidated properly.
 * Added the cellAt(int, int) API for retrieving cells from the table. Retrieved cells can be queried for their validity with isValid.
 * Added QDebug operator<<(...) to TableCell for convenient printing of cells.
 * Added correct updating of the row/column span of cells when rows/columns are removed/inserted, or when several areas of merged cells are united.
 * Added a thorough internal assertValid method to PageItem_Table which checks the consistency of the table. Also added an accompanying ASSERT_VALID macro, which compiles to nothing unless WANT_DEBUG is set.
 * Added scripting methods getCellRowSpan(...) and getCellColumnSpan(...).
 * Added unit test for mergeTableCells(...) to test script.
 * Added scripting API for setting/getting cell border colors and widths.
 * Moved cell related command out of cmdtable.h/.cpp and into their own cmdcell.h/.cpp.
 * Added API for border widths/colors to PageItem_Table.
 * Simplified the cellRect(int, int) and isCovered(int, int) methods in PageItem_Table by using the new cellAt(int, int) API.
 * Moved styles/overview.txt into the wiki and expanded it with a little section on style property lookup.
 * Added getNamedResources(...) and replaceNamedResources to CellStyle and TableStyle.
 * Added two new document-wide style contexts to ScribusDoc, docTableStyles and docCellStyles, along with API that uses them:
 * const TableStyle& tableStyle(QString name)
 * const StyleSet& tableStyles
 * bool isDefaultStyle(const TableStyle& style) const
 * void redefineTableStyles(const StyleSet& newStyles, bool removeUnused = false)
 * void replaceTableStyles(const QMap& newNameForOld)
 * Created default styles for tables/cells in ScribusDoc::init.
 * Added API for handling table/cell style names to ResourceCollection.
 * Added apply and updateStylesCache to SMTableStyle and SMCellStyle, finally allowing the SM to redefine these new styles on the document.
 * Added getCellStyle/setCellStyle(...) to scripting API.
 * Added properties for border width/color to TableStyle and CellStyle.
 * Changed PageItem_Table and TableCell to use an internal style for storing local properties. The style has the new style context added to ScribusDoc as context, and setting a named style on a cell or table will set the parent style of the internal style.
 * Added API and scripting API for background color to tables and cells.
 * Added missing update(...) functions to TableStyle and CellStyle.
 * Added getTableStyle/setTableStyle(...) to scripting API.
 * Added some initial painting of cell / table backgrounds. Color is taken from style and direct formatting also works.
 * Various documentation fixes, bug fixes and cleanups.

Project Status
On the schedule for this past week was


 * More advanced layout.
 * Variable column widths.

But as those were already done, I decided last week that this week I'd instead start working on proper painting of cell borders. However, despite getting a lot of things done, it seems my schedule has finally caught up with me. You could say that I'm now on schedule instead of ahead of it.

The main reason why I'm no longer staying ahead is that it's been more work than anticipated in bringing the styles up to snuff. A lot of legwork to get them integrated into the the document and updates SM<-->ScribusDoc working.

Things are pretty much ready now though. New properties can now easily be added and the tables and cells will pick them up. Work on cell/table border painting can now begin.

Problems / Questions
No specific questions this time either. I had some during the week, but those were resolved over e-mail/IRC. (Thanks to Craig/avox!).

Next Week
In the timeline for next week it reads


 * Cell formatting properties such as border and padding.
 * Cell content layout; use Scribus' text frames to populate cells.

And for the first time since my first week, I'll actually be doing what the schedule says :) I'll work on the painting of tables/cells. Solid backgrounds are actually already painted, as of a commit some hour ago. Hopefully I'll also be able to also start working on bringing Scribus' text frames into the game. I'm sure that will be a lot of work, which is why it's in my schedule for the week after next as well.

= Report #4 = This report was sent to the scribus-dev mailing list on Monday June 27, 2011. Here is a link to the mail.

Work Report

 * Some small cleanups to insertRows/-Columns and removeRows/-Columns.
 * Added a new empty page "Table" to the PP.
 * Added properties for table border model and border drawing options to the table style / table.
 * Drafted a design of how borders could be collected by "border collectors" inheriting from a common base class and started working on it.
 * Added a drawLine(...) taking QPointF to ScPainter.
 * Realized my initial design mentioned above was probably unnecessarily complex / over-designed and abandoned it.
 * Added a PageItem_Table::collapseBorders(...) which does primitive collapsing of two borders.
 * Added four functions to PageItem_Table for getting the correctly collapsed borders of a cell.
 * Added PageItem_Table::drawBorders(...) for painting a list of borders.
 * Added PageItem_Table::drawTableCollapsed which paints the entire table with collapsed borders. It works by making one iteration across the cells of the table, painting cell backgrounds and collecting borders as it goes. It then paints the collected horizontal and vertical borders in the correct order in a second pass.
 * Switched to using QRectF / QPointF instead of FRect / FPoint.
 * Added PageItem_Table::adjustBorderJoins(...) which adjusts the endpoints of the four borders surrounding a cell according to the border drawing order (horizontal first or vertical first).
 * Added setTableBorderDrawingOptions(...) to scripter API.

Project Status
On the schedule for this past week was


 * Cell formatting properties such as border and padding.
 * Cell content layout; use Scribus' text frames to populate cells.

And quite frankly (and unfortunately) I haven't managed to do half of the things I wanted to. The reasons for this are several and mostly has to do with bad planning on my part, so I take full responsibility for all of them.

Firstly, we had visitors from Singapore coming in on Tue-Wed, and the plan was that Hanna would take care of showing them around town et.c., as she was supposed to have those days off. But they called her in for work in the last minute, leaving me to take care of the guests. Secondly, for Midsummer celebrations we were as usual going up to some friend's summer house up in the northwest. Before I left I made sure that tethering using my Android phone would work in Linux, and looked up the 3G coverage up there, as it's quite a remote location. My hope was to get some hacking done while up there. Tethering worked out of the box and the coverage map I looked said it would be OK. Unfortunately it was only good for certain providers, and coverage was horrible (even regular calls were a bit shaky for me). Finally, as the tickets for the train we were going to take up there on Friday rapidly grew outrageously expensive, we had to book tickets for early Thursday instead. This all meant that when I got home late Saturday night, I hadn't gotten much work done at all. Hence the meagre list in the Work Report this time around. It's a combination of unrealistic planning on my part and some bad circumstances.

But enough about that. The project status right now is that basic painting of solid collapsed borders is pretty much done. The approach I finally settled on seems to work out OK, though some things could of course be improved.

Problems / Questions
Nothing at the moment. One thing that gave me a little surprise was that I realized e.g. FRect::right will not return left + width but left + width - 1 (just like that infamous "bug" QRect has for historical reasons). I've since switched to QRectF.

Next Week
In the timeline for next week is


 * More integration of text frames. Fixing of issues/road bumps.
 * UI: Row/column selection/insertion/removal/resizing.

How appropriate that I managed to get the words "road bumps" in there for this week :)

On a more serious note, what I'll be working hard on during next week is of course getting back on schedule. First I'll fix any remaining issues with the basic border drawing I have now. This means specifically


 * Add a vertical and horizontal offset to the table layout. The basic table grid should actually be offset by (max(left borders of first column)/2, max(top borders of first row)/2)
 * Add cell padding (this is easy, it's just a few more properties on the cells).

After that I should have content bounding rectangles for the cells, and can finally turn my head to integrating Scribus' text frames. As the schedule also mentions UI for next week, I'll try to at least start looking at canvas modes and cell selections.