Draft of end-to-end publishing solution

As a start it would be good if users described their use cases, e.g. Matt Donnelly:

---

'I work for a wholesale pet supply company that publishes print catalogs in addition to several websites. We have no workflow, no content management system (CMS), no wikis, and the editors use Word while the designers use InDesign. We're a growing company that struggles to communicate well with each other as well as with our customers.

I'd think it would be terrific to find some way to glue together Scribus, OpenOffice, Gimp, a CMS app, a wiki, etc. to create an end-to-end publishing solution that takes companies through the entire catalog creation lifecycle and gets new info on/from the websites fast. Fold in Web 2.0, and it's a goal.

The closest we've come is the idea of using InCopy for editorial and InDesign for layout. This would at least get editors and designers on the same (virtual) page. But it doesn't solve the problem of effectively sharing information, workflow, modular content that can be reused across catalogs/websites, etc. Microsoft is trying to do some of this with SharePoint/Office integration...

Use Case #1 (Matt Donnelly)

The idea is to get all the information in one place and massaged as quickly and efficiently as possible. There would be workflow engines/CMS in the process to make sure materials are routed properly, quickly, and efficiently. This would also identify bottlenecks, send alerts, etc.

Now our system works this way:

1. We get product information (on paper ) from merchandisers, i.e., those who buy the products. These sheets include photos, product id numbers, data like color, sizing, etc., buyer name, etc.

2. The copywriters (in a different department) take that information and (sometimes using old copy as quasi-templates) create different copy for that sku for different audiences and locations: best buys, insection, dealer, label, etc. This is all written in Word and a copy is saved to the server in a folder.

3. Then the Word documents are printed out, put in folders, and routed to fact checkers and proofreaders

4. Meanwhile, designers are laying out (in InDesign) pages with product images and placeholders for copy

5. The edited copy sheets are then brought back to the original copywriter, who keys in the changes, then marks the finished docs with the color green and writes in each "OK TO PLACE". Sometimes the copy has to route a second time if the changes were extensive.

6. The designers periodically check the server for "green" files and flow them into the placeholders (this is where some thought that giving InCopy-like products to copywriters would shorten the process)

7. Sometimes the text is too long or too short, so the designer goes back to the copywriters for cuts

8. When the page proofs are done (text/images and other assets like bursts/promotions), they are printed out and routed in clear plastic folders (!) so they can be proofed on paper by 4-5 people

9. Once they proofs are done, the designers make the (hopefully) final changes

10. Usually in each production cycle, products are added at the last minute, others are dropped (or colors, sizes are dropped), and some copy is missing, to be written in haste at the 11th hour

This is a rough sketch. Some obvious needs come out of this:

1A. Tracking documents throughout their lifecycle (from docs to text on a page, etc.)

1B. Version control

2. Making sure all assets for each product stay with it (metadata?)

3. Finding a way (a la InCopy and InDesign) for editors to edit copy even in layout, while designers stick to design

4. A workflow tool to track edits, comments, etc.

5. A good reporting tool on the back end to measure efficiency and uncover bottlenecks

Then there are internal issues with any company that we should address in the design:

1. How can we create a system that even technology-fearful editors and designers will use?

2. What commercial products are out there -- and why would a company choose an open source solution? Why would it make good business sense?

3. Would it be worthwhile to produce an open source framework with "adapters" (APIs) flexible enough to interface with existing commercial products like InDesign? (Put another way: How could companies use their existing software/hardware investments with such a system -- without feeling like we'd be asking them to throw it away?)

More could be said, but that's the gist ;-)

--Matt from Boston

---

Some preliminary ideas on how this might work

1. Setting up the workflow. — The CMS administrator (for each workflow) designs the data structure, according to the particular needs of the organisation or project. Generally, this is but defining the proper entries, fields, attributes, their assets and validation. He could do this by means of an intuitive graphical application; from this design XML schemes, DTDs and the database lay-out is generated. The entries, fields etc. are collected and visually depicted as reusable or generic elements, and travel with all users within the project, in each application, feasibly using some sort of widget.

COMMENTS (Matt Donnelly): What would the graphical application look like? Would it make sense to post some sketches for review? I like the idea of "generic elements" -- like assets or modules. (In SharePoint, they use the image of books in a library.) A lot of copywriting would be easier if there could be things like template libraries...

2. Designing the forms. — Designers use the application of their choice to respectively create web forms, PDF forms (or yet another filing tool). They pick-up the available elements from the widget, by simple drag-and-drop into the application, after which the element is shaped graphically. E.g. the administrator created an entry "product", which has attributes "product name", "price", "color", "description". The designer drags the "product"-icon into Scribus and finds that he has to shape the elements "product name", "price", "colour" and "description". He thus determines font, position, color etc. Since he is not creating the final catalogue template, but a PDF form, the application knows (through the CMS) that the designer also has to shape form fields, so that he is presented with those.

COMMENTS (Matt Donnelly): Could OpenOffice Base (or the KDE equivalent) be used to create these forms? The key is to keep everything in one place and constantly enrich an item's associated metadata (which catalogs it was in, page numbers, version history, etc.). I think InfoPath is what Micro$oft uses??

3. Collecting the data. — Users (data providers, copy writers) could input the data by several means: PDF forms, webpage formulars, stand-alone text editors or online editing applets, or even a custom-made GUI stand-alone application. The data is extracted (PDF formulars) or directly routed to the databases, using an XML scheme. The XML validator tells the user if his data is compliant (min/max amount of characters, image quality etc.), if not, rejects the submission and forces the user to re-edit the content.

COMMENTS (Matt Donnelly): XML would work nicely. For the end user, this could be invisible. That's key for editors and designers who have an inordinate fondness for paper ;-)

4. Designing the templates. — Similar to [2] the designer creates the final template; the XML scheme tells him about the features of an element (length, least, highest and intermediate amount of characters, etc.) so that he takes this into account while designing and positioning the elements.

COMMENTS (Matt Donnelly): Good points. A template library -- and, within that, even a copy library -- would be ideal. No sense starting from scratch ever time, especially since some text is repeated across catalogs with only minor variations.

5. Proofing the data. — At each occasion the CMS provides all users and applications involved with the most up-to-date content and templates on the server. Users have respectively read-only or edit access to the data, according to the rights that are accredited to their account. Proofreaders may edit the data using a web application that uses the data from the server, or by using editable PDF forms. Designers could have Scribus generate a draft from the final document by applying the current data to the template; Scribus collects the XML and flows the data into the design elements, adding extra pages when necessary.

COMMENTS (Matt Donnelly): I like the image of a library where someone checks out/checks in a book. Is there any way to replicate the interaction between InCopy and InDesign, where designers and editors can work in tandem on the same page at the same time? Or is there a better way?

Anyway, thanks, Ludwig, for enriching the discussion. Anyone else? -- Matt

Ludwig

P.S. This is my first contribution and I'm not familiar with Wiki's. Please excuse me if I should have put this elsewhere.

---

[EDIT: I've seen that there are some publishers using Scribus: http://www.tuxmachines.org/node/11855. Would any of them be interesting in working with us on this project? --Matt]

---

I propose a separation in four sections:

1. CMS / Database

Do we want to choose one system, eg. Exist, MySql, Zope or whatever or do we want to be able to use different CMS / database systems? Database connections are already well normed, but CMS APIs would need some work to adapt.

What do we want to store in the CMS? Just text and images? Or do we also want workflow support? What about versioning?

[EDIT: Here are a few commercial products that might be useful as reference: http://www.apsiva.com/, http://www.technicon.com/solutions_catalog.html

These raise an interesting question about the relationship between print and web catalogs. It seems that the print catalogs can be generated as a subsection of the web site/web catalog. --Matt]

2. Scribus

No real choice here! :-)

The work would be to allow external linking to text and maybe provide a mechanism to pull content from the CMS automatically.

Another nice idea would be if Scribus could save its docs, templates, scrapbook and styles in an XML database like Exist.

3. Content Editors

Plain text, OOo, LyX, ...

Scribus already supports import from plain text, html, OOo and others. It would be nice to have an import plugin for XML+CSS, with an option to substitute CSS fonts and colors automatically.

[EDIT: I guess having Scribus be able to manage XML data sources is the main asset at stake here. It really shouldn't matter which input tool was used, or how and where the data was saved, as long as it is provided through a validated XML feed. Image paths should be included in this XML feed as well, not separately linked to. Scribus would be used as a template design app, with XML-tag placeholders. A source path (either locally or on a server) is applied to the template and Scribus generates the final document, by flowing the validated data into the template. I think it would be useful to look at ConTeXt. Perhaps Scribus and the ConTeXt/TeX developers could collaborate in that Scribus might come to generate TeX document classes (templates), and, vice versa, ConTeXt XML data would be used natively within Scribus text frames. — Ludwig]

I'm not familiar with ConTeXt/XML. Could you elaborate? - avox

[ConTeXt is probably the most adavanced TeX distribution, natively supports 16-bit Unicode encoding, and inetgrates with XML workflows. ConTeXt uses XML feeds to import database content material, while automating the typesetting according to predefined templates. There is a ConTeXt wiki, here. XeTeX on the other hand also suppirts unicode and has a fully operational OpenType engine, so that OT features are applied. Both the teams of ConTeXt and XeTeX are planning to work together on a new fullfeatures TeX. I think it would be a major leap foreward, if these TeXers and the Scribus team would collaborate, or at least would exhange thoughts and strategies. Scribus could benefit from the code already done in ConTeXt for XML-import and batch processing, while ConTeXt could use, in a way, the WHYSIWIG GUI that Scribus offers. I'm sure Hans Hagen would be interested in collaboration of any king. Here is his website. — Ludwig]

4. Image Editors

InkScape, Gimp, Cinepaint, Xara, ...

Once those *all* support CMYK it's a mere question of linking those apps to the CMS.